RADIOELECTRONIC AND COMPUTER SYSTEMS

In recent times, machine learning algorithms have shown great performance in solving problems in different fields of study, including the analysis of remote sensing images, computer vision, natural language processing, medical issues, etc. A well-prepared input dataset can have a huge impact on the result metrics. However, a correctly selected hyperparameter combined with neural network architecture could highly increase the final metrics. Therefore, the hyperparameters optimization problem becomes a key issue in a deep learning algorithm. The process of finding a suitable hyperparameter combination could be performed manually or automatically. Manual search is based on previous research and requires enormous human efforts. However, there are many automated hyperparameter optimization methods have been successfully applied in practice. The automated hyperparameter tuning techniques are divided into two groups: black-box optimization techniques (such as Grid Search, Random Search) and multi-fidelity optimization techniques (HyperBand, BOHB). The most recent and promising among all approaches is BOHB which, which combines both Bayesian optimization and bandit-based methods, outperforms classical approaches, and can run asynchronously with given GPU resources and time budget that plays a vital role in the hyperparameter optimization process. The previous study proposed a convolutional deep learning neural network for solving land cover classification problems in the EuroSAT dataset. It was found that adding spectral indexes NDVI, NDWI, and GNDVI with RGB channels increased the result accuracy (from 64.72% to 84.19%) and F1 (from 63.89 % to 84.05%) score. However, the convolutional neural network architecture and hyperparameter combination were selected manually. The research optimizes convolutional neural network architecture and finds suitable hyperparameter combinations applied to land cover classification problems using multispectral images. The obtained results must increase result performance compared with the previous study and given budget constraints.

hyperparameter optimization; EuroSAT; BOHB; convolutional neural network; land cover; remote sensing

Yaloveha, V., Hlavcheva, D., Podorozhniak A., Kuchuk, H. Spectral Indexes Evaluation for Satellite Images Classification using CNN. Journal of Information and Organizational Sciences, 2021, vol. 46. no. 2, pp. 95-113. DOI: 10.31341/jios.45.2.5.

Dergachov, K., Кrasnov, L., Bilozerskyi, V., Zymovin, A. Data pre-processing to increase the quality of optical text recognition systems. Radioelectronic and Computer Systems, 2021, no. 4, pp. 183-198. DOI: 10.32620/reks.2021.4.15.

Kuchuk, H., Podorozhniak, A., Liubchenko, N., Onischenko, D. System of license plate recognition considering large camera shooting angles. Radioelectronic and Computer Systems, 2021, no. 4, pp. 82-91. DOI: 10.32620/reks.2021.4.07.

Liubchenko, N., Podorozhniak, A., Oliinyk, V. Research of antispam bot algorithms for social networks. CEUR Workshop Proceedings, 2021, vol. 2870, pp. 822-831.

Pal, A. Rathi, Y. A review of deep learning methods for MRI reconstruction. Preprint arXiv: 2109.08618, 2021.

Jordan, M. I., Mitchell, M. I. Machine learning: Trends, perspectives, and prospects. Science, 2015, vol. 349, no. 6245, pp. 255-260. DOI: 10.1126/science.aaa8415.

Podorozhniak, A., Lubchenko, N., Balenko, O., Zhuikov, D. Neural network approach for multispectral image processing. 14th International Conference on Advanced Trends in Radioelecrtronics, Telecommunications and Computer Engineering (TCSET), 2018, pp. 978-981. DOI: 10.1109/TCSET.2018.8336357.

Ieremeiev, O., Lukin, V., Okarma, K. Combined visual quality metric of remote sensing images based on neural network. Radioelectronic and computer systems, 2020, no. 4, pp. 4-15. DOI: 10.32620/reks.2020.4.01.

Hlavcheva, D., Yaloveha, V., Podorozhniak A., Kuchuk, H. Application of convolutional neural network for histopathological analysis. Advanced Information Systems, 2019, vol. 3, no. 4, pp. 69-73. DOI: 10.20998/2522-9052.2019.4.10.

Hlavcheva, D. Yaloveha, V., Podorozhniak A., Kuchuk, H. Comparison of CNNs for Lung Biopsy Images Classification. 2021 IEEE 3rd Ukraine Conference on Electrical and Computer Engineering (UKRCON), 2021, pp. 1-5. DOI: 10.1109/UKRCON53503.2021.9575305.

Wang, S. C. Interdisciplinary computing in Java programming. Springer Science & Business Media, 2003, pp. 81-100.

Yang, L., Shami, A. On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 2020, vol. 415, pp. 295-316. DOI: 10.1016/j.neucom.2020.07.061.

Wu, J., Chen, X. Y., Zhang, H., Xiong, L. D., Lei, L. D., Deng S. H. Hyperparameter optimization for machine learning models based on Bayesian optimization. Journal of Electronic Science and Technology, 2019. vol. 17, no. 1, pp. 26-40. DOI: 10.11989/JEST.1674-862X.80904120.

Koutsoukas, A., Monaghan, K. J., Li, X., Huan, J. Deep-learning: investigating deep neural networks hyper-parameters and comparison of performance to shallow methods for modeling bioactivity data. Journal of cheminformatics, 2017, vol. 9, no. 1, pp. 1-13. DOI: 10.1186/s13321-017-0226-y.

Elshawi, R., Maher, M., Sakr, S. Automated machine learning: State-of-the-art and open challenges. arXiv preprint arXiv:1906.02287, 2019.

Wang, C., Wang, H., Zhou, C., Chen, H., Li, J., Gao, H. Experiencethinking: Hyperparameter optimization with budget constraints. Preprint arXiv arXiv: 1912.00602, 2019.

Bergstra, J., Bengio, Y. Random search for hyper-parameter optimization. Journal of machine learning research, 2012, vol. 13, no. 2, pp. 281–305.

Bellman, R. E. Adaptive control processes: A Guided Tour. Princeton, Princeton university press, 2015. DOI: 10.1515/9781400874668.

Witt, C. Worst-case and average-case approximations by simple randomized search heuristics. Annual Symposium on Theoretical Aspects of Computer Science, Springer, 2005, pp. 44-56. DOI: 10.1007/978-3-540-31856-9_4.

Hutter, F. Kotthoff, L., Vanschoren, J. Automated machine learning: methods, systems, challenges. Springer Nature, 2019. 219 p. DOI: 10.1007/978-3-030-05318-5.

Zöller, M. A., Huber, M. F. Benchmark and survey of automated machine learning frameworks Journal of artificial intelligence research, 2021, vol. 70, pp. 409-472. DOI: 10.1613/jair.1.11854.

Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 2017, vol. 18, no. 1, pp. 6765-6816.

Krizhevsky, A., Sutskever, I., Hinton, G. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012, vol. 25, pp. 1097–1105.

Jamieson, K., Talwalkar, A. Non-stochastic best arm identification and hyperparameter optimization. Artificial intelligence and statistics, PMLR, 2016, pp. 240-248.

Falkner, S., Klein, A., Hutter, F. BOHB: Robust and efficient hyperparameter optimization at scale. International Conference on Machine Learning, PMLR, 2018, pp. 1437-1446.

Bergstra, J., Bardenet, R., Bengio, Y., Kégl, B. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 2011, vol. 24, pp. 2546–2554.

Yuan, Q., Shen, H., Li, T., Li, Z. et al. Deep learning in environmental remote sensing: Achievements and challenges. Remote Sensing of Environment, 2020, vol. 241. DOI: 10.1016/j.rse.2020.111716.

Zhang, C., Sargent, I., Pan, X., Li, H. Joint Deep Learning for land cover and land use classification. Remote sensing of environment, 2019, vol. 221, pp. 173-187. DOI: 10.1016/j.rse.2018.11.014.

Chlingaryan, A., Sukkarieh, S., Whelan, B. Machine learning approaches for crop yield prediction and nitrogen status estimation in precision agriculture: A review. Computers and electronics in agriculture, 2018, vol. 151, pp. 61-69. DOI: 10.1016/j.compag.2018.05.012.

Mazzia, V., Khaliq, A., Chiaberge , M. Improvement in land cover and crop classification based on temporal features learning from Sentinel-2 data using recurrent-convolutional neural network (R-CNN). Applied Sciences, 2020, vol. 10, no. 1, article no. 238. DOI: 10.3390/app10010238.

Kussul, N., Lavreniuk, M., Skakun, S., Shelestov, A. Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters, 2017, vol. 14, no. 5, pp. 778-782. DOI: 10.1109/LGRS.2017.2681128.

Yaloveha, V., Hlavcheva, D., Podorozhniak A., Kuchuk, H. Fire hazard research of forest areas based on the use of convolutional and capsule neural networks. 2019 IEEE 2nd Ukraine Conference on Electrical and Computer Engineering (UKRCON), 2019, pp. 828-832. DOI: 10.1109/UKRCON.2019.8879867.

Helber, P., Bischke, B., Dengel, A., Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, vol. 12, no. 7, pp. 2217-2226. DOI: 10.1109/JSTARS.2019.2918242.

Abreu, S. Automated architecture design for deep neural networks. Preprint arXiv, arXiv: 1908.10714, 2019.

Liaw, R., Liang, E., Nishihara,R., Moritz, P. et al. Tune: A research platform for distributed model selection and training. Preprint arXiv, arXiv: 1807.05118, 2018.

Carneiro, T., Da Nóbrega, R. V. M., Nepomuceno, T. Performance analysis of google colaboratory as a tool for accelerating deep learning applications. IEEE Access, 2018, vol. 6, pp. 61677-61685. DOI: 10.1109/ACCESS.2018.2874767.

Prechelt, L. Automatic early stopping using cross validation: quantifying the criteria. Neural Networks, 1998, vol. 11, no. 4, pp. 761-767. DOI: 10.1016/S0893-6080(98)00010-0.

Chong, E. EuroSAT Land Use and Land Cover Classification using Deep Learning. Available online: https://github.com/echong/Remote-Sensing. (Accessed: 20.07.2021).

Hinz, T., Navarro-Guerrero, N., Magg, S., Wermter, S. Speeding up the hyperparameter optimization of deep convolutional neural networks. International Journal of Computational Intelligence and Applications, 2018, vol. 17, no. 02, pp. 1-15.