Development of software for data segmentation by photo and video information

Dmytro Kritskiy, Nadiya Shkurenko, Oleksii Popov, Olexandra Kravtsova

Abstract


The object of this research is the process of perception of individual objects in photos and videos for further analysis of the situation in the city. The subject of the research is image processing for selection, classification, and further use of the obtained information about the objects in the photo. The goal is to create information technology for the segmentation of individual segments in photos and videos obtained with the help of unmanned aerial vehicles (in particular, drones) in urban scenes. Main tasks: analysis of existing methods of data segmentation; implementation of an algorithm that would perform segmentation of data by various terrain objects; software testing; and formation of research results. The following results were obtained: existing models of image segmentation were examined for limitations and shortcomings. On the basis of identified shortcomings, the requirements for the developed system were formed. During the analysis of the subject area, the main problems were identified and described and the task to be automated was determined. During the design of algorithms for solving problems, the sequence of the execution of each process was determined. The design of the software included a description of the information space of the system and the user interface. The user interface was chosen after analyzing the main types of user interfaces. To assess the quality of the data processing, existing metrics were used – IoU, pixel accuracy, and the discrepancy matrix. During testing, errors that occur during data segmentation and their causes were determined, after their elimination, the correct result of image segmentation was shown. Conclusions. The scientific novelty of the obtained results is that although there is still no segmentation method that fully satisfies all needs for the analysis of the category of images obtained by drones, the developed software allows uploading images and templates to them as benchmarks for artificial intelligence. perform segmentation and verify the correctness of this segmentation visually and using the IoU metric, the mismatch matrix. The software can be improved by automating the process of creating a reference pattern, training the model on more data, and integrating the created software into a larger system that would work directly with the drone.

Keywords


semantic segmentation; algorithm; metric; UAV; template; mismatch matrix; software

References


Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Benenson, R., Franke, U., Roth, S. & Shiele, B. The cityscapes dataset for semantic urban scene understanding. arXiv, 2016, article no. 1604.01685, pp. 1-29. DOI: 10.48550/arXiv.1604.01685.

Brostow, G. J., Shotton, J., Fauqueur, J. & Cipolla, R. Segmentation and recognition using structure from motion point clouds. Computer Vision – ECCV 2008. ECCV 2008. Lecture Notes in Computer Science, Springer, Berlin, Heidelberg, 2008, vol. 5302, pp. 44-57. DOI: 10.1007/978-3-540-88682-2_5.

Byungju, K., Junho, Y. & Junmo, K. Highway driving dataset for semantic video segmentation. arXiv, 2011, article no. 2011.00674, pp. 1-12. DOI: 10.48550/arXiv.2011.00674.

Pohudina, O., Kritskiy, D., Bykov, A. N. & Szalay, T. Method for Identifying and Counting Objects. Integrated Computer Technologies in Mechanical Engineering. Advances in Intelligent Systems and Computing, Springer, Cham, 2020, vol. 1113, pp. 161-172. DOI: 10.1007/978-3-030-37618-5_15.

Topchiyev, B. S. Alhorytmichno-prohramnyy metod koloryzatsiyi zobrazhen'. Mahistersʹka dys. [Algorithmic-program method of image colorization. Master's thesis]. Kiyv, 2020. 148 p. Available at: https://ela.kpi.ua/handle/123456789/33832 (accessed March 29, 2023).

Chyzh, I. H. & Lisnyak, I. H. Trykomponentni linzovi zum-afokal'ni systemy transfokatoriv [Three-component lens zoom-afocal zoom systems]. Naukovi visti KPI – Scientific news of KPI, 2019, no. 3, pp. 73-79.

Pipko, A. S. Intelektual'na systema fotorealistychnoho perenesennya styliv mizh zobrazhennyamy. Mahistersʹka dys. [Intelligent system of photorealistic transfer of styles between images. Master's thesis]. Kiyv, 2018. 72 p. Available at: https://ela.kpi.ua/handle/123456789/23756 (accessed March 29, 2023).

Patel, M. J., Kothari, A. M. & Koringa, H. P. A novel approach for semantic segmentation of automatic road network extractions from remote sensing images by modified UNet. Radioelektronni i komp'uterni sistemi – Radioelectronic and computer systems, 2022, no. 3, pp. 161-173. DOI: 10.32620/reks.2022.3.12.

Vasil'eva, I. K. & Popov, А. V. Metod avtomaticheskoi klasterizatsii dannykh distantsionnogo zondirovaniya [Remote sensing data automatic clustering method]. Aviacijno-kosmicna tehnika i tehnologia – Aerospace technic and technology, 2019, no. 3 (155), pp. 64-75. DOI: 10.32620/aktt.2019.3.08.

Lin, Ts., Marie, M., Belongie, S., Bourdev, L. & Girshick, R. Microsoft COCO: Common Objects in Context. arXiv, 2015, article No. 1405.0312, pp. 1-15. DOI: 10.48550/arXiv.1405.0312.




DOI: https://doi.org/10.32620/aktt.2023.3.07