Markov modelling of human-machine interaction in an augmented reality environment for UAV/UGV-based hazardous area monitoring systems

Yevhenii Kanarskyi, Vyacheslav Kharchenko, Oleksandr Orekhov, Yuriy Ponochovnyi

Abstract


The subject of the study is Markov processes used for the formal description of the dynamics of states of unmanned vehicles controlled by an operator through augmented reality–based human–machine interfaces. Within the scope of the research, unmanned aerial and ground vehicles are considered as complex multi-state technical systems whose functioning is determined both by their technical characteristics and by the specific features of human interaction with the control interface. The aim of the study is to assess the impact of augmented reality–based human–machine interaction interfaces on the error-free decision-making of unmanned system operators, as well as on their responsiveness during control and reaction to changes in system states. The objectives of this study are to develop Markov models for the following scenarios: (a) without considering system failures and operator errors, with full recovery; (b) without considering failures, but allowing for operator errors, with full recovery; (c) considering system failures without operator errors, with full recovery; (d) considering both system failures and operator errors, with full recovery; and (e) without considering failures, allowing for operator errors, with the presence of a redundant unmanned system. The resulting Markov chains are intended to be used for modelling and subsequent comparison of the impact of different operating conditions on the system. As a result of the study, the following were obtained: (a) a classifier of states of unmanned aerial vehicles within a hazardous environment monitoring system based on the possible presence of failures, operator errors, and system redundancy; (b) Markov models for various system operation scenarios; and (c) simulation results of system operation based on the developed Markov models. Conclusion. The scientific novelty is as follows: a method for assessing the availability of monitoring systems with augmented reality–based human–machine interaction interfaces is proposed, which is based on single- and multi-fragment Markov models that take into account operator actions, partial failures, and the availability of reserve unmanned aerial vehicles. The proposed method quantitatively evaluates the impact of augmented reality not only on subjective indicators but also on the overall system availability and reliability indicators.

Keywords


augmented reality; unmanned aerial vehicles; Markov chains; human-machine interaction

Full Text:

PDF

References


Zatserkovnyі, V., Tsiupa, I., De Donatis, M., Nikoliuk, I., Kravchenia, V., Tsvyk, O., & Mironchuk, T. Methods to detect explosive hazards in agricultural areas. Visnyk of Taras Shevchenko National University of Kyiv. Geology, 2025, vol. 3, no. 110, pp. 127-138. DOI: 10.17721/1728-2713.110.14.

Hameed, Q. A., Hussein, H. A., Ahmed, M. A., Salih, M. M., Ismael, R. D., & Omar, M. B. UXO-AID: A New UXO Classification Application Based on Augmented Reality to Assist Deminers. Computers, 2022, vol. 11, iss. 8, article no. 124. DOI: 10.3390/computers11080124.

Blachnik, M., Przyłucki, R., Golak, S., Ściegienka, P., & Wieczorek, T. On the Development of a Digital Twin for Underwater UXO Detection Using Magnetometer-Based Data in Application for the Training Set Generation for Machine Learning Models. Sensors, 2023, vol. 23, iss. 15, article no. 6806. DOI: 10.3390/s23156806.

Jefry, N. F. S., & Rambli, D. R. A. A review of augmented reality systems and their effects on mental workload and task performance. Heliyon, 2021, vol. 7, iss. 3, article no. e06277. DOI: 10.1016/j.heliyon.2021.e06277.

Fesenko, H., Illiashenko, O., Kharchenko, V., Kliushnikov, I., Morozova, O., Sachenko, A., & Skorobohatko, S. Flying sensor and edge network-based advanced air mobility systems: reliability analysis and applications for urban monitoring. Drones, 2023, vol. 7, iss. 7, article no. 409. DOI: 10.3390/drones7070409.

Fedorenko, G., Fesenko, H., Kharchenko, V., Kliushnikov, I., & Tolkunov, I. Robotic-biological systems for detection and identification of explosive ordnance: concept, general structure, and models. Radioelectronic and Computer Systems, 2023, no. 2, pp. 143-159. DOI: 10.32620/reks.2023.2.12.

Sharma, S., Muley, A., Singh, R., & Gehlot, A. UAV for surveillance and environmental monitoring. Indian Journal of Science and Technology, 2016, vol. 9, iss. 43, pp. 1-4. DOI: 10.17485/ijst/2016/v9i43/104396.

Misse, E. S., Villacrés, S. A., Velasco, P. M., & Andaluz, V. H. Augmented reality system for the assistance of unmanned aerial vehicles. 15th Iberian Conference on Information Systems and Technologies (CISTI), Seville, Spain, IEEE, 2020, pp. 1-6. DOI: 10.23919/CISTI49556.2020.9140958.

Ruano, S., Cuevas, C., Gallego, G., & García, N. Augmented reality tool for the situational awareness improvement of UAV operators. Sensors, 2017, vol. 17, iss. 2, article no. 297. DOI: 10.3390/s17020297.

Mutzenich, C., Durant, S., Helman, S., & Dalton P. Updating our understanding of situation awareness in relation to remote operators of autonomous vehicles. Cognitive Research: Principles and Implications, 2021, vol. 6, article no. 9. DOI: 10.1186/s41235-021-00271-8.

Kalatzis, A., Prabhu, V. G., Stanley, L., & Wittie, M. P. Effect of augmented reality user interface on task performance, cognitive load, and situational awareness in human–robot collaboration. 32nd IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Busan, Republic of Korea, IEEE, 2023, pp. 1252-1259. DOI: 10.1109/RO-MAN57019.2023.10309468.

Seeling, P. Augmented reality device operator cognitive strain determination and prediction. AIMS Electronics and Electrical Engineering, 2017, vol. 1, iss. 1, pp. 100-110. DOI: 10.3934/ElectrEng.2017.1.100.

Kaufeld, M., Mundt, M., Forst, S., & Hecht, H. Optical see-through augmented reality can induce severe motion sickness. Displays, 2022, vol. 74, article no. 102283. DOI: 10.1016/j.displa.2022.102283.

Costa, C., Gomes, E., Rodrigues, N., Gonçalves, A., Ribeiro, R., Costa, P., & Pereira, A. Augmented reality mobile digital twin for unmanned aerial vehicle wildfire prevention. Virtual Reality, 2025, vol. 29, article no. 71. DOI: 10.1007/s10055-025-01145-w.

Sautenkov, O., Asfaw, S., Yaqoot, Y., Mustafa, M. A., Fedoseev, A., Trinitatova, D., & Tsetserukou, D. FlightAR: AR flight assistance interface with multiple video streams and object detection aimed at immersive drone control. IEEE International Conference on Robotics and Biomimetics (ROBIO), Bangkok, Thailand, IEEE, 2024, pp. 614-619. DOI: 10.1109/ROBIO64047.2024.10907428.

Bagassi, S., Fadda, T., & Corsi, M. Advanced human machine interfaces for drone monitoring: assessment of the technological framework for the design of an augmented reality interface. Available at: https://www.icas.org/icas_archive/icas2024/data/papers/icas2024_1059_paper.pdf (accessed: 27 September 2025).

Azuma, R.T. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 1997, vol. 6, iss. 4, pp. 355-385. DOI: 10.1162/pres.1997.6.4.355.

Mugruza-Vassallo, C. A., Granados-Domínguez, J. L., Flores-Benites, V., & Córdova Berríos, L. L. Different Markov chains modulate visual stimuli processing in a Go–Go experiment in 2D, 3D, and augmented reality. Frontiers in Human Neuroscience, 2022, vol. 16, pp. 1-13. DOI: 10.3389/fnhum.2022.955534.

Kliushnikov, I., Fesenko, H., Fedorenko, G., Rudakov, S., Mikhalevskyi, V., & Kompaniiets, O. Swarm of unmanned aerial vehicles as a multistate queueing system with non-controlled and controlled degradation. 12th International Conference on Dependable Systems, Services and Technologies (DESSERT), Athens, Greece, IEEE, 2022, pp. 1-7. DOI: 10.1109/DESSERT58054.2022.10018784.

Rodríguez-Fernández, V., Gonzalez-Pardo, A., & Camacho, D. Finding behavioral patterns of UAV operators using multichannel hidden Markov models. IEEE Symposium Series on Computational Intelligence (SSCI), Athens, Greece, 2016, IEEE, pp. 1-8. DOI: 10.1109/SSCI.2016.7850101.

Abakumov, A., Kharchenko, V., & Ponochovnyi, Y. UAV Cyber Resilience Assessment Method: Combining IMECA, Penetration Testing and State-space Markov Modeling. International Journal of Computing, 2025, vol. 24, iss. 4, pp. 790-801. DOI: 10.47839/ijc.24.4.4346.

Kharchenko, V., Kliushnikov, I., Rucinski, A., Fesenko, H., & Illiashenko, O. UAV Fleet as a Dependable Service for Smart Cities: Model-Based Assessment and Application. Smart Cities, 2022, vol. 5, iss. 3, pp. 1151-1178. DOI: 10.3390/smartcities5030058.

Kabashkin, I., Iskakov, D., Topilskiy, R., Tlepiyeva, G., Sultanov, T., & Sansyzbayeva, Z. Communication Infrastructure Design for Reliable UAV Operations in Air Mobility Corridors. Drones, 2025, vol. 9, iss. 6, article no. 401. DOI:10.3390/drones9060401.

Kliushnikov, I. Safety and security assessment of unmanned aerial vehicles application using Markov models. Systemy ozbroiennia i viiskova tekhnika – Systems of Arms and Military Equipment, 2023, no. 4(76), pp. 51-57. DOI: 10.30748/soivt.2023.76.05. (In Ukrainian).

Kharchenko, V., Ponochovnyi, Y., Ivanchenko, O., Fesenko, H., & Illiashenko, O. Combining Markov and Semi-Markov Modelling for Assessing Availability and Cybersecurity of Cloud and IoT Systems. Cryptography, 2022, vol. 6, iss. 3, article no. 44. DOI: 10.3390/cryptography6030044.




DOI: https://doi.org/10.32620/reks.2025.4.03

Refbacks

  • There are currently no refbacks.