Development of a partially supervised Markov decision-making model for a 3-link collaborative robot-manipulator

Vladyslav Yevsieiev, Dmytro Gurin, Sergii Kulish, Yuliia Voloshyn

Abstract


The subject are mathematical models of decision-making under uncertainty in a production environment with human presence. The research objectives: is to form a safe and effective policy for controlling the motion of a three-link collaborative robot-manipulator, by developing a mathematical model of a partially observable Markov decision process (POMDP). Methods: methodology of partially observable Markov processes (POMDP), numerical modeling, approximation of the expected reward, comparative analysis of scenarios with different risk parameters. Results: the implemented model is able to form an adaptive policy for the manipulator's behavior taking into account incomplete information about the person's position; the dependence of optimal actions on the probability distribution of the human position and the intensity of the risk penalty is demonstrated; the influence of the difference in rewards between the fast movement and stop modes on the choice of actions is shown. Conclusions: the developed POMDP model can be used as a basis for building high-level adaptive control of a collaborative manipulator in a shared workspace with a human. The proposed approach has the prospect of being implemented in flexible production systems and cyber-physical complexes, in particular, taking into account dynamic risk reassessment and integration with computer vision algorithms.

Keywords


Industry 5.0; POMDP; Collaborative Robot; Robot-Manipulator; Decision-Making; Partial Observation; Human Safety; Risk-Based Policy; Reward Function; Flexible Manufacturing; Robotic Interaction; Adaptive Control; Collaborative Work

Full Text:

PDF

References


Narkhede, G. B., Pasi, B. N., Rajhans, N., & Kulkarni, A. Industry 5.0 and sustainable manufacturing: a systematic literature review. Benchmarking: An International Journal, 2025, vol. 32, iss. 2, pp. 608–635. DOI: 10.1108/BIJ-03-2023-0196.

Nevliudov, I., Yevsieiev, V., Baker, J. H., Ahmad, M. A., & Lyashenko, V. Development of a cyber design modeling declarative language for cyber physical production systems. Journal of Mathematics and Computer Science, 2020, vol. 11, iss. 1, pp. 520–542. DOI: 10.28919/jmcs/5152.

Saleem, Z., Gustafsson, F., Furey, E., McAfee, M., & Huq, S. A review of external sensors for human detection in a human robot collaborative environment. Journal of Intelligent Manufacturing, 2025, vol. 36, iss. 4, pp. 2255–2279. DOI: 10.1007/s10845-024-02341-2.

Nevliudov, I., Yevsieiev, V., Maksymova, S., & Filippenko, I. Development of an architectural-logical model to automate the management of the process of creating complex cyberphysical industrial systems. Eastern-European Journal of Enterprise Technologies, 2020, vol. 4, iss. 3(106), pp. 44–52. DOI: 10.15587/1729-4061.2020.210761.

Ben Hazem, Z., Guler, N., & Altaif, A. H. A study of advanced mathematical modeling and adaptive control strategies for trajectory tracking in the Mitsubishi RV-2AJ 5-DOF robotic arm. Discover Robotics, 2025. vol. 1, iss. 1, article no. 2. DOI: 10.1007/s44430-025-00001-5.

Tang, J., Li, S., & Shi, L. Lie-algebra adaptive tracking control for rigid body dynamics. 2025, arXiv preprint arXiv:2502.05491. DOI: 10.48550/arXiv.2502.05491.

Tinoco, V., Silva, M. F., Santos, F. N., Morais, R., Magalhães, S. A., & Oliveira, P. M. A review of advanced controller methodologies for robotic manipulators. International Journal of Dynamics and Control, 2025, vol. 13, iss. 1, pp. 1–17. DOI: 10.1007/s40435-024-01533-1.

Son, V. N., Van Cuong, P., Minh, N. D., & Nha, P. H. Optimize the parameters of the PID controller using genetic algorithm for robot manipulators. 2025, arXiv preprint, arXiv:2501.04759. DOI: 10.48550/arXiv.2501.04759.

Arcieri, G., Papakonstantinou, K. G., Straub, D., & Chatzi, E. Deep belief Markov models for POMDP inference. 2025, arXiv preprint, arXiv:2503.13438. DOI: 10.48550/arXiv.2503.13438.

Wertheim, O., & Brafman, R. I. Model-based AI planning and execution systems for robotics. 2025, arXiv preprint, arXiv:2505.04493. DOI: 10.48550/arXiv.2505.04493.

Jung, K., & Yang, J. S. Mitigating safety challenges in human-robot collaboration: the role of human competence. Technological Forecasting and Social Change, 2025, vol. 213, article no. 124022. DOI: 10.1016/j.techfore.2025.124022.

Albeaino, G., Jeelani, I., Gheisari, M., & Issa, R. R. Assessing proxemics impact on human-robot collaboration safety in construction: a virtual reality study with four-legged robots. Safety Science, 2025, vol. 181, article no. 106682. DOI: 10.1016/j.ssci.2024.106682.

Qian, L., Hao, L., Cui, S., Gao, X., Zhao, X., & Li, Y. Research on motion trajectory planning and impedance control for dual-arm collaborative robot grinding tasks. Applied Sciences, 2025, vol. 15, iss. 2. DOI: 10.3390/app15020819.

Peta, K., Wiśniewski, M., Kotarski, M., & Ciszak, O. Comparison of single-arm and dual-arm collaborative robots in precision assembly. Applied Sciences, 2025, vol. 15, iss. 6, article no. 2976. DOI: 10.3390/app15062976.

Ma, X., Zhen, S., Meng, C., Liu, X., Meng, G., & Chen, Y. H. Robust approximate constraint‐following control design based on Udwadia–Kalaba theory and experimental verification for collaborative robots with inequality constraints and uncertainties. International Journal of Robust and Nonlinear Control, 2025. DOI: 10.1007/s11071-023-09133-y.

Pey, J. J. J., Samarakoon, S. B. P., Muthugala, M. V. J., & Elara, M. R. A decentralized partially observable Markov decision process for complete coverage onboard multiple shape changing reconfigurable robots. Expert Systems with Applications, 2025, article no. 126565. DOI: 10.1016/j.eswa.2025.126565.

Lepers, S., Thomas, V., & Buffet, O. Observer-aware probabilistic planning under partial observability. 2025, arXiv, preprint arXiv:2502.10568. DOI: 10.48550/arXiv.2502.10568.

Liang, J., Miao, H., Li, K., Tan, J., Wang, X., Luo, R., & Jiang, Y. A review of multi-agent reinforcement learning algorithms. Electronics, 2025, vol. 14, iss. 4, article no. 820. DOI: 10.3390/electronics14040820.

Feng, Z., Xue, R., Yuan, L., Yu, Y., Ding, N., Liu, M., & et al. Multi-agent embodied AI: advances and future directions. 2025, arXiv, preprint arXiv:2505.05108. DOI: 10.48550/arXiv.2505.05108.

Chaabani, A., Cherif, R., & Yaddaden, Y. Automating quality control: real-time defect detection and automated decision-making with AI and Doosan robotics. International Journal of Intelligent Robotics and Applications, 2025, pp. 1-15. DOI: 10.1007/s41315-024-00417-z.

Gargioni, L., Fogli, D., & Baroni, P. Preparation of personalized medicines through collaborative robots: a hybrid approach to the end-user development of robot programs. ACM Journal on Responsible Computing, 2025. DOI: 10.1145/3715852.

Nishat, A. Revolutionizing robotics with AI: deep learning for smart navigation and manipulation. Journal of Big Data and Smart Systems, 2025, vol. 6, iss. 1. DOI: 10.51219/JAIMLD/premkumar-ganesan/263.

Tejada, J. C., Toro-Ossaba, A., López-Gonzalez, A., Hernandez-Martinez, E. G., & Sanin-Villa, D. A review of multi-robot systems and soft robotics: challenges and opportunities. Sensors, 2025, vol. 25, iss. 5, article no. 1353. DOI: 10.3390/s25051353.

Ma, W., Duan, A., Lee, H. Y., Zheng, P., & Navarro-Alarcon, D. Human-aware reactive task planning of sequential robotic manipulation tasks. IEEE Transactions on Industrial Informatics, 2025. DOI: 10.1109/TII.2024.3514130.

Zhao, X., Chen, P., & Tang, L. C. Condition-based maintenance via Markov decision processes: a review. Frontiers of Engineering Management, 2025, pp.1-13. DOI: 10.1007/s42524-024-4130-7.

Mustafa, S. K., Yevsieiev, V., Nevliudov, I., & Lyashenko, V. HMI development automation with GUI elements for object-oriented programming languages implementation. International Journal of Engineering Trends and Technology (IJETT), 2022, vol. 70, iss. 1, pp. 139–145. DOI: 10.14445/22315381/IJETT-V70I1P215.

Rokita, M., Modrzejewski, M., & Rokita, P. PyBrook –A Python framework for processing and visualising real-time data. SoftwareX, 2025, vol. 30, article no. 102116. DOI: 10.1016/j.softx.2025.102116.

Rouhandeh, H., & Behroozsarand, A. Simulation and optimization of methanol production process via bi-reforming of methane: a novel genetic algorithm-based approach in Python. International Journal of Hydrogen Energy, 2025, vol. 101, pp. 1161–1171. DOI: 10.1016/j.ijhydene.2025.01.00.




DOI: https://doi.org/10.32620/reks.2025.4.06

Refbacks

  • There are currently no refbacks.