Using the proximal policy optimization and prospect theory to train a decision-making model for managing personal finances

Vladyslav Didkivskyi, Dmytro Antoniuk, Tetiana Vakaliuk, Yevhen Ohinskyi

Abstract


The subject of this article is the development of a decision-making model that can, in the future, be incorporated into a personal finance simulator to improve personal finance literacy. The goal of this study is to develop decision-making models tailored to different investor profiles to provide personalized financial advice on asset allocation. This article employs reinforcement learning techniques and behavioral economics to achieve this objective, thereby contributing to the advancement of practical algorithms and approaches for financial decision-making. The tasks can be formulated as follows: 1) design a reinforcement learning environment featuring different investment options with varying average returns and volatility levels; 2) train the reinforcement learning agent using the Proximal Policy Optimization algorithm to learn recommended investment allocations; 3) implement a reward function based on Prospect Theory, incorporating parameters that reflect different investor risk profiles, such as loss aversion and diminishing sensitivity to gains and losses. The results reveal the development of distinct models for 3 investor profiles: risk-averse, rational, and wealth-maximizing. A graphical analysis of the recommended allocation percentages revealed significant patterns influenced by the value function parameters of Prospect Theory. The practical implications of this research extend to the development of simulation tools based on the model, which will enable individuals to practice and refine their financial strategies in a risk-free environment. These tools bridge the gap in personal finance education by providing experiential learning opportunities. Conclusions. The developed model effectively generates personalized financial advice that reflects individual risk preferences. Future work will focus on creating interactive simulation tools to enhance personal finance management skills. This study underscores the importance of integrating psychological and behavioral insights into financial decision-making models.

Keywords


personal finance; decision-making; reinforcement learning; prospect theory

Full Text:

PDF

References


Thomas, A., & Gupta, V. Social Capital Theory, Social Exchange Theory, Social Cognitive Theory, Financial Literacy, and the Role of Knowledge Sharing as a Moderator in Enhancing Financial Well-Being: From Bibliometric Analysis to a Conceptual Framework Model. Frontiers in Psychology, 2021, vol. 12. DOI: 10.3389/fpsyg.2021.664638.

Antoniuk, D. S., Vakaliuk, T. A., Didkivskyi, V. V., & Vizghalov, O. Development of a simulator to determine personal financial strategies using machine learning. In: Proceedings of the 4th Workshop for Young Scientists in Computer Science & Software Engineering (CS&SE@SW 2021), 2022, Virtual Event, Kryvyi Rih, Ukraine, December 18, 2021. Edited by Arnold E. Kiv, Serhiy O. Semerikov, Vladimir N. Soloviev, Andrii M. Striuk. CEUR Workshop Proceedings (CEUR-WS.org, ISSN 1613-0073). Vol. 3077, pp. 12-26. Available at: http://ceur-ws.org/Vol-3077/paper02.pdf. (accessed 12.08.2024).

Xu, S., Yang, Z., Ali, S. T., Li, Y., & Cui, J. Does Financial Literacy Affect Household Financial Behavior? The Role of Limited Attention. Frontiers in Psychology, 2022, vol. 13. DOI: 10.3389/fpsyg.2022.906153.

Corecco, S., Adorni, G., & Gambardella, L. M. Proximal Policy Optimization-Based Reinforcement Learning and Hybrid Approaches to Explore the Cross Array Task Optimal Solution. Machine Learning and Knowledge Extraction, 2023, vol. 5, iss. 4, pp.1660–1679. DOI: 10.3390/make5040082.

Lobanchykova, N. M., Pilkevych, I. A., & Korchenko, O. Analysis and protection of IoT systems: Edge computing and decentralized decision-making. Journal of Edge Computing, 2022, vol. 1, iss. 1, pp. 55–67. DOI: 10.55056/jec.573.

Ruggeri, K., Alí, S., Berge, M. L., Bertoldo, G., Bjørndal, L. D., Cortijos-Bernabeu, A., Davison, C., Demić, E., Esteban-Serna, C., Friedemann, M., Gibson, S. P., Jarke, H., Karakasheva, R., Khorrami, P. R., Kveder, J., Andersen, T. L., Lofthus, I. S., McGill, L., Nieto, A. E., & Pérez, J. Replicating patterns of prospect theory for decision under risk. Nature Human Behaviour, 2020, vol. 4, iss. 6, pp.622–633. DOI: 10.1038/s41562-020-0886-x.

Lusardi, A., & Mitchell, O. S. The Importance of Financial Literacy: Opening a New Field. Journal of Economic Perspectives, 2023, vol. 37, iss. 4, pp.137–154. DOI: 10.1257/jep.37.4.137.

Koskelainen, T., Kalmi, P., Scornavacca, E., & Vartiainen, T. Financial literacy in the digital age – A research agenda. Journal of Consumer Affairs, 2023, vol. 57, iss. 1, pp.507-528. DOI: 10.1111/joca.12510.

Lin, S. -Y., & Beling, P. A. An end-to-end optimal trade execution framework based on proximal policy optimization. Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, 2021, pp. 4548-4554. DOI: 10.24963/ijcai.2020/627.

Yang, H., Liu, X.-Y., Zhong, S., & Walid, A. Deep reinforcement learning for automated stock trading. Proceedings of the First ACM International Conference on AI in Finance, 2020, vol. 31, pp. 1-8. DOI: 10.1145/3383455.3422540.

Cabedo-Peris, J., González-Sala, F., Merino-Soto, C., Pablo, J. Á. C., & Toledano-Toledano, F. Decision Making in Addictive Behaviors Based on Prospect Theory: A Systematic Review. Healthcare (Basel, Switzerland), 2022, vol. 10, iss. 9, article no. 1659. DOI: 10.3390/healthcare10091659.

Shrader, R. C., Simon, M., & Stanton, S. Financial forecasting and risky decisions: an experimental study grounded in Prospect theory. International Entrepreneurship and Management Journal, 2021, vol. 17, pp. 1827–1841. DOI: 10.1007/s11365-020-00697-4.

Srivastava, S., Aggarwal, A., & Bansal, P. Efficiency Evaluation of Assets and Optimal Portfolio Generation by Cross Efficiency and Cumulative Prospect Theory. Computational Economics, 2024, vol. 63, pp.129–158. DOI: 10.1007/s10614-022-10334-7.

Wang, X., Wang, B., Li, T., Li, H., & Watada, J. Multi-criteria fuzzy portfolio selection based on three-way decisions and cumulative prospect theory. Applied Soft Computing, 2023, vol. 134, article no. 110033. DOI: 10.1016/j.asoc.2023.110033.

Kumar Shakya, A., Pillai, G., & Chakrabarty, S. Reinforcement Learning Algorithms: A brief survey. Expert Systems with Applications, 2023, vol. 231, article no. 120495. DOI: 10.1016/j.eswa.2023.120495.

Brown, S., Sinha, S., & Schaefer, A. J. Markov decision process design: A framework for integrating strategic and operational decisions. Operations Research Letters, 2024, vol. 54, article no. 107090. DOI: 10.1016/j.orl.2024.107090.

Wang, J., Wu, C., & Zhong, X. Prospect theory and stock returns: Evidence from foreign share markets. Pacific-Basin Finance Journal, 2021, vol. 69, article no. 101644. DOI: 10.1016/j.pacfin.2021.101644.

Gisbert-Pérez, J., Martí-Vilar, M., & González-Sala, F. Prospect Theory: A Bibliometric and Systematic Review in the Categories of Psychology in Web of Science. Healthcare, 2022, vol. 10, iss. 10, article no. 2098. DOI: 10.3390/healthcare10102098.

Li, Y., Liu, P., & Wu, X. Failure mode and effect analysis approach considering risk attitude of dynamic reference point cumulative prospect theory in uncertainty contexts. Artificial intelligence review, 2023, vol. 56, iss. 12, pp.14557–14604. DOI: 10.1007/s10462-023-10501-8.

Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., & Dormann, N. Stable-Baselines3: Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 2021, vol. 22, iss. 268, pp. 1–8. Available at: https://jmlr.org/papers/v22/20-1364.html. (accessed 12.08.2024).

Fahad Mon, B., Wasfi, A., Hayajneh, M., Slim, A., & Abu Ali, N. Reinforcement Learning in Education: A Literature Review. Informatics, 2023, vol. 10, iss. 3, article no.74. DOI: 10.3390/informatics10030074.

van Dolder, D., & Vandenbroucke, J. Behavioral Risk Profiling: Measuring Loss Aversion of Individual Investors. Journal of Banking and Finance, 2024, article no. 107293. DOI: 10.2139/ssrn.4199169.




DOI: https://doi.org/10.32620/reks.2024.4.20

Refbacks

  • There are currently no refbacks.