Analysis of the impact of the contextual embeddings usage on the text classification accuracy

Olesia Barkovska, Anton Havrashenko, Vitalii Serdechnyi, Vladyslav Kholiev, Patrik Rusnak

Abstract


This study aims to improve the accuracy of text classification, which is critical in fields such as medical diagnostics and law. In addition, accuracy requirements are increasing constantly with the development of information technologies and increasing volume of text data. The subject of this paper is to study the impact of text vectorization methods on the accuracy of text data classification. The goal of this paper is to evaluate the effectiveness of different word vectorization methods (Word2Vec, GloVe, BERT, and GPT) in the context of text classification based on different embedding strategies - Word and Contextual Embedding. The primary focus was on studying the effect of the number of training epochs (by systematically increasing the number of training epochs) on text classification accuracy. The task of this study was to systematically compare the effectiveness of each type of embedding following the formed matrix of experiments, which controls the equality of the conditions of the experiment, and to further evaluate the key metrics of text classification (on the example of the IMDB dataset) using a neural network classifier (LSTM) with a recurrent architecture. Machine learning methods, including neural network methods, methods of vector representation of words, and statistical analysis, were used in this study. The results demonstrate that the best Word Embedding model was GloVe, which demonstrated a final accuracy of 87.73%. In the context of Contextual Embedding, BERT proved to be more effective than GPT, with a final accuracy of 92.97% compared to 91.65% of GPT. In general, the results demonstrate the superiority of Contextual Embedding in natural language processing tasks and confirm its potential use in modern applications and text analysis systems. Conclusions. The results demonstrate that no universal model is suitable for all types of NLP tasks. It is important to choose an embedding method that matches the specific task, available resources, and specific research goals. The results of this study can be extended to other NLP tasks, such as tone analysis, named entity detection, and machine translation.

Keywords


classification; NLP; context; model; neural network; Word2Vec; GloVe; embedding; BERT; GPT

Full Text:

PDF

References


Chowdhary, K. R. Natural Language Processing. In: Fundamentals of Artificial Intelligence. Springer. New Delhi, 2020, pp. 603-649. DOI: 10.1007/978-81-322-3972-7_19

Khurana, D., Koli, A., Khatter, K., & Singh, S. Natural language processing: state of the art, current trends and challenges. Multimed Tools Appl., 2023, vol. 82, pp. 3713–3744. DOI: 10.1007/s11042-022-13428-4.

Habib, M. A., Ema, R. R., Islam, T., Arafat, M. Y., & Hasan, M. Automatic text summarization based on extractive-abstractive method. Radioelectronic and Computer Systems, 2023, no. 2, pp. 5-17. DOI: 10.32620/reks.2023.2.01.

Patil, R., Boit, S., Gudivada, V., & Nandigam, J. A Survey of Text Representation and Embedding Techniques in NLP. IEEE Access, 2023, vol. 11. pp. 36120-36146. DOI: 10.1109/ACCESS.2023.3266377.

Birunda, S. S., & Devi, R. K. A Review on Word Embedding Techniques for Text Classification. In: Raj, J. S., Iliyasu, A. M., Bestak, R., Baig, Z. A. (eds) Innovative Data Communication Technologies and Application. Lecture Notes on Data Engineering and Communications Technologies, Springer, Singapore, 2021, vol. 59, pp. 267-281. DOI: 10.1007/978-981-15-9651-3_23.

Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., & Gao. J. Deep learning--based text classification: a comprehensive review. ACM computing surveys, 2021, vol. 54, iss. 3, article no. 62, pp. 1-40. DOI: 10.1145/3439726.

Dogra, V., Verma, S., Kavita, Chatterjee, P., Shafi, J., Choi, J., & Ijaz, M. F. A complete process of text classification system using state-of-the-art NLP models. Computational Intelligence and Neuroscience, 2022, vol. 2022, article no. 1883698. 26 p. DOI: 10.1155/2022/1883698.

Rakib Mollah, M. A., Jahangir Kabir, M. M., Reza, M. S., & Kabir, M. Adapting Contextual Embedding to Identify Sentiment of E-commerce Consumer Reviews with Addressing Class Imbalance Issues. 2024 International Conference on Advances in Computing. Communication. Electrical. and Smart Systems (iCACCESS), Dhaka, Bangladesh, 2024, pp. 1-6. DOI: 10.1109/iCACCESS61735.2024.10499554.

Lavanya, P., & Sasikala, E. Deep Learning Techniques on Text Classification Using Natural Language Processing (NLP) In Social Healthcare Network: A Comprehensive Survey. 2021 3rd International Conference on Signal Processing and Communication (ICPSC), Coimbatore, India, 2021, pp. 603-609. DOI: 10.1109/ICSPC51351.2021.9451752.

Spasic, I., Nenadic, G. Clinical Text Data in Machine Learning: Systematic Review. JMIR Med Inform, 2020; vol. 8, iss. 3, article no. e17984. DOI: 10.2196/17984.

Gupta, A., Dengre, V., Kheruwala, H. A., & Shah, M. Comprehensive review of text-mining applications in finance. Financ Innov, 2020, vol. 6, article no. 39. DOI: 10.1186/s40854-020-00205-1.

Lan, L., Huang, T., Li, Y., Song, Y. A Survey of Cross-Lingual Text Classification and Its Applications on Fake News Detection. World Scientific Annual Review of Artificial Intelligence, 2023, vol. 1, article no. 2350003. DOI: 10.1142/S2811032323500030.

Padalko, H., Chomko, V., Yakovlev, S., & Chumachenko, D. Ensemble machine learning approaches for fake news classification. Radioelectronic and Computer Systems, 2023, no. 4, pp. 5-19. DOI: 10.32620/reks.2023.4.01.

Li, R., Liu, M., Xu, D., Gao, J., Wu, F., & Zhu, L. A review of machine learning algorithms for text classification. In: Lu, W., Zhang, Y., Wen, W., Yan, H., Li, C. (eds) Cyber Security. CNCERT 2021. Communications in Computer and Information Science, Springer, Singapore, 2022, vol. 1506, pp. 226-234. DOI: 10.1007/978-981-16-9229-1_14.

Zhou, Y. A review of text classification based on deep learning. 3rd international conference on geoinformatics and data analysis, New York, USA, Association for Computing Machinery Publ., 2020, pp. 132-136. DOI: 10.1145/3397056.3397082.

Nandwani, P., & Verma, R. A review on sentiment analysis and emotion detection from text. Soc. Netw. Anal. Min., 2021, vol. 11, article no. 81. DOI: 10.1007/s13278-021-00776-6.

Pandya, S., & Mehta, P. A review on sentiment analysis methodologies, practices and applications. International Journal of Scientific and Technology Research, 2020, vol. 9, iss. 2, pp. 601-609. Available at: https://www.researchgate.net/publication/344487215_A_Review_On_Sentiment_Analysis_Methodologies_Practices_And_Applications (accessed 12/02/2024).

Barkovska, O. Performance study of the text analysis module in the proposed model of automatic speaker’s speech annotation. Computer systems and information technologies, 2022, no. 4, pp. 13-19. DOI: 10.31891/csit-2022-4-2.

Barkovska, O., Kholiev, V., Havrashenko, A., Mohylevskyi, D. & Kovalenko,A. A Conceptual Text Classification Model Based on Two-Factor Selection of Significant Words //COLINS (2). – 2023. – С. 244-255.

Barkovska, O., Khomych, V., & Nastenko, O. Research of the text processing methods in organization of electronic storages of information objects. Innovative technologies and scientific solutions for industries, 2022, no. 1(19), pp. 5-12. DOI: 10.30837/ITSSI.2022.19.005.




DOI: https://doi.org/10.32620/reks.2024.3.05

Refbacks

  • There are currently no refbacks.