Face autistic classification based on thermal using image ensemble learning of VGG-19, ResNet50v2, and EfficientNet

Junidar Junidar, Melinda Melinda, Dinda D. Diannuari, Donata D. Acula, Zulfan Zainal

Abstract


The subject of this paper is the detection of Autism Spectrum Disorder (ASD) traits using facial recognition based on thermal images. The goal of this study was to evaluate and compare the performance of various Convolutional Neural Network (CNN) architectures in classifying thermal facial images of children with ASD, thereby facilitating the early identification of autistic traits. The tasks addressed include preprocessing a dataset of thermal facial images to prepare them for model training; conducting classification using three CNN architectures VGG-19, ResNet50V2, and EfficientNet; and assessing their performance based on accuracy, precision, recall, and F1-score metrics. The methods employed involved training these CNN models on a balanced dataset of 4,120 thermal facial images and splitting them into training, validation, and test sets. Each model underwent extensive training to determine its ability to effectively classify autism and non-autism classes. The results revealed that ResNet50V2 achieved the highest accuracy of 98.82%, followed by VGG-19 and EfficientNet with accuracies of 96.47% and 96.07%, respectively. ResNet50V2 also demonstrated superior generalizability due to its lower validation loss and higher classification accuracy compared to other architectures. Conclusion. The scientific novelty lies in: 1) introducing thermal imaging as an effective tool for detecting ASD traits; 2) demonstrating the superior performance of ResNet50V2 in classifying thermal facial images with high accuracy and generalization; and 3) exploring EfficientNet for the first time in this domain, highlighting its potential for improving autism diagnostic systems. This study contributes to advancing noninvasive methods for ASD detection and paves the way for further applications of deep learning in clinical diagnostics.

Keywords


Autism Spectrum Disorder (ASD); Thermal Image; EfficientNet; ResNet50V2; VGG-19

Full Text:

PDF

References


Chiarotti, F., & Venerosi, A. Epidemiology of Autism Spectrum Disorders: A Review of Worldwide Prevalence Estimates Since 2014. Brain Sciences, 2020, vol. 10, no. 5, article no. 274. DOI: /10.3390/brainsci10050274.

Melinda, M., Juwono, F. H., Enriko, I. K. A., Oktiana, M., Mulyani, S., & Saddami, K. Application of Continuous Wavelet Transform and Support Vector Machine for Autism Spectrum Disorder Electroencephalography Signal Classification. Radioelectronic and Computer Systems, 2023, vol. 3, no. 107, pp. 73–90. DOI: 10.32620/reks.2023.3.07.

Afrin, M. F. N., Freeda, S., Elakia, S., & Kannan, P. AI Based Facial Expression Recognition for Autism Children. International Journal of Emerging Technology and Innovative Engineering, 2019, vol. 5, iss. 9, pp. 645–651. Available at: https://ssrn.com/abstract=3446639. (accessed 11.11.2024)

Lyu, K., Li, J., Chen, M., Li, W., Zhang, W., Hu, M., Zhang, Y., & Feng, X. A Bibliometric Analysis of Autism Spectrum Disorder Signaling Pathways Research in the Past Decade. Frontiers in Psychiatry, 2024, vol. 15, article no. 1304916. DOI: 10.3389/fpsyt.2024.1304916.

Alkahtani, H., Aldhyani, T. H. H., & Alzahrani, M. Y. Deep Learning Algorithms to Identify Autism Spectrum Disorder in Children-Based Facial Landmarks. Applied Sciences, 2023, vol. 13, no. 8, 4855 p. DOI: 10.3390/app13084855.

Goulart, C., Valadão, C., Delisle-Rodriguez, D., Caldeira, E., & Bastos, T. Emotion Analysis in Children Through Facial Emissivity of Infrared Thermal Imaging. PLoS ONE, 2019, vol. 14, no. 3, article no. e0212928. DOI: 10.1371/journal.pone.0212928.

Rusli, N., Sidek, S. N., Yusof, H. M., Ishak, N. I., Khalid, M., & Dzulkarnain, A. A. A. Implementation of Wavelet Analysis on Thermal Images for Affective States Recognition of Children with Autism Spectrum Disorder. IEEE Access, 2020, vol. 8, pp. 120818–120834. DOI: 10.1109/ACCESS.2020.3006004.

Ramasamy, M. P., Krishnasamy, V., & Ramapackiam, S. S. K. Transfer Learning Based Convolutional Neural Network for Classification of Remote Sensing Images. Advances in Electrical and Computer Engineering, 2023, vol. 23, iss. 4, pp. 31–40. DOI: 10.4316/AECE.2023.04004.

Alamri, J., Harrabi, R., & Ben Chaabane, S. Face Recognition based on Convolution Neural Network and Scale Invariant Feature Transform. International Journal of Advanced Computer Science and Applications, 2021, vol. 12, no. 2, pp. 644–654. DOI: 10.14569/IJACSA.2021.0120281.

Teoh, K. H., Ismail, R. C., Naziri, S. Z. M., Hussin, R., Isa, M. N. M., & Basir, M. S. S. M.. Face Recognition and Identification Using Deep Learning Approach. Journal of Physics: Conference Series, 2021, vol. 1755, no. 1, article no. 012006. DOI: 10.1088/1742-6596/1755/1/012006.

Melinda, M., Ahmadiar, A., Oktiana, M., Nugraha, M. S. A., Qadrillah, M. A. L., & Yunidar, Y. A Novel Autism Spectrum Disorder Children Dataset Based on Thermal Imaging. 2023 9th International Conference on Computer and Communication Engineering (ICCCE). Kuala Lumpur, Malaysia, IEEE, 2023. DOI: 10.1109/ICCCE58854.2023.10246072.

Alam, M.S., Rashid, M. M., Roy, R., Faizabadi, A. R., Gupta, K. D., & Ahsan, M. M. Empirical Study of Autism Spectrum Disorder Diagnosis Using Facial Images by Improved Transfer Learning Approach. Bioengineering, 2022, vol. 9, no. 11, article no. 710. DOI: 10.3390/bioengineering9110710.

Alam, M. S., Rashid, M. M., Faizabadi, A. R., & Zaki, H. F. M. Power of Alignment: Exploring the Effect of Face Alignment on ASD Diagnosis Using Facial Images. IIUM Engineering Journal, 2024, vol. 25, no. 1, pp. 317–327. DOI: 10.31436/iiumej.v25i1.2838.

Yunidar, Y., Roslidar, R., Oktiana, M., Yusni, Y., Nasaruddin, N., & Arnia, F. Classification of Stunted and Normal Children Using Novel Facial Image Database and Convolutional Neural Network. Radioelectronic and Computer Systems, 2024, vol. 109, no. 1, pp. 76–86. DOI: 10.32620/reks.2024.1.07.

Hazra, S. K., Ema, R. R., Galib, S. M., Kabir, S., & Adnan, N. Emotion Recognition of Human Speech Using Deep Learning Method and MFCC Features. Radioelectronic and Computer Systems, 2022, vol. 104, no. 4, pp. 161–172. DOI: 10.32620/reks.2022.4.13.

Oumoulylte, M., Alaoui, A. O., Farhaoui, Y., El Allaoui, A., & Bahri, A. Convolutional Neural Network-Based Skin Cancer Classification with Transfer Learning Models. Radioelectronic and Computer Systems, 2023, vol. 108, no. 4, pp. 75–87. DOI: 10.32620/REKS.2023.4.07.

Yaloveha, V., Podorozhniak, A., Kuchuk, H., & Garashchuk, N. Performance Comparison of CNNs High-Resolution Multispectral Dataset Applied to Land Cover Classification Problem. Radioelectronic and Computer Systems, 2023, no. 2(106), pp. 107–115. DOI: 10.32620/reks.2023.2.09.

Alsaade, F. W., & Alzahrani, M. S. Retracted: Classification and Detection of Autism Spectrum Disorder Based on Deep Learning Algorithms. Computational Intelligence and Neuroscience, 2022, vol. 2022, article no. 9893015. DOI: https://doi.org/10.1155/2023/9893015.

Sudha, V., & Ganeshbabu, T. R. A Convolutional Neural Network Classifier VGG-19 Architecture for Lesion Detection and Grading in Diabetic Retinopathy Based on Deep Learning. Computers, Materials and Continua, 2021, vol. 66, no. 1, pp. 827–842. DOI: 10.32604/cmc.2020.012008.

Melinda, M., Oktiana, M., Nurdin, Y., Pujiati, I., Irhamsyah, M., & Basir, N. Performance of ShuffleNet and VGG-19 Architectural Classification Models for Face Recognition in Autistic Children. International Journal on Advanced Science, Engineering and Information Technology, 2023, vol. 13, iss. 2, article no. 674. DOI: 10.18517/ijaseit.13.2.18274.

Hastari, D., Winanda, S., Pratama, A. R., Nurhaliza, N., & Ginting, E. S. Application of Convolutional Neural Network ResNet-50 V2 on Image Classification of Rice Plant Disease. Public Research Journal of Engineering, Data Technology and Computer Science, 2024, vol. 1, iss. 2, pp. 71–77. DOI: 10.57152/predatecs.v1i2.865.

Fitriasari, H. I., & Rizkinia, M. Improvement of Xception-ResNet50V2 Concatenation for COVID-19 Detection on Chest X-Ray Images. 2021 3rd East Indonesia Conference on Computer and Information Technology (EIConCIT). Surabaya, Indonesia, IEEE, 2021, pp. 343–347. DOI: 10.1109/EIConCIT50028.2021.9431916.

Bayunanda, E., Utami, E., & Ariatmanto, D. Facial Expression Classification Analysis Using Facial Images Based on Resnet-50V2. 2022 6th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE). Yogyakarta, Indonesia, IEEE, 2022, pp. 368-372. DOI: 10.1109/ICITISEE57756.2022.10057622.

Sukegawa, S., Tanaka, F., Hara, T., Yoshii, K., Yamashita, K., Nakano, K., Takabatake, K., Kawai, H., Nagatsuka, H., & Furuki, Y. Deep Learning Model for Analyzing the Relationship Between Mandibular Third Molar and Inferior Alveolar Nerve in Panoramic Radiography. Scientific Reports, 2022, vol 12, article no. 16925. DOI: 10.1038/s41598-022-21408-9.

Yang, E. H., Amer, H., & Jiang, Y. Compression Helps Deep Learning in Image Classification. Entropy, 2021, vol. 23, iss. 7, article no. 881. DOI: 10.3390/e23070881.

Wang, L., Shi, C., Lin, S., Qin, P., & Wang, Y. Convolutional Sparse Representation and Local Density Peak Clustering for Medical Image Fusion. International Journal of Pattern Recognition and Artificial Intelligence, 2020, vol. 34, no. 7, article no. 2057003. DOI: 10.1142/S0218001420570037.

Wang, J., Yang, L., Huo, Z., He, W., & Luo, J. Multi-Label Classification of Fundus Images with EfficientNet. IEEE, 2020, vol. 8, pp. 212499–212508. DOI: 10.1109/ACCESS.2020.3040275.

Ajmera, P. K., Kabra, S. M., Mall, A., & Lhila, A. AMaizeD: An End to End Pipeline for Automatic Maize Disease Detection. 2023 16th International Conference on Sensing Technology. Hyderabad, India, IEEE, 2023, pp. 1–6. DOI: 10.1109/ICST59744.2023.10460787.

Hridoy, R. H., Akter, F., & Rakshit, A. Computer Vision Based Skin Disorder Recognition Using EfficientNet: A Transfer Learning Approach. 2021 International Conference on Information Technology. Amman, Jordan, IEEE, 2021, pp. 482–487. DOI: 10.1109/ICIT52682.2021.9491776.

Alam, I. N., Kartowisastro, I. H., & Wicaksono, P. Transfer Learning Technique with EfficientNet for Facial Expression Recognition System. Revue d’Intelligence Artificielle, 2022, vol. 36, no. 4, pp. 543–552. Available at: DOI: 10.18280/ria.360405.

Ahmed, T., & Sabab, N. H. N. Classification and Understanding of Cloud Structures via Satellite Images with EfficientUNet. SN Computer Science, 2022, vol. 3, article no. 99. DOI: 10.1007/s42979-021-00981-2.

Kaewlek, T., Tanyong, K., Chakkaeo, J., Kladpree, S., Chusin, T., Yabsantia, S., & Udee, N. Classification of Pneumonia, Tuberculosis, and COVID-19 on Computed Tomography Images Using Deep Learning. Trends in Sciences, 2023, vol. 20, no. 11, article no. 6974. DOI: 10.48048/tis.2023.6974.

Ishida, T., Yamane, I., Sakai, T., Niu, G., & Sugiyama, M. (2020). Do We Need Zero Training Loss After Achieving Zero Training Error? http://arxiv.org/abs/2002.08709.




DOI: https://doi.org/10.32620/reks.2025.1.11

Refbacks

  • There are currently no refbacks.