Explainable artificial intelligence for multimodal sentiment analysis in revitalization project management

Abstract

This study focuses on the development and evaluation of an explainable artificial intelligence (XAI) framework for multimodal sentiment analysis, specifically applied to territorial revitalization project management. The research addresses the critical problem of “black box” AI models, whose lack of transparency hinders their adoption by project managers who require trustworthy information for high-stakes decision-making in complex social environments. The goal of this study is to propose and rigorously validate a novel framework for multimodal sentiment analysis that is tailored to provide transparent, trustworthy, and actionable insights for decision-making in territorial revitalization project management. The tasks to be solved include developing a hybrid XAI technique that fuses insights from cross-modal attention and gradient-based attribution, designing a cohesive, user-centric explanation format combining highlighted text and image heatmaps, constructing a custom RevitalizeSent-MM dataset for this specific domain, and empirically evaluating the framework’s predictive accuracy and, crucially, the fidelity of its explanations. The methods used involve a transformer-based Multimodal Sentiment Analysis (MSA) model using BERT and ViT with cross-modal attention for information fusion. The explainability component is a hybrid XAI technique that integrates cross-modal attention analysis with Integrated Gradients to assign importance scores to input features. Evaluation was performed using standard classification metrics for performance and the “Accuracy Drop on Perturbation” metric for explanation fidelity. The results confirmed the efficacy of the framework. The multimodal model demonstrated superior accuracy over unimodal baselines, and the proposed XAI method achieved significantly higher fidelity than naive explanation approaches, demonstrating its ability to accurately reflect the model’s internal reasoning. The scientific novelty lies in three areas: the development of a fused, hybrid XAI technique specifically for transformer-based multimodal models, creation of a unique, domain-specific dataset for revitalization analysis, and validation of a methodology for adapting advanced XAI to solve critical trust and adoption barriers, thereby confirming its practical significance in project management.