Sentiment Analysis with Multiple Modalities in the Age of Generative AI using Multilevel Fusion

Abstract

Introduction: Sentiment analysis is moving from predominantly text-centered classification toward multimodal understanding in which language, acoustic behavior, visual evidence, contextual metadata, and generated semantic descriptions jointly influence interpretation. Generative AI and large language models (LLMs) add a further layer of capability by producing contextual representations, synthetic training signals, semantic summaries, and explanations.

Objectives: This paper develops a conceptual framework for multilevel fusion that separates local modality evidence, intra-modal context, pair wise cross-modal interactions, global multimodal representations, and LLM-guided semantic reasoning. It also identifies design requirements for robustness, interpretability, data efficiency, and responsible deployment.

Methods: A structured narrative synthesis of recent journal literature is combined with architectural analysis. The paper organizes fusion strategies into five levels and maps Generative AI functions to each level. A reference architecture, comparative taxonomy, implementation workflow, and evaluation framework are proposed.

Results: The analysis indicates that multilevel fusion can reduce the information loss associated with single-stage fusion by preserving fine-grained modality cues while progressively modeling cross-modal dependencies. LLM-guided semantic supervision can provide a higher-level context signal, while generative augmentation can address sparse or imbalanced multimodal data. These benefits are accompanied by risks related to hallucination, synthetic-label inconsistency, computational cost, privacy, and domain shift.

Conclusions: A GenAI-era sentiment system should therefore be designed as a layered evidence-integration pipeline rather than as a single fusion operation. Multilevel fusion, uncertainty-aware modality weighting, controlled generation, and explanation auditing provide a coherent direction for future multimodal sentiment systems.