Blog · arXiv Analysis · Last reviewed July 10, 2026

The Affect Classifier Becomes the Fusion Receipt

A multimodal emotion model is not only a prediction. It is a translation from text, voice, and face into a social label, and the fusion rule decides which channel gets to speak for the person.

Published: July 10, 2026 Modified: July 10, 2026 Last reviewed: July 10, 2026 Tags: arXiv, affective computing, emotion recognition, SHAP, surveillance, governance

The Paper

The paper is SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits, arXiv:2607.08573 [cs.AI]. The arXiv record lists Adis Alihodzic and Selma Skopljakovic Hubljar as authors and records submission on July 9, 2026. The HTML version lists the Department of Mathematical and Computer Sciences, Faculty of Science, University of Sarajevo, in Bosnia and Herzegovina.

This belongs near affective safety, emotion recognition and biometric categorization, AI evaluations, explainable AI, AI audit trails, valence-axis warnings, and driver-camera attention judging. The fresh angle is the fusion receipt: when a system infers affect from several channels, the governance record must show how the channels were combined.

The Fusion

Affective computing often arrives in administrative language: sentiment analysis, engagement, safety monitoring, customer experience, tutoring analytics, workplace wellness, driver attention, or call-center quality. Beneath that language is a harder claim. The system says that some combination of words, voice, and facial signals can be converted into an emotion or sentiment label.

The paper is useful because it refuses to make that translation look more magical than it is. It compares early fusion, where modality features are concatenated before classification, with late fusion, where unimodal predictors are combined after training. It then studies XAI-guided adaptive fusion, or XGAF: a tree-based mixture of unimodal and cross-modal XGBoost experts whose sample-level weights are derived from TreeSHAP attribution magnitudes.

The Method

The method trains experts on subsets of three modalities. The full pool includes text-only, voice-only, face-only, three bimodal experts, and one trimodal expert. Early fusion is the special case where only the trimodal classifier is used; late fusion is probability averaging over unimodal classifiers. The proposed gate turns each expert's TreeSHAP attributions into one scalar score, then uses a temperature-scaled softmax to combine the experts' predicted probabilities.

The paper's central technical point is that the scalar reduction matters. Mean absolute SHAP values put each expert on a typical-per-feature scale. Median absolute SHAP values are robust to outlier attributions but behave similarly in the reported MELD experiment. Summed absolute SHAP values preserve total attribution mass, which lets high-dimensional cross-modal experts receive enough weight. In the MELD setup, text and voice experts each have 768 features, the face expert has 512 features, bimodal experts have 1280 or 1536 features, and the trimodal expert has 2048 features. A governance reviewer should hear the warning: the explanation statistic is not neutral plumbing.

Evidence

The experiments use MELD for seven-class emotion recognition and CMU-MOSEI for three-class sentiment recognition. For MELD, the paper reports 9,660 training samples, 1,067 validation samples, and 2,525 test samples after filtering for required modality features. For CMU-MOSEI, it reports 16,326 training samples, 1,871 validation samples, and 4,659 test samples. MELD uses BERT-base text embeddings, wav2vec 2.0 voice embeddings, and a face-emotion pipeline with 15 frame embeddings aggregated by mean pooling, BiLSTM, or Transformer. CMU-MOSEI uses preprocessed aligned text, acoustic, and visual features.

On MELD with the Transformer face aggregator, early fusion reaches 0.6018 weighted-F1, late fusion reaches 0.4598, mean-abs XGAFv2 reaches 0.5714, and sum-abs XGAFv2 reaches 0.5983. McNemar testing shows no significant difference between sum-abs XGAF and early fusion on MELD, with p = 1.000, while sum-abs XGAF is significantly better than late fusion, with p below 0.0001. On CMU-MOSEI, sum-abs XGAF reaches 0.6519 weighted-F1, slightly above early fusion at 0.6485 and above late fusion at 0.5696; the paper gives p = 0.0452 for the small early-fusion comparison.

The most important finding is the negative one. The authors say the current SHAP gate does not show rich adaptive routing across emotions or samples. Mean-abs and median-abs weights are nearly uniform; sum-abs weights concentrate on the trimodal expert. The gain is better described as cross-modal expert dominance than as a sophisticated per-person routing story.

Limits

The limitations make the paper more useful for this site. The experiments use pre-extracted features rather than end-to-end fine-tuning. The MELD setup is per-utterance and does not model dialogue context, speaker state, or conversation graphs. The CMU-MOSEI improvement over early fusion is statistically significant but numerically small, and the authors call for additional seeds, bootstrap confidence intervals, and more datasets before treating it as a robust practical improvement.

The paper also does not test missing, noisy, degraded, or asynchronous modalities, even though those are central deployment problems for multimodal affect systems. That is the governance line. A model may look interpretable on clean features while failing the messy cases that matter most: masked faces, accents, camera angle, microphone quality, disability, cultural display rules, sarcasm, translation, fatigue, grief, or strategic self-presentation. The page should not be read as evidence that emotion recognition is ready for high-stakes inference. It is evidence that even a modest fusion method needs a record of its own fusion behavior.

The Receipt

An affect-fusion receipt should record the task, label set, training corpus, consent basis, demographic and language coverage, modality list, sensor quality, feature extractor, face-detection pipeline, text encoder, acoustic encoder, expert pool, SHAP implementation, reduction rule, temperature, validation metric, per-class performance, modality weight distribution, entropy, missing-modality stress tests, noisy-modality stress tests, calibration, human-review rule, and downstream decision limit.

The audit question is not "did the model recognize emotion?" It is "which channel was allowed to define the person, under what feature pipeline, with what uncertainty, and with what right to contest the label?"

Sources

Return to Blog