Missing Modality Benchmarks Test Multimodal AI Robustness Claims

A wave of benchmarks for broken inputs

Multimodal AI systems combine text, images, audio, video or physiological signals, and they are often marketed as more capable than single-input models. In practice, the inputs are not always there. A microphone fails, a caption gets truncated, or a sensor drops out mid-recording. Two new benchmarks, MissMAC-Bench and MuteBench, try to measure how well models cope when a modality goes missing. A separately published framework, GRaFiT, offers one approach to building models that degrade gracefully.

Read together, the three efforts point in the same direction. Robustness to missing data is real but conditional. It depends on which modality disappears, how the model is built, and whether the failure looks like anything the model saw in training.

MissMAC-Bench: affective computing is text-heavy

MissMAC-Bench focuses on multimodal affective computing, which covers sentiment analysis and emotion recognition. Its authors argue the field lacks fair, unified evaluation standards for incomplete inputs, so they built the benchmark around two principles. Models should not be told during training what will be missing, and a single model should handle both complete and incomplete inputs 1. The protocol tests fixed and random missing patterns at both the dataset and instance levels. The goal is to better reflect deployment conditions 1.

The benchmark draws on four established datasets: CMU-MOSI and CMU-MOSEI for sentiment, and IEMOCAP and MELD for emotion. It standardizes audio and visual features with ImageBind and compares three language backbones, BERT, sBERT and DeBERTaV3 3. The authors released their code publicly 1.

The headline finding is blunt. Across every dataset, removing text caused the largest performance drop, which suggests current affective models are heavily "text-centric" 3. Classification-style emotion recognition also generally held up better under missing data than sentiment analysis did 3. The original abstract frames the experiments as validating that diverse approaches can tackle the problem 1. A more cautious reading of the text-dominance result is that many "multimodal" models may lean on one modality while the others play a supporting role.

MuteBench: architecture beats scale

MuteBench moves the question into clinical settings, where missing data is routine and the stakes are higher. It spans nine datasets from seven clinical domains and more than 125,000 samples, with individual datasets ranging from 515 to 64,726 recordings. It tests six fusion models from three families: channel-independent, Mixture-of-Experts, and shared-specific decomposition designs 2. The authors describe it as the first benchmark to jointly evaluate multiple missingness conditions across both fusion architectures and clinical datasets 2.

Its findings are the most useful for anyone weighing robustness claims. Architecture family predicted robustness better than parameter count did 2. Channel-independent models handled an entire modality going missing well but could stumble when data was missing within a modality, especially on short sequences 2. Most tellingly, curriculum modality dropout, a common training trick, protected models reliably only up to the highest dropout rate used during training 2. Beyond that point, the protection could not be counted on.

GRaFiT: a design aimed at many failures at once

GRaFiT, published in Machine Learning with Applications by Guangyou Lu and Chiwen Qu, approaches the problem from the model side. The authors center it on what they call "reliability drift": as input quality degrades, how much each modality can be trusted shifts in systematic ways 4. They argue that earlier methods tend to target a single type of degradation, lack fine-grained adaptive control, or cannot detect when reliability changes 4. GRaFiT is designed to handle text degradation, missing modalities and cross-modal mismatch together, with a controlled fallback as reliability declines 4.

Two of its components stand out. The first is a multi-view text constructor. It masks spans of each training caption at four intensity levels, from 0.15 to 0.60, which yields five text views ranging from complete to severely degraded. Mask positions are regenerated on every iteration 4. The second is an asymmetric cross-modal attention module. Text acts as the query, and the model retrieves the image regions most relevant to it. The system uses a ResNet-50 image encoder and DistilBERT for text 4.

What it adds up to

GRaFiT was not evaluated on either benchmark, so no direct verdict on it follows. Still, the benchmarks offer a lens for reading claims like its own. MuteBench's dropout result raises an obvious question for any training-time augmentation scheme, including GRaFiT's capped masking schedule. Protection may hold within the trained range and fade beyond it. MissMAC-Bench's text-dominance finding matters too. GRaFiT anchors attention on text as the query, so its behavior when text degrades past the trained levels deserves particular scrutiny.

The broader takeaway is that "robust" is not a single property. A model can tolerate one kind of missingness and be fragile to another. Standardized benchmarks that vary the type, level and pattern of failure are what turn individual robustness claims into comparable evidence. Until new methods are tested that way, their gains are best treated as promising rather than general.