MissBench Benchmark Targets Imbalanced Missing Modalities in AI

What happened

A new benchmark called MissBench aims to fix a blind spot in how multimodal emotion and sentiment models are tested. Its target is a quiet assumption behind many evaluations: that text, audio, and video inputs are all equally likely to be present at inference time.

In practice, that assumption rarely holds. Some input channels are systematically more fragile or more costly to capture than others. The researchers argue this produces imbalanced missing rates, which in turn create training biases that standard task-level scores fail to reveal 2.

MissBench standardizes two kinds of evaluation protocol. In the first, all modalities share the same missing rate. In the second, missing rates differ by modality. The protocols are applied across four widely used sentiment and emotion datasets 2. The framework also fixes data splits and masking seeds, so other groups can reproduce results exactly rather than approximating them 1.

Two diagnostics beyond accuracy

The more interesting contribution is the pair of diagnostic metrics layered on top of ordinary performance numbers.

• Modality Equity Index (MEI): measures how fairly each modality contributes to predictions across different missing-modality configurations 1.
• Modality Learning Index (MLI): quantifies optimization imbalance by comparing gradient norms tied to each modality during training, aggregated across the modules associated with each modality 1.

The MLI is the piece that moves evaluation from outputs to training dynamics. The underlying concern is "gradient dominance," one of the paper's stated keywords 2. Gradient dominance is the tendency for one strong modality, often text, to absorb most of the learning signal. The weaker channels end up undertrained.

When the dominant modality then goes missing, the model has little to fall back on. A headline accuracy figure can hide this fragility entirely. A gradient-level measure is designed to surface it.

How it fits among rival benchmarks

MissBench arrives in a crowded field, and its authors position it against several peers [1]:

• MERBench: covers affective tasks but focuses on noise and robustness.
• MissMAC-Bench: handles fixed and random missing rates and reports performance means and standard deviations.
• BalanceBenchmark: is a general-purpose effort built around an imbalance index rather than affective computing specifically.

By the authors' own comparison, MissBench's distinguishing feature is combining three things in one public framework 1[2]:

• affective tasks
• explicit imbalanced missing-rate protocols
• diagnostics at both the modality level and the gradient level

The broader reckoning on robustness claims

Separate 2026 benchmarking work suggests MissBench is part of a wider correction in how robustness to missing inputs gets evaluated.

MuteBench ran 810 experiments across nine clinical datasets 3. It found that architecture family, not clever gating mechanisms, was the strongest predictor of how well a model tolerates missing modalities. It also found that curriculum dropout training only protects a model up to the exact missing rate it saw during training 3.

MissMAC-Bench, for its part, reported text-centric degradation and reliability that varied by dataset across affective-computing models 3. That finding about text dependence lines up closely with the imbalance problem MissBench was built to diagnose.

Together, these results cast doubt on single-framework claims of general robustness, including those made for fusion methods like GRaFiT. They point toward standardized, multi-architecture testing as a prerequisite for such claims 3.

Why it matters

The practical stakes are straightforward. Emotion recognition and sentiment analysis systems deployed in real settings will routinely encounter muted microphones, obscured cameras, or missing transcripts. Some of these failures are more common than others.

A model that looks strong when all channels are present may be quietly leaning on one modality. Benchmarks that randomize missingness uniformly, or that only report aggregate accuracy, can certify such a model as robust when it is not.

MissBench's bet is that the field needs to measure why models fail, not just how much. That bet appears well placed. The MuteBench finding that training-time dropout protection stops at the trained missing rate is a direct warning about evaluation protocols. Clean or uniform testing conditions can flatter models.

Testing with deliberately unequal missing rates, and inspecting gradient balance during training, are reasonable responses to that warning.

Caveats

The material available here describes MissBench's design and metrics but not detailed findings on which models score well or poorly. Whether MEI and MLI become standard reporting tools will depend on uptake by other researchers.

There is also an open question about whether gradient-norm comparisons transfer cleanly across very different architectures. That concern is sharpened by MuteBench's conclusion that architecture family matters so much 3.

Still, the direction is clear. Robustness to missing modalities is increasingly being treated as something to stress-test under realistic, uneven conditions rather than assume.