FP8 LLM Training: Delta-Matching Targets Attention Gradient Gap

What happened

A new arXiv paper, submitted on 29 September 2026, takes aim at what its authors call the last obstacle to fully native 8-bit training of large language models: reliable FP8 attention. 3 The paper, Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs, is credited to Haozhan Tang, Hao Kang, Han Cai, Song Han and Chenyan Xiong. 3 It argues that inconsistencies between the forward and backward passes produce a "stale delta" that distorts training dynamics. The proposed fix, Delta-Matching, is presented with a proof that it restores a key mathematical property of the softmax gradient. 3

Tech Times describes the work as coming from MIT, Carnegie Mellon and NVIDIA Research. It calls the flaw a stale scaling-factor mismatch that corrupts attention gradients, and calls the fix a drop-in method that reaches parity with BF16. 4 The outlet's own framing is inconsistent: its headline credits MIT and NVIDIA, while its body text names MIT and Carnegie Mellon. 4 That is minor, but it shows how quickly a preprint can become a "solved" story in coverage.

Why the stale-delta finding is interesting

The most useful part of the paper may be its diagnosis, not its fix. The authors report that hybrid runs affected by stale delta show only a modest loss gap at 569M parameters. At 1.67B and 5.29B parameters, the losses grow substantially and downstream performance degrades. 3 Known stabilizers such as QK normalization, removing positional encoding (NoPE), and extending context at a lower learning rate can soften or delay the damage, but they do not remove it. 3

The authors read this as accumulated optimization error that small models and short runs can hide. 3 If that holds up, it is a caution for the whole field. An FP8 recipe that looks clean in small ablations is not necessarily safe at frontier scale.

Why it matters

The economic case for FP8 is simple. Tech Times notes that NVIDIA's H100 delivers 3,958 teraflops of FP8 compute with sparsity, compared with 1,979 for BF16. 4 Closing the remaining accuracy gap would let pretraining teams use that roughly 2× arithmetic headroom without a quality penalty. Tech Times adds the important qualifier that this depends on the method holding at production scale. 4

A crowded field

Delta-Matching joins an active line of research. DeepSeek-V3 is widely treated as the anchor of the 2025 FP8 wave. It was a roughly 671B-parameter mixture-of-experts model trained natively in FP8, using per-group activation and per-tile weight quantization, at a cost of 2.664M H800 GPU-hours over 14.8T tokens. 1 DeepSeek later published a retrospective on hardware-software co-design. It covered FP8 accumulator precision and offered recommendations for future FP8 and FP6 tensor cores. 1

Other groups have taken different routes:

• Towards Fully FP8 GEMM LLM Training at Scale moves both forward and backward matrix multiplications fully into FP8, using tensor-wise dynamic scaling for gradients, and reports validation at 70B scale. 1 Its FOG architectures replace learned gains with a constant and slightly raise the attention softmax scale to compensate. 2 The authors also found that kurtosis signals trouble well before loss diverges, and they continued pretraining one variant out to 450B tokens to test long-run stability. 2
• μnit Scaling, from Cerebras and Bloomberg, replaces per-tensor scaling with μP-style unit scaling. The goal is for FP8 to work across model sizes without the overhead of delayed scaling. 1

These approaches split along a clear line. FOG and μnit Scaling mostly reshape the architecture or the scaling scheme so that FP8 behaves well. Delta-Matching instead claims to correct a specific numerical inconsistency inside attention backpropagation. 123 In principle, the two approaches could complement each other. The Delta-Matching results suggest that architectural stabilizers alone leave some residual error behind. 3

The reading

The stale-delta analysis is a credible and useful contribution. The scaling trend it reports, where a gap is small at 569M and large at 5B, is the kind of evidence that deserves attention from anyone running long FP8 jobs. 3

The "final gap closed" framing should still be treated as provisional. The experiments described reach about 5.29B parameters. 3 DeepSeek-V3 and the 70B FOG work operate at much larger scales. 1 Production adoption will depend on independent replication at tens or hundreds of billions of parameters, and on accessible kernels that labs can drop into existing stacks. Until then, Delta-Matching is best seen as a promising new entry in a busy field, not the end of it.