FP4 Pretraining Recipes Challenge NVFP4's Hadamard Approach

A crowded field of 4-bit recipes

Four-bit floating point has moved from an inference trick to a serious option for training large language models. In recent months, several independent groups have published FP4 pretraining recipes that differ from NVIDIA's NVFP4 playbook in important ways. One technical report argues that a core NVFP4 stabilization step, the randomized Hadamard transform (RHT), can be dropped. In a related ablation, removing it along with a BF16 exemption raised measured throughput by 21.2%. 3 Another paper argues that the dominant 4-bit element format itself is the wrong primitive. 2 A third group is building out Huawei's own FP4 format for Ascend NPUs. 1

The question underneath all three is the same: which numerical design best preserves training quality while keeping the efficiency gains that made FP4 attractive?

Why FP4 is worth the trouble

The Huawei-focused paper summarizes the case. FP4 formats such as MXFP4 and NVFP4 have been applied to the linear GEMM operations in LLMs. Compared with higher-precision baselines, they offer up to 4× improvements in compute throughput and memory efficiency. 1 The difficulty is numerical. The standard E2M1 payload can only represent a narrow range of magnitudes, which makes stable pretraining hard. 3

The main formats take different approaches to that problem. MXFP4 and NVFP4 both keep independent scale factors for fine-grained tensor blocks. Their block sizes and scale hierarchies trade dynamic range, precision, and hardware efficiency against one another. 2 NVFP4 generally trains more accurately than MXFP4 because it uses smaller blocks and a more refined scaling scheme. 2 NVIDIA's Transformer Engine recipe adds further safeguards: current-tensor scaling, the randomized Hadamard transform, and BF16 handling for a final block of layers. 3

Three alternative bets

Wider block scales, fewer safeguards. The UE5M3 report keeps E2M1 payloads but pairs them with unsigned E5M3 block scales, which gives the scales a wider range. 3 With that change, the recipe omits RHT and applies selective stochastic rounding only to backward gradients. It also runs FP4 in every eligible internal linear layer instead of reserving some layers for BF16. 3 The authors also ran an ablation using native NVFP4. Removing both RHT and the BF16 final-block exemption increased measured model-body token throughput by 21.2%. 3

That figure needs some care. The 21.2% comes from the native NVFP4 ablation. The UE5M3 pretraining itself was software-emulated, and the report explicitly calls for native hardware support for UE5M3 block scaling. 3 So the result shows that removing these safeguards has real speed value. It does not yet show UE5M3 delivering that speed on current silicon.

Rethinking the grid. The UFP4 paper goes further and targets the E2M1 element format itself. It notes that current FP4 hardware paths center on E2M1, including NVIDIA's Blackwell- and Rubin-class systems and AMD's MI350-series GPUs. 2 The authors trace a "shrinkage bias" in FP4 pretraining to geometric causes. Their UFP4 recipe consistently showed lower BF16-relative loss degradation than strong E2M1 baselines on three long-run pretraining setups: a 1.5B dense model, a 7.9B MoE, and a 124B MoE. 2 They support this with scaling-law analysis and ablations. They conclude that future accelerators should treat E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1. 2

A vendor-specific format. On Huawei's side, researchers studied HiFloat4, a format designed for next-generation AI accelerators, and compared it directly with MXFP4 in large-scale dense and MoE training. 1 They report that FP4-specific stabilization techniques can keep relative error within 1% of full-precision baselines while retaining 4-bit efficiency. 1

The volume of work keeps growing. October 2026 machine-learning listings include a paper titled "Format-Aware Fusion for Fast FP4 Pretraining," which suggests kernel-level optimization is becoming its own line of research. 4

Where they agree and where they split

All three papers accept that FP4 pretraining works and is worth pursuing. They differ on where the remaining inefficiency lies:

• UE5M3: the scale format and the extra stabilization steps are the problem. 3
• UFP4: the element grid is the problem. 2
• HiFloat4: a format co-designed with specific hardware is the answer. 1

Two of the three explicitly ask hardware makers to change what their chips support natively. 23

The takeaway

NVFP4 is not being replaced. Instead, it is now a baseline that others measure against and try to simplify. The most practical finding is the throughput gain from dropping RHT and the BF16 exemption, because it was measured on native NVFP4 hardware. 3 The more consequential claims, about UE5M3 scales or uniform 4-bit grids, depend on silicon that does not yet support them natively. 23 Their influence will therefore show up in future chip designs more than in today's training runs. These papers are effectively arguing for a place in the next generation of accelerator specifications.