Mistral Large 4 benchmarks: Le Chonk trails Chinese open models

A trillion parameters, and a familiar gap

Mistral has put its new flagship, Mistral Large 4, into public preview. The French lab nicknames it "le Chonk." The company's pitch is direct: this is the most capable open model built outside China, and it competes with the strongest open systems anywhere 5. Early independent measurements support only part of that claim. Le Chonk does well in cybersecurity and some specialist domains. On broad aggregates and on coding, Chinese open-weight models still lead.

The model is a multimodal mixture-of-experts. It has roughly 1 trillion total parameters (1.05 trillion on the model page) and 49 billion active per token. It also has a 1.6-billion-parameter vision encoder, a 1-million-token context window, and training data in more than 160 languages 1. Activating only a small slice of the network per token is meant to keep serving costs closer to a mid-size dense model than the headline parameter count suggests 4.

API pricing is $1.36 per million input tokens and $4.18 per million output tokens. That is cheap compared with closed frontier systems 24. Mistral says it trained the model in its own European datacenters on 3,800 Nvidia Grace Blackwell GPUs. Some reports put the figure at 4,000 4. Accounts of the launch date also differ slightly: one describes a Monday announcement and another a Tuesday one 45.

Weights are coming, but not yet

For a model sold on openness, the weights are not yet available. Mistral says they will arrive by the end of October 2026 12. The company says it is first red-teaming the model with cybersecurity partners and state authorities, and is offering preview access to "cybersecurity leaders" 25. The license terms are also unclear. The model page lists the license only as "Open" without naming specific terms 1. Until both questions are settled, claims about how open the model is remain provisional.

Where independent numbers diverge from the pitch

Mistral benchmarks itself directly against DeepSeek V4 Pro, Qwen3.8 Max, Kimi K3 and GLM-5.3, and claims significant outperformance 4. Third-party results are less flattering.

Broad aggregates:

• Artificial Analysis Intelligence Index: the preview scored 38 and was listed 25th of 25 models tracked 2.
• Vals Index: 32nd of 44 overall and 9th of 16 among open-weight models 3. One analysis observed that these results are difficult to reconcile with Mistral's "competitive with the strongest open-source models globally" framing, at least on aggregate 3.

Coding:

• DeepSWE v1.1: Mistral's published score is 61.7% 1, reported elsewhere as 62 2. That ranks second among open-weight models in one tally, behind Kimi K3 paired with Kimi Code CLI at 68 2.
• Cross-checks: VentureBeat's independent check put GLM-5.3 at 69% on DeepSWE 4. Kimi K3 and DeepSeek V4.1 Flash also post higher published scores than Large 4 1.

The pattern is consistent across outlets: on coding at launch, Chinese open-weight models hold the edge.

The cybersecurity case is real

Security is where le Chonk performs best:

• Artificial Analysis Cyber Index: tied for first with GLM 5.3 Flash 2.
• CyberGym: ranks in the top five globally on one reading 2.
• CyberGym-E2E: 82%, first of 18, ahead of MiMo-V2.6-Pro at 79% and GPT-6 Luna at 78%. This task requires reproducing and patching real vulnerabilities 3.

Mistral's self-reported security figures are also strong, including 93% on Cybench and 93.3% on B3 prompt-injection resistance. Those are company numbers rather than independent verification 3. Outside security, the model placed first among 31 open-weight models on Harvey's Legal Agent benchmark, though with a modest absolute score of 15.83%. It was mid-pack on Finance Agent v2, at 22nd of 75 3.

The cyber results help explain the rollout strategy. A model good at finding and patching vulnerabilities has obvious defensive value and obvious misuse potential. That makes a staged release to security partners and governments easier to justify than an immediate weight drop.

Why it matters

The geopolitical frame is central to this launch. The most capable open models have been Chinese, they are seeing growing global adoption, and access to open AI has become a flashpoint in the U.S.–China technology race 5. Western developers want a credible alternative that can be modified and self-hosted, unlike closed systems from OpenAI and Anthropic 5.

On the evidence so far, Large 4 is a strong specialist rather than a general leader. It is arguably the best open option for security work, competitive in legal tasks, and inexpensive to run. It is not yet a model that beats Kimi, GLM or DeepSeek on general capability or coding. For European and Western buyers who weigh provenance and data residency heavily, that may be enough. For anyone choosing purely on capability, the gap Mistral set out to close remains open. The final weights, the license terms and fuller independent testing will show whether the release narrows it.