Magnitude Inference Engine: Unpacking the 2x llama.cpp Claim

What launched

Magnitude, a Y Combinator S25 startup founded by Anders and Tom, released an open-source inference engine on September 30. It is built to run open-weight models locally as the backend for coding agents. 23 Its central idea is that most local runtimes ship kernels precompiled for broad hardware classes. Magnitude instead compiles and tunes its kernels on the user's own machine before a model runs. 13 The project is licensed under Apache 2.0. It ships as a desktop app for macOS, Linux and Windows and connects to agents developers already use, including OpenCode, Codex and Claude Code. 1

The launch drew attention quickly. The Hacker News post reached 178 points and 87 comments, and the GitHub repository had roughly 6,000 stars and 401 forks at one count. A post about the project also topped r/LocalLLaMA the same day. 1

The pitch: a gap between existing engines

The founders describe the current landscape as a set of trade-offs. 2

• vLLM and SGLang are tuned for batched datacenter serving, at the cost of single-session speed.
• llama.cpp and Ollama favor broad compatibility over hardware-specific optimization.
• Narrower projects such as oMLX and ds4 target particular hardware or models but lack a complete engine.

They argue that none of these were designed for local agent workloads. Those workloads involve long sessions and several concurrent agents, all on a machine the user still wants to use for other work. 2

Magnitude's answer is kernels written with flexible parameters that get tuned on the actual device. The team says this delivers broad compatibility with the performance ceiling of hand-specialized kernels. 2 It also concentrates its effort on a limited set of model architectures rather than chasing every format. 2 Supported targets include Apple Silicon, Nvidia and AMD GPUs, and CPU-only systems. 3

Reading the "2x" number carefully

The headline claim is that Magnitude is "up to 2x faster than llama.cpp." 2 The phrase "up to" matters. In the company's own benchmarks, decode throughput was 92% faster on Apple's Metal backend but only 19% faster on CUDA. 3

So the near-doubling applies mainly to Macs. On Nvidia hardware, which a large share of serious local-inference users run, the reported gain is a modest fraction of that. 3 A 19% decode improvement is still meaningful for long agent sessions. It is a different proposition from "twice as fast," though, and buyers of the headline should know which platform they are on.

A plausible explanation, offered here as analysis rather than something the company has stated, is that llama.cpp's CUDA path has been heavily optimized by a large contributor base. That leaves less headroom than its Metal path. Device-level autotuning would then produce the biggest wins wherever the incumbent's kernels are least specialized.

These figures all come from Magnitude's own testing. Results from independent, reproducible benchmarks across a range of GPUs, especially recent Nvidia and AMD cards, have not yet been established. Self-reported numbers from a launch-day repository deserve the usual caution. The size of any advantage may shift as both projects update and as testers try different models, quantizations and context lengths.

Why it matters

The more interesting question is not one benchmark ratio. It is which engine should sit beneath a coding agent. 1 Developers running agents locally have mostly defaulted to llama.cpp or Ollama because they run nearly everywhere. Magnitude is betting that this generality leaves performance unused, and that agent workloads in particular reward a runtime built around them. 12

The on-device compilation approach carries trade-offs the launch material does not fully address. Tuning presumably adds setup time before first use. Supporting fewer architectures also means some models users want may not be available. 2 Whether Magnitude can keep pace with the rapid release of new open-weight architectures, which llama.cpp's community absorbs quickly, will likely decide its long-term relevance.

The verdict for now

Magnitude is a credible, well-received entry with a coherent technical thesis and genuine early traction. 1 The fair summary of its performance claim is narrower than its marketing. It shows a large reported advantage on Apple Silicon and a smaller one on CUDA, all from in-house benchmarks. 3

Mac users running local agents have the clearest reason to try it today. Users on Nvidia hardware should treat the 2x figure as a ceiling rather than an expectation, and wait for third-party measurements before switching runtimes.