Magnitude Inference Engine Takes On llama.cpp With 2-Person Team

A tiny team picks a fight with the local-inference giants

Magnitude, a Y Combinator Summer 2025 startup, has released an open-source inference engine aimed at people who want to run AI agents on their own machines rather than in the cloud. 35 The company says the engine tunes itself to the exact hardware it runs on. It also claims up to twice the throughput of llama.cpp, the widely used open-source runtime. 35 The launch landed on September 30. 5

The company is very small. Y Combinator's directory lists Magnitude as an active, San Francisco–based company with two employees. 4 The founders describe themselves as software engineers who previously built an open-source browser agent. That project passed 4,000 GitHub stars and 100,000 downloads. 3 They say the new engine grew out of frustration: they wanted to run that agent on local models and couldn't find an inference engine that suited the workload. 3

What Magnitude actually does differently

The core technical idea is on-device compilation. Most runtimes ship kernels that were compiled ahead of time. Magnitude instead compiles and tunes its kernels on the target machine before a model runs. 35 The engine supports Apple Silicon, Nvidia and AMD GPUs, and CPU-only systems, across macOS, Linux and Windows. 5

The founders frame the pitch around a tradeoff they see in existing tools. 3

• vLLM and SGLang are built for batched datacenter serving and give up single-session speed.
• llama.cpp and Ollama favor broad compatibility over hardware-specific optimization.
• Niche projects such as oMLX and ds4 specialize but lack completeness.

They argue that none of these were designed for local agents. Agent sessions run long, several often run in parallel, and the user still needs the computer for other work. 3

The headline numbers need careful reading. The "up to 2x" claim comes mostly from Apple hardware. The company's own benchmarks show decode running 92% faster on Metal but only 19% faster on CUDA. 5 That is a meaningful gain on Nvidia, but nowhere near double. All of these figures are self-reported, and independent benchmarking will matter a great deal here.

A market that has already matured

Magnitude is arriving in a crowded field. Quantization formats such as AWQ, GPTQ and newer GGUF variants have combined with specialized runtimes to make local inference practical on ordinary developer workstations. 2 Developers now run multi-billion-parameter models locally to protect proprietary code, avoid API subscription costs and run low-latency agent loops. 2 One industry write-up argues that local inference has moved from a nice-to-have to a deal-breaker for teams facing strict client or regulatory demands for data control. 1

The incumbents are strong and well funded. Ollama is widely described as the de facto standard for frictionless local execution. It bundles model downloads, quantization management, GPU offloading and HTTP serving into a single tool. 2 It reportedly raised $88 million and ships updates almost daily. 1 One recent release caches model metadata between runs, which cut time-to-first-token in Ollama's benchmarks from roughly 995ms to 524ms. 1 Alongside Ollama, vLLM, llama.cpp and LM Studio round out the set of engines developers commonly rely on. 2

The accounts also differ on agents specifically. One overview credits Ollama with leading on agent orchestration. 2 Magnitude's founders say no existing engine was designed for running agents locally. 3 Both positions can be partly true. Ollama makes it easy to wire models into agent tooling, while Magnitude is targeting the runtime behavior underneath: long sessions, concurrency and coexisting with other workloads.

The read: a credible wedge, not yet a contender

Magnitude's bet is sensible on paper. The big runtimes optimize for breadth or for datacenter batching. That leaves room for a tool tuned to the single-user, multi-session pattern that local coding and browser agents create. On-device kernel tuning is a defensible technical angle, especially on Apple Silicon, where the claimed gains are largest. 5

The obstacles are equally clear. Two engineers are competing against a rival with $88 million and a near-daily release cadence. 14 Ollama is already cutting latency in ways that narrow the room for differentiation. 1 Performance claims tied mostly to one platform will face scrutiny, and the smaller CUDA advantage may not be enough to pull Nvidia users away from mature tools. 5

The most plausible path for Magnitude is as a specialist, the engine power users pick for agent-heavy work on Macs, rather than a general replacement for llama.cpp or Ollama. Its open-source release lets the community verify the benchmarks quickly. If the numbers hold up under outside testing, a two-person team could earn real attention. If they don't, the launch will mostly confirm how hard it is to unseat incumbents in local inference.