HART Image Generator Runs Up to 9x Faster Than Diffusion Models

A hybrid generator that does the rough work first

Researchers at MIT, NVIDIA and Tsinghua University have built an image generator that tackles generative AI's biggest everyday cost: how long it takes to make each image. The system is called HART, short for Hybrid Autoregressive Transformer. MIT says it matches or beats leading diffusion models on image quality while running about nine times faster11. The team also says it needs little enough computing power to run locally on a commercial laptop or smartphone, and a user only has to type one plain-language prompt to get an image11.

The work was accepted at ICLR 20251. The preprint first appeared on arXiv in October 20242, and MIT News covered it on March 21, 20251. That makes it about 18 months old as of this writing. It still deserves a close look, because the design addresses a constant problem for anyone producing visual content at scale: you usually have to choose between fast output and good output.

How the trick works

Today's image generators mostly follow one of two approaches. Diffusion models such as Stable Diffusion and DALL-E start from random noise and clean it up over and over, often for 30 or more steps. Each step lets the model fix earlier mistakes, which is why the images look so good. It is also why the process is slow and expensive15. Autoregressive models, the kind behind large language models, build an image piece by piece in sequence. That is much faster, but they cannot go back and correct errors. They also compress images into discrete tokens, which throws away detail3.

HART uses both. An autoregressive transformer predicts compressed tokens that set the overall composition. Then a small diffusion model predicts "residual" tokens, which hold the fine detail lost in compression3. Co-lead author Haotian Tang says these residual tokens capture the details that discrete tokens tend to get wrong, such as object edges, hair, eyes and mouths13. The diffusion part only fills in what is missing, so it needs just eight steps instead of the usual 30 or more12.

The team says the order of the stages mattered. When they brought in the diffusion model early in the autoregressive process, errors piled up. Applying it only at the end, and only to the residuals, gave much better results12. The final system pairs a 700-million-parameter autoregressive transformer with a 37-million-parameter diffusion module. MIT says it produces images comparable to those from a 2-billion-parameter diffusion model3.

Where the numbers diverge

Coverage of HART mostly repeats MIT's press release, and the widely shared summary figures differ in places from the technical paper. Readers comparing tools should know where.

The headline claim of "nine times faster" is accurate but comes from one specific comparison. The project page reports 9.3 times higher throughput than SD3-medium at 512×512 resolution1. At 1024×1024, the paper reports smaller gains: 4.5 to 7.7 times higher throughput and 3.1 to 5.9 times lower latency on an A100 GPU, compared with state-of-the-art diffusion models9. A summary of the paper gives HART's latency at 1024px as 0.75 seconds, against 2.3 seconds for SDXL and 4.4 seconds each for PixArt-Σ and SD3-medium5.

The compute claim needs more care. MIT News and outlets that republished it say HART uses about 31 percent less computation than state-of-the-art models314. The paper uses 31 percent for something else: a 31 percent improvement in generation FID, a quality score, from 7.85 to 5.38 compared with a discrete-only baseline9. The paper's compute figure is much larger. It reports 6.9 to 13.4 times fewer multiply-accumulate operations than leading diffusion models1. It appears the 31 percent figure was attached to the wrong metric somewhere between the paper and the press summary, and that version spread. If the paper is right, HART's efficiency gain is bigger than most coverage says.

Other coverage overstates things or gets them wrong. One aggregator put "quantum" in its headline even though nothing in the work involves quantum computing. The same piece described vision-language integration as a current feature6, while MIT presents it as a future goal3. The ICLR listing calls HART "the first" autoregressive model to generate 1024×1024 images directly8. The project page uses the more careful word "early"1. Readers should treat the claim to be first as contested.

Why it matters for content production

MIT's own examples are research-focused: training robots, building simulated environments for self-driving cars, and helping designers create scenes for video games1114. The bigger commercial effect, in this analysis, is likely in fields MIT does not mention: ad creative, user-generated content apps, and high-volume media production.

In these fields, cost per image and speed of each attempt matter more than peak quality. A marketing team testing dozens of ad variations, or a social app offering a generation feature to millions of users, pays for every second of GPU time. A buyer's guide to commercial image generators says many platforms already finish standard-resolution images in 5 to 15 seconds. It notes that tools like Runway charge in GPU credits, with faster models using more credits per image. The same guide already mentions hybrid autoregressive designs as a speed advantage over standard diffusion. That suggests the idea has spread from research into vendor marketing, though the guide gives no evidence of which products, if any, use HART specifically.

Running on a device may matter even more for user-generated content. If a phone can make a good image from a single prompt11, generation moves off company servers. Platforms would pay less for inference, and users could create content without uploading prompts or photos anywhere. For consumer creative apps, that cuts costs and helps with privacy.

HART is also open source. The code is on GitHub, and a 0.7-billion-parameter 1024px model is on Hugging Face4. This lowers the barrier for smaller ad-tech and creator-tool companies that cannot train frontier models. Early uptake looked modest, though. One snapshot showed the repository with 42 GitHub stars5.

The video question

The researchers want to extend HART beyond still images. They plan to build vision-language models on the architecture and apply it to video generation and audio prediction3. They argue that because the autoregressive part does most of the work, the same type of model behind LLMs, HART fits better with the new unified models that handle both vision and language11.

This is the most important part for video tools. Video is where diffusion costs grow fastest, because every frame multiplies the denoising work. A design that sets structure cheaply and then refines detail with a small, few-step module would, in principle, address the main cost of AI video. The source material makes no claims about video results, though, and readers should treat this as a research direction, not a capability.

Context: one of several speed-ups

HART is one of several efforts to cut the cost of diffusion. A year earlier, another MIT team, at CSAIL, introduced distribution matching distillation (DMD). DMD compresses multi-step diffusion into a single step and reached quality comparable to Stable Diffusion v1.5 while running 30 times faster. That approach has limits of its own: the fast model inherits the weaknesses of the model it learns from, including trouble with text and small faces.

The two methods work differently. DMD speeds up existing diffusion models through distillation, while HART changes the architecture so a language-model-style transformer does the heavy lifting. Both point the same way. In this analysis, the long-standing belief that quality requires many slow denoising steps is losing ground, and generation cost will keep falling.

The takeaway

HART's real contribution is that it shows a mostly autoregressive system can reach diffusion-level quality at 1024px, using a 37-million-parameter module to clean up what compression leaves behind1. The "nine times faster" headline is accurate but describes the best case, and the widely quoted 31 percent compute figure looks like a mislabeled quality metric. The paper's own numbers point to larger efficiency gains9. For teams producing ads, social content and media at scale, the implication is that good-quality generation will keep getting cheaper and more portable, including onto phones. That shift is likely to matter more than any single benchmark.