Open-Source Vision Models Rival CLIP as FDA Holds Back LLMs

Open vision catches up to closed labs

For roughly four years, the visual front end of most multimodal AI systems has come from one place: OpenAI's CLIP. Released in early 2021, CLIP became the default vision encoder for builders of multimodal foundation models 1. The approach was elegant. CLIP trained an image encoder and a text encoder together on 400 million image-text pairs scraped from the web, pulling matching pairs closer in a shared representation space and pushing mismatched pairs apart 5. That contrastive objective gave models a basic form of grounded understanding, and CLIP still serves as the visual layer for many multimodal large language models 5.

That dominance is now being contested from the open-source side. Researchers behind OpenVision describe a family of vision encoders that match or beat CLIP when plugged into multimodal frameworks such as LLaVA 1. Their central argument is about openness rather than raw performance. Alternatives such as Google's SigLIP have challenged CLIP, but the authors say none are fully open, because their training data is proprietary, their training recipes are unpublished, or both 1. OpenVision is built on existing open work, using the CLIPS training framework and the Recap-DataComp-1B dataset, and the team releases the full pipeline. The paper also reports findings on what improves encoder quality 1.

The Allen Institute for AI (Ai2) is pushing in a parallel direction with Molmo 2, an open model family that can watch video, track objects, count events, and identify where and when things happen in a clip 2. GeekWire presents it as an open alternative to closed video systems such as Google's Gemini, OpenAI's GPT-4o, and Meta's Perception LM 2. Ai2 is a nonprofit, so it is not chasing commercial market share. Its stated goal is to advance the field and release the results freely. Molmo 2 joins its earlier open text (OLMo) and image (Molmo) models in a progression toward a unified model that reasons across modalities 2.

Why the broader picture matters

A new open-access review in Discover Informatics places these releases in context. Authors Azhar A. Hadi and K. P. Supreethi narrowed more than 400 candidate papers to 120 studies and compared thirteen pretrained multimodal models released between 2020 and 2025 5. They argue that earlier surveys tended to look at single domains and lacked a side-by-side view of architectures, datasets, and training objectives 5. They also trace how these models are reaching fields as different as radiology and wildlife monitoring 5.

Radiology is where the gap between research progress and real-world deployment is easiest to see.

The clinical bottleneck

Regulators have been approving narrow imaging AI at a rapid pace. The Imaging Wire, reporting on the FDA's update covering authorizations through September 2025, says the agency has cleared 1,356 AI-enabled devices since it began tracking, an 8.5% rise from its previous report 4. Of those, 1,039 are radiology devices, about 77% of the total 4. Radiology's lead goes back to 1998, when the first authorization went to a mammography CAD tool 4.

An IntuitionLabs overview gives different figures. It describes 115 radiology algorithms added by mid-2025 and roughly 873 total 3. The two reports cover different time windows and appear to count differently, so the exact numbers should be treated with some caution. Both point the same way, though: medical imaging is the specialty where regulated AI is most firmly established 34.

The more important point concerns what has not been cleared. IntuitionLabs notes that generative and foundation models, including GPT-4V-style multimodal systems, could support uses such as automated report generation and multimodal analysis. These capabilities have not been validated or approved for routine clinical use, and current LLM use is characterized as "unauthorized" under medical regulations 3. It is more accurate to say the FDA has not yet cleared these tools than to say it is actively blocking them. The practical result is similar either way.

Reading the moment

These developments are related. The open-source movement is mostly responding to a reproducibility problem. When a field depends on a vision encoder whose training data is unknown, researchers cannot fully audit what the model learned 1. That opacity is a scientific frustration for academic labs, and it becomes a serious obstacle in medicine, where validation requires knowing what a system was trained on.

One reasonable interpretation is that fully open encoders such as OpenVision, and transparent model families such as Molmo, could eventually make clinical validation easier, because regulators and hospitals could examine the full pipeline. That outcome is speculative. None of these releases is a medical device, and matching CLIP on general benchmarks is very different from passing clinical review. In the near term, the evidence describes two tracks moving at different speeds. Open multimodal research is quickly catching up to closed labs. Clinical medicine continues to approve narrow, task-specific imaging tools by the hundreds while keeping general-purpose multimodal LLMs out of routine use.