LLM Evaluation Gap: Teams Trace Agents but Rarely Judge Output

Enterprise AI teams have become good at watching their language-model systems. They are much less consistent at checking whether those systems give correct answers. New survey data and recent industry commentary point to the same imbalance. Observability has become standard practice. Systematic evaluation has not.

What the numbers show

The clearest evidence comes from LangChain's State of Agent Engineering survey. One account describes it as covering more than 1,300 professionals 1. Another puts it at 1,340 responses and dates its publication to June 2026 2. Both report the same headline figures:

• About 57% of respondents run agents in production.
• Roughly 89% have implemented some form of observability for those agents 12.

Evaluation tells a different story:

• 52.4% run offline evaluations.
• 37.3% run online evaluations.
• 29.5% say they run no evaluations at all 12.

The more detailed breakdown makes the gap sharper. Among teams already in production, observability rises to 94%, and 71.5% have detailed step and tool-call tracing 2. In practice, nearly every production team can replay what its agent did. Far fewer have a structured way to decide whether the result was any good.

This matters because quality, not speed, is what respondents say holds them back. A third cited quality as their top barrier to production. Latency came second at 20% 2. A separate summary of the same survey puts the quality figure at 32% 1.

Why monitoring alone falls short

The core problem is that the most damaging LLM failures do not look like failures. Traditional application performance monitoring was built to catch crashes, timeouts and error codes. Hallucinations, prompt injection, decaying retrieval quality and gradual prompt drift produce none of those signals 2. A system can be fully up, fast and cheap while confidently giving wrong answers. Obsium's framing captures the issue: the task is monitoring "when nothing is technically broken." 2

This is where the newer class of tooling comes in. Platforms such as Langfuse, LangSmith, Braintrust, Arize and Opik treat the LLM trace as their main unit of analysis. They record:

• prompts and completions
• retrievals and tool calls
• token counts, latency and cost

They then layer automated evaluators on top to score output quality 1. MarkTechPost describes the category as having moved from optional tooling to core infrastructure in 2026. It cites a Gartner projection that LLM observability investment will account for 50% of generative AI deployments by 2028, up from 15% in early 2026 1.

The plumbing is still not settled. OpenTelemetry's conventions for generative AI telemetry are not yet stable, according to Obsium 2. Teams that build heavily on any one vendor's trace format may face migration work later.

How enterprise roadmaps frame it

At the CIO and CTO level, the conversation centers on moving from pilots to production systems tied to business outcomes. Lumenalta argues that governance, data quality and integration maturity are now the pillars of scaling 3. It describes investment shifting toward platforms that standardize data access, controls and evaluation. It also expects retrieval-augmented generation to be the dominant architecture, because grounding answers in trusted sources supports citations and provenance 3.

Evaluation appears in that list. But when the same analysis names the shared services IT roadmaps will include, it lists prompt routing, vector search, feedback stores and cost observability 3. Evaluation is not singled out as a service of its own. That is a small detail, but it fits the survey pattern. The parts of the stack that are easy to measure and budget for get built first. The harder question of whether answers are correct gets folded into broader "governance."

The reading

Taken together, the three perspectives suggest the industry has solved visibility before it has solved judgment. Tracing tools are mature and widely adopted. Vendors are already competing on evaluation features. Analysts expect spending in the category to grow sharply.

The practical gap sits inside organizations. Almost a third of teams run no evaluations, even as quality is their biggest obstacle to shipping 12. That is not a tooling shortage. It is a process and ownership problem.

RAG-heavy enterprise architectures add to the pressure 3. Retrieval quality can decay silently, which is exactly the kind of failure monitoring misses 2.

For enterprise leaders, the takeaway is that evaluation should be planned as its own workstream. That means:

• offline test sets before release
• online scoring in production
• clear owners for quality metrics

Treating evaluation as a side effect of observability is not enough. Teams that make that shift are likely to be the ones that turn production quality from a blocker into a measurable target.