Splunk Observability Report: 56% Can't Measure AI Agent Quality

Companies are deploying AI agents faster than they can check them

Cisco's latest Splunk observability survey describes an awkward position for enterprise AI. Companies are putting autonomous agents into production faster than they can tell whether those agents are working, what they cost, or whether someone has tampered with them.

On October 20, Cisco published Splunk State of Observability 2026: Restoring Predictability in the AI Era, the sixth edition of the series. It is based on responses from 2,050 IT operations and engineering professionals in 10 countries.28 The report's main conclusion is that the push to deploy AI agents is moving faster than organizations' ability to monitor them.1 Oxford Economics ran the survey between February and March 2026. Respondents ranged from developers and SREs to CIOs and CTOs, and they were based in Australia, France, Germany, India, Japan, New Zealand, Singapore, Saudi Arabia, the UK and the US.3

The report deserves attention less for its broad message, which no one will dispute, than for where it says the failures happen. Those are the evaluation layer, the billing layer, and the boundary between operations and security.

Most teams can't score what their agents produce

The most useful finding for anyone running LLM systems in production is a measurement gap. Eighty-seven percent of respondents said monitoring AI is an organizational priority. Yet 54% said they have no clear business outcomes tied to that monitoring, and 56% said they cannot measure AI workload metrics such as LLM response quality.3 Hallucinations show the same pattern: 85% said visibility into hallucinations matters, but only 58% said they are effective at getting it.3

Put simply, more than half of the organizations that call AI monitoring a priority cannot check the quality of what their models return. Coverage at Neoteo made the same point, noting that teams may prioritize monitoring without any way to judge whether a system's responses meet their needs.

The report treats this as a different category of problem from classic application monitoring. Cisco argues that tracking agent behavior needs a new class of behavioral metrics tied to business outcomes and to the decisions agents make. Traditional uptime and latency dashboards do not capture these.3 Cisco also warns that when agents run in workflows such as software development and customer experience, one wrong action can turn into a serious business failure.8

This looks like the real bottleneck. Logging prompts and token counts is a solved problem. The unsolved part is the evaluation pipeline: deciding automatically, at production scale, whether an agent's answer or action was correct. A team that cannot score quality cannot tie monitoring to outcomes, which explains why the 56% and 54% figures sit so close together.

Regional results suggest the gap can be closed. In India, 51% of respondents called their monitoring of AI model quality very effective, against 35% globally, and only 46% said they could not measure metrics like response quality.27 The UK pairs heavy adoption with heavy worry. Thirty-two percent of UK respondents use agents all the time to diagnose root causes, against 20% globally, and 55% strongly worry about unintended consequences.

The cost problem is unpredictable bills, not big ones

The headline framing of hidden costs comes from the financial section, and the most telling number is about volatility rather than size. Fifty-five percent of respondents said they cannot control AI costs, and 60% said unexpected billing spikes cause more operational strain than high baseline spending.3 Cisco links this to the way organizations now run dozens of agents across several models, which makes usage and spend hard to untangle.5

This matches how agentic systems actually behave. One user request can set off a variable chain of model calls, tool calls and retries. Token use per task is therefore hard to predict, and a misbehaving agent loop can show up as a budget shock before anyone sees it as a reliability incident. Kamal Hathi, Splunk's general manager at Cisco, described the situation as AI enthusiasm running into the reality of the bill. He argued that building observability into an "AI tokenomics framework" can turn AI into a measurable investment rather than an unpredictable expense.8

The report also contains a figure that cuts against its own message. Fifty-seven percent of respondents said their telemetry costs now exceed the cost of the infrastructure being monitored.3 Monitoring is supposed to make AI spending predictable, yet for most respondents the monitoring itself costs more than the systems it watches. Agents produce far more traces, logs and evaluation data than conventional services, so this pressure is likely to grow. Cisco answers with consolidation, noting that 67% of respondents use a single observability platform to see where telemetry costs come from.7 That is a reasonable answer, but it is also the one that suits Splunk's business.

Security and observability are merging

The third theme is that agents are erasing the line between performance monitoring and security. Eighty-three percent of respondents said they worry about the unintended consequences of AI agents. Cisco says this is pushing ITOps and engineering teams closer to security teams, because agents can now reach across the whole technology stack.1

The year-over-year numbers support this more than most of the report. Eighty-one percent said monitoring AI workloads has increased collaboration between observability and security teams. The share that can accurately trace performance problems to security root causes rose from 65% to 73%.2 Joint troubleshooting with security teams rose from 62% in 2025 to 77%, and data sharing between the teams went from 74% to 84%.2

Jeetu Patel, Cisco's president and chief product officer, said that whether an agent has been compromised and whether it is doing its job well are inseparable questions that move at machine speed. He predicted that most companies will soon have far more agents than people.1 In his view, shared data platforms that support real-time detection and response are what make it possible to trust agents at scale.7

On this point the argument holds up on its own terms. An agent that suddenly calls an unfamiliar API, reads unusual data or burns through tokens could be buggy, prompt-injected or compromised. The telemetry that would expose each of those cases is mostly the same. Running separate tools and teams for each case is likely to slow diagnosis just when speed matters most.

What the "leaders" do differently

Like earlier editions, the report singles out a group of leaders. They share three practices: they let agents carry out remediation often, they closely track model quality, cost, usage, security and privacy, and they tie telemetry to business planning.2 Ninety-one percent of leaders can accurately attribute AI costs to specific agents, models and workloads, against 69% of everyone else. Leaders are also 2.4 times more likely to say their observability platform's ROI far exceeds expectations. Fifty-four percent of leaders say observability has generated at least $5 million in annual business revenue, against 28% of non-leaders.1

The first practice stands out. The organizations most worried about AI agents still trust agents to fix production problems, which suggests that confidence comes from instrumentation rather than restraint. Splunk's report page adds that 86% of respondents reported faster mean time to detect thanks to agentic AI. Neoteo noted that this figure is self-reported, not an independent performance measurement.

Where the coverage agrees, and what it leaves out

Nearly all the coverage of the report reproduces Cisco's numbers and framing closely, across trade, regional and financial outlets.1910 Regional outlets differ mainly in emphasis. Indian publications reported the India-specific figures.7 Middle East coverage led with comments from Splunk's regional vice president, Ahmed el Saadi, on managing performance, cost and risk.6 TechEdgeAI was more measured, observing that the report does not identify one approach that works for every organization, while still concluding that AI observability is becoming a core operating requirement.

Hardly any coverage raised three caveats readers should keep in mind. First, this is a vendor-sponsored survey whose recommendations, such as unified platforms, shared observability and security data, and tokenomics, line up with what Splunk and Cisco sell. Second, almost every figure is self-reported perception, not measured performance. A respondent saying they "can't control" costs is not an audited overrun. Third, the fieldwork ended in March but the results came out in late October, a long gap in a market where agent tooling changes every quarter.3

The main finding still holds up. The specific percentages may shift with different samples, but the pattern is internally consistent and fits how agentic systems behave in practice: high priority, weak measurement, volatile bills and converging security needs. Today's AI operations bottleneck is not model capability but evaluation and attribution, meaning the ability to say whether an agent did the right thing and what it cost. Companies that build that layer will scale their agents with confidence. Those that don't will keep discovering their agents' behavior through surprise invoices.