Amazon AGI Director: Reliability Blocks AI Agents, Not

Eighty-five percent of enterprises are piloting AI agents, but only 5% have shipped them to production — and Amazon’s AGI director says the bottleneck is reliability, not raw capability. Bryan Silverthorn, Director of AGI Autonomy at Amazon, made that case Tuesday at VB Transform 2026, drawing on both field failures and a Princeton-sourced framework to explain why agents that pass internal evaluations routinely collapse in production.

The 85-to-5 Gap Defining Enterprise AI in 2026

The chasm between AI agent pilots and production deployments is the defining tension in enterprise AI right now. Cisco data cited by VentureBeat shows 85% of enterprises are running agent pilots, while just 5% trust them enough to ship. That 80-point gap is not closing because models lack raw intelligence — it’s closing slowly because production environments expose failure modes that controlled evaluations never surface.

Silverthorn joined Amazon through its acquisition of Adept AI and now leads multimodal agent training inside Amazon’s AGI lab. His central argument at VB Transform: the industry is measuring the wrong things, and better benchmarks alone will not close the deployment gap.

Four Dimensions of Reliability That Evals Miss

Silverthorn’s proposed fix is a reliability framework, credited to Princeton research, that splits the concept into four distinct dimensions: consistency, robustness, predictability, and safety. Each addresses a different failure mode that current evaluations tend to collapse into a single pass/fail score.

“It unpacks different factors that I see tangled together in almost every eval I’ve ever seen,” Silverthorn said at the session.

The distinction matters practically. An agent can score high on consistency — producing the same output given the same input — while scoring poorly on robustness, meaning it degrades when inputs shift slightly. Treating these as one metric masks the specific weakness a production team needs to fix.

How a Serial Number Scanner Broke After Two Months

Silverthorn illustrated the framework with a real customer case. An enterprise deployed an agent for software QA work: the agent’s job was to extract serial numbers from on-screen displays. According to VentureBeat’s reporting, the system worked without incident for two months before it began intermittently reading wrong numbers.

The root cause was a robustness failure, not a capability failure. The underlying vision encoder behaved differently depending on where the serial number appeared on screen. A software update — imperceptible to human operators — shifted that position enough to trigger errors the original evaluation never tested for.

The lesson Silverthorn drew was about measurement scope, not model quality. “The models have to be better. Obviously, we’re working hard on making the models better,” he said. But the more actionable takeaway, he argued, is that teams must map their specific dimensions of variability and calibrate measurement rigor to match the stakes of the application.

Matching Measurement Rigor to Application Stakes

Not every agent deployment carries the same risk profile. A customer service chatbot that occasionally gives a wrong store hour is a different failure category than a QA agent that logs incorrect serial numbers into a compliance database. Silverthorn’s framework implies that teams should tier their evaluation depth accordingly — investing heavier measurement infrastructure where errors are costly or hard to reverse.

VentureBeat’s own proprietary research, presented at the same session, reinforced the point: half of surveyed enterprises reported that production failures, not pilot failures, were their primary reason for stalling deployment. The implication is that organizations are passing internal gates and then discovering variability only after real users encounter edge cases.

What This Means

Silverthorn’s argument reframes the AGI milestone conversation in a way that is more useful to practitioners than to benchmarkers. The field has spent years tracking performance on reasoning tasks, coding challenges, and multimodal comprehension — and models have improved substantially on all of them. But the 85-to-5 deployment ratio suggests that capability scores are not the binding constraint for enterprise adoption.

If reliability — specifically the four-dimensional version Silverthorn outlined — is the actual bottleneck, then the next meaningful AGI milestone for business may not be a model that scores higher on a reasoning benchmark. It may be a model, or a measurement methodology, that gives operators enough confidence to ship. Amazon’s positioning here is notable: by framing the problem as a measurement and reliability challenge rather than a raw capability gap, the company is signaling where its AGI lab is directing effort post-Adept acquisition.

For the broader industry, the implication is that evaluation infrastructure — not just model training — needs to mature before the pilot-to-production gap closes meaningfully.

FAQ

Why are only 5% of enterprises shipping AI agents to production?

According to Cisco data reported by VentureBeat, the gap between the 85% piloting agents and the 5% shipping them stems from reliability failures that internal evaluations don’t catch. Production environments expose variability in inputs, software dependencies, and edge cases that controlled benchmarks never surface.

What is the four-dimension reliability framework Bryan Silverthorn described?

Silverthorn credited Princeton research for a framework that breaks reliability into consistency, robustness, predictability, and safety. Each dimension captures a distinct failure mode — for example, an agent can be consistent under identical inputs but fail on robustness when inputs shift slightly, a distinction that single-score evals obscure.

Who is Bryan Silverthorn and what is his role at Amazon?

Bryan Silverthorn is Amazon’s Director of AGI Autonomy. He joined the company through Amazon’s acquisition of Adept AI and currently leads multimodal agent training inside Amazon’s AGI lab.