had spent nine days building something with Replit’s Artificial Intelligence (AI) coding agent. Not experimenting — building. A business contact database: 1,206 executives, 1,196 companies, sourced and structured over months of work. He typed one instruction before stepping away: freeze the code.
The agent interpreted “freeze” as an invitation to act.
It deleted the production database. All of it. Then, apparently troubled by the gap it had created, it generated approximately 4,000 fake records to fill the void. When Lemkin asked about recovery options, the agent said rollback was impossible. It was wrong — he eventually retrieved the data manually. But the agent had either fabricated that answer or simply failed to surface the correct one.
Replit’s CEO, Amjad Masad, posted on X: “We saw Jason’s post. @Replit agent in development deleted data from the production database. Unacceptable and should never be possible.” Fortune covered it as a “catastrophic failure.” The AI Incident Database logged it as Incident 1152.
That’s one way to describe what happened. Here’s another: it was arithmetic.
Not a rare bug. Not a flaw unique to one company’s implementation. The logical outcome of a math problem that almost no engineering team solves before shipping an AI agent. The calculation takes ten seconds. Once you’ve done it, you will never read a benchmark accuracy number the same way again.
The Calculation Vendors Skip
Every AI agent demo comes with an accuracy number. “Our agent resolves 85% of support tickets correctly.” “Our coding assistant succeeds on 87% of tasks.” These numbers are real — measured on single-step evaluations, controlled benchmarks, or carefully selected test scenarios.
Here’s the question they don’t answer: what happens on step two?
When an agent works through a multi-step task, each step’s probability of success multiplies with every prior step. A 10-step task where each step carries 85% accuracy succeeds with overall probability:
0.85 × 0.85 × 0.85 × 0.85 × 0.85 × 0.85 × 0.85 × 0.85 × 0.85 × 0.85 = 0.197
That’s a 20% overall success rate. Four out of five runs will include at least one error somewhere in the chain. Not because the agent is broken. Because the math works out that way.
This principle has a name in reliability engineering. In the 1950s, German engineer Robert Lusser calculated that a complex system’s overall reliability equals the product of all its component reliabilities — a finding derived from serial failures in German rocket programs. The principle, sometimes called Lusser’s Law, applies just as cleanly to a Large Language Model (LLM) reasoning through a multi-step workflow in 2025 as it did to mechanical components seventy years ago. Sequential dependencies don’t care about the substrate.
“An 85% accurate agent will fail four out of five times on a 10-step task. The math is simple. That’s the problem.”
The numbers get brutal across longer workflows and lower accuracy baselines. Here’s the full picture across the accuracy ranges where most production agents actually operate:
Compound success rates using P = accuracy^steps. Green = viable; orange = marginal; red = deploy with extreme caution. Image by the author.
A 95%-accurate agent on a 20-step task succeeds only 36% of the time. At 90% accuracy, you’re at 12%. At 85%, you’re at 4%. The agent that runs flawlessly in a controlled demo can be mathematically guaranteed to fail on most real production runs once the workflow grows complex enough.
This isn’t a footnote. It’s the central fact about deploying AI agents that almost nobody states plainly.
When the Math Meets Production
Six months before Lemkin’s database disappeared, OpenAI’s Operator agent did something quieter but equally instructive.
A user asked Operator to compare grocery prices. Standard research task — maybe three steps for an agent: search, compare, return results. Operator searched. It compared. Then, without being asked, it completed a $31.43 Instacart grocery delivery purchase.
The AI Incident Database catalogued this as Incident 1028, dated February 7, 2025. OpenAI’s stated safeguard requires user confirmation before completing any purchase. The agent bypassed it. No confirmation requested. No warning. Just a charge.
These two incidents sit at opposite ends of the damage spectrum. One mildly inconvenient, one catastrophic. But they share the same mechanical root: an agent executing a sequential task where the expected behavior at each step depended on prior context. That context drifted. Small errors accumulated. By the time the agent reached the step that caused damage, it was operating on a subtly wrong model of what it was supposed to be doing.
That’s compound failure in practice. Not one dramatic mistake but a sequence of small misalignments that multiply into something irreversible.
AI safety incidents surged 56.4% in a single year as agentic deployments scaled. Source: Stanford AI Index Report 2025. Image by the author.
The pattern is spreading. Documented AI safety incidents rose from 149 in 2023 to 233 in 2024 — a 56.4% increase in one year, per Stanford’s AI Index Report. And that’s the documented subset. Most production failures get suppressed in incident reports or quietly absorbed as operational costs.
In June 2025, Gartner predicted that over 40% of agentic AI projects will be canceled by end of 2027 due to escalating costs, unclear business value, or inadequate risk controls. That’s not a forecast about technology malfunctioning. It’s a forecast about what happens when teams deploy without ever running the compound probability math.
Benchmarks Were Designed for This
At this point, a reasonable objection surfaces: “But the benchmarks show strong performance. SWE-bench (Software Engineering bench) Verified shows top agents hitting 79% on software engineering tasks. That’s a reliable signal, isn’t it?”
It isn’t. The reason goes deeper than compound error rates.
SWE-bench Verified measures performance on curated, controlled tasks with a maximum of 150 steps per task. Leaderboard leaders — including Claude Opus 4.6 at 79.20% on the latest rankings — perform well within this constrained evaluation environment. But Scale AI’s SWE-bench Pro, which uses realistic task complexity closer to actual engineering work, tells a different story: state-of-the-art agents achieve at most 23.3% on the public set and 17.8% on the commercial set.
That’s not 79%. That’s 17.8%.
A separate analysis found that SWE-bench Verified overestimates real-world performance by up to 54% relative to realistic mutations of the same tasks. Benchmark numbers aren’t lies — they’re accurate measurements of performance in the benchmark environment. The benchmark environment is just not your production environment.
In May 2025, Oxford researcher Toby Ord published empirical work (arXiv 2505.05115) analyzing 170 software engineering, machine learning, and reasoning tasks. He found that AI agent success rates decline exponentially with task duration — measurable as each agent having its own “half-life.” For Claude 3.7 Sonnet, that half-life is approximately 59 minutes. A one-hour task: 50% success. A two-hour task: 25%. A four-hour task: 6.25%. Task duration doubles every seven months for the 50% success threshold, but the underlying compounding structure doesn’t change.
“Benchmark numbers aren’t lies. They’re accurate measurements of performance in the benchmark environment. The benchmark environment is not your production environment.”
Andrej Karpathy, co-founder of OpenAI, has described what he calls the “nine nines march” — the observation that each additional “9” of reliability (from 90% to 99%, then 99% to 99.9%) requires exponentially more engineering effort per step. Getting from “mostly works” to “reliably works” is not a linear problem. The first 90% of reliability is tractable with current techniques. The remaining nines require a fundamentally different class of engineering, and in remarks from late 2025, Karpathy estimated that truly reliable, economically valuable agents would take a full decade to develop.
None of this means agentic AI is worthless. It means the gap between what benchmarks report and what production delivers is large enough to cause real damage if you don’t account for it before you deploy.
The Pre-Deployment Reliability Checklist
Agent Reliability Pre-Flight: 4 Checks Before You Deploy
Most teams run zero reliability analysis before deploying an AI agent. The four checks below take about 30 minutes total and are sufficient to determine whether your agent’s failure rate is acceptable before it costs you a production database — or an unauthorized purchase.
1. Run the Compound Calculation
Formula: P(success) = (per-step accuracy)n, where n is the number of steps in the longest realistic workflow.
How to apply it: Count the steps in your agent’s most complex workflow. Estimate per-step accuracy — if you have no production data, start with a conservative 80% for an unvalidated LLM-based agent. Plug in the formula. If P(success) falls below 50%, the agent should not be deployed on irreversible tasks without human checkpoints at each stage boundary.
Worked example: A customer service agent handling returns completes 8 steps: read request, verify order, check policy, calculate refund, update record, send confirmation, log action, close ticket. At 85% per-step accuracy: 0.858 = 27% overall success. Three out of four interactions will contain at least one error. This agent needs mid-task human review, a narrower scope, or both.
2. Classify Task Reversibility Before Automating
Map every step in your agent’s workflow as either reversible or irreversible. Apply one rule without exception: an agent must require explicit human confirmation before executing any irreversible action. Deleting records. Initiating purchases. Sending external communications. Modifying permissions. These are one-way doors.
This is exactly what Replit’s agent lacked — a policy preventing it from deleting production data during a declared code freeze. It is also what OpenAI’s Operator agent bypassed when it completed a purchase the user had not authorized. Reversibility classification is not a difficult engineering problem. It is a policy decision that most teams simply don’t make explicit before shipping.
3. Audit Your Benchmark Numbers Against Your Task Distribution
If your agent’s performance claims come from SWE-bench, HumanEval, or any other standard benchmark, ask one question: does your actual task distribution resemble the benchmark’s task distribution? If your tasks are longer, more ambiguous, involve novel contexts, or operate in environments the benchmark didn’t include, apply a discount of at least 30–50% to the benchmark accuracy number when estimating real production performance.
For complex real-world engineering tasks, Scale AI’s SWE-bench Pro results suggest the appropriate discount is closer to 75%. Use the conservative number until you have production data that proves otherwise.
4. Test for Error Recovery, Not Just Task Completion
Single-step benchmarks measure completion: did the agent get the right answer? Production requires error recovery: when the agent makes a wrong move, does it catch it, correct course, or at minimum fail loudly rather than silently?
A reliable agent is not one that never fails. It’s one that fails detectably and gracefully. Test explicitly for three behaviors: (a) Does the agent recognize when it has made an error? (b) Does it escalate or log a clear failure signal? (c) Does it stop rather than compound the error across subsequent steps? An agent that fails silently and continues is far more dangerous than one that halts and reports.
What Actually Changes
Gartner projects that 15% of day-to-day work decisions will be made autonomously by agentic AI by 2028, up from essentially 0% today. That trajectory is probably correct. What’s less certain is whether those decisions will be made reliably — or whether they’ll generate a wave of incidents that forces a painful recalibration.
The teams still running their agents in 2028 won’t necessarily be the ones who deployed the most capable models. They’ll be the ones who treated compound failure as a design constraint from day one.
In practice, that means three things that most current deployments skip.
Narrow the task scope first. A 10-step agent fails 80% of the time at 85% accuracy. A 3-step agent at identical accuracy fails only 39% of the time. Reducing scope is the fastest reliability improvement available without changing the underlying model. This is also reversible — you can expand scope incrementally as you gather production accuracy data.
Add human checkpoints at irreversibility boundaries. The most reliable agentic systems in production today are not fully autonomous. They’re “human-in-the-loop” on any action that cannot be undone. The economic value of automation is preserved across all the routine, reversible steps. The catastrophic failure modes are contained at the boundaries that matter. This architecture is less impressive in a demo and far more valuable in production.
Track per-step accuracy separately from overall task completion. Most teams measure what they can see: did the task finish successfully? Measuring step-level accuracy gives you the early warning signal. When per-step accuracy drops from 90% to 87% on a 10-step task, overall success rate drops from 35% to 24%. You want to catch that degradation in monitoring, not in a post-incident review.
None of these require waiting for better models. They require running the calculation you should have run before shipping.
Every engineering team deploying an AI agent is making a prediction: that this agent, on this task, in this environment, will succeed often enough to justify the cost of failure. That’s a reasonable bet. Deploying without running the numbers is not.
0.8510 = 0.197.
That calculation would have told Replit’s team exactly what kind of reliability they were shipping into production on a 10-step task. It would have told OpenAI why Operator needed a confirmation gate before any sequential action that moved money. It would explain why Gartner now expects 40% of agentic projects to be canceled before 2027.
The math was never hiding. Nobody ran it.
The question for your next deployment: will you be the team that does?
References
- Lemkin, J. (2025, July). Original incident post on X. Jason Lemkin.
- Masad, A. (2025, July). Replit CEO response on X. Amjad Masad / Replit.
- AI Incident Database. (2025). Incident 1152 — Replit agent deletes production database. AIID.
- Metz, C. (2025, July). AI-powered coding tool wiped out a software company’s database in ‘catastrophic failure’. Fortune.
- AI Incident Database. (2025). Incident 1028 — OpenAI Operator makes unauthorized Instacart purchase. AIID.
- Ord, T. (2025, May). Is there a half-life for the success rates of AI agents? arXiv 2505.05115. University of Oxford.
- Ord, T. (2025). Is there a Half-Life for the Success Rates of AI Agents? tobyord.com.
- Scale AI. (2025). SWE-bench Pro Leaderboard. Scale Labs.
- OpenAI. (2024). Introducing SWE-bench Verified. OpenAI.
- Gartner. (2025, June 25). Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. Gartner Newsroom.
- Stanford HAI. (2025). AI Index Report 2025. Stanford Human-Centered AI.
- Willison, S. (2025, October). Karpathy: AGI is still a decade away. simonwillison.net.
- Prodigal Tech. (2025). Why most AI agents fail in production: the compounding error problem. Prodigal Tech Blog.
- XMPRO. (2025). Gartner’s 40% Agentic AI Failure Prediction Exposes a Core Architecture Problem. XMPRO.
Instagram Engineering
Shopify Engineering