By Shireen Nagdive and Riya Suri.
In our Engineering Energizers Q&A series, we highlight the engineering minds building technology across Salesforce. Today, we spotlight Shireen Nagdive, Engineering Manager at Salesforce, whose team faced a terrifying engineering problem: deliberately trigger more than 600 test failures, and still detect any genuine regression camouflaged among them. The catalyst was the retirement of Salesforce’s Simple Object Access Protocol (SOAP) API login mechanism. The failures could produce similar exceptions, span multiple teams, and appear expected even when something else had broken. One wrong classification could bury a real defect inside the blast radius or send engineers chasing the wrong problem for days.
Explore how Shireen’s team built a Model Context Protocol (MCP)-based AI tool to distinguish expected failures from real regressions in ambiguous stack traces, route related failures to the correct owners across teams, and overcome a 50 failure reassignment interface limit without creating duplicate work items or misrouting failures, reducing an estimated two or three days of triage to approximately 90 minutes.
One platform change could unleash hundreds or thousands of test failures, and somewhere inside that blast radius, a genuine regression could be hiding in plain sight. Expected failures and real problems could produce similar exceptions, originate from different test classes, and span multiple scrum teams. Classify them incorrectly, and engineers could either spend days chasing an intentional behavior change or allow an unrelated defect to disappear inside the noise.
The team’s mission is to retire SOAP login and help customers migrate to safer authentication mechanisms. The team needed to introduce restrictions in phases while communicating the change and giving customers time to migrate. Internally, however, enforcing the new behavior intentionally broke tests across Salesforce teams and services that still depended on the previous authentication model.
That left the team with a brutal requirement: distinguish anticipated failures from unrelated problems, recognize shared causes, identify the correct owners, and reassign hundreds of failures without compounding the duplicate and misrouting problems engineers already faced. AI-Assisted Test Failure Buddy had to eliminate days of operational work without removing the engineering judgment required to determine what had actually broken.
The existing workflow turned every failure into a sequence of manual decisions and individual clicks. Engineers had to open stack traces, determine ownership, and repeatedly reassign work through an interface that could process only 50 failures at a time. That limit was not the only failure point. The team-assignment workflow could create multiple work items for the same team and, at high volume, sometimes route failures incorrectly. Engineers then had to determine whether similar work items represented separate problems or duplicate reports of the same issue.
As a result, triage became a team activity rather than a routine task. Teams sometimes held dedicated meetings or divided the work among two or three engineers simply to understand what had failed and where it belonged. The team built AI-Assisted Test Failure Buddy to replace that fragmented process with one coordinated, automated test-failure triage workflow.
The architectural danger was giving the model access to powerful operations without clearly defining when and how each should run. The system needed to retrieve failures, obtain stack traces, analyze patterns, and reassign work while preserving authentication, input validation, and the identifiers required to act on the correct records.
In response, the team structured AI-Assisted Test Failure Buddy as a Python and FastMCP process exposing eight tools to Claude Code or Cursor. Those tools connect to the APIs behind GUS, the internal Salesforce workflow system engineers use to manage work items and test failures. The team separated what the model could request from what deterministic code would execute. An interface layer defines typed, self-describing MCP tools so the model understands the available capabilities and required arguments. A core logic layer retains control over API calls, identifiers, response parsing, validation, batching, clustering, pattern matching, and reassignment, while a support layer manages authentication and token handling.
Together, those layers support four stages: retrieve failures, gather context from stack traces, detect patterns or apply categories, and reassign failures individually, in bulk, or by team. Engineers can move through that entire workflow using natural-language requests instead of manually opening, analyzing, and routing hundreds of records.

Architectural Diagram (MCP Workflow)
Raw stack traces gave the system no stable machine-readable key that directly identified a failure’s root cause especially at scale. AI-Assisted Test Failure Buddy had to derive useful signatures by walking through each trace and extracting exception types, messages, assertion lines, and other recognizable patterns. Even a strong signature could not resolve every case. Different test classes could share the same underlying failure, while one error pattern could span several scrum teams. Conversely, the same general exception type could describe an anticipated Salesforce SOAP API login failure in one context and an unrelated problem in another. Grouping failures by superficial similarity could therefore send the wrong work to the wrong team.
To address that ambiguity, the team combined deterministic processing with AI reasoning. Code handles API operations, identifier lists, validation, and regular-expression parsing. The model helps interpret novel stack traces, examine unmatched clusters, and reason through ambiguous ownership. For example, the SOAP login preset can recognize supported strings such as INSUFFICIENT_ACCESS and other operation-specific patterns associated with the enforcement change. This gives the tool a defined path for known failures while reserving AI-assisted analysis for unmatched cases that require deeper investigation.
Speed would have meant little if AI-Assisted Test Failure Buddy simply reproduced the existing workflow’s errors at a larger scale. The tool needed to reduce manual effort while accurately grouping related failures, preserving the correct identifiers, targeting the right owners, and avoiding duplicate work items. To control those risks, the MCP tool assigns related failures to one specified existing work item rather than allowing each batch to create another. Explicit failure IDs replace the broader interface action that had sometimes misrouted results, while a shared message lets engineers communicate the reason and available mitigation consistently across an entire group.
The result was a substantial reduction in time and engineering effort. One work item containing 600 failures was estimated to require several engineers and two or three days of manual triage. Using AI-Assisted Test Failure Buddy, one engineer processed those failures in approximately an hour to an hour and a half. The tool has since been used by approximately three or four teams. Although it does not yet include adoption telemetry, Slack responses from other engineers have confirmed that the workflow is helping teams manage their own test failures.
Bulk retrieval and reassignment were mechanical problems, but deciding what should move, and where, was not. A conventional script could call APIs, join failure IDs, parse known patterns, and submit a reassignment. It could not reliably determine what an unfamiliar stack trace meant or resolve unclear ownership across teams.
That limitation established the boundary between automation and AI. Deterministic code handles API calls, identifiers, validation, and reassignment mechanics. AI interprets novel stack traces, examines unmatched clusters, resolves ambiguous ownership, and connects a detected cause with the appropriate mitigation. Engineers then review the output and determine whether its conclusions and proposed actions are correct.
The team also used AI-assisted software development to accelerate development of the tool. The work began with a detailed markdown file describing the problem, desired design, and required behavior, then used Claude Code to generate and refine the implementation while applying engineering judgment, keeping human in the loop for review and testing the results. Ultimately, code controls the mechanics, AI interprets the ambiguity, and engineers retain authority over the decisions that determine where the work belongs.