How agentic AI reduced test failure triage time by 80%

Every QA engineer worth his/her salt knows the drill: a regression suite finishes, several tests fail, and someone must sift through screenshots, traces, network calls, and logs to understand what went wrong.

Across multiple pipelines, this repetitive work can consume hours of engineering time. Manual triage takes roughly ten minutes per failed test case, pulling teams away from building and improving the product.

We explored whether an agentic AI workflow could handle the evidence gathering and first-pass diagnosis instead. In this implementation, it reduced triage time from approximately ten minutes to two minutes per failed test, while keeping human review, cost controls, and governance firmly in place.

This blog explains how the agentic approach saves time, reduces triage costs, and frees engineers to focus on strategic work.

Why does manual test failure triage not scale?

1. Every failure requires investigation: Whether the cause is a genuine regression, a flaky test, or an environment issue, an engineer must still review the evidence and classify it.

2. The quality and speed of diagnosis vary: An engineer familiar with the test may identify the cause quickly, while a newer team member may need considerably more time.

3. The evidence is fragmented: Screenshots, traces, network logs, console output, and stack traces often sit across different tools, making it time-consuming to reconstruct what happened.

The cost rarely appears as one major incident. Instead, it accumulates quietly across pipelines and sprints, consuming engineering time one failed test at a time.

How does an agentic AI pipeline analyze test failures?

Our approach employs a purpose-built workflow that leverages a large language model to analyze logs in a structured and reliable manner. 
A scheduled (or manually triggered) CI workflow does the following for every failed test in a run:

1. Extract failed tests: Parse the test report (HTML summary or JSON) to identify the failed tests and pull their metadata.

2. Navigate the report with a browser: Open each failed test's detail page and extract links to its trace and screenshots.

3. Collect visual evidence: Prioritize the last trace screenshot captured before the failure, along with any step-level screenshots that add context.

4. Extract network data: Pull the API calls, HTTP status codes, and failed requests/responses associated with the test.

5. Collect logs and errors: Gather console output, stack traces, and error messages.

6. Run smart processing: Group similar failures before sending them to the model and adjust wait times based on each test’s typical duration.

Once the system collects the evidence, it sends the complete context to the model. The model then analyzes screenshots, network activity, and logs together, just as an engineer would. It identifies the root cause, classifies the issue, recommends a fix, and generates clear action items linked to the original evidence.

Laid out end-to-end, the pipeline looks like the one in the image below:

Figure 1: input sources and the CI workflow assemble an evidence bundle; the AI analysis engine reasons over it under an approved inference profile and guardrail; outputs feed reports, dashboards, and downstream tooling — all wrapped in the same governance band described below.

How does failure grouping reduce the cost of AI test triage?

One of the most impactful optimizations in this pipeline is similarity grouping, significantly reducing redundant analyses and improving the efficiency of AI-assisted failure triage. When ten tests fail for the same underlying reason, such as a slow-loading modal, an expired test account, or a downstream API outage, there is little value in running ten separate model analyses.

Grouping similar failures before sending them to the model allows the system to analyze one representative case per group, reducing token usage while preserving coverage of the main failure patterns.

The next question that we need to address is: What guardrails are needed for safe AI test triage?

Sending test evidence including screenshots, API payloads, and error messages to an LLM requires careful consideration of security, privacy, and governance. Test data can contain fragments of production-like information, internal URLs, and system internals that shouldn't leave a controlled boundary.

Any team implementing an AI-assisted test triage solution should address governance considerations before integrating it into a production pipeline:

1. Use only approved models and access routes: Send every request through an approved connection. Do not allow direct, unmanaged access to model endpoints.

2. Apply content safeguards automatically: Check and remove sensitive information before sending content to the model and before returning the response.

3. Encrypt and record every request: Protect all data while it is being sent and keep a clear audit trail of what was sent, what was returned, and why.

4. Keep data only as long as necessary: Store evidence and analysis files for a limited period, just long enough to support tools such as dashboards or ticketing systems. Do not keep anything permanently by default.

Responsible AI testing starts with strong controls, followed by performance and trust. Skipping those controls may speed things up at first, but it can quickly turn a seemingly useful internal tool into a security risk.

What does an AI-generated test triage report include?

The output has to be immediately usable by an engineer, not text that needs further interpretation. A representative analysis looks like this:

A complete triage report including reason, root cause hypothesis, a concrete code-level fix, and a checklist of follow-up work is generated automatically with links back to the original screenshots and logs, so an engineer can verify the reasoning rather than take it on faith.

How much time can agentic AI save in test triage?

Manual triage runs at roughly ten minutes per failed test case; an agent-driven pass brings that down to about two minutes, an 80% reduction because the evidence-gathering and first-pass diagnosis happen automatically, leaving the engineer to review and act rather than investigate from scratch.

This impact compounds further at scale. Take a team running 500 test executions a week at a 10% failure rate, making it 50 failed tests to triage. Under a manual process, that's roughly 500 minutes (about 8.3 hours) of engineering time every week.

With an agent handling first-pass analysis, the same 50 failures take about 100 minutes (1.7 hours), a savings of around 6.5 hours every single week. Multiply that across a quarter, and it's multiple resource-days returned to actual engineering work rather than log-reading.

The agentic intervention identifies issues and categorizes in about two minutes instead of for hours, helping catch, understand and fix regressions in time. 

What does an 80% reduction mean for engineering teams?

Efficiency impact: manual triage vs. agentic triage

Let’s consider a team running 500 test executions a week at a 10% failure rate (50 failed test cases to triage weekly) as a working example:

Does agentic test triage replace quality engineers?

The pattern generalizes well beyond a single test framework: any pipeline that produces structured failure evidence (screenshots, traces, network logs, console output) is a candidate for this kind of agentic triage layer, whether the underlying tests are written in Playwright, Cypress, Selenium, or something else entirely.

The real opportunity for AI in testing lies in identifying the work that consumes human time without requiring human judgment. With the right evidence, guardrails, and cost controls, that work can be handed to an agent while engineers retain oversight of the decisions that matter.

Test failure triage is a strong example of this shift. By automating evidence collection and first-pass diagnosis, teams can shorten feedback loops, reduce repetitive effort, and return valuable engineering time to building better products.