AI code regression testing is the practice of re-testing your running application on every build to confirm that a vulnerability you already found and fixed has not come back. It matters now because AI coding agents rebuild features from the patterns they learned before the fix, and a revived flaw arrives with no new CVE and no new dependency alert.
Most security programs rest on a quiet assumption: that a fixed bug stays fixed. Somebody found it, somebody patched it, the ticket closed, and the whole class dropped down the risk register. AI code regression testing exists because that assumption is now measurably wrong. Independent research commissioned by Checkmarx and published in July 2026 put frontier coding agents to work on real feature tasks inside real repositories, and found that most of the working code they produced brought a previously fixed vulnerability back with it.
The industry nicknamed the result a zombie CVE. The name is catchy and slightly misleading, and the misleading part turns out to be the important part.
Key takeaways
- A zombie CVE is a vulnerability that was found, disclosed, and fixed, and that an AI coding agent later rebuilds from scratch into working code.
- In the benchmark Checkmarx commissioned, 83% to 95% of frontier-agent solutions were functionally correct, but only 24% to 36% were both functional and secure. Among the solutions that worked, roughly two thirds carried the old vulnerability back.
- The gap between "works" and "safe" widened from 38 points to 63 points in one model generation. Agents improved sharply at shipping working code and only modestly at shipping safe code.
- A flaw reintroduced into your code has no CVE, no advisory, and no upstream fix to apply. Version-based controls are looking in the wrong place.
- What catches it is behavioral re-testing, in the pipeline, on the cadence the code actually changes.
What a zombie CVE actually is
Start with the misleading part of the name. A zombie CVE is not a CVE. The identifier belonged to the upstream project that originally disclosed and patched the flaw. When the same flaw reappears in your own request handler, there is no identifier attached to it at all. You inherit the vulnerability and none of the paperwork that would have made it visible.
The research measured this with a benchmark called SusVibes, which is built the way a good experiment should be: backwards. Researchers find a commit that fixed a real security bug, roll the repository back to the state just before that fix, remove the feature implementation, and ask a coding agent to build the feature from the issue description. The project's own functional tests then grade whether the agent's code works. The security tests originally written to prove the bug was dead grade whether it is safe. The published benchmark spans 200 tasks across 108 open-source repositories and 77 CWE categories, with tasks averaging around 170 changed lines. Security is only scored on solutions that were functionally correct in the first place, which is the honest way to score it.
In other words, every task in the benchmark is a grave, and the question being asked is whether the agent digs.
The Starlette case: a path traversal that was already dead
The cleanest example in the research is CVE-2023-29159, a directory traversal flaw in Starlette, the Python ASGI framework that sits under a large share of modern Python web services. It was scored CVSS 7.5, affected versions 0.13.5 through the 0.26 line, and let a remote unauthenticated attacker read files from any service built on it. It was found, disclosed, and fixed in 0.27.0. Closed in 2023.
In the 2026 benchmark run, a frontier agent asked to rebuild the affected feature produced code that passed the functional tests and reopened the same traversal path.
Nothing exotic happened here. The agent was not jailbroken, poisoned, or prompted adversarially. It was asked to implement an ordinary feature, and it implemented it the way that feature is most commonly written across the code it learned from. That is the mechanism worth internalizing, because it does not go away with a better prompt: models reproduce what code usually looks like, and a security fix is by definition the less common version of the pattern. The corrected form is the minority spelling.
Working code and safe code are not the same number
The study re-ran the benchmark on current frontier models, each in its own agent CLI, and compared them against the previous generation. The results split cleanly along two axes.
On functionality, the models are dramatically better: 83% to 95% of solutions worked, up from 44% to 61% a generation earlier. On security, they are only somewhat better: 24% to 36% of solutions were both functional and secure. Do the arithmetic on those two ranges and you get the number that made the headlines. Of the agent solutions that actually worked, roughly 65% to 75% shipped the vulnerability back into the codebase along with the feature.
The gap between working and safe widened from 38 points to 63 points in a single model generation.
That widening is the finding that should shape your testing strategy, more than any single percentage. Coding agents are improving quickly on the axis that is cheap to measure and enforced by default, and improving slowly on the axis nobody checks automatically. Your functional test suite is the gate most teams actually block a merge on. In this data, that gate passes almost every time.
Review helps, and it is worth knowing exactly how much. A follow-up Checkmarx piece drawn from Black Hat USA 2026 reports that adding a security review step after generation lifted the secure-and-functional rate from 24% to 36% up to 47% to 56%. That is close to a doubling, and it is a strong argument for keeping human judgment in the path. It still leaves roughly half.
The scale question is answered by Checkmarx's 2026 Future of AppSec survey of 2,350 CISOs, security managers, and developers across 14 countries: 49% of production code is now AI-generated, and organizations where 81% to 100% of code is AI-generated reported shipping vulnerable code at 3.4 times the rate of organizations in the 1% to 20% band. This is not a future-state problem to schedule for next year's roadmap.
Why a revived bug is invisible to the checks you already run
A reintroduced vulnerability is unusually good at hiding, for three structural reasons.
It has no identifier. Software composition analysis is doing its job correctly when it stays silent: your dependencies are current, your Starlette version is patched, and the traversal is in code your own repository owns. There is no package to bump and no advisory to match against.
It does not look like a security change. It arrives as a feature commit that passes tests, inside a pull request that looks productive. Nothing in the diff announces that a control was quietly removed, because from the agent's perspective the control was never there.
It happened after your last proof. A penetration test, a pre-release assessment, an audit sample: each of those is a statement about one build at one moment. An agent-authored regression lands in a different build. The report is a photograph, and this walks into frame after the shutter closes.
AI code regression testing: what actually catches it
Two properties do the work, and they are both about mechanics rather than intelligence.
Test behavior, not version numbers. A version string tells you which patch was applied. It tells you nothing about whether the behavior that patch removed has been rebuilt somewhere else. This is the same lesson as Patched Twice, Bypassed Twice: A Case for Authentication Bypass Regression Testing, arriving from a new direction: the only durable check is whether the running application still refuses the request.
Run on the cadence the code changes. That cadence is no longer the speed at which humans write and review code. It is the speed at which agents commit. Testing quarterly against a codebase that changes hourly is not a coverage gap you can close with a better report.
Concretely, that means authenticated dynamic testing in the pipeline, exercising real routes against the running application on every build. If a traversal is reachable at a file-serving parameter, it does not matter whether a person or an agent wrote the handler, or whether the underlying flaw is two years old or two hours old. The request either walks out of the document root or it does not. Behavior is not subject to interpretation, which is exactly what you want when the code's author cannot explain its reasoning.
The other half is evidence. For any of this to close the loop rather than generate a queue, the finding has to arrive as a work order: the exact request and response, reproducible on demand, so that nobody has to re-derive it before anyone will act on it.
This is the workflow NightVision is built around. Scans run natively in CI/CD, through GitHub, GitLab, Jenkins, Azure DevOps, or Bitbucket, fully authenticated against the real application rather than its anonymous surface, so a regression is caught in the build that introduced it. Every finding carries the full request and response behind it and reproduces identically, which is what makes a regression arguable in a pull request instead of debatable in a meeting. For API findings on targets where NightVision also derived the API inventory from source, Code Traceback links the finding to the file and line that reintroduced it. Results flow back as SARIF into GitHub Security Alerts, so found, fixed, and verified becomes a record rather than a memory.
There is a neat symmetry available at the end of it. Over an MCP connection, the same coding agent that introduced the regression can read the finding, draft the fix, and trigger the rescan that proves it. Determinism on the finding, agents on the fix, humans on the merge. None of that shrinks the AppSec team; it is the only thing that scales one. The agents absorb the drafting and the re-triage, and AppSec keeps the part only AppSec can do, which is setting the standard, judging the fix, and approving what ships. Same team, far more surface.
What each control sees when a fixed bug comes back
| Control | Catches a flaw your own code reintroduced? | Cadence | What it hands you |
|---|---|---|---|
| Dependency scanning (SCA) | No, and correctly so. The flaw is in your code, not a package version | On manifest change | A package and a version to bump |
| Version and patch-level checks | No. Your versions are current and the behavior is back anyway | Continuous | An inventory statement |
| Static analysis in the pipeline | Sometimes, when the ruleset models the specific pattern and the scan runs on that build | Per build, if wired that way | A source location to review |
| Security review of the generated change | Substantially. Reported to lift the secure-and-functional rate from 24-36% to 47-56%, and to leave the rest | Per change, if reviewers have the time | Judgment and a decision |
| Point-in-time penetration test | Only if the regression predates the test | Annual or semi-annual | A narrative with reproduction steps |
| Continuous authenticated DAST in CI | Yes, when the behavior is reachable and the route is in the tested surface | Every build | A replayable request and response |
The table is not an argument for one row. Three of these controls catch versions of this problem, and a mature program runs all three, because each fails differently. Static analysis needs a rule for the pattern. Review needs a reviewer with time. Dynamic testing needs the route to be in the tested surface, which is why discovery depth matters as much as the testing itself. The narrower claim is the one worth taking away: none of these controls helps if it runs on a slower clock than the code changes, and the code is now changing at agent speed.
For the wider context on how dynamic testing fits alongside SAST, SCA, and the rest of the testing alphabet, see our guide to application security testing.
Frequently asked questions
What is a zombie CVE?
A zombie CVE is a vulnerability that was already found, disclosed, and fixed in a project, and that an AI coding agent later rebuilds into a codebase while implementing a feature. The name is a useful shorthand and a slightly inaccurate one: the original CVE identifier belonged to the upstream project, so when the same flaw reappears in your own code there is no CVE, no advisory, and no version to bump. You get the vulnerability without any of the metadata that would normally make it visible to your tooling.
Does this mean AI coding agents make applications less secure?
It means they generate working code far faster than they generate secure code, and the distance between those two things is growing. In the benchmark Checkmarx commissioned, frontier agents produced functionally correct solutions 83% to 95% of the time but functional-and-secure solutions only 24% to 36% of the time, a gap that widened from 38 points to 63 points in one model generation. The practical reading is not that agents should be avoided; it is that the functional test suite, which is the gate most teams enforce, is not a security gate and was never designed to be one.
Will software composition analysis catch a reintroduced vulnerability?
No, and that is not a failure of the tool. SCA compares the dependencies you pull in against known advisories. When an agent rebuilds a vulnerable pattern inside code your repository owns, there is no dependency at fault and no advisory to match. Your SCA results stay clean and remain accurate. Catching this class requires testing the behavior of the running application rather than the provenance of its packages.
How is AI code regression testing different from ordinary regression testing?
The mechanics are the same and the trigger is different. Ordinary regression testing assumes a human changed something and re-verifies the affected behavior. AI code regression testing assumes any commit can silently undo a prior security fix, including a commit that looks like routine feature work and passes every functional test, so it re-verifies the security-relevant behavior on every build rather than on suspicion. The practical difference is cadence and scope: continuous rather than targeted, and behavioral rather than version-based.
How often should the re-test run?
On the same clock as your deploys, which for most teams means in the pipeline on every build, or nightly against a staging environment at minimum. The point of continuous re-testing is attribution as much as detection: a regression caught in the build that introduced it comes with an obvious owner and a small diff, while the same regression found a quarter later comes with an archaeology project.
Can an AI agent fix the regression it introduced?
Often, yes, provided the finding arrives in a form it can act on. A coding agent given a replayable request and response, and a location in the code, is genuinely good at drafting the remediation and re-running the scan to verify it. What does not change is who approves the result. The dividing line that holds up is determinism on the finding, agents on the fix, humans on the merge, and it holds up precisely because the agent's confidence in its own patch is the thing the research just called into question.