Source-linked DAST is dynamic application security testing that can connect request-level findings to the endpoint source file and line on supported source-discovered API scans. The result combines runtime evidence with a code location an agent or developer can investigate.
That last word is where AI-native security testing is quietly struggling. A finding that appears on one scan and vanishes on the next cannot be triaged, cannot be assigned, and cannot be fixed with confidence. The industry has spent two years asking whether AI can find vulnerabilities. The harder question is whether it can find the same one twice.
Key takeaways
- Large language models are non-deterministic by design. Even at temperature zero, the model vendor's own documentation says identical inputs can produce different outputs.
- Run the same AI security review over unchanged code twice and the set of findings can shift; independent research has measured accuracy swings of up to 15% across identical runs, with a best-to-worst gap as large as 70%.
- A finding you cannot reproduce is a notification, not a result. Deterministic, source-linked evidence is what makes it actionable.
- AI is generating more code, and more attack surface, than teams can verify by hand: 45% of AI-generated code samples in Veracode's 2025 research introduced an OWASP Top 10 vulnerability.
- The fix is not "trust the AI less." It is to anchor probabilistic analysis to deterministic evidence: determinism on the finding, agents on the fix, humans on the merge.
The reproducibility problem: why AI findings change between runs
Run the same static or dynamic scanner over unchanged code twice and you expect the same output. Run an AI security reviewer over the same code twice and you often do not get it.
This is not a rumor, it is a documented property of the technology. Anthropic's own documentation states that "even with temperature set to 0, the results will not be fully deterministic and identical inputs may produce different outputs across API calls." Independent academic work quantifies the effect: one study measuring repeated runs under supposedly deterministic settings found accuracy swings of up to 15%, with a gap between best and worst runs as large as 70%. Non-determinism is not a bug in one product. It is how these systems sample tokens, compounded by hardware-level floating-point behavior.
Point that same non-deterministic engine at your source code as a security reviewer and the consequence follows directly: run the review twice over unchanged code and the set of findings can differ. A vulnerability the model raises on one pass may be gone on the next. The code did not change; the reviewer did.
For a marketing chatbot, run-to-run variance is a curiosity. For a security program, it is a workflow failure. If a critical finding surfaces on Monday's pipeline run and disappears on Tuesday's, which one do you staff, patch, and report?
What source-to-finding linkage actually means
The answer is not to distrust AI. It is to give each actionable finding observed runtime evidence and, where supported, a stable source reference.
A supported source-linked API finding can carry four useful pieces of context:
- the endpoint that was exercised,
- the request that triggered the behavior,
- the response that proves it, and
- for supported source-discovered API findings, the source file and line that defines the affected endpoint.
The first three are the core of dynamic testing: evidence from the running application. The fourth narrows the investigation. On supported source-discovered API scans, Code Traceback identifies the file and line that defines the affected endpoint. That handler location is a starting point, not a guarantee that the vulnerable sink lives on the same line.
This is the same principle behind fixing API security problems at their source rather than at the perimeter. Runtime tells you a door is unlocked. Source linkage tells you which door, in which wall, and who framed it.
Why deterministic evidence beats a confidence score
Confidence scores are an attempt to make probabilistic output feel decisive. They rarely survive contact with a triage queue.
A confidence score answers "how sure is the model?" It does not answer "will I see this again?" or "where do I go to fix it?" When the underlying finding is unreproducible, a high confidence score is worse than useless, because it invites you to act on something that may not exist on the next run. That is the trap: a model can report high confidence in a finding that will not survive the next scan.
Observed runtime evidence answers questions a confidence score cannot. It gives a second engineer, ticketing system, or coding agent a concrete behavior to investigate. On supported source-discovered API scans, file and line context further narrows the search. Dynamic testing complements static analysis: runtime shows application behavior, while source context helps route the fix.
There is a coverage dimension too. A crawler only tests the endpoints it can reach. Deterministic local source discovery can add recognized routes from supported REST frameworks that documentation or observed traffic missed, and supported findings on those targets can retain endpoint source context.
A finding you cannot reproduce is a notification, not a result.
What a source-linked finding looks like in the workflow
Here is where the two halves of the argument meet: the same determinism that makes a finding trustworthy for a human is what makes it safe to hand to an agent.
| Property | LLM-only review | Crawl-only DAST | Source-linked DAST |
|---|---|---|---|
| Reproducibility | Varies run to run; identical inputs can produce different findings | Observed request and response behavior | Observed behavior plus stable endpoint source context where supported |
| Evidence delivered | Natural-language explanation plus a confidence score | Endpoint, request, response | Endpoint, request, response, plus file and line context on supported source-discovered API findings |
| Endpoint coverage | Depends on prompt and context window | Only what the crawler can reach | Crawler plus a source-derived API spec, reaching unlinked and undocumented routes |
| Triage | Re-verify each finding by hand | Reproducible proof of concept | Reproducible proof mapped to the owning code |
| Remediation handoff | A suggestion to re-check | Points at the endpoint | Points at the endpoint definition where supported, ready for investigation |
| Fit for AI agents | The agent must validate its own output | The agent gets runtime evidence | The agent gets runtime and supported source context; a human approves the merge |
That last column is the model we build toward: agent-ready, not autonomous. Through the NightVision MCP server and our Agent Skills, a coding agent can consume observed runtime evidence and source context where supported, propose a change, and rescan. The pull request still lands in front of a human.
This is not an argument against AI in security. It is an argument for giving it a deterministic floor to stand on. The probabilistic layer proposes; deterministic, source-linked evidence is what confirms it, assigns it, and fixes it. Agents move fast on top of facts, not on top of findings that might evaporate.
Related readingPart of our series on modern application security testing. Start with the pillar, What You Should Know About Application Security Testing, and see the wider trust argument in Autonomous Penetration Testing: A CISO Reality Check.
Frequently asked questions
What is source-linked DAST?
Source-linked DAST is dynamic application security testing that can connect request-level findings to the endpoint source file and line on supported source-discovered API scans. You get runtime evidence plus a code location to investigate; the reported handler line is not always the underlying sink.
Why do AI security findings change between runs?
Large language models sample their output probabilistically, and even with temperature set to zero they are not fully deterministic, a limitation the model vendors document themselves. Hardware-level floating-point behavior adds further variance. Run the same review twice and the set of findings can differ.
Does this mean AI is useless for security?
No. AI is genuinely useful for surfacing candidate issues and reasoning about code at scale. The problem is not that it finds nothing; it is that its raw output is not reproducible on its own. The lesson is not to stop using AI, it is to anchor its findings to deterministic, reproducible evidence so the ones worth acting on hold still from one run to the next.
How is source-linked DAST different from SAST?
SAST reads code statically and can point at a line, but it cannot prove a vulnerability is reachable and exploitable at runtime. DAST proves exploitability by exercising the running application. Source-linked DAST adds the line-level pointer back onto that runtime proof, giving you both the exploit and the address.
Can a DAST tool really map a runtime finding to a source line?
Yes, for supported source-discovered API scans. The source-derived route map can help the scanner reach endpoints a crawler would miss and can add endpoint file and line context to request-level findings.
How does this help AI coding agents fix vulnerabilities?
An agent is only as reliable as the evidence it reasons from. Handed a finding that may not reproduce, it makes a guess. Handed a reproducible finding tied to a specific line, it can draft a precise fix, which a human then reviews and merges.
Is this the same as autonomous pentesting?
No. NightVision is agent-ready, not autonomous. It supplies deterministic, source-linked evidence that agents and humans act on. For the fuller picture, see Autonomous Penetration Testing: A CISO Reality Check.
The bottom line
The last two years of AI in security were a story about generation. The story that matters now is reproducibility. A finding you cannot reproduce is a notification, not a result, and no confidence score fixes that. Give agents observed runtime evidence and, where supported, endpoint source context, and the proposed fix becomes something a human can evaluate. Determinism on the finding, agents on the fix, humans on the merge.