ManagedQA PartnersSee plans

AI code · 2026-09-12 · 3 min read

What the measured samples say about AI-generated code

Five studies with stated samples, read for what a tester should do differently. Every figure links to the evidence page with its source, its verification date and, for percentages, the denominator.

Every figure below is on the evidence page with a source URL, a publication date and the day we read it; every percentage states what it is a percentage of. Percentages without a denominator are not on this site.

Security tests

Veracode benchmarked more than 100 language models on an 80-task set across Java, Python, C# and JavaScript. 45% of the generated code samples failed security tests. (Source: Veracode, 2025-07-30 · verified 2026-09-21; denominator: code samples generated by 100+ LLMs on an 80-task benchmark) Cross-site scripting (user-typed text rendered as markup, so a script in a name or comment runs in other users' browsers) was the worst class: generated code failed to defend against it in 86% of relevant samples. (Source: Veracode, 2025-07-30 · verified 2026-09-21; denominator: code samples where cross-site scripting was relevant, in the Veracode 2025 benchmark) The October 2025 update on the same benchmark found the best model at a 72% pass rate, meaning more than one in four tasks still introduced a known vulnerability. (Source: Veracode, 2025-11-18 · verified 2026-09-21; denominator: 80 coding tasks in the Veracode benchmark, best-performing model)

What a tester does with that: AU-07 (user content renders inert) and LM-03 (model output treated as untrusted) run on every release, and no model upgrade is assumed to have fixed them.

Review findings

CodeRabbit compared 320 AI-co-authored pull requests with 150 human-only ones from open-source repositories. The AI-co-authored set averaged 10.83 issues per pull request against 6.45. (Source: CodeRabbit, 2025-12-17 · verified 2026-09-21) Relative to the human-only set in that sample, cross-site scripting issues were up to 2.74 times more frequent, error-handling gaps nearly twice as common, and logic issues 75% more common. (Source: CodeRabbit, 2025-12-17 · verified 2026-09-21)

What a tester does: EH-01 through EH-05 inject faults instead of reading for them, and the logic items (cascade deletes, concurrent writes, empty results) are run against data rather than reviewed in the diff.

Whole applications

Tenzai had five coding agents build 15 identical applications from the same prompts. The applications contained 69 vulnerabilities; every agent produced at least one server-side request forgery (the server made to fetch an attacker-chosen URL, reaching internal services); none of the 15 applications had CSRF (cross-site request forgery: another site making a user's browser send a state-changing request with the user's cookies) protection or security headers (HTTP response headers (HSTS, content-type, frame policy) that switch on the browser's built-in protections); every login page shipped without rate limiting. (Source: Tenzai, 2026-01 · verified 2026-09-21)

What a tester does: SC-05, AU-04 and AU-05 exist because those three were absent in every one of 15 apps.

Deployed apps

Escape scanned more than 5,600 publicly available applications built with prompt-to-app platforms and found more than 2,000 vulnerabilities, over 400 exposed secrets, and 175 instances of exposed personal data. (Source: Escape, 2025-10-29 · verified 2026-09-21) GitGuardian counted 28.65 million new hardcoded secrets on public GitHub in 2025. (Source: GitGuardian, 2026-03-17 · verified 2026-09-21)

What a tester does: SC-01 and SC-02 are automated, run on every release, and scan the built bundle (the JavaScript your app sends to the browser or ships inside the mobile app; anyone can read it) and the full history rather than the diff.

What the numbers do not say

They do not say which tool to use; the samples span tools, and none of the studies was designed to rank them. They do not say how much of your code is affected; they describe their own samples. The failure catalogue maps each finding to a pattern and a check, which is the only use we make of them.

More in AI code

Start with the audit

5 business days, $1,200, credited to the first month.

Blog · ManagedQA Partners