LLM features
Model or prompt changed without a regression run
A provider upgrade or prompt edit changes outputs across the product and nobody measures it before release.
TermsLLM feature (a product feature that calls a large language model, such as an assistant or a summariser) · golden set (a fixed list of model inputs with recorded expectations, re-run on every model or prompt change) · fault injection (making a dependency fail on purpose in staging to see what the user and the logs show)
highseverity if found in a releasesources verified 2026-09-21
Why AI-assisted code produces it
Veracode's benchmark shows security behaviour varies by model and did not improve uniformly with newer models; the same variance applies to product behaviour.
On the same 80-task benchmark, the best model evaluated through October 2025 reached a 72% security pass rate, so more than one in four coding tasks still introduced a known vulnerability.
Source: Veracode, 2025-11-18 · verified 2026-09-21 · denominator: 80 coding tasks in the Veracode benchmark, best-performing model
The OWASP Top 10 for LLM Applications 2025 lists prompt injection (LLM01), sensitive information disclosure (LLM02), improper output handling (LLM05), excessive agency (LLM06), system prompt leakage (LLM07) and unbounded consumption (LLM10) among the ten risks.
Source: OWASP GenAI Security Project, 2025-03-12 · verified 2026-09-21
How we detect it
Maintain a golden set and diff outputs on every change.
Most associated with
Tools: Cursor, GitHub Copilot, Claude Code, Windsurf, Lovable, Bolt
Stacks: Next.js
Checklist items that catch it
Model or prompt change runs the regression set
automated Maintain a golden set of inputs with expected properties; run it on every model, prompt or temperature change and diff against the recorded baseline.
The golden set runs on the change and the diff is reviewed and signed before release; regressions are listed in the verdict.
Dots: Starter, Growth, Scale. The item runs on every release for the plans marked.
More in LLM features
- critical
- critical
- critical
- high
- high
Check your codebase for this pattern
The coverage audit lists which catalogue patterns apply to your critical flows. 5 business days, $1,200, credited to a first month.