ManagedQA PartnersSee plans

Situation

Adding an LLM feature

A model call adds three surfaces: what it can spend, what it can be told by user content, and what its output can do. Each has a checklist item with a pass criterion.

Termsverdict (a one-word release result (pass, pass with findings, fail) with the findings and the checklist version behind it) · two-user test (sign in as one user, create a row, then try to read or change it as another user; pass is nothing comes back) · webhook (a message a provider such as Stripe sends to your server when something happens, like a payment) · fault injection (making a dependency fail on purpose in staging to see what the user and the logs show) · CI (the pipeline that builds and tests every change before it can be merged or deployed) · protected branch (the git branch that cannot be pushed to directly and is the only source of production deploys) · migration (a scripted change to the database schema that ships with a release and should be reversible) · store sandbox (Apple's and Google's test environments where purchases run without real money) · LLM feature (a product feature that calls a large language model, such as an assistant or a summariser)

The situation

The feature works in the demo. In production, a loop or a hostile user runs up the provider bill, a document with instructions in it steers the assistant, model output lands in the page as HTML, and a provider upgrade changes answers across the product. The OWASP Top 10 for LLM applications names each of these; the LLM category of the checklist tests them.

What to do this week

An ordered sequence from the checklist. Each item links to its pass criterion.

  1. Day 1

    Set and test the spend cap per user and globally. Exceed both in staging and confirm refusal, message and log entry.

    • LM-01Spend cap enforced per user and globallycritical
  2. Day 2

    Place instruction text in every input the model reads (messages, documents, retrieved records) and watch tool calls. No tool action may execute from injected text.

    • LM-02User content cannot trigger tool actions without checkscritical
  3. Day 3

    Force outputs containing markup, links and shell-like strings through every render and execution path.

    • LM-03Model output is treated as untrusted before rendering or executionhigh
    • AU-07User-supplied content renders inerthigh
  4. Day 4

    Read the assembled prompt for secrets and private data; inject provider errors, rate limits and 30-second delays.

    • LM-04System prompt and retrieved context contain no secretshigh
    • LM-06Model call failures and timeouts degrade gracefullymedium
  5. Day 5

    Build the golden set (60 cases is a good start) and record the baseline, so the next model or prompt change has something to diff against.

    • LM-05Model or prompt change runs the regression sethigh

Which plan

Growth $5,900/mo

Growth includes the six LLM items and runs the regression set on every model or prompt change. Starter does not include the LLM category.

Limits, inclusions and what is not included.

Email about Growth

What happens without it

OWASP's 2025 list ranks prompt injection first and includes improper output handling, excessive agency and unbounded consumption. None of them is exotic; each is the default behaviour of a model call wired straight to user input and to the page. The LLM items exist because the defaults are the failure.

The OWASP Top 10 for LLM Applications 2025 lists prompt injection (LLM01), sensitive information disclosure (LLM02), improper output handling (LLM05), excessive agency (LLM06), system prompt leakage (LLM07) and unbounded consumption (LLM10) among the ten risks.

Source: OWASP GenAI Security Project, 2025-03-12 · verified 2026-09-21

Questions

How to test an LLM feature before release?
Six items: spend cap (LM-01), injection through user content (LM-02), output treated as untrusted (LM-03), no secrets in the prompt (LM-04), regression set on model or prompt change (LM-05), failure and timeout handling (LM-06). Each has a pass criterion on the checklist page; Growth runs them every release.
LLM prompt injection testing checklist
Enumerate every input the model reads, including retrieved records and page titles. Place instruction text in each. Observe outputs and, more importantly, tool calls. Pass is no tool action from injected text and every tool call passing the same authorization as a direct request (LM-02).
What breaks in AI written code?
For LLM features specifically: no spend cap, injection through user content, tool actions with the app's credentials, output rendered as trusted, secrets in the system prompt, and model changes shipped without a regression run. Six patterns in the catalogue's LLM category, each with a source.

Start with the audit

5 business days, $1,200, credited to the first month.

Start with the auditRead the checklist