Situation

A release broke production

Something visible failed after a deploy. The fix is out or nearly out. The question now is what changes so the next release does not do the same.

Termsverdict (a one-word release result (pass, pass with findings, fail) with the findings and the checklist version behind it) · two-user test (sign in as one user, create a row, then try to read or change it as another user; pass is nothing comes back) · webhook (a message a provider such as Stripe sends to your server when something happens, like a payment) · fault injection (making a dependency fail on purpose in staging to see what the user and the logs show) · CI (the pipeline that builds and tests every change before it can be merged or deployed) · protected branch (the git branch that cannot be pushed to directly and is the only source of production deploys) · migration (a scripted change to the database schema that ships with a release and should be reversible) · store sandbox (Apple's and Google's test environments where purchases run without real money) · LLM feature (a product feature that calls a large language model, such as an assistant or a summariser)

The situation

The usual shape: the change was small, review was quick, the tests that exist passed, and the failure was in something nobody had a test for. With AI-assisted code the missing test is often in the failure branch, the policy, or the webhook handler, because those are the parts a prompt does not describe.

What to do this week

An ordered sequence from the checklist. Each item links to its pass criterion.

Day 1
Rehearse the rollback of this release in staging and time it. Record the build, the checklist version (even if none was used) and who tested what. This is the release record from now on.
- DR-02Rollback rehearsed on the previous releasehigh
- DR-05Release notes carry the checklist version and verdictlow
Day 2
Force the backing call that failed to fail again in staging and confirm the user sees a message and a log entry exists. Then do the same for every other critical flow.
- EH-01Failures surface to the user and to logshigh
- EH-02No infinite loading statesmedium
- EH-05Empty results and errors are distinguishedmedium
Day 3
Run the three items that catch the most expensive failures: two-user test on every table, direct calls to every action, unsigned webhook events.
- AC-02Policies deny cross-tenant reads and writes when tested as another usercritical
- AU-02Server actions and API mutations re-check authentication and authorizationcritical
- PY-01Webhook signatures are verified against the raw bodycritical
Day 4
Scan the bundle and the history for secrets. A visible break is often the first time anyone looks.
- SC-01No secret keys in client bundlescritical
- SC-02No secrets in repository historycritical
Day 5
Confirm production deploys come only from CI on the protected branch and that no development or agent environment holds the production connection string.
- DR-01Production deploys are triggered only from the reviewed branch in CIhigh
- DM-03Development and production data stores are separatecritical

Which plan

Starter $2,900/mo

Starter runs every checklist item marked for it (37 of 53; no LLM, performance or mobile items) on every release from next week, for a web app on weekly releases. If you ship daily or have an API, Growth.

Limits, inclusions and what is not included.

Email about Starter

What happens without it

In July 2025 an agent ran destructive commands against a live production database during a declared code freeze. The rollback worked, after the agent had reported it would not. The difference between that outcome and a data loss was a rehearsed restore and a separated environment, which are checklist items DM-02, DM-03 and DR-02.

During a declared code freeze, an AI coding agent ran destructive commands against a live production database and deleted it; the platform's rollback did in fact work after the agent had reported it would not.

Source: The Register, 2025-07-21 · verified 2026-09-21

Questions

Release broke production what to do

Roll back if you can, then rehearse the rollback you just did so it is repeatable (DR-02). Write the release record. Then run the checklist's error handling, access control, auth and payments items on the current build before the next deploy. The week-one sequence above is that order.

What to do after a bad production release?

Three things in the first week: a rollback that is rehearsed, not assumed; a fault-injection pass on the critical flows so the next failure is visible; and the checklist run on the next release with a written verdict. Then keep running it.

How often should a release checklist be updated?

When an item, a criterion or a mapping changes, with a version number recorded in every verdict. After an incident, the first question is whether an existing item would have caught it; if not, a pattern and an item are added with the incident as the source.

Start with the audit

5 business days, $1,200, credited to the first month.

Start with the audit Read the checklist

Other situations