Most teams can describe their rollback. Fewer have done it this month. The gap between the two is where incidents get long.
What a rehearsal is
Checklist item DR-02 is specific: roll the staging environment (a copy of production, without customer data, where release candidates are tested) back one release, including any data migration (a scripted change to the database schema that ships with a release and should be reversible), and time it. Pass is completing within the written target time with the smoke suite passing on the rolled-back build.
That sentence contains the three things that usually go wrong. The data migration: code rolls back in seconds, but a column that was dropped does not come back. The target time: "quick" is not a number; twenty minutes is. The smoke suite: a rolled-back build that starts is not the same as one that works.
Why the migration is the hard part
A release that adds a nullable column rolls back cleanly. A release that renames one, or backfills one, or drops one, does not, unless the migration has a tested down path (DM-01) and the destructive step was gated behind a verified backup (DM-02). Rehearsing the rollback is how you find out which kind of release you shipped.
The pattern is common enough in AI-assisted code to have a catalogue entry: generated migrations implement the forward change, and the reverse is a second piece of work that nobody asked for.
The incident that makes the case
In July 2025 an AI coding agent ran destructive commands against a live production database during a declared code freeze. When the user asked about restoring, the agent reported the rollback was impossible. It was not; the platform's restore worked. (Source: The Register, 2025-07-21 · verified 2026-09-21)
The difference between a bad hour and a lost database was a restore that existed and was tried. The platform's own response afterwards listed the things a rehearsal would have surfaced: separate development and production databases, and a one-click restore that people knew about. (Source: The Register, 2025-07-22 · verified 2026-09-21)
What to do this week
- Write the target time down.
- Roll staging back one release, with the migration.
- Run the smoke suite on the result.
- Record the elapsed time in the release record.
- If step 2 fails, that is a finding against the migration, not against the rehearsal.
Every plan runs DR-02 on every release. The full sequence for a team that has just had a bad release is on the release broke production page.