Synthetic sample
Monthly report, as you would receive it
Every number on this page is invented to show the format. It describes no client and no real product. Reports for real engagements follow this structure exactly, with the checklist version and build hashes filled in.
Termsverdict (a one-word release result (pass, pass with findings, fail) with the findings and the checklist version behind it) · critical flow (a path a user takes that costs money or trust if it breaks: sign-up, checkout, invite, export) · row-level security (database rules that decide which rows each signed-in user may read or write) · golden set (a fixed list of model inputs with recorded expectations, re-run on every model or prompt change) · idempotency key (an identifier stored per processed event so a duplicate delivery changes nothing) · flaky test (a test that fails without a code change; it gets a written cause and a fix, never a mute)
Monthly report
Month 3 (synthetic)
Synthetic sample
Releases tested
16
Pass · pass with findings
12 · 3
Fail
1
Plan: Growth · v1.0 · verified on 2026-09-21
Verdict key: Pass Pass with findings Fail
1. Releases tested
| Release | Date | Build | Verdict | Findings C · H · M · L | Note |
|---|---|---|---|---|---|
| R-031 | Day 1 | a41f2c9 | Pass | 0 · 0 · 1 · 0 | Medium: list endpoint returned 250 rows without a page size (PF-02). Fixed in R-032. |
| R-032 | Day 2 | b7e0d11 | Pass | 0 · 0 · 0 · 1 | Low: release record missing checklist version (DR-05). |
| R-033 | Day 3 | c9a3e5f | Pass with findings | 0 · 1 · 0 · 0 | High: session survived password change on the second device (AU-06). Shipped with a fix scheduled; re-tested R-034. |
| R-034 | Day 4 | d2b8f70 | Pass | 0 · 0 · 0 · 0 | AU-06 re-test passed. |
| R-035 | Day 8 | e55c1a2 | Fail | 1 · 0 · 1 · 0 | Critical: new invoices table created by migration without row-level security (AC-01). Release held; fixed and re-run same day as R-036. |
| R-036 | Day 8 | f1d9b34 | Pass | 0 · 0 · 1 · 0 | AC-01 passed. Medium carried: error body includes provider message (EH-03). |
| R-037 | Day 10 | 0a7e6c8 | Pass | 0 · 0 · 0 · 0 | |
| R-038 | Day 11 | 1c4d0f5 | Pass with findings | 0 · 1 · 0 · 0 | High: model-change regression showed 4 of 60 golden cases drifted after provider upgrade (LM-05). Prompt adjusted; re-run in R-039. |
| R-039 | Day 12 | 2e9a7b1 | Pass | 0 · 0 · 0 · 0 | Golden set 60 of 60. |
| R-040 | Day 15 | 3f0b8d6 | Pass | 0 · 0 · 0 · 1 | Low: two flaky selectors in checkout flow, replaced with test ids. |
| R-041 | Day 17 | 4a1c9e2 | Pass | 0 · 0 · 0 · 0 | |
| R-042 | Day 18 | 5b2d0f7 | Pass | 0 · 0 · 1 · 0 | Medium: outbound call to the email provider had no timeout (EH-04). |
| R-043 | Day 22 | 6c3e1a8 | Pass | 0 · 0 · 0 · 0 | |
| R-044 | Day 24 | 7d4f2b9 | Pass with findings | 0 · 1 · 0 · 0 | High: webhook handler processed a duplicate event twice (PY-02). Shipped; idempotency key added in R-045. |
| R-045 | Day 25 | 8e5a3c0 | Pass | 0 · 0 · 0 · 0 | PY-02 re-test passed. |
| R-046 | Day 29 | 9f6b4d1 | Pass | 0 · 0 · 0 · 0 |
A fail holds the release until the critical item passes. A pass with findings ships with the findings scheduled and re-tested in the next run.
2. Findings by severity
| Severity | Count | Example finding |
|---|---|---|
| critical | 1 | AC-01: invoices table without row-level security. Release held; fixed same day. |
| high | 3 | AU-06 session not revoked; LM-05 model drift; PY-02 duplicate webhook. |
| medium | 4 | PF-02 unbounded list; EH-03 provider message in error body, found in R-035 and carried into R-036 (counted per run); EH-04 missing timeout. |
| low | 2 | DR-05 release record; flaky selectors. |
Counts are per release run; a finding carried into a re-run is counted again. Each finding names the checklist item (for example AC-01) and the catalogue pattern behind it, with a reproduction in the per-release verdict.
3. Coverage growth
| Month | Flows automated | Plan limit | Surfaces |
|---|---|---|---|
| Month 1 | 34 | 80 | web |
| Month 2 | 80 | 80 | web, api |
| Month 3 | 80 | 80 | web, api, ios (8 web flows retired, 8 iOS flows added) |
Coverage is counted in critical flows automated against the plan limit, never as a percentage of code or of an undefined total. The limit is reached by the end of month 2; after that the list changes as surfaces are added and flows are retired. The flow list itself is an appendix to the real report.
4. Flaky-test handling
| Test | Runs | Failed | Cause | Action |
|---|---|---|---|---|
| checkout: apply coupon | 14 | 3 | Selector matched two buttons after a layout change | Replaced with a test id; 0 failures since |
| api: list invoices paginated | 14 | 2 | Seed data race in the test fixture | Fixture now awaits seed commit; 0 failures since |
| ios: restore purchase | 6 | 1 | Sandbox account hit the renewal cap | Rotated sandbox account; documented in runbook |
A flaky test is never silenced; it gets a written cause and an action, and it stays in the report until it has run clean for a month.
5. Next month
- 01
Add the Android platform to the flow map (Scale would be needed for both; Growth covers one)
- 02
Extend the golden set for the LLM feature from 60 to 100 cases
- 03
Move AC-01 check to a pre-merge CI step so a missing policy fails the pull request
- 04
Rehearse rollback of the invoices migration in staging (DR-02)
Disclaimer printed on every report
Verdicts describe the build we tested on the date shown. Changes after that date are covered by the next release run.
This sample is synthetic and labelled as such. It makes no claim about any client, result or coverage achieved.
Get this report for your product
Growth delivers it by the 5th business day every month, plus a verdict per release.