Cycle time with Change failure rate
SpeedCycle time
37 h
−29%better vs baseline
QualityChange failure rate
10.4%
−1.6 ptsbetter vs baseline
Engineering reports
For engineering leads: every metric with its pair, and the rules that act on them.
M12Steady state · months 7–12
No speed metric is reported alone
Healthy
Deploys with Uptime
SpeedDeploys
4.7 / svc / wk
+52%better vs baseline
QualityUptime
99.95%
+0.02 ptswithin noise vs baseline
Agent-authored share with Rework rate
SpeedAgent-authored share
46%
+43 ptsbetter vs baseline
QualityRework rate
6.8%
−0.8 ptsbetter vs baseline
Throughput with Review load
SpeedThroughput
18.7 / team / wk
+34%better vs baseline
QualityReview load
6.0 h / wk
no changewithin noise vs baseline
baseline (months 0–1)SLO or thresholdChanges inside each metric's noise band count as no change.
Decision rules
If change failure rate above baseline two weeks running
Fired, clearedthen Expansion pauses
Now 10.4%, baseline 12.0%
Fired at month 4; cleared at month 6. The last two teams waited. Reviews were fixed before anyone else onboarded.
If error budget spent on a service
Fired, clearedthen Feature work stops on that service until it recovers
Now 99.95%, limit 99.90%
Fired at month 5; cleared at month 6. Feature work on the service stopped; the team worked only on reliability.
If eval pass rate below the release threshold
Fired, clearedthen Release blocked
Now 95%, limit 92%
Fired at month 8; cleared at month 9. The release candidate with the new model was held. Prompts were fixed against the failing evals.
If review latency over one day for two weeks
Fired, clearedthen Review is the bottleneck: cap PR size, add AI first-pass review, rebalance reviewers
Now 16 h, limit 24 h
Fired at month 4; cleared at month 6. PR size capped at 400 lines, an AI reviewer took the first pass, and reviewers were shared across teams.
If agent code reworked more than human code
Fired, clearedthen Fix specs and context before blaming the tool
Agent code 6.8%, human code 7.7%
Fired at month 4; cleared at month 7. Spec templates and repository context files were rewritten; agents got the team's conventions up front.
If a metric is used to rank individuals
Idlethen Stop. The metric comes off that report
0 reports slice a metric by person
Not fired in months 0–12.
Spend rises first, then falls as work is routed
Month
AI spend per merged change, month 12
$22.00
+$19.90worse vs baseline · target ≤ $25.00
Large · design, hard bugs20%
Medium · most changes, review45%
Small · lint fixes, test scaffolds35%
Prompt-cache hit rate70%
+55 ptsbetter
The levers
Right model per job · Prompt caching · Batch runs for evals · Ephemeral cloud sandboxes, torn down after each run.
Full scorecard and timeline
Phase 4 of 4
Steady state months 7–12
Routing, caching and evals tuned; the reports become the monthly operating review.
Exit test
Pairs hold for a quarter and AI spend per merged change is under target.
- Baseline0–1
- Pilot2–3
- Expand4–6
- Steady state7–12
What happened, up to month 12
M12 One year on (Good news)This month
Every pair Healthy. Cycle time down 29%, throughput up 34%, deploys up 52%, change failure rate 10.4%. AI spend about $22 per merged change, roughly $270 per engineer per month.
M9 Upgrade fixed (Good news)
Prompts adjusted against the failing evals; pass rate back above threshold and the release ships.
M8 Model upgrade (Trouble)
A new model version drops the eval pass rate below the release threshold. The release is blocked.
M7 Routing by job (Good news)
Small models for lint fixes and test scaffolds, medium for most changes, large only for design and hard bugs. Prompt-cache hit rate passes 50%.
M6 Responses land (Good news)
PR size capped, AI first-pass review before people, reviewers rebalanced, spec and context templates rewritten.
M5 Rules fire (Trouble)
Change failure rate at 15.8%, review load peaks, and the booking service spends its error budget. Expansion pauses and feature work on bookings stops.
M4 Expansion starts (Trouble)
Two more teams onboard and the pilot gains partly slip. Agent pull requests grow larger, review latency passes one day, change failure rate jumps to 15%. Spend per change peaks at $38.
M3 Pilot result (Good news)
Pilot teams' cycle time down about 20%; rolled up across all six teams, 8%. Change failure rate and rework flat. Exit test passed.
M2 Pilot: two teams (Note)
Agents write tests and evals first, then the change. The largest model does nearly everything, so spend per merged change across the org goes from about $2 (autocomplete only) to $12, nearly all of it on the two pilot teams.
M0 Baseline starts (Note)
Dashboards built from Git, CI, deploy and incident data. No change to how teams work yet.
The full scorecard
| Metric | Definition | Baseline | Month 12 | Target | Status |
|---|---|---|---|---|---|
| Delivery speed | |||||
Cycle time (median)Speed | First commit to running in production, median across merged changes. | 52 h | 37 h −29%better | ≤ 40 h | On target |
Deploy frequencySpeed | Production deploys per service per week. | 3.1 / svc / wk | 4.7 / svc / wk +52%better | ≥ 4.5 / svc / wk | On target |
Review latency (p90)Speed | Pull request opened to first human approval, 90th percentile. | 20 h | 16 h −20%better | ≤ 24 h | On target |
PR size (median) | Lines changed per merged pull request, median. Generated files excluded. | 210 lines | 160 lines −24%better | ≤ 200 lines | On target |
| Stability | |||||
Uptime against SLOQuality | Successful booking requests over all requests. SLO is 99.9%; the gap is the error budget. | 99.93% | 99.95% +0.02 ptswithin noise | ≥ 99.90% | On target |
Change failure rateQuality | Share of deploys that cause an incident, a rollback or a hotfix. | 12.0% | 10.4% −1.6 ptsbetter | ≤ 11.0% | On target |
Time to restore (median) | Incident opened to service restored, median. | 58 min | 43 min −26%better | ≤ 45 min | On target |
Escaped defects | Bugs found in production per 100 merged changes. | 3.2 / 100 | 2.7 / 100 −16%better | ≤ 2.8 / 100 | On target |
| AI effectiveness | |||||
Agent-authored shareSpeed | Share of merged lines first written by an agent, then reviewed by a person. | 3% | 46% +43 ptsbetter | ≥ 40% | On target |
First-pass rate | Agent pull requests merged without a person rewriting the change. | 38% | 66% +28 ptsbetter | ≥ 60% | On target |
Rework rate (agent code)Quality | Share of agent-written lines rewritten or deleted within 21 days of merge. | 7.6% | 6.8% −0.8 ptsbetter | ≤ 8.0% | On target |
Eval pass rate | Share of the eval suite (booking assistant and agent prompts) passing on the release candidate. | 93% | 95% +2 ptsbetter | ≥ 92% | On target |
| Cost | |||||
AI spend per merged change | Model, sandbox and eval spend for the month over merged changes. Baseline is autocomplete only. | $2.10 | $22.00 +$19.90worse | ≤ $25.00 | On target |
Prompt-cache hit rate | Input tokens served from the prompt cache over all input tokens. | 15% | 70% +55 ptsbetter | ≥ 60% | On target |
| PeopleReported per team, never per person. | |||||
Developer experience survey | Monthly pulse, team average, scored 0 to 100. Anonymous. | 64 / 100 | 72 / 100 +8 ptsbetter | ≥ 70 / 100 | On target |
ThroughputSpeed | Merged changes per team per week. | 14.0 / team / wk | 18.7 / team / wk +34%better | ≥ 18.0 / team / wk | On target |
Review loadQuality | Hours spent reviewing per engineer per week, team average. | 6.0 h / wk | 6.0 h / wk no changewithin noise | ≤ 7.0 h / wk | On target |
On-call pages per shift | Pages that woke someone, per on-call shift, team average. | 2.4 | 1.9 −21%better | ≤ 2.0 | On target |
Scroll the table sideways to see every column.
Share of tokens by model tier belongs to the cost group too. It is charted in the cost section above rather than given a single target, because the right mix depends on the work.
Deliberately left off the scorecard
- Lines of code
Agents make lines cheap; more of them is not more value.
- PRs per person
Ranks people, invites splitting work, and breaks the per-team rule.
- “Hours saved” estimates
Self-reported guesses. Cycle time and throughput show the real change.