The 2am Page: Operating Model Tested

Part 8 of “The Landing Zone That Survived” — a year in the life of a New Zealand platform team, told from fakey.xyz. Fictional organisation, aggressively fake people, realistic problems.

Recap. Seven parts in, fakey.xyz has: a landing zone built to 1.6 FTE’s honest capacity, a five-stream operating model with named owners, an on-call rotation now carried by contractor Max Overhead, a two-way platform/workload contract, a Privacy Act decision tree that has run once for real, and a FinOps estate that just survived a CFO audit. Every part of the system has been tested individually. This is the post where all of it gets tested at once, at 2am, on a Friday, by a P1.

Every document in your platform programme is a hypothesis about a future bad night. The runbooks, the escalation table, the two-way contract, the decision tree, the exception register — all of it is a bet that says: when the bad night comes, these artefacts will be enough for the person standing in the dark, who is tired, and who may not be us.

At 2:11am on a Saturday in late autumn, Wellington time, the bet paid out. Mostly.

2:11am: The Page

The sequence began with an Azure Firewall Premium health event in the hub — a control-plane failover that Azure’s own SLA describes as transient and self-healing. In most hub deployments, that’s a footnote. In fakey.xyz’s, it triggered a cascade the team had modelled but never lived:

The firewall failover dropped and re-established its data plane. During the ninety seconds of instability, BGP sessions from the Petone colo VPN flapped. The VPN re-established cleanly — but the failover’s route-table convergence left one of the two colo sites’ routes preferencing a stale path with an asymmetric MTU. Traffic didn’t stop. It degraded — packet fragmentation, retransmits, latency that swung between acceptable and terrible depending on the flow.

By 2:09am, Netty Latency’s fraud scoring service — which draws transaction context from the colo-side transaction stream during the migration transition — was watching its p99 breach its SLO for the first time in production. Her alerting, tuned to a razor’s edge by a team that does not tolerate ambient anxiety, fired.

By 2:11am, three pages fired in sequence:

1. Azure monitor → platform health stream → the pager → Max Overhead.
2. Netty’s SLO alert → her team’s on-call — her data scientist, the former workstation operator, now properly on the workload rotation — who, per the two-way contract from Part 6, checked the platform status channel first, found nothing posted, and paged the platform P1 path.
3. The composite platform alert (firewall health + colo VPN error rates + workspace ingestion anomalies) → the platform P1 path → Max again, now with severity confirmed by a second signal.

The triple-signal design worked. No single page could have told Max what was happening. Three of them, arriving within four minutes, gave him a picture before he’d finished making tea: firewall event, VPN flap, SLO breach on a migrated workload, residency-relevant data in the degraded path.

That last clause is why this incident mattered. The degraded path carried payment transaction data in flight between the colo and the cloud. Which meant, potentially, the Privacy Act clock.

2:11am–2:40am: What “the Operating Model Was Real” Actually Looks Like

Here is the timeline, reconstructed from the incident log — and the reason each step worked is annotated, because the why is the stealable part.

2:11 — Max acknowledges the page. 3 minutes. Within the 15-minute commitment. He is awake because the pager is loud and because contractors who take on-call take it seriously. He opens the platform runbook titled “Hub connectivity degradation” — written by Robbie Deployment, tested once in a game day, and — this is the sentence from the post-incident review that made Fakey’s month — “runnable top-to-bottom without improvisation until step 7.”

2:14 — Max posts to the platform status channel. First public communication, three minutes after ack. The two-way contract’s promise — communicate during platform-caused or suspected-platform incidents — was kept before anyone asked. Netty’s on-call engineer, watching the channel instead of isolated in his own incident, stood down from investigating his own service and began capturing SLO evidence. The boundary between “platform broke” and “workload coped” — rehearsed in the DNS retro — held under pressure at 2am. This is what a two-way contract is for. Nobody had to negotiate roles in the dark.

2:19 — Max reaches step 7 of the runbook: the step that requires judgement. The route table fix is documented, but the runbook honestly flags it: “Applying the corrective route during traffic may cause a second, briefer flap. If workload SLO impact is active, coordinate before applying.” Max did not improvise. He did what the runbook told him to do at step 7: he called Netty’s on-call engineer. Direct, human-to-human, at 2:19am. They agreed: the scoring service’s circuit breaker was holding in degraded mode; a 60-second second flap was survivable; the degradation as it stood was not, because p99 was drifting toward the SLO boundary. Second flap approved. Applied at 2:23. Routes converged by 2:25. Firewall stable. Colo traffic clean by 2:31.

2:26 — Netty’s on-call, unprompted, posted the sentence that the entire pilot-partnership investment had been building toward:

“SLO recovering, p99 back inside budget, circuit breaker re-opening to full feature set. Whatever you did, do more of it. Also: logging the SLO excursion window for our post-incident review — flagging that the degraded path processed transaction context, so your privacy assessment may want this.”

Read that again. A workload engineer, at 2:26 in the morning, voluntarily flagging the compliance dimension of his own incident to the platform team. That reflex — the instinct to hand the privacy question to the people whose job it is — did not exist at fakey.xyz nine months ago. It exists because the Part 5 retro put security in every retro, and because the decision tree from Discovery Week gave everyone a shared map of what “might be notifiable” means.

2:34 — Max, following the runbook’s final section, invoked the incident-to-Privacy-Act decision tree. It was 2am. He was a contractor four months in. And the artefact worked exactly as designed for exactly this person: a competent stranger, tired, with no tribal knowledge. The tree asked: was personal information exposed? accessed? processed in an unauthorised location? His answers, from the monitoring estate: in-flight data was encrypted end-to-end (TLS the whole way — the fragmentation degraded performance, not confidentiality); no unauthorized access in the logs; processing locations unchanged. Preliminary assessment: no notifiable event; assessment recorded; escalation to Serge Secure flagged as required-at-business-hours.

He filled in the assessment form. He paged nobody at Serge’s house. He wrote the incident timeline while it was fresh. At 2:47am he posted: “P1 resolved. Recovery complete 2:31. Privacy preliminary: non-notifiable, recorded, Serge to review Monday. Writing it up now.” Then he went back to bed.

Total platform impact: 22 minutes of degraded colo transit, zero data exposure, zero customer-visible failures — the scoring service’s degradation architecture and the platform’s fast recovery together kept the SLO excursion inside Netty’s design tolerance.

The runbook, the contract, the decision tree, the paging path, the status channel, the monitoring signals: every artefact from Parts 4 through 7 did its job on the same night, most of them simultaneously, most of them for a person who had never lived through the events that created them.

That is the whole argument of this series, in one Saturday morning.

The Almost

It went well. Almost entirely. And the almosts — found in Monday’s review, run by Fakey with her no-blame rule and Serge in the room — are the next layer of the operating model, because a system’s second-order gaps only appear after its first-order gaps are fixed.

Almost #1: The escalation table had a hole where “the platform team is the incident” should be. Max’s P1 escalation path was: runbook → judgement step → call the workload → resolve → escalate to Fakey post-hoc. What it lacked: a threshold for waking Fakey up. The route fix had worked first try — but if it hadn’t, Max’s documented options at 2:30am were “keep troubleshooting” or “wait for business hours.” For a P1 with residency implications and a live SLO breach, neither was right. The fix: an explicit second-escalation trigger in the runbook — “if the corrective action fails, or if impact exceeds 30 minutes, wake the secondary on-call regardless of hour.” Fakey’s comment on the change: “Waking me for nothing costs one bad night’s sleep. Not waking me when you’re stuck costs the platform’s whole credibility. Err toward waking me.”

Almost #2: Serge found out at 9:40am Monday, and that was 8 hours too late for his comfort — but 30 years too early for the process’s. The preliminary assessment said non-notifiable, and it was right. But Serge’s review surfaced the real issue: a contractor had made a preliminary Privacy Act determination alone, at 2am, and the process had no verification step. The assessment was correct this time. The process didn’t guarantee it. The fix — proposed jointly by Serge and Max, which is worth noting — is a second-signature rule: preliminary non-notifiable assessments made under time pressure get a counter-signature from the privacy function within 12 business hours, every time, no exceptions. The counter-signature isn’t distrust; it’s the same principle as the DNS retro’s second reviewer: judgement calls made in the dark get daylight-reviewed.

Almost #3: The status channel was great for the people watching it, and invisible to everyone who wasn’t. Netty’s team watched it because the DNS retro had taught them to. Vera Transaction’s team — whose payments API rides the same colo transit — found out about the incident on Monday. Their workload happened to be unaffected (their traffic takes the direct path, not the corridor), but they couldn’t have known that at 2am, and the contract promises communication to affected workloads. The platform had no way to say “affected: X, Y; assessed-and-unaffected: Z.” The fix: the status channel gained a reach matrix — a living map of which workloads depend on which platform components, maintained as part of vending, so the first communication of any incident includes an honest affected/not-affected call. Fakey’s note: “In an incident, silence is information too. Make sure the silence says what you mean.”

Almost #4 — the small human one: Max, four minutes into the incident, had spent ninety seconds looking for the on-call contact numbers — Netty’s on-call, Fakey’s secondary — because the runbook referenced “call the workload on-call” without saying how. The numbers existed in three places, one of which was out of date. The fix took twenty minutes (a single contacts page, generated from the on-call tooling, linked from every runbook) and the out-of-date entry was, of course, a number that would have mattered.

The Post-Incident Review: The Artefact That Tie It Together

The PIR was published five business days later — per the two-way contract’s promise — and it was structured the way the series has taught: timeline, five findings, owner and date on each fix, and a section that no PIR template in the industry includes and all of them should:

“What this incident validated.” The two-way contract (roles held under pressure, zero negotiation). The decision tree (first 2am use, correct, now with second-signature). The paging path (3-min ack). Cross-redundant signals (the composite alert caught what no single monitor would have). The workload team’s degradation architecture and compliance instinct. The runbook’s honesty about its own limits — step 7 worked because it admitted step 7 was a judgement call.

The final line of the PIR, from Fakey:

“Nine months ago this incident was impossible — not because the network was better, but because there was no platform, no pager, no contract, and no Netty to design a circuit breaker. The incident happened because the platform is real. The outcome happened because the operating model is too.”

Serge Secure’s contribution to the PIR, filed into the CPS 234 evidence pack alongside it:

“Incident detected by automated monitoring at 2:09. Acknowledged at 2:11. Workload impact contained by workload resilience. Privacy assessed same-night with daylight counter-signature. Post-incident report within five business days. This is the control operating as intended, evidenced.”

CPS 234 evidence, it turns out, isn’t a quarterly document. It’s a bad night, handled well, written down.

The Increment Review Metrics

The Steal-This Checklist

• Treat every artefact as a bet on a bad night — runbooks, contracts, decision trees. Then schedule the night: game days find second-order gaps before real 2ams do.
• Runbooks must mark their own judgement steps — and the judgement step’s protocol is coordinate with the affected workload, human to human, not improvise alone.
• A second-escalation trigger in every P1 runbook — “if the fix fails or impact exceeds X, wake the secondary regardless of hour.” Waking someone for nothing is cheap. Not waking them when stuck is not.
• 2am preliminary privacy assessments get daylight counter-signatures — every time. Judgement made in the dark gets reviewed in the light.
• An incident reach matrix — which workloads depend on which platform components, maintained at vending time, so your first status post says who’s affected and who isn’t.
• Communicate before you’re asked — the first channel post inside 5 minutes buys you patience you will otherwise never get back.
• PIRs include “what this incident proved” — not just what it broke. Your controls’ first real test is evidence, and evidence is the product.
• Verify in-flight encryption assumptions in your logs, not your architecture diagrams — the residency/confidentiality assessment at 2am depends on it.

Next in the Series

Part 9 — “The Second Team, and the Difference Between a Platform and a Custom Build.” Hugh Documents’ onboarding workload moves to production with the strictest data classification in the company — and immediately demands things the platform’s golden path never imagined. The temptation to build bespoke for every important customer meets the discipline of the exception register, and the platform team learns the lesson that separates platforms that scale from platforms that become a bespoke agency with a roadmap: some asks you build, some you route, and some you decline — in writing, with love.

Has your platform had its 2am yet — the night the runbooks either held or didn’t? What did the morning-after retro find that no game day had? Comments open at fakey.xyz.