Onboarding the Pilot Team for Real

Part 5 of “The Landing Zone That Survived” — a year in the life of a New Zealand platform team, told from fakey.xyz. Fictional organisation, aggressively fake people, realistic problems.

Recap. Increment One delivered: management groups live, 41 policy initiatives watching in audit mode, vending version zero (a Microsoft Form, a human, a pipeline), Tessa Spreadsheet converted to the platform cause, and Netty Latency’s team deploying test workloads into compliant environments in two days instead of four weeks. The platform works. The title of this post exists because “the platform works” and “the platform is ready for production” are separated by precisely the distance this post covers.

There is a difference between a customer who is using your platform and a customer who is depending on it. The first is a relationship. The second is an obligation — with an on-call roster on one end and a production incident on the other.

Increment Two is where fakey.xyz’s platform team crossed that line, and — this is the honest part — they crossed it without noticing, which is how it happens everywhere.

The Moment They Crossed the Line Without Noticing

It happened on a Tuesday, in a routine conversation that Fakey McFakerson later reconstructed in the post-incident review as “the moment we became accountable for uptime, via email.”

Netty Latency’s team had finished their fraud scoring engine’s first phase and were planning the production cutover. Netty posted in the platform channel:

“Plan: production environment next sprint, cutover the scoring service the week after. One question for the platform crew — what’s your change freeze over Christmas, so we can plan around it?”

Nobody on the platform team had a change freeze, because nobody on the platform team had formally considered that other people’s production workloads would now be frozen around theirs. The platform had gone from being a thing teams deployed into, to being a thing their production depended on — a firewall in the traffic path, a workspace their logs flowed to, a vending pipeline their DR plan would someday reference.

Netty had assumed the platform had production SLAs, change management, and a freeze calendar. The platform had none of those things. It had a sticky note and a really good vending form.

Fakey’s response was to call a meeting titled, honestly, “We Are Now Part of Production”, and the first item was a list she put on the whiteboard of everything the platform team was now implicitly promising:

• Hub network availability — Netty’s scoring traffic transits it. Down hub = down fraud scoring = the 8ms SLO dies in front of customers.
• Firewall and the low-latency corridor — Part 3’s measured exception is now load-bearing production infrastructure.
• Central Log Analytics — Serge Secure’s CPS 234 evidence and the Privacy Act detection capability both live here. Its failure is a compliance incident, not just a technical one.
• The vending pipeline — DR rebuilds, new environments, and the migration waves all flow through it.
• The platform team itself — 1.6 FTE. Who is paged? Paged for what? Paged from where?

Lesson 1 — The platform becomes production-grade the moment the first production workload plans its cutover, not when you finish your checklist. Find that moment deliberately, or it will find you in an incident.

The Corridor Meets Reality, or: Measured Once, Measured Again

The low-latency corridor from Part 3 — the policy-gated, logged, quarterly-reviewed exception for the fraud scoring stack — survived its first production test. Barely, and not the way anyone expected.

The proof-of-concept had measured firewall hop latency: ~1.9ms per hop. What the POC had not measured was the corridor’s behaviour under real production traffic patterns — specifically, Netty’s scoring engine’s behaviour when the transaction stream bursts (peak: 11:45am daily, when NZ’s payment volumes spike before lunch).

The first week of production-load rehearsal surfaced two findings:

Finding one: the corridor itself was fine. Direct peered traffic between the scoring subnets and the transaction stream subnet held p99 well inside budget. Part 3’s decision to measure, gate, and monitor the exception was vindicated. Serge Secure noted this in the exception register’s first quarterly review with the words “exception performing as designed” — four words that did more for the exception register’s credibility than any policy document could have.

Finding two: everything around the corridor was not fine. The scoring engine’s telemetry side-channel — model metrics, feature-store stats, health probes — was configured to flow through the hub firewall to the central workspace, and at peak load that telemetry volume saturated the firewall’s throughput enough to add jitter to other spokes’ inspected traffic. The corridor had protected Netty from her own firewall; it hadn’t protected everyone else from Netty.

The fix took two shapes, and both are the instructive part:

1. Architectural: telemetry moved off the firewall path entirely — diagnostics flow over Azure backbone to the workspace via private endpoints, never via hub-inspected routing. This is now a documented platform standard: observability traffic never rides the inspection path.
2. Process: the exception register’s quarterly review template gained a line item — “adjacent impact assessed?” — because an exception can be healthy while quietly degrading its neighbours.

Netty’s contribution to the review meeting, delivered with her characteristic inability to soften anything:

“Your firewall is a shared road. My corridor is a bus lane. This week my bus was fine and I congested your road for everyone else. If the register only checks the bus lane, you’re measuring the wrong thing.”

Fakey stole the metaphor on the spot. It’s now in the platform documentation.

The Cutover, and the Difference Between a Runbook and a Plan

The production cutover for the scoring service was scheduled for a Wednesday evening, out of NZ business hours, with a rollback plan. What follows is not the incident — that’s Friday’s story. Wednesday went almost smoothly, and the almost is the lesson.

The cutover ran: DNS shifts, traffic ramp, the 11:45am Thursday peak as the real acceptance test. The scoring service held its SLO through the peak — p99 inside budget, corridor stable, telemetry flowing. Netty declared it “production, tentatively, until something surprises us.”

Then the surprise: the scoring service’s scale-out hit a regional quota in Azure NZ North that nobody had checked, because quota checks weren’t in the platform’s environment-provisioning checklist. The service scaled to its ceiling at 2am Thursday, threw alerts, and Netty’s on-call engineer — who was paged, because the workload team had sensibly set up their own alerting — raised it with Robbie Deployment at 2:14am.

Quota raised by 6am. Service healthy. No customer impact. But Fakey’s post-cutover review produced the increment’s most durable artefact — a one-line addition to the vending pipeline’s checklist:

“Quota and capacity check for the requesting team’s stated scale — at vending time, not at 2am.”

The deeper lesson, which Fakey wrote into the design document’s open questions:

A runbook tells you what to do. A plan tells you what you should have checked before you needed the runbook. The cutover had a rollback runbook. What it lacked was a pre-cutover checklist owned by both teams — platform and workload — covering the shared-fate items (quota, scale limits, support entitlements, regional capacity) that neither team owns alone.

The joint cutover checklist — platform sections and workload sections, each team signing the other’s — became increment two’s most-reused artefact, and the template the migration waves will follow.

Friday, 4:52pm: The Incident

And then came Friday.

At 4:52pm — after most of Wellington had mentally clocked off — a pull request merged to the platform repo and the pipeline deployed a change to the DNS private resolver configuration in the hub. The change was small, well-reviewed, and correct against its intent. It was also, due to a Terraform quirk in how the resolver’s inbound endpoint subnets were referenced, applied out of order, briefly removing DNS resolution for two private zones — including the zone the fraud scoring service’s feature store lived on.

For eleven minutes, the scoring service couldn’t resolve its feature store. Netty’s alerting fired at 4:58pm. Robbie was at his desk (a minor mercy), identified the cause by 5:03pm, and rolled the change back by 5:09pm. Full recovery. No customer-visible impact — the scoring service’s circuit breaker degraded gracefully to cached features, which is exactly what Netty’s team had built it to do, and is a story for their blog post, not this one.

Eleven minutes. Here’s what made those eleven minutes the most valuable of the increment — the retro on the following Monday, run by Fakey with a strict no-blame rule, produced five findings that reshaped the platform:

1. The blast radius of platform changes is every workload, simultaneously. A workload team’s bad deploy breaks their app. A platform team’s bad deploy breaks the platform’s promise to everyone. The pipeline now requires platform infrastructure changes to declare — in the PR template — their blast radius, and anything touching DNS, network routing, or identity requires a second reviewer outside the platform team. The reviewer pool: Netty, Vera Transaction, and Hugh Documents. Your customers reviewing your changes is the operating model working, not the operating model failing.

2. “Deployed via pipeline” and “deployed correctly” are different guarantees. The change was exactly what the repo said. The repo was wrong about how it would apply. Terraform’s ordering behaviour earned its own entry in the platform’s known-gotchas document, and the pipeline gained a post-apply smoke test: resolve a canary FQDN in each critical private zone, every deployment, automatically.

3. Nobody outside the platform team knew how to reach the platform team. Netty’s engineer’s first instinct was the platform channel — fine — but their second thought was “if this had happened at 3am, who do I even page?” The answer was: nobody, because no paging path existed. This finding fed directly into the operating model work that becomes Part 6’s crisis, but its first output was simple and immediate: a platform contact policy — channel for business hours, a paging integration (a cheap SaaS pager, stood up in a day) for platform P1s, and an on-call rotation that, given 1.6 FTE, was formally declared to be “Robbie, primary; Fakey, secondary; both of us hoping the bus factor holds until hiring.” Fakey flagged the rotation’s fragility to Barry Bigboss in writing the same week. It’s on the record. Watch what happens to it later in this series.

4. The workload teams’ grace under fire proved the platform’s design. The scoring service’s circuit-breaker degradation, Netty’s alerting firing within six minutes, the clear separation between “platform broke” and “workload coped” — every investment the pilot partnership had made paid for itself on Friday evening. Fakey’s note in the retro, underlined twice: “The pilot team made us look good in our own incident. That is what a design partner is for.”

5. Serge asked the compliance question, and it was the right one. In the retro, Serge Secure’s only input was: “Are these eleven minutes notifiable under the Privacy Act?” The answer — no, no personal information was exposed, only unavailable, and the service degraded without data loss — took twenty minutes to establish confidently, because the platform’s logging covered the failure but the breach-assessment reasoning lived in nobody’s head. The fix: a one-page incident-to-Privacy-Act decision tree, drafted by Serge and Locksley Keymaster, now attached to the platform’s incident process. Detection capability, Discovery Week’s requirement, had been built. Assessment capability had not. Now it exists, and it exists because security was in the room for the technical retro — which is why security is in the room for every retro.

The Increment Review: New Metrics for a New Relationship

The increment review with Barry Bigboss carried a new metrics table, because the platform’s obligations had changed:

Barry Bigboss looked at the “11 minutes” line for a long moment, and then asked — and this is the reason Barry is the CEO in this story and not a villain — “Which of these numbers is the next one to get worse before it gets better?”

Fakey’s answer: “The on-call rotation. Two people on a pager is not a rotation, it’s a countdown. I need it funded in the annual plan, and I need you to hear that the platform’s reliability is now coupled to one person’s continued employment.”

Barry wrote it down. Whether it gets funded is Part 6’s problem.

The Steal-This Checklist

• Identify the moment you became production — the first workload planning a cutover around you. Formalise the implicit promises that moment creates.
• Observability traffic never rides the inspection path — and your exception register assesses adjacent impact, not just the exception itself.
• Joint cutover checklists — platform and workload teams each sign the other’s; shared-fate items (quota, capacity, support) checked at vending time.
• Platform PRs declare blast radius; network/DNS/identity changes get a reviewer from a workload team.
• Post-apply smoke tests in the pipeline — canary DNS resolution, connectivity probes, every deployment.
• A paging path that exists before the first 3am incident — plus a named, written-down rotation, with its fragility escalated in writing.
• Security in every retro — the “is this notifiable?” question gets asked by someone whose job it is to ask it.
• A tabletop-tested incident-to-Privacy-Act decision tree — detection is not assessment.
• Report incidents to the exec honestly, with the “which number gets worse next” answer ready.

Next in the Series

Part 6 — “The Operating Model, or: Who Is Watching the Watchers.” The platform now has production obligations, an on-call rotation of two, a monitoring estate generating real alerts, and no formal operating model. When Tessa Spreadsheet’s cost alerts, Serge’s detection alerts, and genuine platform health alerts all land in the same silent inbox, fakey.xyz discovers the difference between having monitoring and having an operator — and Barry Bigboss finally has to choose between funding the platform and testing Fakey’s resignation letter draft.

When did your platform cross the line from “used by production” to “depended on by production” — and did you notice at the time, or in the retro? Comments open at fakey.xyz.