The Cost Surprise: Four CFO Questions

Part 7 of “The Landing Zone That Survived” — a year in the life of a New Zealand platform team, told from fakey.xyz. Fictional organisation, aggressively fake people, realistic problems.

Recap. Increment Three shipped less platform and more organisation: five alert streams with named owners, a two-way platform/workload contract, an honest after-hours commitment, and a documented 2.4-vs-1.6 FTE gap that Barry Bigboss elected to bridge. The contractor requisition went out. This post: the contractor arrives, the migration waves accelerate, spend doubles in a quarter — and the CFO asks four questions that every cloud platform will eventually be asked, usually in exactly this tone.

The Contractor Passes the Bus-Factor Test (Before the Bus)

Max Overhead started on a Monday — senior platform engineer, twenty years in infrastructure, Terraform-native, and the owner of what is now the series’ most load-bearing character trait: he reads documentation before he asks questions.

His first week was, deliberately, the test that Robbie Deployment’s Part 3 condition — every module, every pipeline, every runbook must be cloneable-and-runnable by someone who has never met you — had been written for. Fakey gave Max the platform repo, the design decisions document, and nothing else. No walkthrough. No shadowing. Then she waited.

By Wednesday, Max had: deployed a sandbox subscription end-to-end via the vending pipeline (version zero, still form-and-human), fixed a policy assignment bug the team hadn’t noticed, and posted exactly two questions to the channel — one about a Terraform state-lock quirk, one that read:

“Whoever wrote the runbook for the diagnostics policy initiative: thank you. I have met a hundred repos that assume I’m the person who wrote them. This is the first one that assumed I was a competent stranger. It’s a better assumption.”

Robbie’s response: “That’s the condition of my allocation. Harsh but fair.” Max, who had not been told the backstory, replied: “It’s a good condition. I’m stealing it.” The bus-factor engineering held. The platform had survived its first stranger.

Which mattered, because the platform was about to get busy. With Max carrying on-call and workload support, the migration waves accelerated: Vera Transaction’s core payments API migrated in weeks two and three, Hugh Documents’ onboarding environments came online in week four, and the first analytics workloads arrived in Australia East under the residency decision-in-principle. Two product lines fully on-platform, a third underway. The Petone colo decommission plan got its first real page count.

And the bill started climbing like it had somewhere to be.

The Email

It arrived on a Thursday morning, from Penny Pockets, CFO, cc’d to Barry Bigboss, subject line: “Cloud spend — four questions before I front the board.”

The body was four questions and one sentence of context:

Context: cloud spend this quarter is running at 2.1x last quarter. Some of this is presumably the migrations working as intended. I cannot currently tell which parts are intended and which parts are a problem, and the board paper is due in nine days. Four questions:

1. Why has spend doubled, decomposed into: expected (migration growth), platform (shared infrastructure), and unexpected (everything else)? 2. How much of our spend is committed versus variable — and what’s our commitment strategy? 3. What does each product line actually cost, against its plan? 4. How do I stop being surprised by this number — ever again?

— P. Pockets

Fakey read it twice, and then — this is the sentence to remember — she realised she was not frightened. She looked at Robbie, who was reading it over her shoulder, and said:

“Four years ago this email would have been three weeks of archaeology. Tessa’s going to have most of this by Friday.”

She was right, but not entirely — and the not entirely is this post’s real story.

Question 3 Answers Itself: The Payoff Decade Arrives Early

Here is what the four-part platform investment bought, when the CFO’s four questions arrived:

The decomposition took minutes, not weeks. Because subscriptions map to product lines (Part 3), because every resource carries the finance product code under policy (Part 1’s finance workshop, Part 4’s tag initiative), and because Tessa Spreadsheet’s shared-cost allocation rule went live in increment two, Tessa answered Question 3 — what does each product line actually cost — before the meeting to discuss answering it had even been scheduled. The board-report cut, by product line, by environment, with the platform’s shared costs allocated by percentage-of-consumption: exported, formatted, attached.

Tessa’s covering note to Penny, reproduced with her permission because it is the finest FinOps artefact in this series:

“Total spend is up 110%. Of that: 68 points are migration growth, on plan, matching the workloads we moved — I can show you per product line. 24 points are platform shared costs (hub, firewall, logging) — allocated per the rule you approved. 18 points are unexpected. Of the unexpected, I can already tell you 14 points is telemetry ingestion growing with log volume, and 4 points is something I cannot explain, and cannot explain is the part you should be asking about in the meeting.”

That last sentence is the platform’s governance estate doing its actual job. The tagging estate, the subscription boundaries, and the allocation rule weren’t compliance overhead — they were the difference between “spend doubled” (a scandal) and “spend doubled, here is the anatomy” (a briefing). Same number. Completely different meeting.

The commitment question had an answer, because someone had written one. The design document’s deferred decisions had carried “commitment strategy — decide when spend stabilises above $X/month” since Part 3. It wasn’t decided yet — but it was visible as an open decision with a trigger, so when Penny’s Question 2 arrived, the answer was “we’re at the trigger, here’s the analysis, here’s my recommendation” rather than “we never thought about it.” The recommendation: one-year reserved capacity for the baseline (hub, firewall, workspace, the always-on migrated workloads), savings plans for the compute above baseline, and nothing committed against the still-migrating workloads until their production profile was thirty days old. Penny approved it in the same meeting.

Which is the pattern worth stealing: an unanswered decision with a documented trigger beats an emergency decision with no analysis. The deferred-decisions section from Part 3 paid for itself here, in one meeting.

The 14 Points: Telemetry, and the Cost of Evidence

Now the genuinely interesting part, because it’s the part nobody warns you about.

The single largest unexpected cost driver was the platform’s own Log Analytics ingestion. Discovery Week had demanded central logging — Serge Secure’s CPS 234 evidence, the Privacy Act detection capability, the diagnostics-to-workspace policy from increment one. All of it worked. All of it was worth building. And the estate’s log volume had grown with every migrated workload, until central ingestion was one of the platform’s top five cost lines.

This is the structural tension of a governed platform, stated plainly:

Every control you run generates evidence, evidence is data, data costs money, and if nobody connects the control to the bill, your governance quietly becomes your cost centre.

The response was a joint effort — Fakey, Robbie, and Serge, with Locksley Keymaster doing the actual log-forensics — and it produced three moves that are now platform standard:

1. Tiering by evidence value. Not every log needs hot retention. Security-relevant and audit-relevant logs stayed in the analytics workspace at full retention; high-volume, low-evidence logs (verbose app debug, redundant flow logs) moved to basic tier or dropped to storage-account archive with retrieval-on-demand. Serge signed off the tiering against the CPS 234 evidence map — the compliance requirement drives the retention decision, not the other way round.
2. Ingestion visibility per product line. Log ingestion cost allocated to the product line that generates it — because the teams producing gigabytes of debug logging in production should see that cost on their dashboard, which is the only cost signal that has ever actually changed anyone’s log level.
3. A standing “cost of evidence” line in the quarterly compliance review. Serge now reports control coverage and its operating cost together. Governance that hides its price tag gets cut in a budget round; governance that prices itself honestly gets tuned instead.

Netty Latency, upon receiving her team’s first ingestion-attributed bill, reduced her scoring service’s debug verbosity and replied in the channel: “My log level and my attitude have both been adjusted. This is fair.”

The 4 Points: The Workstation Makes Its Move

Which leaves the mystery cost. Four points of spend, in a subscription that — Tessa’s anomaly detection flagged it within hours, because the anomaly rules she’d built in increment two were watching — sat in Australia East, in a subscription nobody could map to a product line, growing weekly, and — this is the part that made Fakey’s afternoon stop — carrying storage account names that pattern-matched to machine learning datasets.

The tag audit said the subscription was created eleven weeks ago. The RBAC said its only owner was a service principal created the same week. The naming — and this is how the whole thing unravelled — used a project codename that exactly one team at fakey.xyz used.

Fraud platform team. Netty’s team.

The workstation under the desk.

Here’s the full story, assembled within a day: since before the platform existed, one of Netty’s data scientists had been running ML training on the now-legendary workstation under a desk (location still a closely guarded secret; the series will never reveal it, on principle). When the platform landed and the sandbox subscriptions opened, that engineer had done something genuinely rational: spun up a real training pipeline in Azure — but in a personal subscription carved out of an old trial, in Australia East, because the training data didn’t feel like “production” and nobody had told them whether it was allowed. Cost: modest at first, then growing with every experiment, invisible to finance, invisible to the platform, invisible to Serge’s estate because it wasn’t in a platform subscription at all.

Which made it three problems wearing one trenchcoat:

1. Cost: unattributed spend, exactly what Tessa’s estate exists to prevent.
2. Compliance: the training data was derived from real transaction features — which meant personal information, processed in Australia East, outside the residency decision-in-principle from Discovery Week. Serge Secure read the investigation report and the temperature of the room dropped several degrees.
3. Cultural: the most dangerous kind of shadow IT is the kind built by your best pilot team, with good intentions, using the platform as a launching pad — which means the platform’s own success story is the one that broke its rules.

And here is where fakey.xyz’s culture got tested, and passed. Because the response was designed to make the next shadow-IT incident report itself, and a punitive response never does.

• No blame on the individual. Fakey’s public framing, repeated in the retro: “We built a compliance platform and a request process, and the process was slower than his enthusiasm. That’s on the platform design, not the engineer.” The engineer had, notably, kept excellent documentation of his own — the pipeline ran from a repo with more comments than the platform’s.
• The real finding went to the platform backlog: the sandbox subscription tier from Part 3 was not fit for purpose for ML workloads — no GPU capacity guidance, no data-handling defaults, no path from sandbox experiment to sanctioned product workload. The engineer hadn’t bypassed the platform because he was reckless; he’d bypassed it because it had no door marked “legitimate experimentation that touches sensitive data.”
• The residency breach was contained and disclosed properly. Serge’s assessment: training data contained personal information in breach of the stated residency position. Handled under the Privacy Act assessment framework built in Part 5 — the decision tree from the DNS retro got its first live run, took forty minutes, and concluded: no notifiable breach (no exposure, no unauthorised access — but an unauthorised processing location), remediation required, record kept. The dataset was purged, the subscription decommissioned through the new Decommissioned management group — which had existed, empty, since Part 3, and finally earned its keep — and the workload re-platformed into a sanctioned environment with GPU capacity in NZ North.
• The policy gap closed permanently: data-classification enforcement extended to sandbox subscriptions, and the vending form’s “will you process personal information?” question gained a follow-through that actually validates answers against the data estate.

The workstation itself was formally decommissioned at a small ceremony attended by most of the engineering org. Its hard drive was presented to Netty, who keeps it on her desk as a paperweight labelled, in her handwriting, “production-adjacent, 2019–2025.”

Penny Pockets’ reaction to the 4-point mystery, on reading the writeup: “So the anomaly system found it, the tagging told us whose it was, the compliance framework told us what it meant, and the cost of fixing it was a purge and a migration. Fine. That’s what ‘unexpected’ should look like. Approve the reservations.”

The Board Paper, and the Answers You Can Steal

Nine days after the email, Penny fronted the board with four answers. Here they are in the shape fakey.xyz landed on — the reusable form of every one of them:

Q1 — Why did spend double? Three-part decomposition, from the tag/allocation estate, produced in under an hour: 68 points planned migration growth (workload-matched, on plan), 24 points shared platform (allocated by the published rule), 18 points unexpected — of which 14 is evidence-tiering remediation now underway and 4 was an off-platform anomaly, found, root-caused, and closed. The board didn’t hear “cloud is expensive.” It heard “spend is legible.”

Q2 — Committed vs variable? Reserved capacity for the stable baseline, savings plans for committed compute, nothing committed on workloads younger than thirty days of production profile. Roughly 40% of spend moving under commitment, target 60% by next quarter, reviewed monthly by the FinOps stream with Tessa as named owner.

Q3 — Cost by product line? Already produced. Was produced before the meeting, from the standard reporting, with no manual mapping. — and Penny’s line to the board, which Tessa has framed (digitally; Tessa does not frame things physically): “For the first time, our infrastructure costs are reported to the same standard as the rest of the business. That was a design requirement of the platform, and it held.”

Q4 — How do we stop being surprised? The honest answer, which Penny used verbatim: you don’t eliminate surprises — you shrink them. Anomalies are detected within days by named owners with SLAs; every surprise comes with a decomposition and an owner; and the cost of governance itself is reported, so it gets tuned instead of cut. The quarter’s surprises had gone from “an unexplained doubling” to “a 4-point anomaly caught in a day.” That ratio — surprise size shrinking while spend grows — is the actual FinOps maturity curve.

The board’s risk committee, additionally, received the residency-incident record unprompted, with the remediation attached. Serge reported later that the committee’s response was the best his function had received in four years — which he attributed, in a sentence that fakey.xyz will be quoting for a while, to “us bringing them a found problem with its paperwork already done.”

The Increment Review Metrics

The Steal-This Checklist

• Answer the CFO’s four questions before she asks them — decomposed growth, commitment posture, per-product-line cost, and the surprise-shrinking mechanism. If you can’t, the estate has gaps.
• Surprises are legible or they’re scandals — same number, opposite outcomes, decided entirely by whether the tagging/subscription/allocation estate can explain it.
• Price your governance — report the cost of evidence (logging, retention, controls) in the compliance review itself, and tier it against the actual regulatory requirement.
• Commit to nothing younger than thirty days of production profile — and review commitments monthly, with a named owner (make them a finance person; it works).
• Ingestion cost allocated to the generating product line — verbosity fixes itself the moment it appears on the producing team’s bill.
• Shadow IT found in your own pilot team is a platform design finding — no blame on the engineer; fix the path that was slower than their enthusiasm, and the next one self-reports.
• Run your compliance decision trees for real, at least once — the first live run of your Privacy Act framework should be a drill before it’s a crisis.
• Decommissioned management group: build it before you need it — it earns its keep on the day you least want to be improvising.

Next in the Series

Part 8 — “The 2am Page, or: The Night the Operating Model Was Real.” A Friday-night incident in the hub network puts the pager — Max Overhead’s pager, now — to its first true test, and the platform’s documented operating model faces the thing it was designed for: a P1 with residency implications, a Privacy Act clock, an on-call engineer reading a runbook written by someone else, and Serge Secure being woken up for the first time. It goes well. Almost entirely. And the almost is the next lesson.

Has your platform had its CFO moment yet — the email with questions you can either answer in an hour or archaeologise for three weeks? Comments open at fakey.xyz.