“We need to be an AI-first organization.”

And then someone in the back raised their hand and asked “how will we know if it’s working?” Probably the one PM who actually reads documentation. The answer was a pivot. A reframe. A confident non-answer dressed in the language of momentum.

“We’re tracking token usage and PR velocity.”

Congratulations. You are now measuring your AI transformation the way a restaurant measures quality by how many plates leave the kitchen.

The Vanity Metric Playbook

Token usage. PR velocity. GitHub Copilot adoption rate. Developer sentiment scores.

These are the KPIs of the AI transformation era, and they share one useful property: they go up. You can always make them go up. You can prompt engineers to generate more tokens. You can merge more PRs. You can run a survey after you’ve announced the initiative, when everyone knows the right answer.

What you cannot do with any of these metrics is answer the question the CFO is going to ask in Q3.

What did we get for this?

There is a behavior spreading through engineering orgs right now that deserves a name. Call it tokenmaxxing: the practice of optimizing AI usage metrics without optimizing for outcomes, usually because the metrics are what’s being measured and the outcomes are not.

It looks like this. A team is told their AI adoption will be evaluated by token consumption. So they build a Slack bot that summarizes every channel. They generate AI drafts of documents nobody reads. They pipe ticket descriptions through a model to produce “enhanced” versions that get immediately overwritten by the engineer who wrote the original.

Token count: up. Outcomes: unchanged. Nobody can tell you by how much on either, because nobody set a baseline. That last part is the real story. Your org is generating a number it can present to the board, for a transformation it cannot measure, against a starting point it never recorded. The dashboard is live. The denominator doesn’t exist.

The restaurant is counting plates. The kitchen is on fire.

The Bill Is Coming

Your CFO has a spreadsheet nobody in your product org has been invited to see. It probably doesn’t have a scary name. Something like “FY26 Infra Review,” last modified in Q1.

In April 2025, Anthropic engineer Barry Zhang gave a talk at the AI Engineer Summit called “How We Build Effective Agents.” He published a four-question checklist for evaluating whether a task actually warranted an agent. The second question was: “Is the task valuable enough?”

His answer was a number. Under ten cents per task, build a workflow. Over a dollar, build an agent.

Anthropic was saying this in a public conference talk. This is the company that needs enterprises to build agents to pay its inference bills. The slide was on YouTube. Ten cents doesn’t justify an agent.

Most enterprises were measuring tokens anyway.

In April 2026, Anthropic moved Claude agent access to per-token billing. Flat-rate subscriptions no longer covered third-party agent frameworks. Developers who had been running agents on fixed monthly plans were now on usage-based pricing with no ceiling. Some reported their effective monthly spend had jumped fifty times.

The warning came first. Then the invoice.

Microsoft reportedly canceled most of its direct Claude Code licenses. Six months earlier, it had been handing them out freely to engineers. The tool worked. That was the problem. Engineers used it, adoption skyrocketed, and the token budget evaporated in months rather than years. Microsoft owns a major stake in OpenAI and has spent two years selling AI transformation to enterprise customers. It couldn’t afford its own AI tools at the scale its own employees wanted to use them.

Read that sentence again.

Bryan Catanzaro runs Applied Deep Learning at Nvidia, whose business exists on the premise that AI compute will be cheap enough to be everywhere. He said this out loud: “For my team, the cost of compute is far beyond the costs of the employees.” Not close. Not a rounding error. Far beyond.

More adoption doesn’t mean lower cost. It means more tokens, more often, running through systems that compound spend in ways nobody modeled in 2024. Agentic architectures don’t hit the model once. They hit it dozens of times per task.

Goldman Sachs projects agentic AI will drive a 24-fold increase in token consumption by 2030. Gartner put it differently: cheaper tokens won’t mean cheaper enterprise AI, because agentic models consume so many more tokens per task that falling unit costs won’t offset rising volume. Their senior director:

“Chief product officers should not confuse the deflation of commodity tokens with the democratization of frontier reasoning.”

Nobody in your all-hands will mention the actual reason the pricing felt manageable until recently. Every major AI provider is running inference at a loss, subsidized by venture capital betting on cost curves that haven’t arrived yet. You’re not buying the product at cost. You’re buying it at its fundraising price. When that math changes, the workflows baked into every corner of your product and engineering org aren’t free to unwind.

The cloud lock-in playbook took five years to spring the trap. AI is doing it in eighteen months.

If you're living a version of this inside your org, we want to hear it. The dashboard that hides the denominator. The budget that evaporated. The PRD you had to defend that you didn't write. War stories are published without names, companies, or identifying details. The pattern and the receipts

Tell us what really happened →

The PM Who Will Be Asked to Explain This

When the finance review comes, somebody is going to stand in a room and be asked what they got for this.

That person may be you.

If your only answer is “token usage went up and our developers are excited,” you don’t have an answer.

Track your own before-and-after. Quietly. In a spreadsheet nobody asked you to make. Not because your company will reward you for it. But because the PM who shows up to that meeting with cycle time, rework rates, and feature velocity has something nobody else in the room has. A defensible number.

Everyone else will be arguing over the line item. You’ll be the one explaining what the line item bought.

That’s leverage. Budget crises have a way of turning “we spent Q1 and Q2 warming up the model” into a very specific question about what you personally can account for. The PM with a number survives that conversation. The PM with a vibe doesn’t.

What You Can Actually Do

If you have any say in how your org measures AI, get three things in place before the mandate lands.

Pick one outcome that changes, not one behavior that increases. “Faster time to first draft” is a behavior. You can game it by generating more drafts no one reads. “30% reduction in time from discovery to PRD sign-off” is an outcome. Those are different things.

Establish a baseline before you start. Obvious. Almost never done. If you can’t measure where you started, you can’t prove you moved. And you won’t have an answer when the CFO asks why the infrastructure bill doubled.

Separate AI utilization from AI value. A team using AI on 10% of tasks and shipping 40% faster has better ROI than a team using it on 90% of tasks with no velocity change. Token count measures the first. The board asks about the second when the budget runs out.

If none of that is politically possible, if the fleece vest already won and the dashboard is live, document your baseline anyway. Quietly. In a spreadsheet. You’ll want it when the questions start.

The kitchen is on fire. The chef ordered too much AI.

Metrics Worth Tracking

Did outcomes change? That’s the only question that matters when the bill arrives.

A few things signal the difference:

Time-to-decision, not time-to-draft. Did AI shorten the gap between “we have a problem” and “we made a call”? Or did it produce more documents for no one to read?

Rework rate. Are AI outputs being used as-is, or is every PR getting a full human rewrite that takes longer than writing from scratch? That’s not adoption. That’s AI homework.

Cycle time on real user-facing work. Not stories closed, but features customers interact with, shipped faster.

Quality delta. Bug rate in AI-assisted code versus your baseline. The orgs running this number in high-complexity domains are often not happy with what they find.

Decision acceleration. Is everyone generating their pre-read docs and not reading each other’s? Or are product reviews shorter because the analysis arrived better?

Most orgs can’t answer any of these questions. They didn’t define a baseline before they launched.

The PM’s Actual Problem

When leadership says “just use AI and we’ll figure out the KPIs later,” they need a board story. They don’t want measurement getting in the way of it.

The people who pay for that decision are rarely the ones on the keynote slides.

It’s the PM who is already over-mentoring, over-explaining, and over-documenting. The only woman on the platform team. The Black or brown PM who mysteriously ends up running every “culture” initiative on top of their real job. The person in a different country or a different body who still gets asked to “translate” for leadership.

You didn’t pick “tokens per month” as the KPI. You didn’t negotiate the vendor discounts. When the AI bill shows up, your calendar fills up with “quick chats” and “can you help me understand this?” invites. You are expected to explain a mess you didn’t make, to people who will never admit they don’t understand their own slide deck.

PM Confidential is written for you. The product leader treated like a line cook: essential, invisible, and instantly replaceable.

TL;DR

• Anthropic’s own checklist from April 2025 said don’t build agents unless the task value exceeds $1 per task. Fourteen months later their billing change made it financial reality, not just a guideline. Some developers saw 50× cost increases overnight.

• Microsoft canceled most of its Claude Code licenses because internal adoption was too successful. The company selling AI transformation to enterprises couldn’t afford it at scale internally.

• Token usage and PR velocity are not AI ROI metrics. Before the Q3 budget review arrives, build your own baseline quietly, in a spreadsheet nobody asked you to make.

Next issue: What does a real AI ROI baseline look like? (Hint: almost nobody has one.)