Your AI Transformation Is Invisible (Until You Measure It Like This)

Here's the uncomfortable pattern in enterprise AI adoption. An organisation invests real money in AI-assisted engineering (tools, training, new ways of working) and six months later, nobody can say what changed. Some developers transformed how they work. Others quietly didn't. And nobody can tell which is which.

The executives sponsoring the change are funding something they cannot observe. Every claim of uplift is a story, not evidence. Continued funding rides on those stories.

We've been working on how to fix this, and the answers cut against most of the measurement advice on the market.

Raw output metrics answer the wrong question

The obvious move is to buy developer analytics: lines of AI-generated code, PR counts, suggestion acceptance rates. These measure raw output. What actually matters is something else: how far a team has shifted toward agentic software development, and what that shift means for the business.

That distinction is everything. The gap in the market is not a data gap. Telemetry is nearly free now; modern AI coding tools ship with OpenTelemetry support and hook systems out of the box. The gap is a framing and reporting gap: connecting that exhaust to a transformation narrative an executive can steer by.

Don't impose a metrics regime. Arrive where the client is.

The standard playbook says: establish DORA metrics first, then measure improvement against them. We've watched that fail in practice, and we now believe leading with it is the wrong move for many organisations.

Plenty of successful enterprises run two-month lead times and monthly releases, with no appetite to change that cadence. Others release quarterly. Push a faster-delivery measurement model onto them and you're fighting the client instead of helping them. At best your metrics work "virtually" while reality carries on unchanged. At Propel, we still push for DORA metrics, and faster delivery can remain a gradual long-term goal. It just can't be the entry fee for AI enablement.

The better move is to report against the delivery cadence and metrics the organisation already runs. Every business tracks its own productivity somewhere: a single DORA metric, epics completed per quarter, cycle time in their PM tooling. Which leads to the most useful trick we know:

Retrofit the baseline. Because the organisation's own metrics have history, you can go back in time and build the before/after picture from data that already exists, rather than freezing delivery while you stand up a fresh measurement regime. The "baseline first" instinct is right. The "build new instrumentation first" reflex usually isn't.

Expect a compounding curve, and measure it forward

One more thing trips up impact reporting: AI-assisted delivery is not fastest on day one. Adoption follows a compounding curve. Efficiency gains scale non-linearly as organisational skills, context, and trust build. Evaluate at week four against a promise of instant transformation, and you'll kill programs that were about to pay off.

We're not against uplifting metrics, but measurement shouldn't block the work of starting a transformation. Instead, run two tracks. Retrofit the organisation's existing metrics for the historical before/after, and start an AI-adoption-specific metric that trends forward from day one of the change. Together they tell the story honestly, without requiring anyone to bet on a heavy upfront measurement build.

Extend what's running; don't build a collector

Existing IDEs and agentic developer tools already expose built-in hooks. Enterprises, meanwhile, operate mature observability stacks and maintain the infrastructure to deploy extensions to every developer machine. So the strategy is straightforward: push custom events from those existing hooks directly into the client's current observability platform, then focus on the reporting layer above it. That's where the differentiating value lives.

Aligning an organisation with entirely new metrics tools is never a drop-in process. It's fundamentally a data-governance challenge: deploying a new data-collection product inside a client's ecosystem introduces heavy security, privacy, and compliance obligations. None of this is an argument against advanced tools. It's an argument for implementing them correctly. Measurement should serve the transition, not hinder it, and tracking should never become the bottleneck that delays the actual rollout.

Three practices make this concrete:

• Metrics as code. No AI capability ships without its metrics. A code review agent, a code-generation agent, a skill: each emits its own usefulness signals from day one, versioned alongside the thing it measures.
• Instrument the conversation layer. Default dashboards tell you that developers used an AI tool. Custom metrics and hooks tell you how: which skills, which capabilities, which workflows. That's where adoption actually shows up. We run this in production today.
• Roll AI effectiveness up to one score. As outlined in [AI-Augmented SDLC: The Flywheel of Compounding Delivery](AI-Augmented SDLC: The Flywheel of Compounding Delivery), a custom AI-effectiveness score computed from existing telemetry gives executives a trendline to steer by. Two core questions define it: Of what the AI wrote, what survived? And of the final output, what came from the AI?

The takeaway

If you can't show the baseline, the shift, and the trend, your AI transformation is a leap of faith wearing a business case. Measure the shift toward new ways of working, not out-of-the-box metrics. Measure it against the metrics your organisation already trusts, not an imposed regime. Do it through telemetry you already have, not a new product. And give the compounding curve time to compound.