The AI ROI Trap
In a recent discussion with a CTO, I asked about the metrics they used for measuring the impact of AI on their business. He shared a slide that had been presented to the executive team as a bragging right that boldly claimed 87% Adoption of the company’s new AI co-pilot.
When I pressed him with questions, … what needle did this move? Productivity? Tech-debt reduction? Customer satisfaction? There was a deafening silence for a few seconds. Andy Grove used to say – you cannot improve what you do not measure. That advice still holds true and should be part of any CxO’s dashboard when creating the metrics for AI adoption – what is the impact to the business?
We’ve all seen this movie before – eyeballs, page views, clicks, likes, reach, impressions. All these metrics were made up by the platforms to sell more ads. The only metric that matters is conversion. Did the user take the action that was desired – buy or subscribe? This is why I wrote in one of my books – motion is not progress.
Entire market caps evaporated because activity metrics stood in for value metrics for few years too long. A MIT study making the rounds this year found that roughly 95% of enterprise generative AI pilots never move the P&L needle. That is not a failure of technology. It’s a measurement failure, and it’s entirely self-inflicted.
Here’s the CxO’s real trap – adoption (motion) is easy to measure and easy to report upward, so it becomes the metric, even though it’s the least meaningful. The issue is that such metrics become difficult to defend when the board does not see any solid results from all these investments in the latest technology.
I had developed my own 5-rung measurements that I have used in many transformation initiatives, which can easily be applied to AI ROI metrics. Call it the AI ROI Ladder.
- Rung 1: Awareness – People know the tool exists
- Rung 2: Adoption – People log in to the tool or use it in certain scenarios (often when they are reminded)
- Rung 3: Habituation – People use it without being told to
- Rung 4: Attribution – You can trace the spends to actual results; e.g. cost saved or revenue growth
- Rung 5: Strategic Advantage – Competitors cannot easily replicate what you have built
In my view, all current AI success stories are rung 1 or 2 dressed up as rung 5. Here’s a comment made by one ‘AI Leader’ that I found very amusing – “I don’t care if the chatbot hallucinates or responds in gibberish. As long as we type in a question and the chatbot responds, the PoC is successful”. I didn’t question him. I already knew where this team was headed.
I would go on a limb and say that every enterprise AI initiative should carry a mandatory disclosure of which rung of the ladder their proposal metric sits on. If your IT team, or the consultant, or the sponsor can’t answer that question, you’re not funding a capability, you’re funding a headline.