Companies Wanted Proof Their Teams Were Using AI. They Forgot to Measure Whether It Helped.

A senior engineer who was asked to draft an architecture proposal with AI pushed back with a line worth writing down. "I can get the AI to sound confident about anything. That's the problem, not the feature."

Gaurav Shenolikar, Director of Engineering at Mediaocean, an advertising-technology and ad-infrastructure company, recounted that exchange a few months into his team's Copilot rollout when he recounted the experience on LinkedIn. The engineer wasn't refusing the tool. He was refusing to treat fluent output as finished thinking.

Shenolikar kept circling back to something bigger than that one engineer's caution: the shape the rollout had taken underneath everyone. "Somewhere, 'use AI' quietly became a mandate instead of a tool," he wrote. "Leadership dashboards now track prompt volume." His shop isn't unusual in that. In a September 2025 survey of 1,295 U.S. business leaders, 58% said their company required employees to use AI tools, with about a quarter mandating it across every role. Once a tool becomes a requirement, someone has to report on compliance, and the easiest thing to report is how often people run a prompt.

More prompts, same problems

Counting prompts is where the reasoning slips. The assumption under a prompt-volume dashboard is that more usage means more value, so a leader's job is to push the number up. Shenolikar's point is that the number measures the wrong thing, because AI doesn't pay off evenly across the work. It compounds on the tasks where a mistake is cheap and easy to catch: drafting a first version, writing documentation, generating tests, digging through logs, summarizing a long thread. Those are exactly the places a strong engineer should reach for it, and reach for it often. The trouble starts when the same push gets aimed at the other half of the job.

The damage appears in work where mistakes carry real consequences. “Architecture decisions that carry five years of consequences, a hard conversation with a stakeholder who’s losing patience, performance feedback, conflict between two leads who each think the other is the problem,” Shenolikar wrote. “These moments aren’t slow because we lack tools. They’re slow because they require judgment, context nobody wrote down anywhere, accountability and enough empathy to read what someone isn’t saying.” A prompt-volume dashboard can't tell apart an engineer who used AI to clear an afternoon of documentation from one who used it to shortcut a design call that'll shape the system for years. Both push the counter up the same amount.

Action vs. distraction

Elsewhere, activity and outcomes have already begun to drift apart. In a 2024 study of 2,500 workers by Upwork and Workplace Intelligence, 96% of C-suite leaders expected AI to raise their company's productivity, while 77% of employees said the tools had actually added to their workload, and nearly half didn't know how to reach the productivity gains they were being asked for. The expectation sat on one side and the lived result on the other. A dashboard tracking how much AI got used would have shown a healthy, rising line straight through the gap.

Shenolikar's fix isn't to slow the rollout down, it's to aim it. "AI can help you prepare for those moments," he wrote of the hard calls. "It cannot sit in the room for you." A model can hand an engineer a sharper draft of the architecture proposal, a cleaner set of talking points before the tense stakeholder meeting, a first pass at the performance review. The engineer still has to make the five-year call, hold the conversation, and hear the thing the other person isn't saying. For a leader, that splits the job in two. Aim AI hard at the work where a miss is cheap, and protect the people doing the work where it isn't. The measure that keeps the two straight isn't usage. "I don't measure AI adoption by how many prompts my teams run," Shenolikar wrote. "I measure it by whether outcomes improved. If the answer is no, more AI isn't the fix. It's the distraction."

The people behind the outcomes

Once outcomes replace prompt volume as the measure, the most valuable engineers are the ones making the high-stakes calls the dashboard can’t capture. Every team can buy the same assistants, and routine work like drafting and log-digging will begin to level out across the industry. The difference will show up in the judgment that remains: who makes the architecture call that still holds up five years later, and who can sit through a hard conversation and get it right. Those are the engineers leaders need to keep strong, because making routine work cheaper only raises the cost of getting the consequential work wrong.

A dashboard can count every prompt a team runs. It still can't tell you whether the work got better.