Measuring agentic AI: what to instrument

You need to instrument four things to run an agent programme. Track whether the work passed on its first attempt, where its cycle time went, how much of that time was spent waiting on a person, and how much of the work came back afterwards. Everything else either follows from those four or is activity data that only looks like a metric.

The gap here is large. Arcade.dev's 2026 survey found 52% of gen-AI organisations running agents in production and only 31% with any measurement framework for them. Most teams shipped the worker before they built the instrument.

Why do the usual AI metrics not answer the question?

Because they measure the model, while your question is about the work.

Token counts, latency, tool-call frequency and hallucination rate are all worth having, and all useful for debugging a run. None of them can tell you whether the task got done. An agent can produce a fast, cheap, perfectly grounded answer to the wrong question, and every one of those numbers will look excellent.

Most of the observability tooling built around agents serves that debugging job. Industry reviews say so plainly. The thin capability is policy enforcement at execution time and attribution good enough to stand up, more than the depth of tracing. You cannot attribute a call you did not log. And a log you only read after the fact does not let you manage anything while it happens.

What are the four that matter?

First-pass verification rate. The share of agent tasks that pass on attempt one. It measures the quality of your specifications, and it is the closest thing this work has to a single health number.

Cycle time, split by phase. Break it into spec, build, verify and settle. Agents shrink the build phase, but the time moves to other phases instead of disappearing. So a single total often shows no improvement at all and gets read as failure.

Human wait share. The percentage of total cycle time spent waiting on a person: review pickup age, sample-queue age, blocked-on-owner. It is usually the biggest thing you can improve and the cheapest, because fixing it takes scheduling, not engineering.

Reopen rate. Tasks reopened within thirty days of completion. This one catches a verification system that grades itself too generously. No measure taken at the moment of completion can catch that.

What about throughput?

Count it, but count verified completions only, never raw ones.

Raw completion counts are this field's vanity metric, and agents make them meaningless overnight. A worker that can mark forty tasks done in an hour turns "completions per week" into a measure of how fast workers can mark things done. The chart goes up and to the right whether or not anything was delivered. That makes it worse than no chart, because someone will present it.

Verified throughput is the same count with the numerator fixed. It only counts completions that passed a gate. It moves more slowly and is sometimes embarrassing, but it is the only version that holds up when a stakeholder checks.

Where should the numbers come from?

From the system that observed the work, never from the worker that did it.

This is the main design rule, and it is easy to get wrong because self-reported telemetry is much simpler to build. An agent that writes its own trace is reporting on itself. It is usually not lying. But it reports what it believed it did, and that is exactly what is in question when a task fails.

So record tool calls at the gateway they pass through. Record verdicts from the checker that issued them. Record timestamps from the transitions the tracker wrote. If the party being evaluated can edit a number afterwards, treat it as a claim. Label it as one everywhere it appears.

What should you deliberately not instrument?

• Agent signups and raw job counts. They measure volume, not value, and should never headline a dashboard.
• Per-run token cost as a headline. It is worth watching for budget but useless as a quality signal. As a headline, it pushes teams toward cheaper runs that fail more often.
• Anything a person maintains by hand. If a metric depends on somebody keeping a field accurate, it drifts quietly, and you will not know when it stopped being right.
• Model-level benchmarks as a proxy for delivery. A better score on a public benchmark does not tell anybody whether their tasks are passing.

How long before the numbers mean anything?

Collect four to eight weeks of baseline before you draw conclusions. If you still can, take the baseline before the agents arrive. Teams doing this well agree on the point. Instrument the workflow first, because a pre-agent baseline beats any retrospective benefits model.

The first honest report usually looks worse than the optimistic one it replaces. Verified throughput is lower than raw throughput. First-pass rate on a new programme is often unflattering. The drop does not mean anything got worse. It shows the gap between what was happening and what was being counted, all at once.

In Cognibl these five sit on the project home page. They are computed from the verifier's own records, not from activity. That is why they are smaller and duller than the numbers on a usage dashboard, and why you can show them to somebody who checks.