How much agent work should a human review?

Not all of it. The better question is what should decide the percentage. If you review everything, the queue becomes the bottleneck within weeks. If you review nothing, you have no idea what your gates are missing. What works is a rate tied to evidence. It starts high while a worker is unproven, falls as the worker builds a record, and rises again as soon as the numbers show something changed.

Why does "review everything" stop working?

Because teams stop doing it long before anybody decides to.

It happens quietly. A team reviews every completed task. Then agent volume climbs, and the reviews get faster, then shallower, until they are a scan of the summary. Nobody announces a policy change. The written process still says everything is reviewed, but that is no longer true. This is worse than an honest sampling policy, because nobody can see the gap, including the people doing the reviews.

You can measure this as human wait share, which is the fraction of cycle time spent waiting on a person. Once agents produce work faster than people can absorb it, that share grows without limit, and the queue adds directly to cycle time. You keep paying the full cost of review while less and less reviewing happens.

So you will sample either way. The choice is whether you do it on purpose or by accident.

What should set the rate?

Evidence about that worker, applied by a set rule.

The inputs that matter:

• Track record on this kind of task. A first-pass rate measured over enough work is the strongest signal. It is also specific to the task type. An agent can be excellent at one task template and poor at another.
• How new the worker is. Anything unproven starts at a high rate: a new agent, a new model version or a new task type. Probation is how you collect the data that lets the rate come down.
• What the task can reach. Irreversible or externally visible work carries a higher rate regardless of track record, because a miss there costs more.
• Whether anything changed. A model upgrade, a prompt change or a new tool means your past record no longer describes the worker you have now.

Some things should not set it: how busy the team is, how much somebody trusts a particular vendor, or whether the release is on Friday. If the rate moves for those reasons, it is not a control. Problems tend to show up in the very week it was lowered.

Should the rate be the same for everything?

No. A uniform rate is the most expensive mistake you can make here.

A flat 10% across all work spends the same attention on a documentation fix and a change to billing. The point of sampling is to put scarce review where a miss would cost the most. A uniform rate throws that away.

Set bands by consequence first and by confidence second. Work that touches money, customer-visible output, permissions or data deletion sits in a high band permanently. Routine internal work with a long clean record sits low. The middle, which is most work, moves with measured performance.

A quick check: if the rate never differs between two kinds of task in your system, you have a flat policy, not a sampling strategy.

What should a reviewer look at?

The evidence, and especially the places where the evidence and the claim disagree.

The worst use of a review is to hand somebody a completed task and ask whether it is good. They will read the summary, find it plausible and approve it. Plausible output is what the worker is best at, so plausibility is the wrong thing to test.

A review is productive when the reviewer arrives with the mechanical checks already done and the disagreements already shown. That means the deterministic checks that ran and what they said, the definition of done beside the attached evidence, and any point where the two do not line up. The reviewer then decides the questions that need judgement, instead of redoing work a machine already did.

This is also what makes a sampled review worth more than an unsampled one. If every review is a full manual re-check, you can afford very few. If the reviewer only has to rule on what the automated checks surfaced, you can afford enough reviews to learn something.

What do you do with what the sample finds?

Change the gate, not just the task.

A sampled review that catches a problem tells you two things. This task was wrong, and your automated checks missed it. The second is much more valuable, and it usually gets ignored under time pressure. Every problem found by sampling describes a check that should have existed.

The rate should change too. A failure found in a sample shows that this worker's rate was set too low. The adjustment should happen automatically, without a debate.

To tell whether the system as a whole is working, watch auto-verification share. That is the proportion of work that passed with no human touch. You want it to rise over time, because that means the gates are handling more of the load. You also want the reopen rate to stay stable while it rises. If auto-verification and reopen rate rise together, the gates got more permissive, not better.

Where should you start?

Start high, and lower the rate as evidence comes in.

Starting low and raising it after the first incident gets it backwards. The incident is the thing you were trying to avoid. And a rate set on a bad day tends to become permanent. Instead, review a large share of a new agent's work, wait for the first-pass rate to settle, and let the rate fall as the record grows.

Before you need it, write down what would push the rate back up. If the rate can only ever fall, it is not a control either.

In Cognibl the human sample comes after the deterministic checks. On the AI plan, it also comes after a flow that reads the proof against the definition of done. The sampled share is set by the agent's score, not by a person's judgement on the day. A manual override is possible, but it needs a logged reason and it shows in the audit trail.