Writing a definition of done an AI agent can pass

Every tracker has had a definition-of-done field for a decade, and on most teams it holds something like "works and is tested". That line was a reminder, written for a colleague who already knew what the task was for. It worked because the reader filled in everything it left out.

An agent fills in the gaps too. But it fills them with whatever seems plausible, and that is not the same as correct. It also does this without stopping to ask, which a person would have done.

The reminder and the specification

A reminder is read by someone who shares your context. It only needs to jog their memory. A specification is read by someone who has none of that context. It has to still work when taken literally.

"Handle the edge cases" is a reminder. Taken literally it has no limit, so whatever the reader happens to think of first will satisfy it. "Rejects an empty file, a file over 10MB, and a row whose date does not parse, and reports which row failed" is a specification. It can be wrong, and that is what makes it useful.

So the test to apply to anything you have written is this: could this be marked failed? If no honest reading of the sentence could ever fail, the sentence is doing no work.

Write the check instead of the intent

In practice, write down what someone would observe, rather than the result you are hoping for.

Intent sounds like: the import should be reliable. Robust error handling. Good test coverage. None of these can be checked, so none of them can be refused.

A check sounds like: a malformed row leaves the other rows imported and appears in a rejects file with its line number. The test suite covers the three rejection paths. A run over the sample file completes without manual intervention.

The second version is longer, and that is its cost. The person who understands the task best pays that cost once, at the start. Without it, whoever reviews output that nearly works pays it again and again.

Say which systems may be touched

A definition of done written for an agent has a second job that a human-facing one never needed. It has to limit the blast radius. A person knows not to refactor the billing module while fixing a typo. An agent does not know that unless the task says so.

So name the surface: which repository, which branch, which files or directories are in scope, and what must not be touched. You would scope a pull request the same way. The difference is that here it is written down for a worker who cannot work it out alone.

Watch two numbers together

You can measure whether any of this is working, and you need two metrics to do it.

First-pass verification rate is the share of agent tasks that pass on the first attempt. It measures specification quality. A well-written definition of done passes first time. A vague one shows up here instead of in production.

Spec time is the part of cycle time from creation to first run, including every revision of the definition of done.

Neither means much alone. Long spec time is not a fault by itself. Some tasks are hard to specify, and the thinking has to happen somewhere. Long spec time beside a low first-pass rate is the signal to act on. It means the definitions are being written slowly and are still not working. That is a sign people are describing the work instead of specifying it.

Watch for the reverse pattern too. A very high first-pass rate with almost no spec time usually means the definitions are too loose to fail. The gate is then confirming that something happened. It is not confirming that the right thing happened.

A structure that works

Four parts, in this order:

1. The observable outcome. What is true afterwards that is not true now, stated so it could be checked by someone who was not in the conversation.
2. The checks. What specifically will be run or looked at, including the cases that should fail.
3. The surface. Which systems and files are in scope, and which are not.
4. The evidence. What the task should produce so the outcome can be confirmed later. This is the proof of work.

It does not need to be long. Most tasks fit in a short paragraph and a handful of bullets. It must not be a sentence that no outcome could contradict.

Start where it hurts

You do not need to rewrite your backlog. Take the last five tasks that came back wrong and read what their definitions of done said. You will usually see the pattern straight away, and it is usually the same one. The sentence described the work instead of the result, so anything that looked like the work satisfied it.