How to Audit What an AI Agent Actually Did

Most teams running agents have exactly one control: somebody clicks approve. It is the right first control and it has a blind spot. A person approving forty drafts a week is checking each one against "is this obviously wrong", which is a good filter for individual errors and a poor one for patterns. The problems that actually damage a business are patterns, and nobody sees them from inside the approval queue.

These are two different jobs and it is worth naming them separately.

Approval asks: is this specific message acceptable to send? It happens in the moment, under time pressure, against a queue. It catches the obviously wrong.

Audit asks: looking at everything from the last month, is this working the way we think it is? It happens away from the queue, without pressure, across many items at once. It catches drift, repetition and systematic error.

Most teams log what was sent. That tells you what happened and nothing about why, so when you find a problem you cannot tell whether the model reasoned badly or the data it was given was wrong. Those have opposite fixes.

The output. As sent, not the template.

The input. What context it had. This is what distinguishes a model error from a data error.

The sources. Which record, page or document it drew a claim from. Without this, verifying a specific means re-researching it from scratch, which nobody does.

The human delta. What was changed before approval. The single most useful field, because it is a continuous measure of how well the thing is working.

The outcome. Replied, bounced, complained, ignored.

Small businesses do not need a governance framework. They need one named person and a repeatable twenty minutes.

Read ten at random. Actually random, not the ten most recent and not ten you pick. Recency and selection both bias toward the ones you already had in mind.

Check every specific in those ten. Each number, name, date and claim, against its logged source. This is the bulk of the twenty minutes and it is where fabrication is caught.

Look at the edit rate. What proportion of drafts were changed before approval, and is it moving? Rising edit rate means something changed. Falling means it is learning. Flat and high means the setup is wrong and nobody has said so.

Read one thread end to end. Pick a single customer and read every message they received in sequence. Repetition and incoherence across a thread are invisible per message and obvious in sequence.

Write down what you found. Three lines. The value is almost entirely in being able to compare against last month.

Ten random items is fine at low volume. Past a few hundred a week, the random sample will keep showing you the common case and miss the things worth finding. Stratify instead.

Sample the edges, not the middle. The longest and shortest outputs, the ones where the agent had least context, the ones to your largest accounts, and anything it flagged as uncertain if it can. Failures concentrate at edges; the middle is where it works.

Always sample after a change. New prompt, new data source, new model version. This is where new failures come from, and the output looks completely normal until you check it.

Sample by recipient value. An error to a prospect costs an impression. The same error to your biggest customer costs more. Weight accordingly; it is not a fair sample and it is the right one.

Work out which layer failed before changing anything. Bad output from good input is a model or prompt problem. Bad output from bad input is a data problem, and tightening the prompt will not touch it. This is why the input log exists.

Check whether it happened elsewhere. One instance in a sample of ten usually means many instances in the population. Search for the specific claim across everything sent before deciding it was isolated.

Fix the class, not the instance. Correcting the one message is necessary and insufficient. If a fabricated statistic got through, the question is what stops the next one, and the answer is usually a requirement to cite a source rather than a better prompt.

Narrow the autonomy rather than switching it off. The common overreaction is to stop using agents. The proportionate response is to move that particular task back behind approval while leaving the rest running, which is the whole point of having a gate you can move.

The reason all of this is necessary rather than optional is in what AI still cannot do in marketing: a model produces wrong answers with exactly the fluency of right ones, so the confidence in the output carries no information. Auditing is how you get the signal the output does not give you. How the approval half works is at the approval gate.