Key takeaways

Raw task-success rate is the obvious metric and dangerous alone, because the fastest way to raise it is removing friction that was protecting the human. Always pair it with a legibility or reversibility measure.
Four measures worth tracking: discoverability, unassisted completion, failure recoverability, and human reconstructability.
Failure recoverability is the highest-leverage one, because it is really a measure of whether your error messages name a next action. Fixing errors moves every other number.
Run it as a fixed task set across several agents with repeated runs, and record transcripts rather than just outcomes. The outcome tells you the score; the transcript tells you why.
This is a young discipline with no industry benchmark. Treat these as a working frame to compare against your own baseline, not an external standard.

You have done the agent experience work. There is an API, the docs are machine-readable, auth completes without a browser detour. Now somebody wants to know whether it worked, or you need to decide where the next quarter of effort goes, and you reach for a number and find that the obvious one is not trustworthy.

It is not. Here is why task success alone misleads, and four measures that behave better.

Why task success rate misleads on its own

The obvious metric is the share of attempts where an agent completes the task. Easy to define, easy to instrument, and it moves in the right direction when you do good work.

The problem is that it also moves in the right direction when you do bad work. Remove a confirmation step and success goes up. Grant a broader standing credential so the agent stops getting blocked, and success goes up. Stop producing an artifact a human would have reviewed, and success goes up, because the review was where runs stalled.

Every one of those improves the number while degrading the thing Agent Experience is supposed to serve, which is an agent that is more capable without a human becoming less able to understand and intervene. A metric that rewards removing oversight will eventually get you a system with none.

So the rule is simple: never report task success without a companion measure of whether a person can still see and undo what happened. The fourth metric below exists for that.

Metric one: discoverability

Can an agent find the capability without a human who already knows the vocabulary?

Measure it by running a cold agent at a task with no hints: no documentation links in the prompt, no naming of your product’s internal terms, just the goal. Record whether it finds the right capability, how many wrong paths it tries first, and whether it gives up.

This is the metric most products score worst on and the one teams least expect to. Capability exists; it is simply only findable by someone who knows what it is called. If an agent has to already know your vocabulary to discover your feature, the feature is invisible to every agent that has not been specifically configured for you.

The fix is usually machine-readable description rather than better prose: capabilities declared somewhere structured rather than implied by a dashboard’s layout.

Metric two: unassisted completion

The share of runs that finish without a human stepping in, which is the honest version of task success.

The value is not the headline number, it is the segmentation. Record where the human had to intervene, and the categories tend to be few: the agent could not authenticate, it could not find the capability, it took an action that needed approval, or it got stuck in a loop it could not exit.

Each category points somewhere different. Auth failures are a flow problem. Discovery failures are metric one. Approval stops may be entirely correct and should not be optimized away, which is exactly why you segment rather than tracking a single rate.

The number to watch over time is not “completion went up,” it is “the intervention mix shifted,” because that tells you whether your work landed where you aimed it.

Metric three: failure recoverability

When an agent fails, does it succeed on the next attempt without help?

This is the highest-leverage measure on the list and it is slightly indirect, because it is not really measuring the agent. It is measuring whether your failures are actionable. An error that says what happened and what to do next produces a self-correcting agent. An error that says a status code and a stack trace produces an agent that tries a variation and sounds confident.

Measure it by counting, among failed first attempts, the share that succeed on a retry with no new human input. A low number means your errors are not carrying the information an agent needs.

The reason this is leverage: fixing error messages moves metric two directly and metric one indirectly, costs very little, and improves the human experience at the same time, because a human reading “add it to dependencies or set the build command” is better served than one reading “exit code 127.”

Metric four: human reconstructability

After a run, can a person who was not watching establish what happened?

This is the counterweight that keeps the other three honest, and it is the one teams skip because it feels qualitative. It is not. Give someone the aftermath of a run and a fixed set of questions: what changed, when, which action caused the current state, and how would you undo it. Time them. Record whether they can answer at all.

The mechanisms that make this answerable are the ones already worth having: observability tied to each deploy so effects attach to the change that caused them, function and build logs showing what the code actually did, and artifacts like a preview URL or a diff that a person can open.

If reconstructability is falling while task success rises, you have the exact failure mode this article opened with, and you can now see it in a number rather than discovering it after an incident.

How to run the evaluation

Four practical requirements, none expensive.

A fixed task set. Ten to thirty tasks that represent real work, written down and unchanged between runs. Varying the tasks makes the numbers incomparable, which is the most common way these evaluations become useless.

Several agents. Different models and harnesses fail differently, and a result from one tells you about that one. Netlify’s own agent-facing surface is infrastructure for AI code agents, and the whole premise of an open deploy target is that many different agents arrive, which is also the right assumption for an evaluation.

Repeated runs. Agents are non-deterministic. A single run of a task is an anecdote; five runs is a rate. This is the requirement people cut first and it is the one that makes the numbers mean anything.

Transcripts, not just outcomes. Record what the agent actually did. The score tells you where you stand; the transcript tells you why, and the why is the only part that generates work.

Netlify’s related surface here is netlify.ai, an agent-first entry point and agent-access evaluation, which is the same idea applied to the platform itself: can an agent succeed at a defined goal using what is available.

What the numbers usually say

Three patterns that show up often enough to expect.

Discovery dominates. Most failures happen before the agent does anything, because it could not work out what was available. Teams usually expect execution problems and find navigation problems.

Error quality is the cheapest fix. Rewriting the twenty most common failure messages to name a next action typically moves unassisted completion more than a quarter of API work.

And the uncomfortable one: when task success improves sharply after a change, check reconstructability before celebrating. A sharp improvement often means a safeguard was removed rather than a capability added.

An honest caveat about all of this

There is no industry benchmark for agent experience, no standard task set, and no agreed definitions. Anyone presenting a single AX score is presenting their own construct.

So treat these four as a working frame for comparing against your own baseline over time, which is the comparison that matters anyway. Instrument them, record a baseline now, and measure the delta after your next piece of work. Internal comparability is achievable today and is enough to direct effort, which is what measurement is for.

FAQs

How do you measure agent experience? Track four things rather than one: whether agents can discover your capabilities without prior knowledge, what share of runs complete without human intervention, whether failed attempts self-correct on retry, and whether a person who was not watching can reconstruct what happened. The fourth is what keeps the other three honest.

Why isn’t task success rate enough? Because it rises when you remove oversight. Dropping a confirmation step, granting a broader standing credential, or no longer producing an artifact a human would review all raise task success while reducing a person’s ability to see and undo what happened. Always pair it with a reconstructability measure.

What is the most useful AX metric to start with? Failure recoverability, meaning the share of failed attempts that succeed on retry with no new human input. It is really a measure of whether your error messages name a next action, and fixing those is cheap and moves the other metrics at the same time.

How do I run an agent experience evaluation? Fix a task set of ten to thirty real tasks and do not change it between runs. Run it across several agents, repeat each task enough times to get a rate rather than an anecdote, and record full transcripts rather than just pass or fail. The transcripts are where the actionable findings are.

Is there a standard benchmark for agent experience? Not currently. There is no agreed task set or definition, so any single AX score is the author’s own construct. Internal comparison against your own baseline over time is achievable now and is enough to direct where effort goes.

What usually turns out to be the problem? Discovery, more often than teams expect. Most failures happen before the agent takes any action, because it could not determine what was available. Capabilities exist but are only findable by someone who already knows what they are called.

Record a baseline on the four before your next piece of AX work, so the delta means something. If you want to see what an agent does against a platform built for this, point one at netlify.new and read the transcript rather than the outcome.