# Quality · Loa Arbor

Arbor saves every step of every conversation. Copy one at the step where it went wrong, change a prompt, a model or a tool, run it again, and read the two side by side with what each cost.

Example: `arbor session fork --session s-1 --at 42 --replay --prompt style=./brief.md`

## Fix a bad reply, and check the fix

Terminal recording (https://arbor.getloa.app/casts/06-fork-with-a-changed-prompt-and-compare.cast): Claude Code, in a terminal, finding an Arbor conversation that went badly, changing the agent's instructions, forking the conversation onto the new version and comparing the two.

A real Claude Code session (Sonnet), unedited, on the support desk built on the Examples page. It finds a conversation where the desk gave no next step, deploys changed instructions as revision 4 without moving `latest`, forks the bad conversation onto it and runs `arbor session compare`. Two commands the allowlist refused are on screen too. The cost is $0 on both sides because the local model has no price, so the agent compares tokens instead. Recorded on Arbor's main branch at fb647c60.

## Testing an agent

Without a record of every step:

- A user reports a bad answer. You have a log line and a guess.
- You edit the prompt and try it on three inputs you invented.
- Nobody can say if last week's change made the refund flow worse.
- To rerun a conversation you rerun its tools, and the email goes out twice.

With Arbor's ledger:

- The bad conversation is already saved, step by step. It is the test case.
- You copy it at the step that went wrong, change one thing, and run on from there.
- The two runs are compared step by step, with each side's spend and the first step where they differ.
- For each kind of call you choose who answers, so a tool that sent an email in the original does not send it again.

## Any conversation is a test

A fork is a copy of a session up to a position that carries on from there with the change you choose. The original stays as it was.

### Copy at the step that went wrong

`arbor session fork --session s-1 --at 42` copies the session up to position 42. The copied steps are not run again, so they cost nothing. On a node not in `--dev` mode, a fork needs `--replay`. A replay fork is kept out of the session list and archived when it ends.

Source, in arbor at v0.2.5: cli/session_fork.go

### Change one thing

A prompt (`--prompt id=file`), a whole overlay of settings (`--overlay`), or the revision (`--revision`, or `--label` for one a label names). The fork runs on from the copy with that change.

Source, in arbor at v0.2.5: cli/session_fork.go

### Choose who answers each call

`--intercept` picks, per kind of call (the model, a tool, a module, an outside host), one of `live`, `recorded`, `stub`, `generate`, `fuzz` or `fault`. With none given, tools answer from the original's record and the model answers live.

Source, in arbor at v0.2.5: cli/session_fork.go

### Compare, with the cost

`arbor session compare --session s-1 --with s-2` aligns the two sides' steps by what each step commits (messages consumed, calls made, messages sent, state left) and prints each side's spend and the first step where they differ.

Source, in arbor at v0.2.5: cli/session_fork.go

### Replay a session from the record

`arbor session replay --session s-1 --revision 5` answers the calls the record holds from the record, and they cost nothing. After the first request the record does not hold, the replay goes on live, unless `--strict` stops it there. That is where a changed prompt first changes what the model is asked.

Source, in arbor at v0.2.5: cli/session_fork.go

Forks and replays in the workflow: https://arbor.getloa.app/en/../workflow/

## The loop

1. Probe a live session

   ```
   arbor quality probe add --session s-1 --at 42 --set prompt.style='{"text":"brief"}'
   ```

   A probe forks the session at that position under your change, runs the copy and grades it by every eval whose matcher fires. `--base` runs the unchanged revision beside it, so you can tell the change's effect from the model's own variation.

2. Keep the moment as a case

   ```
   arbor quality probe keep <probe> --benchmark refunds
   ```

   A case is a kept slice of a session. It carries the saved steps before the slice, which are never run again. Or formalize the probe as an experiment with `arbor quality probe formalize`.

3. Run an experiment

   ```
   arbor quality experiment candidate add <experiment> --set prompt.style='{"text":"terse"}'
   arbor quality experiment gather set <experiment> --eval refunds
   arbor quality experiment gather run <experiment>
   arbor quality experiment run <experiment>
   
   ```

   An experiment has a base revision, numbered candidates (each a change laid over the base), a gather that finds more cases by an eval's condition in real sessions, and the evals that decide. A run prints one row per candidate, with its score and its cost.

4. Hold some cases out

   ```
   arbor quality experiment add --question "..." --case <case id> --holdout 0.3
   arbor quality experiment case hold <experiment> <case id>
   arbor quality experiment holdout <experiment>
   
   ```

   `--holdout <share>` on `experiment add` holds out a share of the cases, and `experiment case hold` holds out one more. Held-out cases are not run while you iterate. `experiment holdout` grades them once, so you cannot tune until it agrees with you.

5. Conclude

   ```
   arbor quality experiment conclude <experiment> --promote c2
   ```

   Promoting stages the winning change into the draft. It publishes nothing, so a person still reviews and publishes.

6. Run a benchmark on each new revision

   ```
   arbor quality benchmark run refunds r3 r4 r5
   ```

   A benchmark is a standing set of cases. Run it against the revisions you name. It exits non-zero when a replay could not run, so it fits a CI step. `benchmark history` and `benchmark heatmap` show the score by revision against two bands, good and acceptable.

7. Grade live traffic

   ```
   arbor quality monitor set web-refunds --evals refunds --trigger after:24h --sample 0.25 --ceiling 40 --ingress web
   ```

   A monitor runs evals over a sample of live sessions. A filter picks the sessions, the sample says how many, and the trigger says when to grade. The ceiling caps grading at forty an hour.

Every `arbor quality` command, in the workflow: https://arbor.getloa.app/en/../workflow/

## How a conversation gets a grade

An eval is a question you can ask of any conversation, live or copied. When this happens, was it good?

### A condition, not a step number

An eval's matcher finds the moments to grade: an actor of a definition ran, a tool was called, a message arrived, a label was set, the session closed. You do not point at step 42. You say what to look for, and every session that has it can be found with `arbor quality eval matches`.

Source, in arbor at v0.2.5: docs/2026-11-08-design-eval-v2.md

### Checks first, then one judge

An eval has any number of checks, which are plain assertions and cost nothing, and at most one judge: an agent with a rubric. A failed check scores 0. Otherwise the judge's score is the grade. With no judge, the grade is 1.

Source, in arbor at v0.2.5: docs/2026-11-08-design-eval-v2.md

### Every grade is one number from 0 to 1

So grades add up. A mean and a count, by revision, by arm or by time, mean the same whether the grades came from live sessions or from copies. `arbor quality eval grades --by revision` shows it.

Source, in arbor at v0.2.5: docs/2026-11-08-design-eval-v2.md

### Judging costs money, so it is counted

An eval with a judge costs one judge call per match. Every grade records what its judge cost. A monitor's sample and hourly ceiling keep the bill in bounds, and what was matched, graded, sampled out and clipped is counted, not guessed.

Source, in arbor at v0.2.5: docs/2026-11-08-design-eval-v2.md

### Arbor can play the user

In a copy, an agent can sit in the participant's seat, to see how a change holds up over a whole conversation and not one reply. You turn it on with a case that has a simulated feed, which needs a persona and a goal.

Source, in arbor at v0.2.5: module/quality/case.go

How the commands are named: https://arbor.getloa.app/en/../workflow/

## What to know before you rely on it

### A copy is not a replay of the model

The model answers live in a plain fork, and it can say something different each time. That is why a probe can run the base beside the change, and why an experiment can repeat each run.

Source, in arbor at v0.2.5: cli/quality.go

### A benchmark is run by you

A benchmark can be marked to run on every new revision (`--every-revision`), but the run is yours to start, from CI or by hand. Nothing triggers it for you.

Source, in arbor at v0.2.5: module/quality/benchmark.go

### A low grade does not fail a run

A run exits 1 when a replay could not run. A low score is a result you read in the history, not a failure.

Source, in arbor at v0.2.5: cli/quality.go

### The console pages are still being built

The commands exist today. The console's quality pages are built against them and are not finished.

Source, in arbor at v0.2.5: docs/2026-11-08-design-eval-v2.md

## Arbor holds itself to the same standard

- 4,172: mutants in the first sweep, over ten kernel and runtime files. Each is a small break in the code that some test should catch. (docs/2026-11-12-report-mutation-triage-runtime.md)
- 74%: of the 3,310 mutants that built and that a test covered were caught (2,450 caught, 860 survived). The survivors and the 256 uncovered mutants were sorted by hand, and tests were written for the ones that guard commits, leases, retry and park rules. (docs/2026-11-12-report-mutation-triage-runtime.md)
- 121: assumptions about the first eval design, since replaced, each checked against the real code. Six turned out to be bugs, and are fixed. (docs/2026-10-22-report-eval-verification.md)

How Arbor is tested, and what it found: https://arbor.getloa.app/en/../history/

## How the same ideas test Arbor's own runtime

The runtime is held to a written standard of its own, laid out in `docs/quality-report.md` and `docs/2026-11-12-plan-quality.md`.

For each part of the runtime that has been through it, three steps are done. The part's states, events and waits are written down from the code. Each decision it makes is made visible as an event. Then an explorer drives it through every ordering of its events, with faults injected, and checks each against the written table. A combination with no entry in the table fails. An ordering that breaks its row is reported and never fixed in the same change.

```mermaid
%% The three steps each part of the runtime goes through
flowchart TB
  table["1 Write down states,<br/>events and waits<br/>from the code"] --> visible["2 Make each decision<br/>visible as an event"]
  visible --> explorer["3 An explorer drives every ordering<br/>of events, with faults injected"]
  explorer --> check{"Matches the<br/>written table?"}
  check -- "yes" --> done["Part passes"]
  check -- "no entry, or breaks its row" --> fail["Reported, not fixed<br/>in the same change"]
```

Other checks run beside that:

- **Mutation testing** breaks the code in small ways, one at a time, and checks that a test fails for each break. The first full sweep is the number above. Four reports record the triage of its survivors (`docs/2026-11-12-report-mutation-triage-runtime.md` and the three of 15 November).
- **The race detector** runs under the Go suite on every pull request.
- **Stress and soak runs** look for wrong answers: lost messages, two turns of one actor committing, a parked actor waking. `arbor soak` drives scripted conversations through a running node.

It is not finished. Only the wake path has been through all three steps, by the quality report. Held calls and reading while writing are partly done: held calls have the written table and their fixes have tests, but their decisions are not yet visible and no explorer has run; reading while writing has the table and visible decisions, and no explorer yet. Ending a session has not been begun. The sweep covered ten files, not the whole runtime. [Benchmarks](../benchmarks/) shows what the load and stress tests found, and [History](../history/) lists every defect, with the open ones marked.

- [Workflow](https://arbor.getloa.app/en/../workflow/): Every `arbor quality` and `arbor session` command, in the order you would use them.
- [Benchmarks](https://arbor.getloa.app/en/../benchmarks/): The load, stress, soak and race testing, with the limits they found.
- [History](https://arbor.getloa.app/en/../history/): Every defect found, how it was found and what is still open.

Loa Arbor. Licensed under the Functional Source License 1.1, with MIT as its future licence (FSL-1.1-MIT): each release becomes MIT two years after it comes out. Coming soon: the download opens when the repository goes public.
