# History · Loa Arbor

A dated changelog, how Arbor is tested, and every notable defect we found, with the symptom, the cause and the fix. Open problems are marked open.

This page is the project's own record, written from its changelog, its bug list and its flake log. It is meant to be uncomfortable in places. A runtime that promises "a step commits whole or not at all" should show you the times that promise was found broken.

Dates are 2026. The repository's history here starts on 3 October; `v0.1.0`, "the first published build", was tagged on **5 November**. At the point this page was written (20 November) the history holds 3,574 commits and 507 numbered pull requests.

A few words you will meet:

- An **actor** is the basic unit: an inbox, a ledger, a function that handles messages, and an outbox. An **agent** is an actor whose function is a loop around a language model.
- A **step** is the unit of commit. A turn is made of steps, and each step commits together or not at all.
- A **node** is one running Arbor process. A **lease** is a node's time-limited claim on an actor.
- A **store** holds the data. Arbor ships three that must behave the same: memory, SQLite and Postgres.

## How Arbor is checked

The rule behind everything below: **a fix is its own change, with a test that fails without it.** A bug is not closed because the symptom went away.

### What runs, and when

| Check | What it does | When |
|---|---|---|
| Unit and end-to-end tests | The Go suites, run with the race detector (`-race`) | Every pull request, and again on `main` after every merge |
| Postgres integration suite | Runs the stores against a real embedded Postgres | Since 19 November, on `main` and on demand, never on a pull request |
| Contract suites | One suite per store abstraction; memory, SQLite and Postgres must all pass the same cases, failure modes included | Every pull request |
| Explorers | Drive a part of the runtime (the wake path, held calls) through every ordering of its events and injected faults, and compare each with a written table | Every pull request, on the in-memory stores; the reduced sets on SQLite and Postgres run in the integration suite, so on `main` and on demand |
| Stress runs | A seeded random walk of two nodes sharing one store, checked against invariants such as "nothing sent is lost at a close" | A small budget per pull request, a large one nightly |
| Shuffled runs | The unit and integration suites, once through, in random order, to expose tests that depend on order | Sundays |
| Mutation sweep | Breaks the code in small ways (a comparison flipped, a statement removed) and checks that some test fails for each break | When its inputs change |
| Plan survey | Checks that database statements keep a good query plan as tables grow | The full grid runs in `main`'s full run and in dispatched runs, when the stores changed |
| Load test | 200 sessions of 20 turns each on two nodes sharing one Postgres, with the model answering at once and after 200 ms | On a dedicated runner |
| Field test | A person builds a real app with the shipped binary on a second machine | Occasionally; the first was 28 October |

```mermaid
%% Which checks run when
flowchart LR
  subgraph pr["Every pull request"]
    t1["Unit and end-to-end tests"]
    t3["Contract suites"]
    t4["Explorers"]
    t5["Small stress run"]
  end
  subgraph night["Nightly, and Sundays"]
    n1["Shuffled runs<br/>Sundays only"]
    n2["Large stress run"]
  end
  subgraph other["Other times"]
    o0["Postgres integration suite<br/>on main and on demand"]
    o1["Mutation sweep<br/>when its inputs change"]
    o2["Load test<br/>on a dedicated runner"]
    o3["Field test<br/>occasionally"]
  end
  pr --> night --> other
```

### The numbers

- **Mutation testing, first full sweep (13 November):** 4,172 mutants over ten kernel and runtime files. 606 did not build and 256 sat on lines no test runs. Of the 3,310 that built and that a test covered, 2,450 were caught (2,282 killed, 160 timed out and 8 ran out of memory) and 860 survived. Score: 74% of those 3,310 (`docs/2026-11-12-report-mutation-triage-runtime.md`). All ten files were then triaged by hand, every survivor sorted as equivalent, cosmetic, a missing assertion, untested behaviour, or a possible defect, and tests were written for the groups that guard commits, leases, retry and park rules. Four reports record it (12 and 15 November).
- **Database plans:** the plan survey's full grid went from 58 red cells to 41 after the 17 November cure for kept plans (below). A test now asserts that none of 176 measured statements in the catalog, directory and modules is slower on a long-lived connection than on a fresh one. It had been 121 on the default setting.
- **Nightly stress, past failures:** 40 kept nightlies (5 October to 13 November) were read, every failed seed was traced to a cause, and each cause was fixed. Ten failures of the stores' harness and five of the runtime's two-node test.
- **Data races:** 1,322 race reports in 41 job logs (16 October to 5 November) were all one race, in the test suite's own id counter, not in product code.
- **Held-call listing vs. who may answer:** a check of every combination found the two disagreed in 37 of 112 (12 November, below).
- **The bug list, 22 November** (`docs/bugs.md` as last changed on that day; the count line carries no date of its own): 0 open P1, 17 open P2 and 46 open P3 rows, and one row with no priority. On 20 November the same file counted 0 open P1, 47 open P2 and 68 open P3. The two counts are made the same way, from the file's rows, but they are not one series: the file now holds fewer rows. On 20 and 21 November the left-out lists of built designs and other features were moved from `docs/bugs.md` to `docs/roadmap.md` (which counts 39 open P2 and 37 open P3 rows of its own), and some rows were closed. So the fall in P2 is mostly a move and a few closures, not 30 defects fixed. One P1 was opened and closed in between: `arbor session chat --until reply` could end before the reply, found on 21 November and fixed on 22 November. P0 breaks a promise the kernel makes; P1 is a known hole in the foundation's proof; P2 is real but not urgent; P3 is later.

### How defects are tracked

- `docs/bugs.md` holds every open defect, gap and test gap, each with a priority and a type. Closed rows move, word for word, to `docs/tasks-history.md`.
- `docs/flakes.md` logs every intermittent failure with the sequence that produced it, its cause and its fix. Its rule: a flake on a loaded runner that passes locally is a test that depends on an order or on time, and the fix is the test, "never a longer wait and never a skip". A failed required check is read before it is rerun, because a rerun that passes proves nothing about the first attempt.
- A second model read the whole kernel cold, after the October refactor, against its documented promises (8 November, below).
- The checks are themselves audited. Several of the bugs below are bugs in the test machinery.

### Honest limits of the checks

- Not every part has been through the full program of written table, visible decisions and explorer. Only the wake path has been through all three steps (`docs/quality-report.md`). Held calls and reading while writing are partly done: held calls have the written table and fixes with tests, but their decisions are not yet visible and no explorer has run; reading while writing has the table and visible decisions, and no explorer yet. Ending a session has not been begun.
- The load test ran on a Redis stand-in and then a real Valkey server on a runner. Its ceilings, the numbers a run must stay under, are not set yet.
- The criterion "two weeks of nightly stress green on every store" is **not met**, and the count restarts whenever a store changes. A store freeze began on 19 November so the count can run.

## Changelog

Newest first. Only changes that matter to someone using or running Arbor; the full record is the history.

```mermaid
%% Milestones from the first commit to the store freeze
flowchart TB
  a["3 Oct: first commit in the repository"] --> b["28 Oct: first field test"]
  b --> c["5 Nov: v0.1.0, the first published build"]
  c --> d["7 Nov: gateway removed, relay added"]
  d --> e["8 Nov: cold review of the whole kernel"]
  e --> f["10 Nov: command tree reshaped around nouns"]
  f --> g["16 Nov: first load runs on a dedicated runner"]
  g --> h["19 Nov: store freeze began"]
```

### 21 to 22 November: v0.2.5

- **22 Nov.** v0.2.5 released. `session follow --until reply` and `session chat` now end at the reply to your own message, for every caller: a reply from a turn already running when the message landed no longer ends the wait. A model call in flight now counts as activity, so a slow answer that does not stream is no longer taken for a stalled turn; such a call is bounded by a deadline of its own (`model.strategy.call_timeout`, 10 minutes). The per-attempt `timeout` is now `idle_timeout`, the node's `--model-timeout` flag is now `--model-idle-timeout`. `arbor dev watch` prints a revision's publish warnings. A chat template's control tokens that a model prints as text (`<|im_end|>`, `<end_of_turn|>`) are now stripped from its replies and never sent back to it, and a paired `<|channel>` … `<channel|>` is kept as the model's reasoning, not shown as text.

### 17 to 20 November: speed under growth, and a store freeze

- **20 Nov.** The backlog was split in two: defects in `docs/bugs.md`, features in `docs/roadmap.md`. A pass read all 57 open P2 rows against the code: 11 were already fixed and closed, 13 were stale (still open for a smaller remainder), 30 stayed open and 3 could not be decided by reading.
- **19 Nov.** A **store freeze** began: no schema, procedure or store changes, so the nightly stress count can run. Fork statements stopped keeping bad plans. A model that answered a conversation-compaction request with a tool call instead of a summary no longer triggers another paid model call every turn. A slow ledger page on long actors was fixed. The renewal-time metric landed.
- **18 Nov.** The Sessions view's filters, paging and counts moved into the stores, so every count is exact instead of capped at the 2,000 newest sessions. At 100,000 sessions it was then brought under its targets.
- **17 Nov.** Two fixes to how a node scales with the size of its database: the poll for due work no longer reads every actor, and Go-side statements no longer keep a young query plan. A crashed turn now backs off before it is retried. The Redis-compatible store passed its contract on a real server and on a three-primary cluster.

### 13 to 16 November: the quality program, the load test and the notice fixes

- **16 Nov.** The first load runs on a dedicated runner found three defects in a day (below) and then a serialized turn start. Retried writes are now keyed, so a retry whose first attempt landed no longer applies twice. A dead database listener connection is now noticed by a ping instead of after about 150 seconds. Agent2Agent callers who are anonymous got a first fix; on 21 November it was decided that Agent2Agent requires an identified caller (see Known open issues).
- **15 Nov.** The mutation triage finished (four reports). Every failed stress seed since 6 October was traced. The Redis-compatible store's suites were first run against a real Valkey server and a cluster (measured here, closed 17 Nov).
- **13 to 14 Nov.** The quality report began, with the wake path taken through written tables, visible decisions and explorers, and held calls and reading-while-writing taken part of the way (`docs/quality-report.md`). The mutation sweep ran for the first time in five weeks. An audit of every CI workflow found the nightly could report green over failed tests (below). The tests that skip were all read.
- **12 Nov.** A cap's stop that could be spent past was closed. A lost store watch is now reopened. A lease lapse can no longer double-run a tool call. Held-call answers and crashes got their rules and tests.

### 8 to 11 November: reviews, the configuration model and the coding toolset

- **11 Nov.** The **Activity view**: a session told as activities and acts, served by the API and printed by `arbor session activity`.
- **10 Nov.** The **command tree was reshaped** around nouns, with no aliases kept. The **secret plane**: secrets resolved through a chain, with envelope keys per scope and erase-by-key-destruction. The telemetry read side: any node can read a turn another node ran.
- **9 Nov.** The **configuration model** (settings keys are tagged, resolved by one function, with per-actor overrides). The **coding toolset**: a workspace actor, git operations, file operations and typed tools. Console views served by the node.
- **8 Nov.** The cold review of the whole kernel, and the fixes for every promise it found broken the same day (below). The P1 defect pass.

### 5 to 7 November: v0.1.0 and the defect pass

- **7 Nov.** The **gateway was removed**: one session declaration per app, exposure as the only surface the outside reaches, roots addressed by name. A **relay** behavior (re-emit each message to the audience it calls for), including private breakouts.
- **6 Nov.** A **typed final reply**: an agent declares a JSON Schema, the reply is checked and asked for again with a note naming each problem. An **inbox control**: an actor can refuse user messages at the door and later accept them.
- **5 Nov. `v0.1.0`, the first published build.** The same day: a send to a closed actor now comes back to its sender as an `undeliverable` message, a tool call's id became its idempotency key, and a tool declares whether it is read-only, idempotent or destructive. The `run` tool followed that night: a short JavaScript program over an agent's tools, every call a step of its own.
- **4 Nov.** A pass closed every known defect in the backlog before the next feature (seven groups).

### October: the first build

- **28 Oct.** The first field test (below).
- **22 to 25 Oct.** Verification passes for evals, ingress, overlays and the CLI walkthrough. Configuration became one document.
- **21 Oct.** A fork now shares its origin's stored bytes instead of copying them; step entries were about nine tenths of a ledger's bytes.
- **18 Oct.** The first sample application's flavor was retired in favour of a service that talks to a node through the API and `@arbor/client`.
- **10 to 11 Oct.** A large batch of capabilities landed: `@arbor/client`, the single `/v1` API with the console at `/ui/`, the package registry, semver releases with a contract hash, recorded outbound HTTP (`arbor:http`), `arbor:node`, partition addressing with the modes `one`, `many` and `once`, schedules, successor sessions, retry on a tool, outbound webhooks, and the project CLI.
- **3 Oct.** The first commit in this repository. The design before it (September) is summarized in `docs/history.md`.

## Bugs found and fixed

Each entry: when, what was wrong, how it was found, and the fix. "Found by" matters: it says which check earned its keep.

### Wrong results and broken promises, found by the cold review (8 November)

A second model read the whole kernel against its documented promises, with five readers and probe tests run outside the tree. What held up: a step commit is atomic in all three stores, the fence and consumed checks agree, and no crash point between a claim and a commit loses or double-consumes a message. What did not: **fourteen findings broke a documented promise.** All were fixed the same day, one commit each, each with a test that failed first. The worst:

- **A refused commit re-ran the turn forever.** If a store refused a commit, the turn crashed without writing a marker, so the next claim recorded it as a lost lease, which never counts toward `lifetime.max_retries`. The turn, model calls included, re-ran about once a second with no bound. Fixed: a refused commit is a crash with its marker, and a refusal that can never succeed parks the actor at once.
- **A re-hosted actor could hold its lease for ever.** A Go tool that never returned kept its actor pinned to the node, and the node kept renewing the lease. No other node could take it; it needed a person. Fixed: a waiting re-host is never renewed.
- **Postgres turned "limit 0" into 1,000 on ledger reads.** In memory and SQLite, 0 means "all". Forking a session with more than 1,000 spawned children silently lost the children after the thousandth. The suite never wrote more than 1,000 rows. Fixed, with a scenario above 1,000.
- **A purge could lose a session's spend record.** The session row was deleted first and the spend recorded after; if that failed, the retry could never find the session. A session's cost is meant to survive its purge. Fixed: a purge records the spend first.
- **An operator scope read whole ledgers.** A caller with `admin:read` but not `sessions:read_all` could read full ledgers through the compare and archive-file routes. Fixed: both check the caller.
- **Postgres numbered send ids and stored "opaque" state differently** from the other two stores (so ids and bytes differed between stores), and `ClaimDue` did not order unfinished turns first as documented. All three stores now agree, and the contract suite asserts it.

The review also noted that the rule "tests never sleep" was broken and not enforced, and listed rules claimed in the project notes that no check enforced.

### Found by stress runs

- **A message stored for an actor that had already closed was never consumed (4 to 7 Nov).** The stress driver reads the open actors, then posts; an actor terminated from inside a turn could close in between. The store accepted the message, a closed actor is never claimed, and nothing consumed it. It showed as "posted message consumed 0 times" in the nightly: on 11 October, 18 October and 5 November. Found by reproducing it three runs in ten with a 3 ms delay; 20 of 20 passed after the fix. Now the store refuses it with `conflict.agent_closed`, and a closing actor too (7 Nov). Closing the loop: every message is now consumed by a turn, refused to its caller, or bounced to its sender.
- **A terminate to a closing or closed child left its live descendants open** on SQLite and Postgres (seed 2076800113, November). Product defect, fixed.
- **A turn could run past a cap's stop (seed 37, 7 Oct; fixed 12 to 13 Nov).** A commit that crosses a cost stop parks the actor, but the turn only learned of it by reading the actor back. If that read failed, the next commit was accepted on the parked actor. The cause is inferred from the code, not shown from the log, and the report says so.
- **Not product bugs, but traced:** six of the stores' stress failures were the harness folding two close positions into one (a false alarm, corrected, then each of the six replayed and confirmed). One was a wrong lease rule in the harness, one a test driver returning before goroutines had run, one an embedded Postgres whose data directory vanished under it. Every failed seed in the kept logs now has a cause.
- **1,322 data-race reports** across 41 job logs were all one race in the contract suite's id counter, fixed on 5 November. No report had a frame of product code.

### Found by mutation testing

- **A cap's stop could be spent past (13 Nov).** The triage of `runtime/turn.go` found the same hole by reading a surviving mutant. A scratch test showed a stop at 100 ending with 300 spent. Fixed two ways: a failed read now cuts the turn as the stop would, and every store refuses a further commit in the cut epoch (`conflict.cap_stopped`).
- **A negative `lifetime.maxRetries` meant "never park".** A turn that crashed every time would be retried for ever. Now refused as `req.invalid.params` at the bundle's manifest and at the settings; 0 still takes the default of three.
- **Rules the stores did not agree on (15 Nov, open, P3).** Four places where memory, SQLite and Postgres differ: Postgres keeps a parent id on a root instance; two messages with an empty id are two rows in memory and one elsewhere; an outbox row with no body stores a blob on Postgres only; and memory hands back a mutable pointer to an archived outcome. None is reachable by a caller today, by reading. Each is written as a contract case that was red on one store.
- **A step can be committed with no live claim (15 Nov, open, P3).** A commit at epoch 0 on an actor no node has claimed is accepted by all three stores. Nothing in the runtime commits without a claim, so nothing reaches it, but the store should refuse it.
- **The mutation sweep itself had failed every scheduled run for five weeks,** each time on the runner and not on a result, and nobody was told.

### Found by the quality program: held calls, the wake path, reading while writing

Held calls are tool calls that wait for an answer, such as an approval. The wake path is how a message becomes a turn.

| Date | What was wrong | Harm | Fix |
|---|---|---|---|
| 10 Nov | A call's timeout still fired after the call was answered | An empty turn each time | A step may withdraw a message it sent itself |
| 10 Nov | A join on children named by name never completed | A two-minute wait on every such request | Each child resolved once, at the join |
| 12 Nov | The waiting-calls list and who may answer disagreed in 37 of 112 combinations | An operator shown nothing it could answer; callers shown calls they could not answer | One rule for both |
| 13 Nov | An answer accepted between the decision and the final step, then a crash, decided the call again | An approved call ran twice | Fixed |
| 13 Nov | A resumed turn with no answer in its batch took the call as decided | A call needing approval ran unapproved | Fixed |
| 13 Nov | A park left a decided call in state with its timeout withdrawn | A fresh approval ran a destructive call again, or the actor passed every turn for ever | Fixed |
| 13 Nov | A park at the retry limit consumed the answering turn's messages | An answer lost; a call that could never time out | Fixed |
| 11 Nov | A message due between the wake policy's read and the idle window sat behind the lease for the whole window | A late turn | The policy is asked again |
| 12 Nov | A node whose store watch ended never opened another | Messages for an idle actor waited up to five minutes for the rest of the node's life, with nothing logged | The watch reopens with backoff; residents also re-read at each poll |
| 12 Nov | With renewals failing for a lease's time, another node claimed while the first still ran its turn | A tool call with an outside effect could run twice | A margin rule for giving the lease up; the first rule was too weak and the walk found it the same day |
| 13 Nov | A live tail carried on silently when its stream lacked what was asked | A live view missing messages; the record intact | The tail heals from the durable record |
| 13 Nov | An Agent2Agent stream and push config began at the end of the task's cache | An event lost to a trim; after a restart the task's whole history pushed again as new events | Both start from the task's read |
| 13 Nov | A ledger follow paged `step` frames by commit time | A step skipped for good | One cursor, a position |

### Found by the load test (16 November)

These are the product's, found on a runner, not a laptop.

- **A message due "now" was not due.** An instant reached Postgres procedures two ways, one cut to the microsecond by the driver and one rounded by the database. On a node whose clock has nanoseconds (every Linux node; a Mac's has none, which is why no local run showed it), about half of the posts raised no wake notice. An idle actor then waited out its idle hold, up to five minutes. 113 to 182 of 4,000 turns per run got no reply in two minutes. Nothing was lost. Fixed in the store; the next run was clean, 4,000 of 4,000 replies in each of four runs.
- **A node's own streaming tails starved its Redis connection pool.** Each open tail held a connection in a blocking read. With more tails than connections, every command (a turn's streamed text, delivery, the breaker) waited behind blocked readers. At 200 sessions and 240 tails, a turn's mean was 3.1 s while the model call itself answered in 200 ms. Any node with more open tails than ten per CPU had it, with any model provider. Fixed.
- **A work watch dropped notices when its 256-slot buffer was full.** A runner lost 72, then 251, of 4,200 on one node, silently, on all three stores. A dropped notice left a message unanswered for the actor's idle hold. Fixed: the buffer is a set that never blocks and never drops, merging duplicate notices and counting them.
- **Every turn start on a node waited at one lock.** The lock was held across a store call; on an 8-core runner the wait was 847 ms a turn against a 786 ms mean turn. Reworked. What it left, a pause once per poll interval, is open (below, as the poll's pause).
- **A client's write whose reply was lost was applied twice on retry.** A plain post, a post to a partition (which under the default mode could open a second session), an open without an external id, and the shipped client's own "retryable" flag all did it. What a caller saw: a 5xx or dropped connection for a message that had landed, then a second message and a second turn on the retry. The stores were not the cause; a write with no key cannot be told from a new one. Fixed with keyed posts.

### Found by measuring: slow in a way that grew with the database

These produced right answers slowly. Honest numbers:

- **A query plan made while a table was tiny, then kept (13 to 17 Nov).** A connection that prepared a statement while a table held a page or two kept its plan, a sequential scan, as the table grew. The pump's read took 0.40 to 0.46 s at 100,000 bodies (0.9 ms fresh); a heal 100 rows behind took 2.5 to 3 minutes; an actor-ledger page took 4 minutes. The first cure (13 Nov) put `enable_seqscan = off` on the 49 procedures that read tables, guarded by a test. The second (17 Nov) fixed the statements the planes and modules send from Go (176 measured in `docs/2026-11-17-report-k23-cure.md`), by using pgx's `cache_describe` mode. It cost about 0.014 ms a statement, and no round trip was added.
- **The poll scanned every actor (17 Nov).** Measured at 20,000 actors: 45.6 ms per poll on Postgres, and on SQLite about 65% of every second spent holding the store's one lock for the poll alone. In a later before-and-after run on the same machine, a poll with nothing due fell from 28.4 ms to 0.68 ms at 20,000 actors, and a poll reads 35 to 47 agent rows instead of all 2,000.
- **A conversation-compaction request answered with no summary was asked again every turn (19 Nov).** A paid model call per turn, about 0.93 fold requests a turn over 200 turns. Fixed with a back-off, then by folding without the model.
- **The ledger page for a long actor (19 Nov).** With no or old statistics the planner made a nested loop that re-read a join per returned row: 1.43 s at 8,000 rows and 18.9 s at 30,000 rows, on every turn of that actor. Fixed.
- **The Sessions view was partial past 1,000 sessions (18 Nov).** Counts were silently capped. Now every count is exact. At 100,000 sessions it first cost 311 ms on Postgres and 724 ms on SQLite; two follow-ups brought it to its targets.
- **A fork's statements, a poll, a crashed-turn backoff:** similar fixes, with figures in the reports.

### Found in CI and the test machinery

- **The nightly stress job reported green over failed tests (found 13 Nov).** Its steps piped `go test` to `tee` under a shell without `pipefail`, so the step took `tee`'s exit status. A real failure (`TestStressConcurrent` on the memory store) went unreported for an unknown time. Fixed, followed by an audit of every workflow and script in four repositories against eight families of the same hole. The audit found, among others: a manual mutation run naming an unlisted file mutated nothing and passed; a rerun script that swallowed its own failure; a formatting check that hid a failing `git ls-files`; and a file-list limit at 3,000 paths that was not said.
- **A test that skipped on every run (found 14 Nov).** `TestDeclarationsTypecheck`, which typechecks a bundle against the declarations the node serves, skipped whenever the declarations were not built, and no CI job built them. It had been skipping unseen. Found by reading every skip in the tests. CI now builds them first, and a test fails, not skips, when CI says a tool should be there.
- **Outbound webhooks could deliver nothing (17 Nov).** A catch-up added that day opened a cursor before the kernel wrote the session; a pass in the gap got `not_found.session`, which the cursor treated as a store failure and held back by a poll. On a test node whose clock never moves, the hold never ended. On a real node it delayed a session's first entry by up to 5 seconds. This was caught by a CI run on `main` and not called a flake: the log was read first, and 150 local runs at one and two CPUs had not shown it. Fixed the same day.
- **Test flakes with a product cause.** The flake log separates "a test that acted too early" from product bugs it exposed. Two came from the separate workspace engine: two processes started at once were given the same id, and a stop that closed on an already-spent context left processes dead but recorded as running. Both fixed with tests that fail without the fix. One plan-test failure showed that a query plan depends on the size of an unrelated table by about two cost units; that test now holds autovacuum off.
- **A 48-hour count of lost CI jobs:** 20 spot evictions in 421 launches (4.8%), costing 15 jobs. Not a defect in Arbor, but recorded so a timeout on a slow runner is not sent round as a flake.
- Smaller: conflict markers reached `main` once in a document (a check now fails a pull request that carries one); embedded database servers left running by tests that died are now stopped by the next run.

### Found by early verification and the first field test (October)

- **Eval verification (22 Oct):** six bugs fixed. A trial's delivery span lacked its session kind, so a metric would have counted trials. A mock tool ignored the scenario. A `to: "all"` check passed on a fact published to no one. A cell that failed after its trial started left the trial running. A typo in an `arbor eval` flag exited 1, the same code as a failed check, so CI could not tell a typo from a regression. A fork refusal showed as a bare error in the console.
- **The field report (28 Oct):** a developer took the binary to a second machine and built a support-handoff app for two days. Eighteen findings came back, each restated against the code: for example, a foreign server on the dev port reported as `internal`; two different refusals for one version bump; `arbor admin nosuch` printing help and exiting 0; no command to join an existing session; and one person unable to act as two participants on a development node. The report records which were confirmed by reading the code and which were guesses until a test showed them.
- **The defect pass (4 to 7 Nov):** the speaker rule (a turn no posted message woke ran as the creator's seat), an end that missed a root opened after its read, `receiving` meaning "holds a live lease", owed lifecycle calls with no cursor, and more. Each fixed with a test, or closed with its reason.

## Known open issues

From `docs/bugs.md` as last changed on 22 November: 0 open P1, 17 open P2, 46 open P3 (the bug list above says how this cut differs from the 20 November one). These are the ones a user could notice or an operator should know. Each is open.

| Topic | Priority | What | Where it stands |
|---|---|---|---|
| The poll's pause | P2 | Once per poll interval, no turn starts on a node from the poll's lock to its unlock. After the 17 November change to the poll, at 20,200 actors the pause was 15.6 ms at the median and 156.8 ms at p99, on a loaded development machine (`docs/bugs.md`). A later series put the pause down to a saturated machine, not to the poll. | Parked; one measurement with the database on its own machine is the only next step |
| A heal far behind | P2 | After `ANALYZE`, a heal far behind on a long actor walks the whole outbox primary key: 32.8 ms at 100,000 rows, where the kept plan took less. | Open; three rewrites of the page tried on 17 November, none kept; the cure left is a loop per actor, not built |
| Cost of a turn | P2 | What a turn costs: a one-message, one-reply turn on Postgres is 26.7 round trips and 185 statements, about 21 ms of server time. | Open; two items built on a branch |
| A lost wake notice | P2 | An idle resident could read its inbox each poll, which would bound a lost wake notice by the poll, not the idle hold (up to five minutes). | Open; measured on 19 November |
| Running-sessions page | P2 | On SQLite with no statistics, a partition's page of running sessions reads the whole app's sessions: 13.3 ms at 40,000 sessions against 0.06 to 0.13 ms with statistics. | Cure on a branch; open on `main` |
| Retried writes | P2 | Several more retried writes were refused although the first attempt landed. | Rows 1 to 5 built on a branch; open on `main` |

Closed since 20 November, so no longer in the table: the context cache's growing write was measured and neither design was worth building (each moves a 1,000-turn turn by about 2% at most; closed 20 November). A partition post's open now takes its own steps cap (built 20 November). SQLite now has plan tests for the ledger reads past a head (`TestTheLedgerReadPastAHeadScansNoTable`, closed 20 November; what those plans read past the rows they return is a new P2 row).

Decided on 21 November: Agent2Agent requires an identified caller. A2A has no way to carry an anonymous caller's session between requests, so Arbor doesn't accept anonymous A2A callers; one is refused at the door with the reason. For a public, anonymous surface, use an http ingress that admits the anonymous audience.

Open design questions and gaps that are not yet scheduled include: the stores' four disagreements and the epoch-0 commit above (both still open P3 rows); and a place computed for actors more than 64 parents deep depends on list order (found 15 Nov, P3). Two items this page listed on 20 November are closed. A streamed text append inside the model call's span, which lets a slow stream store look like a slow model, was measured on 21 November against a local model and a local Valkey at 0.55 to 1.00% of the call, so nothing was built; a slower store would raise the share. A reader agent's wiretap hearing a list-audience message once per entry was accepted on 20 November as latent, since only a root relay can have readers.

### What is next

The stability milestone has eight conditions; as of 20 November three are done and five are open:

1. No open P1 or P2 in the kernel. **Open.** Not yet.
2. Every failing stress seed traced and fixed. **Done.**
3. Two weeks of nightly stress green on every store. **Open;** the count starts after the store freeze and restarts on any store change.
4. The data-race series explained. **Done.**
5. No test fails only on a slow machine. **Open.**
6. The mutation sweep's survivors on the ten kernel files triaged and weak tests strengthened. **Done.**
7. The load test at 200 sessions on two Postgres nodes within its targets. **Open;** the ceilings are not set yet.
8. A 24-hour soak on a long-running test node, clean: no leak, no stuck session. **Open,** not started.

- [Benchmarks](https://arbor.getloa.app/en/../benchmarks/): The load and latency figures behind several fixes below.
- [Reference](https://arbor.getloa.app/en/../reference/): The commands, settings and guarantees the checks protect.
- [Home](https://arbor.getloa.app/en/../): Back to what Arbor is.

Loa Arbor. Licensed under the Functional Source License 1.1, with MIT as its future licence (FSL-1.1-MIT): each release becomes MIT two years after it comes out. Coming soon: the download opens when the repository goes public.
