Benchmarks
Every number comes with its setup, its date and what it found.
This is the report behind the four numbers on the home page, plus ten people in one session, and the load, real-Redis, soak, stress and race testing. Where a test found a limit, you will see the limit, the fix and the rerun. What has not been measured is listed at the end.
What is measured, and why
An agent’s time is spent waiting for a model. A model call takes hundreds of milliseconds or more, and a tool call can take longer. So the runtime’s job is not to be fast in the way a database is fast. Its job is to add as little as possible to that wait, and to keep adding little when many conversations run at once.
Arbor saves every step of a conversation before the next one starts. Three words you will meet below:
- A step is the unit that is saved. One step is one commit: one database transaction that records what the actor consumed, what it did and what it sent.
- A turn is one activation of an actor. It handles a batch of messages and ends when it has nothing more to do for them. A turn is usually several steps, and an answer from a model takes two (the model’s response, then the final reply).
- A session is one conversation. Its actors share one ordered record, the ledger.
So the questions are: how long does a commit take, how many commits a second can the store take, and what happens to turns when hundreds of sessions run on one node or two? Latency numbers use p50 (the median: half of the measurements were faster) and p99 (the slowest 1% start here).
Two warnings apply to everything on this page.
- Almost every load number uses a scripted model. It answers at once, or after a fixed 200 ms, with a one-line answer. Nothing calls a network and nothing is spent. That isolates what the runtime costs, and it is also why these are not predictions for an app with real models and long answers.
- The machines are named beside each number, and numbers from different machines are never compared here. Most early runs were on one laptop that also ran the database and the load driver. Later runs were on rented cloud machines. The sources themselves refuse to read a ceiling across two machine types, and so does this page.
The commit test
This is the test behind the home page’s numbers. It measures the commit procedure alone, not a whole node.
| Commit test | Commits per second | Median commit | Slowest 1% |
|---|---|---|---|
| 1 session, 1 actor | 1,196 | 0.60 ms | 0.78 ms |
| 1 session, 4 actors | 1,437 | 2.36 ms | 5.01 ms |
| 1 session, 16 actors | 1,324 | 11.56 ms | 18.55 ms |
| 16 sessions, 1 actor each | 10,414 | 1.18 ms | 1.58 ms |
Measured on 5 October 2026 against Postgres 17.5 on an Apple M5 Pro laptop (18 cores, 64 GB, Go 1.27.1). The test is TestCommitCeiling in store/postgres/commit_ceiling_test.go.
How it was set up:
- Each committing actor had its own database connection.
- Each step wrote about 2 kB: a recorded entry, one outbox entry (a message going out) and one message to a sibling actor.
- Steps came in turns of four. The first step consumes what the claim handed it, then the actor claims again.
- Each case ran for 20 seconds.
%% The commit path of one step, as the commit test drives it
flowchart LR
claim["Actor claims<br/>its due messages"] --> turn["A turn of four steps"]
turn --> lock["Take the session's row<br/>commits of a session queue here"]
lock --> commit["One transaction:<br/>consumed, did, sent"]
commit --> claim
What it says:
- A single actor commits in about 0.6 ms and about 1,200 times a second. The slowest 1% take under 0.8 ms.
- One session tops out at about 1,300 to 1,500 commits a second, however many of its actors commit. The commits of one session queue on the session’s row in the database. With 16 actors that is about 12 ms each, about 80 commits a second per actor.
- Separate sessions do not share that limit. Sixteen sessions reached 10,414 commits a second.
- The queue is deliberate. Taking the session’s row first is what makes the session’s spend add up exactly inside the commit. This cost was accepted on 4 November 2026. It matters only once one session’s actors together commit far faster than their model calls let them.
This is the commit procedure on a laptop. It is not a load test of a whole node, and it says nothing about a managed Postgres with replicas, where each commit is slower.
Whole-node load tests
The commit test leaves out everything else a node does: accepting the request, claiming the actor, running the agent loop, writing to the live stream, delivering the reply. The load test runs all of it.
How the load test works
arbor soak --load opens many sessions at once, drives each through a fixed number of turns, and reports throughput and latency. make load runs it against nodes booted inside the test process. The numbers come from two clocks, and the report says which is which:
- The driver’s clock gives turn latency: from sending the message to receiving the reply frame on the driver’s own live stream. Quantiles are exact.
- The node’s own metrics (
GET /v1/admin/metrics, read before and after each phase) give turns, steps and commits per second, and the latency of each step and each commit. These quantiles are interpolated inside histogram buckets, so each is good to its bucket. - Followers are extra passive live streams on the first sessions. Their lag is the time from the start of a row’s commit to the frame arriving, so it holds the commit, the push to the stream and the network write.
The matrix of the first reports:
- Sessions: 10, 50 and 200 opened at once (64 connecting at a time), each driven through 20 turns, a turn posted only after the last one’s reply.
- Model: scripted, answering at once, or after 200 ms.
- Repeats: every row ran twice on nodes booted fresh. The tables show both runs.
- App: one agent answering in one model call.
- Configurations (2026-11-09): one node on SQLite (a file, WAL mode,
synchronous NORMAL, one connection); one node on embedded Postgres 17.5 at its default configuration (fsync and synchronous commit on); two nodes on one Postgres database. - Redis stand-in (2026-11-09): miniredis, a Go imitation of Redis that runs inside the test, because the live streams need a Redis-like store. It was not real Redis. See below for why that mattered.
- Machine: an Apple M5 Pro, 64 GB, Go 1.27.1,
GOMAXPROCS18. The nodes, the driver and the database shared it.
Reading a row: “turns/s” is the node’s own count of finished turns over the drive phase’s wall time.
The first run, 2026-11-09, and the limit it found
First run, model answering at once. Turn latency is the driver’s, from run 1 only. Both runs’ throughput is shown.
| Configuration | Sessions | Turns/s (run 1 / run 2) | Turn p50 / p99 (run 1) |
|---|---|---|---|
| SQLite | 10 | 152 / 158 | 59.9 / 125.5 ms |
| SQLite | 50 | 34 / 37 | 1.44 / 3.08 s |
| SQLite | 200 | 7 / 7 | 27.1 / 62.3 s |
| Postgres | 10 | 539 / 578 | 17.3 / 29.2 ms |
| Postgres | 50 | 732 / 748 | 59.2 / 110.2 ms |
| Postgres | 200 | 181 / 172 | 1.05 / 1.57 s |
| Two nodes, one Postgres | 10 | 581 / 591 | 15.9 / 30.5 ms |
| Two nodes, one Postgres | 50 | 585 / 562 | 72.6 / 153.2 ms |
| Two nodes, one Postgres | 200 | 173 / 168 | 954 ms / 2.06 s |
On Postgres at 50 sessions the node committed about 1,464 to 1,496 times a second. This is where the architecture page’s “about 730 turns a second” comes from: the mean of 732 and 748 is 740, and the later rerun’s mean is 731.
Two things were wrong, and neither was the commit.
- Throughput fell as sessions rose. SQLite went from 152 turns a second at 10 sessions to 7 at 200. Postgres peaked at 50 sessions and fell to about 175 at 200. Meanwhile the median commit stayed at 2 to 4 ms on Postgres at 200 sessions, so the commit was not what stopped it.
- What fell was one read. Before each model call the agent loop reads the actor’s own ledger, at least twice a turn, to rebuild its history and find an unfinished turn. That read’s cost grew with every row in the whole database, not with the actor’s own rows.
- On SQLite the query plan materialized two views (
turn_step_readandoutbox_read), so every read scanned every step and outbox row of every session. A profile at 50 sessions put 79% of the node’s CPU in it. - On Postgres a join to the processed messages could not use its partial index, because the join did not say
processed_pos > 0. It scanned the wholemessagetable once per row of the actor’s ledger: 3.2 ms of one 3.5 ms read at 50 sessions (EXPLAIN ANALYZE). It was the active query in 82% ofpg_stat_activitysamples at 200 sessions, with the Postgres server using about ten cores and the node two and a half.
- On SQLite the query plan materialized two views (
The fix, and the rerun
The fix is the commit “store: an actor’s ledger read seeks its own rows on both SQL stores” (docs/model.md §2.4, §2.5). Both stores now read the actor’s ledger by seeks on its own rows.
The rerun on the same machine and matrix is docs/load/2026-11-09b.md. It was run with extra care: each of the 36 runs was started alone after 15 quiet seconds, with ps sampled every second, and nothing else ran during any of them.
Model answering at once, after the fix:
| Configuration | Sessions | Turns/s (run 1 / run 2) | Turn p50 / p99 (run 1) |
|---|---|---|---|
| SQLite | 10 | 345 / 354 | 28.6 / 43.9 ms |
| SQLite | 50 | 348 / 335 | 143 / 195 ms |
| SQLite | 200 | 297 / 299 | 647 / 959 ms |
| Postgres | 10 | 558 / 540 | 17.0 / 30.9 ms |
| Postgres | 50 | 765 / 697 | 61.3 / 93.6 ms |
| Postgres | 200 | 182 / 180 | 1.05 / 1.65 s |
| Two nodes, one Postgres | 10 | 588 / 569 | 16.0 / 28.3 ms |
| Two nodes, one Postgres | 50 | 760 / 646 | 50.4 / 125.3 ms |
| Two nodes, one Postgres | 200 | 204 / 217 | 929 ms / 1.89 s |
%% SQLite turns per second by sessions, run 1: first run (bars) and after the fix (line)
xychart-beta
title "SQLite, run 1: bars first run, line after the fix"
x-axis "Sessions" ["10", "50", "200"]
y-axis "Turns per second" 0 --> 400
bar [152, 34, 7]
line [345, 348, 297]
%% Postgres turns per second by sessions, run 1: first run (bars) and after the fix (line)
xychart-beta
title "Postgres, run 1: bars first run, line after the fix"
x-axis "Sessions" ["10", "50", "200"]
y-axis "Turns per second" 0 --> 800
bar [539, 732, 181]
line [558, 765, 182]
Model answering after 200 ms, before and after the fix (turns per second, run 1 / run 2; turn p50 / p99 from run 1):
| Configuration | Sessions | Before: turns/s | Before: turn p50 / p99 | After: turns/s | After: turn p50 / p99 |
|---|---|---|---|---|---|
| SQLite | 50 | 36 / 36 | 1.29 / 2.97 s | 215 / 215 | 216 / 270 ms |
| SQLite | 200 | 7 / 7 | 26.3 / 60.7 s | 294 / 293 | 610 ms / 1.01 s |
| Postgres | 50 | 201 / 202 | 241 / 275 ms | 205 / 207 | 236 / 266 ms |
| Postgres | 200 | 117 / 133 | 1.34 / 2.35 s | 134 / 136 | 1.21 / 2.11 s |
| Two nodes, one Postgres | 50 | 206 / 211 | 233 / 272 ms | 205 / 207 | 236 / 273 ms |
| Two nodes, one Postgres | 200 | 218 / 254 | 366 ms / 1.34 s | 166 / 173 | 838 ms / 2.33 s |
What changed:
- SQLite at 200 sessions went from 7 to about 298 turns a second with the model answering at once. Its median commit went from 142.6 ms to 3.6 ms, and its slowest-1% commit from 489 ms to 14.6 ms (run 1 of each). With the 200 ms model it keeps up at 50 sessions: 215 turns a second against an ideal 220.
- Postgres did not move, and the reason was the interesting part. After the fix, at 200 sessions the Postgres server ran at half a core to three (it had been about ten), and the node at one to four of its eighteen cores. Neither was busy. A mutex profile of the node’s process put 60% of the contended wait in the test harness’s clock bridge for miniredis and 37% in miniredis’s own dispatch and blocking reads. So the Postgres ceiling at 200 sessions was the stand-in for Redis, not Arbor.
- Two nodes did not raise that ceiling. The near-twofold gain of the first run with the 200 ms model at 200 sessions (a mean of 236 against 125 on one node) did not reproduce in the guarded rerun (170 against 135). The docs say the shared miniredis was the likely limit of both and that it was not profiled further. The rows above show the same: 218 and 254 before, 166 and 173 after.
Where this stands: the architecture page’s “ceiling today” paragraph is this run. It is correct for what it measured, with one caveat the later sections give: the stand-in for Redis turned out to matter more than that paragraph says.
The runner runs, 2026-11-16
The test was then run on GitHub Actions runners, as the baseline for a “stability” milestone. Its targets, set on 15 November 2026, judge the two-node configuration at 200 sessions of 20 turns, both models, two runs each:
- (a) No timeout and no error. A run that has one fails.
- (b) Every session finished: every turn posted, consumed and answered, and no work left in any session’s ledger.
- (c) The two runs of a row agree: their turns a second are within 20% of the larger.
- (d) Ceilings on the turn’s p99 and the followers’ lag p99. Not set. They are to be set from a run on a real Redis-compatible server, not on miniredis.
At the pinned source, (a), (b) and (c) are met and (d) is not set.
Nine reports were written that day. Some ran with miniredis and some with Valkey, a Redis-compatible server, as the stream store. They read like a bug hunt, and they are worth reading in order.
| Report | Machine and stream store | Result | What it found |
|---|---|---|---|
2026-11-16 |
m8g.2xlarge, miniredis | Failed (a) and (b) | 113 to 182 of 4,000 turns a run got no reply within the driver’s two minutes. The first reading blamed the wake path’s dropped notices. |
2026-11-16b |
m8g.2xlarge, miniredis | Failed the same way | The suspicion was wrong. The Postgres watch dropped nothing, but the server sent only about 2,190 notifications for 4,000 posts. 123, 105, 160 and 138 timeouts. |
2026-11-16c |
m8g.2xlarge, miniredis | Passed (a), (b), (c) | Cure for the missing notifications. 4,200 sent and 4,200 taken a run. |
2026-11-16d |
m8g.2xlarge, real Valkey 7.2.15 | Passed | The turn’s time was not the harness’s. It was a pool of connections the node itself shared. |
2026-11-16e |
m7g.2xlarge, real Valkey 7.2.15 | Passed | After that cure, the node’s own turn fell to about 50 ms. The driver still saw about 800 ms. |
2026-11-16f to i |
m7g.2xlarge, real Valkey 7.2.15 | Passed, four draws | The remaining wait was one lock on the node. |
The missing notifications
When a message is saved and its time has come, the Postgres procedure insert_message raises a notification so a node wakes the actor at once. In the failing runs, about half of the posts raised none.
- The cause: the procedure notifies only when the message’s delivery time is at or before the call’s “now”. A post is due at that “now”. But the delivery time reaches Postgres in a JSON row, rounded to the microsecond, and the “now” as a parameter, truncated to it. On a clock with nanoseconds, such as the runner’s, an instant with more than half a microsecond past the microsecond read as due one microsecond later, and no notice was raised.
- The harm: an idle actor then waited for the rest of its idle hold. The driver’s two minutes ran out first.
- The finding came from counting. The second run counted notifications at the server, on the node, and in the node’s event window. It showed the node’s counter equal to what the server sent, which cleared the watch, and showed the server sending half as many as posts.
- The fix, and its proof: the third run, 2026-11-16 (c), on commit
31fd117b,TestLoadtook 219 s (the two runs before it, 1,938 s and 2,031 s). Every one of four runs had 4,000 posts and 4,000 replies, no timeout, no error, every session finished. Turns a second rose from about 8 to about 80 (79 and 83 with the model at once, 76 and 81 with 200 ms), simply because no turn waited two minutes any more. - A contract case now pins it:
Inbox/AMessageDueNowRaisesANoticeWhateverItsNanoseconds.
Also found that day: the node’s event window, a queue of recent events, drops its oldest silently under load (it held 1,423 to 2,447 of 4,200 notices in the third run). That is a gap in what a report can read, not in what the node does.
A wrong reading, corrected
The first three reports said the slow turn time (a median of 1.8 to 2.4 s with a model that answers at once, beside a 4 ms commit) was the harness’s, because miniredis is slow under many followers. The fourth report ran the same test against a real Valkey server and says that reading was wrong. The turn median there was 1.9 to 3.3 s too.
Fourth run, real Valkey 7.2.15 (m8g.2xlarge, commit 40d36974, persistence off) |
Turns/s (run 1 / run 2) | Driver’s turn p50 (run 1 / run 2) |
|---|---|---|
| 200 sessions, model at once | 76 / 68 | 1.92 s / 2.71 s |
| 200 sessions, model 200 ms | 57 / 58 | 3.31 s / 3.31 s |
A block profile on a development machine held to 8 cores (200 sessions, the 200 ms model, one run) reproduced the run: a turn’s mean 3,192 ms, the model call’s span 2,306 ms, 58 turns a second. The profile summed delay across all goroutines for 4,000 model calls:
| Where the delay was | Total delay |
|---|---|
| Inside the model call’s span, per call | 2.53 h (2.28 s a call: the whole span) |
| Of that, waiting for a pooled Redis connection | 2.30 h |
| Of that, appending the turn’s streamed text inside the call | 2.02 h |
| Whole process waiting for a pooled connection | 5.05 h |
| Of that, the live tails’ own blocking reads | 1.70 h |
The node had one Redis client with go-redis’s default pool, 10 connections per CPU (80 there). Each open live tail held one in a blocking read, and the run had about 120 tails a node. So every append, get and read waited its turn behind blocked readers. It is the product’s behaviour, not the harness’s: a node with more open tails than ten a core would have it with any model provider.
The cure gives the blocking reads a pool of their own. In the fifth report, on a different machine type (m7g.2xlarge, commit fb7608e7, Valkey 7.2.15), TestLoad took 90.5 s against 299.6 s on the m8g before the cure.
| Fifth run (m7g.2xlarge, real Valkey 7.2.15) | Turns/s (run 1 / run 2) | Driver’s turn p50 | Node’s own turn, mean | Model call’s span, mean |
|---|---|---|---|---|
| 200 sessions, model at once | 232 / 225 | 794 / 823 ms | 52 / 49 ms | 8 ms |
| 200 sessions, model 200 ms | 206 / 197 | 950 / 975 ms | 265 / 287 ms | 207 / 208 ms |
Before the cure on the m8g the same means were: node’s turn 1,992 to 2,542 ms and model-call span 1,195 to 1,747 ms with the model at once, and 3,100 to 3,134 ms and 2,298 to 2,313 ms with the 200 ms model. The report itself says the two machine types differ, so no ceiling is read from the fifth run.
What was left: the driver sees 800 ms, the node’s turn is 50 ms
In the fifth run about 700 ms of a turn, as the driver counts it, was outside the node’s own turn. It was not delivery (follower lag p50 was 13 to 33 ms) and not the post itself (9 to 39 ms at the median). The sixth to ninth reports are four draws to find it, two nodes on one Postgres and one Valkey 7.2.15 on an m7g.2xlarge (8 cores), 200 sessions of 20 turns, the model answering at once, one run each:
| Report | What differed | Driver’s turn, mean | Message committed to first commit of its turn, mean | Node’s turn, mean | Turns/s |
|---|---|---|---|---|---|
2026-11-16f |
each node’s Postgres pool 4 | 824 ms | 788 ms | 65 ms | 215 |
2026-11-16g |
pool 32 | 781 ms | 750 ms | 88 ms | 242 |
2026-11-16h |
pool 32, block, mutex and CPU profiles on | 786 ms | 757 ms | 75 ms | 238 |
2026-11-16i |
pool 32, per-statement counts on; 10, 50 and 200 sessions, both models | see below |
What the draws showed:
- The Postgres pool was not it. With 4 connections, nearly every acquire waited (40,935 of 43,399 on one node, 4.6 ms at the mean). With 32 almost none did (2,230 of 38,481, 0.26 ms). The wait from a message’s commit to its turn’s first commit only went from 788 to 750 ms. Neither Redis pool waited in any draw.
- One mutex was. A block profile (draw 3) counted 3,389 s of waiting on a
sync.Mutexover the run, 99.99% of it at three callers in the node:nextTurn2,137 s,claimOne1,231 s,poll21 s. Over 4,000 turns that is 847 ms a turn, close to the 757 ms the chain table gave. The lock wasclaimMu, which the node held across a store call, so a node started its turns one at a time however many cores it had. Every other mutex of the process waited under a second in sum. - A correction the report makes against itself. Before the profile was read, the explanation was that 200 sessions were queuing for a machine at its limit. That is true of any closed load, so it explained nothing, and the profile ruled it out as the mechanism.
- The gap by load (draw 4, 200 ms and at-once models, mean milliseconds; the per-statement counting was on, so these are not compared with the other draws):
| Model | Sessions | Driver’s turn | Node’s turn | Difference | Turns/s |
|---|---|---|---|---|---|
| 200 ms | 10 | 240 | 232 | 7 | 41 |
| 200 ms | 50 | 274 | 248 | 26 | 177 |
| 200 ms | 200 | 883 | 324 | 559 | 216 |
| at once | 10 | 55 | 44 | 11 | 173 |
| at once | 50 | 226 | 75 | 151 | 215 |
| at once | 200 | 823 | 77 | 746 | 226 |
The gap is absent at 10 sessions and whole at 200, and two nodes turned over 215 to 226 turns a second whatever was asked. Those rows are the “before” of the cure. The report says the cure’s proof is this table again after it.
- What a turn costs in Postgres (16 Nov, m7g.2xlarge, draw 4; the figures are summed at the head of
docs/load/2026-11-16f.md; the server’s own count, 200 sessions, model at once): 26.7 round trips a turn, 185 statements inside them, 21.0 ms of the server’s execution time a turn.commit_stepruns 2.05 times a turn at 5.2 ms (10.7 ms a turn).actor_ledger_pageruns 3.00 times a turn at 1.2 ms. The claim runs 1.61 times at 0.44 ms. A CPU profile of the test process (both nodes and the driver) put the agent loop’s history load at 10%, loading the app’s code bundle at 9 to 11%, JSON decoding at 13% and the garbage collector at 12%. The report says nothing there is a finding yet. - The machine (sampler, 2 s a sample): 80 to 92% of the 8 cores busy during the drives, Postgres using 3.3 to 4.2 cores, the test process 2.4 to 3.0, Valkey 0.2 to 0.4.
- Notices dropped again. In draw 1 one node took 3,949 of the 4,200 notifications the server sent (251 dropped). In the fifth report one node took 4,128 of 4,200 once. Both are the Postgres watch’s 256-notice buffer overflowing. No turn timed out for it. The wake path’s own document says the store’s work watch no longer drops a notice for want of room as of 2026-11-16, the same day as these reports.
- A flaw in the earlier two-node rows. Reading the harness showed both nodes had been given one Postgres pool, not one each, though the notes said one each. With pgx’s default, the earlier two-node rows (
2026-11-16to2026-11-16e) ran on 8 connections between the two nodes. They did not print the pool. From the sixth report on each node has its own and the head prints its size, so the earlier two-node rows are not directly comparable.
What was changed after, and how far it is measured
The lock was made a read-write lock on 2026-11-16: a resident’s re-claim and a linger’s claim take it shared, so a node starts turns side by side, and only the poll takes it exclusively. What remained was the poll’s own section, once a poll interval (one second by default), during which no new turn starts on that node. Until 17 November the poll read every actor that is not closed and has no live lease, so it grew with the database. The claim_due_agents figures from before that change (docs/read-write.md) are 16 to 19 ms at 10,000 actors and 152 to 186 ms at 100,000, nothing due. “Stage two”, moving the poll outside any node-wide lock, is not built.
The table below is from 16 November, before the poll changed. It was measured on a development machine, two nodes on one Postgres, 200 sessions of 300 turns, about 300 polls a size, with the machine at a load average of 20 to 38, so the figures are for a busy machine:
| Actors in the database | Wait for the lock, p50 / p99 | Held, p50 / p99 | Whole pause, p50 / p99 | Held part before the change, p50 |
|---|---|---|---|---|
| 200 | 43.6 / 133.8 ms | 23.9 / 98.4 ms | 71.9 / 179.6 ms | 2.2 ms |
| 2,000 | 27.9 / 131.5 ms | 19.8 / 78.0 ms | 53.4 / 172.3 ms | 6.7 ms |
Before the change the poll waited 360 ms at the median for the turns ahead of it (364 ms at 200 actors). The held part is longer after the change, and why is not shown: no profile of it was taken. At 20,000 actors, after the lock change only, the pause was 102.2 / 290.9 ms.
What changed on 17 November. The poll now reads only the actors that are due (docs/2026-11-17-report-k30-step2.md). A poll with nothing due went from 28.4 ms to 0.68 ms at 20,000 actors. docs/bugs.md gives the pause measured after that, on a loaded development machine (two nodes on Postgres and Valkey, 200 sessions of 300 turns, one run a cell). At 20,200 actors the whole pause was 15.6 ms at the median and 156.8 ms at p99, and at 2,200 actors 22.2 ms at the median. What is left of the pause is the wait to drain claims already in the store and the poll’s decisions, not the scan.
A later series, at a load average of 19 to 36, found that the poll waits for a database connection like every other caller. The pool was full, with 80 to 105 goroutines a node waiting. The doc’s conclusion is that the pause is a symptom of a saturated machine, not a defect of the poll. That is inferred: the database’s side was not measured. The work is parked. Its one next step is a measurement with the database on its own machine.
At the pinned source there is no runner report that reruns the load matrix after the lock change. So the two nodes’ throughput after it is not recorded here.
The stand-in for Redis was bounding the results, and then was removed
The model document (docs/model.md §2.4) says that since 2026-11-17 the load test has no configuration on miniredis. It reports that miniredis bounded the Postgres figures: one node, 200 sessions, 33 turns a second on miniredis against 1,086 on a real Valkey.
The measurement itself is in docs/tasks-history.md, in the entry on the miniredis limit (“Read 2026-11-16”). One node on the embedded Postgres, with one real Valkey 7.2.15 as the stream store and nothing else changed, at 200 sessions at once, LOAD_PG_POOL=32, one run each, on the development machine at a load average of 3 to 5:
| Configuration | Turns a second | A turn as the driver saw it, mean | A turn inside the node, mean | Errors |
|---|---|---|---|---|
| Postgres with miniredis | 33 | 6,035 ms | 5,984 ms | 0 |
| Postgres with Valkey | 1,086 | 179 ms | 12.1 ms | 0 |
| Postgres with Valkey, again | 1,093 | 178 ms | 12.0 ms | 0 |
The stream store is the only difference between the first two rows. The model is the test’s scripted one, so these are Arbor’s own costs, not a model’s. The machine, the database and the driver are one machine, and no report file for these runs is in docs/load/.
So the configurations are now sqlite, postgres-valkey (one node) and two-nodes-valkey, and the Valkey server is one the test starts on loopback, with persistence off.
This is the home page’s number. The home page quotes the 1,086. The architecture page’s older “about 730 turns a second at 50 sessions” is the 2026-11-09 miniredis-era measurement above: correct for what it measured, but bounded by the stand-in, not by Arbor.
Real Redis
Arbor’s live streams and short-lived state sit in a Redis-compatible store, which a deployment can lose without losing a conversation. Until then every test of the Redis store ran against miniredis. The plan for fixing that is docs/2026-11-15-plan-redis-store-on-a-real-server.md. Step 1 was measured and reported on 15 November 2026.
What ran, and where:
- Engine: Valkey 7.2.15, built from the project’s release source (its checksum checked against the project’s published list, and again on the runner), without TLS. The engine is provisional: no engine has been chosen yet.
- Shapes: one real server and a three-primary cluster, both on loopback, started and ended by the test harness (
store/integtest/valkey.go). A machine withoutARBOR_VALKEY_SERVERskips the new tests with the reason. - Suites: the Redis store’s own contract suite and its fault suite, and the trace and series suites of the fleet’s telemetry.
| Suite | Cases | Unchanged on a real server | Through a bridge | Cannot run |
|---|---|---|---|---|
ephemeraltest.Run |
13 | 8 | 5 | 0 |
ephemeraltest.RunFaults |
1 | passes, with the server stopped and started (all three nodes on the cluster) | 0 | |
telemetrytest.RunTraces |
13 | 13 | 0 | 0 |
telemetrytest.RunSeries |
9 | 9 | 0 | 0 |
The results are the same on the single server and the cluster, and no case was edited. The “bridge” exists because the suites use a synthetic clock. Where a case moves that clock past a key’s expiry, the bridge has the real server expire the key at once (PEXPIREAT in the past); where it compares expiry, it reads the server’s own PTTL; where it needs to know a blocking read is parked, it reads the server’s blocked_clients. No case failed on the real server for any reason other than the clock, an ordering hook, or the fault switch. No answer, error or order differed from miniredis in what ran.
The hash write’s Lua script was also exercised on a real server: on a fresh one, after SCRIPT FLUSH, after the server was killed and restarted, and on the cluster across three slots. Forced refusals were READONLY, OOM, a connection cut after the server received the call, CLIENT PAUSE, and MOVED and ASK. Every one left the connection clean.
It also found five things in the store’s behaviour. None was fixed in that report. Two were cured the next day:
- A read that spans two sessions fails on a cluster (
CROSSSLOT) and is called retryable. It falls back to its poll interval. Open at that report. - Every server error reply was reported as “unavailable, retryable”, so a refusal that never succeeds looked like a server that was down. Cured 2026-11-16.
- A down server cost about 1.5 s a call with the client’s defaults (the fault case’s 14 calls took 20.8 s on one server, against 0.03 s with no retry and one dial). Open at that report.
- A cancelled context did not end a call the server was holding: it returned at the client’s read timeout, 5.0 s. Blocking reads cured 2026-11-16 (the store’s part): a slice now runs under a 2.25 s deadline.
- A call whose connection is cut after the server received it is applied and reported unavailable. The hash write is idempotent. The report says an append with an automatic id, retried, would append twice. That was inferred from the code, not run.
Time cost of running all this in CI:
| Development machine (M5, load 2.3 to 3.0) | A runner (m8g.2xlarge, one run) | |
|---|---|---|
| One server ready | 5 to 6 ms | 3 ms |
| The cluster converged | 2.0 to 3.0 s | 2.0 s |
| All the new tests, no race detector | about 48 s | 51 s |
All the new tests, with -race |
about 55 s | 54 s of tests |
What the report did not do: replicas and failover, TLS and credentials, the -LOADING and CLUSTERDOWN errors. At the time, the node’s own client could not talk to a cluster of more than one shard, and a cluster created with TLS needed a client change. Those were listed as the plan’s next step, and no later report in the docs at this commit closes them.
Soak: driving real conversations
arbor soak is a different kind of test. It is a client command that drives scripted conversations through a running node’s API and reports what happened. It replaced shell scripts that found about twenty real bugs in the first two days of live runs: crashed and parked turns, runaway tool loops, cache-defeating prompt churn and budget surprises.
arbor --server https://127.0.0.1:8787 --token "$TOKEN" soak --app seed-memory
arbor ... soak --app care --root front --params '{"customer_id":"C-1001"}' --sessions 3 --live
- What it sends: one user message per line of a file (
--turns), or a built-in 47-turn customer-care conversation with memory, research that spawns children, a long paste and recall questions late in the run. - What it measures: per session, the reply latency (p50 and max), spend against the budget, every model and tool call per turn, ledger event counts by kind, and prefix stability: how many leading bytes of the model request each call shares with the previous one. An append-only history scores just under 1. A rewrite scores low and counts as a break below 0.9, because it throws away the model provider’s cached prefix.
- What it flags on its own: a crashed or parked turn, a guardrail halt, a turn left unanswered, a reply slower than 60 s, a turn with more than 10 model calls, a turn that cost more than three times the median, and instructions that changed between consecutive calls.
- A guard against spending money by accident: without
--live, the soak refuses when it can tell the model is a paid provider (req.live_required, exit 2). A local model such as Ollama on loopback, and thescriptedtest provider, pass.
No soak report appears in the docs at this commit with figures to quote, and the built-in conversation is 47 turns, so it is not an endurance test. Its job is to find bugs, not to produce a number.
Ten people in one session
The recording at the top of this page is a different test from the load runs: one session, a real model, and people talking at once. A script starts one node on a SQLite file, with its model claude-sonnet-5-5 through the Cloudflare AI Gateway and a $10 budget on the app. A driver then posts as 10 participants, 8 customers and 2 staff, each with their own token, role and timer.
| Measure | Result |
|---|---|
| People in the session | 10 (8 customers, 2 staff) |
| Actors | 40, all live at once at the peak: 27 agents and 13 plain actors |
| Posts answered | 45 of 45 |
| Turns, commits, model calls | 482, 599, 117 |
| Tokens | 169,553 in, 10,324 out |
| A model turn | 1.7 s at p50, 4.6 s at p95 |
| A post’s reply | 5.2 s at p50, 8.1 s at p95 |
| Crashes, parks, errors | none |
| Cost | $0.44, priced by Arbor’s meter |
The run used no prompt caching: Arbor’s Anthropic provider does not yet mark any prompt for caching, an open defect.
A turn that called the model took as long as the model call inside it: 1.7 s at the median for both. The turns that called no model, 369 of 482, took close to no time. A reply takes longer than one call because one post can take several model turns in a row, as agents hand work to each other. It is one run, on one machine, so read it as what one session of this size looks like, not as a ceiling.
The same session on a slower model
We ran this demo first on an open model, Qwen 3.8 27B on Workers AI, with everything else the same: the app, the driver, the 10 people and the 40 actors, on the same build and the same day. Every post was answered and nothing failed, but replies were slow: 36 s at the median and 130 s at p95. With that many people and agents in one session, we had to ask whether Arbor was adding the time.
It wasn’t. On both models, a turn that called the model took the same time as the model call inside it, and the turns that called no model took close to none.
| p50 / p95 | Qwen 3.8 27B on Workers AI | Claude Sonnet 5.5 |
|---|---|---|
| A model call | 5.4 s / 35.2 s | 1.7 s / 4.2 s |
| A turn that called the model | 5.4 s / 35.2 s | 1.7 s / 4.6 s |
| A post’s reply | 36.0 s / 130.1 s | 5.2 s / 8.1 s |
| Turns that called no model | 634 of 749 | 369 of 482 |
So a reply’s time is the model’s time, multiplied by how many calls the reply waits on in a row. Arbor gives you two levers for it. Compaction keeps each call’s input short: what leaves an agent’s window is folded into a summary, which a cheaper model can write (.context() in the Reference). The topology is how many agents a post passes through, one after another, before someone answers, and that is your design.
In the same days we found a defect in how Arbor timed model calls, though it did not touch these runs, whose longest call was 86 s. A model call that sent nothing back for 2 minutes was taken for a stalled turn, then cancelled and made again, so it was paid for twice. That happened with a call that does not stream, such as a typed reply or a child agent’s call. Any call over 3 minutes was cut the same way, streaming or not. Since v0.2.5, a call in flight counts as activity, the 3-minute limit applies only to a stream that has gone quiet, and each call has a deadline of its own, 10 minutes.
Everything else that was measured
The cost of keeping the outbox’s indexes tidy
Each reply an actor sends goes into the outbox table, and a pump delivers it to the live stream and marks it delivered. Postgres keeps two small partial indexes (indexes over only the undelivered rows) for the pump to read. Under churn those indexes grow with dead entries, and the pump’s read can flip to a plan that reads the whole delivered history.
A measurement report on 15 November 2026 asked how fast that happens and whether a periodic rebuild would cure it. The machine was an M5 Pro with embedded Postgres 17.5 and Go 1.27.1, at a load average of 2.2 to 7.1. The tests are under the measure build tag in tests/load/ and run in no per-push job.
The report says two of its six stop conditions were met. The one that matters is that ordinary traffic trips the rebuild rule again within the 30 minutes the plan allows:
- A rebuild cures only briefly. After rebuilding both indexes, the rule trips again at 3,000 rows passed. At the load test’s Postgres rates of the time (765 turns a second at 50 sessions, 182 at 200, 134 at 200 with the 200 ms model, all from
2026-11-09b) that is 3.9 s, 16.5 s and 22 s, if a turn puts one row through the indexes (the rows were not counted). - In the slow-follower shape the page really went bad: at 15,000 rows with autovacuum off, which is 20 to 112 s at those rates. In 330,000 rows of the pump’s own traffic it did not.
- A rebuild every 30 minutes would therefore cure about 1 to 6 percent of the gap in that shape.
| Run | Rows to the rule | To a plan reading the history | Index at the end (agent / position pages) |
|---|---|---|---|
| pump, autovacuum off (1,985 rows/s, load 6.0) | 3,000 | none in 330,000 | 4,237 / 5 |
| pump, autovacuum on (2,265 rows/s, load 7.1) | 3,000 | none in 330,000 | 3,250 / 7 |
| slow follower, autovacuum off (1,608 rows/s) | 3,000 | 15,000 for two plans, 17,000 to 30,000 for the others | 307 / 63 at 30,000 |
| slow follower, autovacuum on (1,259 rows/s, load 4.7) | 3,000 | none in 330,000 (statistics stale) | 2,076 / 379 |
Other measurements from the same report:
| What | Result |
|---|---|
| Commits during a rebuild (300 other sessions, one writer to a burst of 100 replies a commit) | median 1.5 to 1.9 ms during against 0.7 to 1.5 ms before for one writer and for six; a burst of 100 replies a commit 4.55 to 4.84 ms against 3.34 to 4.09 ms; worst 3.8 to 7.7 ms. At 120,000 undelivered rows a rebuild of both indexes takes 79 to 100 ms. The stop line (median over 100 ms) is two orders away |
lock_timeout |
bounds a rebuild’s wait on an old snapshot: error 55P03 after 10.00 s, leaving an invalid index that must be dropped |
| The capped count the upkeep would use | 0.003 ms at 5 live rows, 1.3 ms at 10,001 |
| The pump’s read, bloated against rebuilt | 0.82 to 0.83 ms against 0.73 ms at 1,200 delivered; 1.37 to 1.44 ms against 0.69 to 0.73 ms at 12,000 |
| SQLite | no such defect: the seek stays 5 index entries, the pump’s read is 0.28 to 0.43 ms |
What it did not measure: traffic at 765 rows a second in real time (the harness ran at 1,259 to 2,265), a managed Postgres’s own policies, a transaction-mode pooler, and whether the plan flips in the autovacuum-on runs on fresh statistics. The report proposes changes to the plan and does not edit it. No document at this commit says what was built afterwards.
A related guard is already running: the nightly “plan survey” (scripts/plan-survey.sh 100m all) checks the plans of key reads, including the actor’s ledger page at 10,000 and 30,000 rows on every state of the table’s statistics. A failure there is read as a changed plan.
A very long session in memory
A session’s ledger can grow without bound, and the node’s views of it keep a folded in-memory copy, called a book, under a budget of 256 MB. The views are the activity view and the ledger view. A measurement on 14 November 2026 on an M5 Pro (18 cores, 64 GB, Go 1.27.1, load average 2.1 to 3.4) fed it sessions of 100,004 and 300,004 rows.
- A kept book retains a few kilobytes a row. Kept whole, the fold’s entry holds about 3,504 bytes a row including the rows (350 MB at 100,000 rows); the smaller guard’s book holds 2,224 bytes a row (222 MB at 100,004 rows).
- The old estimate under-counted for most shapes. It kept books past what 256 MB holds for every shape but the long ledger and large payloads. The counted estimate, built 15 November, stops them earlier, by 5 to 25%.
- At 100,000 rows assembling a book took 0.604 to 0.608 s and 524 MB of allocation. At 300,000 rows it took 3.29 to 3.32 s and 3.27 GB, and read all 300,014 rows on every read, because that book was not kept.
- Memory across sessions was unbounded: reading 1, 2, 4 and 8 sessions of 100,004 rows at once peaked at 383, 599, 1,215 and 2,432 MB over baseline.
Built, 15 November: a gate on concurrent assemblies, and a shared flight so readers of one session share one assembly. Measured after, load average 4.4 to 5.3 (busier than before), cold first reads, peak heap including uncollected garbage:
| Sessions at once | Before: peak heap, wall | After: peak heap, wall |
|---|---|---|
| 1 | 330 to 384 MB, 1.19 s | 373 to 386 MB, 1.18 s |
| 2 | 601 to 654 MB, 1.21 to 1.23 s | 623 to 641 MB, 2.36 to 2.37 s |
| 4 | 1,140 to 1,264 MB, 1.27 s | 880 to 899 MB, 4.73 to 4.75 s |
| 8 | 2,383 to 2,543 MB, 1.62 s | 807 to 866 MB, 9.50 to 9.54 s |
The peak no longer grows with the number of sessions, at the cost that the last of N readers waits N-1 assemblies (1.18 s each here). Four readers of one 300,004-row session went from latencies of 3.60 to 14.6 s to all four at 3.57 to 3.62 s, with one assembly instead of four.
The rest of the plan is parked, with the stated reason that no session of that size is in use. The store’s own read cost was not measured. The report says so.
Stress and race testing
To see one crash survived end to end, the Architecture page has a recording of a node killed with kill -9 in the middle of a turn, and the other node finishing it.
Load tests find limits. Stress tests look for wrong answers: lost messages, two turns of one actor committing, a parked actor waking. Arbor runs a number of them, mostly on a schedule, always under Go’s race detector.
What runs, and when
The architecture page puts the total at about 2,900 Go tests, run with the race detector, plus an integration suite against an embedded Postgres. The checks are grouped by what they guard (docs/quality-report.md §2):
| Check | When it runs |
|---|---|
| Explorers on the in-memory stores: every ordering of one part’s events, to a stated depth, checked against a written table that is total (a combination with no entry fails) | every pull request |
| The reduced explorer sets on SQLite and Postgres | the integration job. Since 2026-11-19 it runs on main and on demand, never on a pull request |
| A seeded random walk over two nodes | a small budget on every pull request, a large one nightly |
| The contract suites: every implementation of a store keeps one contract, its failure modes included | every pull request |
| The race detector | the go suite on every pull request (make test is go test -race) |
| A shuffled test order, under the race detector | the unit and integration suites, once a week (Sundays) |
| The stores’ stress harness and the runtime’s two-node stress test | nightly at 03:17 UTC |
The stress budget is a number of seeds (the per-store scales are in scripts/scheduled-inputs.sh). A seed makes one deterministic random run of operations, kills and restarts, and a failing seed can be replayed. The default is 2,000, divided by a scale per store: memory 4 (500 seeds a night), Postgres 4 (500), the runtime’s two-node test 20 (100), and SQLite 10 (200, at 150 operations, on Sundays only). The first seed is numbered by the day, so every night reads new ground. A test never sleeps and never reads the wall clock: that is a check of its own (TestTestsNeverSleep, internal/boundaries).
The wake path, the part that decides which node takes a due message, is the most checked. TestWakePathOrderings runs about 4,700 orderings on every pull request, and 325 each on SQLite and Postgres in the integration job. TestStressTwoNodesWakePathWalk runs nightly.
The data race series
Between 16 October and 5 November 2026 the nightlies’ logs held 1,322 WARNING: DATA RACE reports across 41 job logs. All were the same race, in the contract suite’s own helper (coordtest.id(), an id counter two scenarios drew from at once), and none had a frame of product code. It was fixed on 5 November (“the suite’s id counter is atomic”). The ten nightlies kept after the last report have none. The report counted all 2,644 racing lines.
Ten failed seeds, each traced
Ten seeds of the stores’ stress harness failed in the kept nightlies, all between 6 and 13 November. The report went through 40 nightlies (5 October to 13 November) by counting and by replay.
| Cause | Seeds | What it was |
|---|---|---|
| The harness | 8 of the 10 | Six had one cause: the harness’s position normalizer folded two close positions that hold no ledger row into one, so the check “its sender closed first” read false and demanded a bounce nobody owed. A seventh seed was the same false alarm, fixed as a fixed case, and an eighth was the harness’s lease rule, also fixed |
| A product defect | 1 | Seed 2076800113: a terminate sent to a closing or closed child left its live descendants open on SQLite and Postgres. Fixed |
| The runner | 1 | The embedded Postgres’s data directory was removed under it (docs/flakes.md, 9 November) |
The six seeds that looked alike were each reproduced at their nightly’s own commit, and their trace, not their seed, was replayed. A seed means the same run only on the generator of its own commit, so replaying old seeds on a corrected harness proved nothing. In every trace the sender did close first, so none hides a real loss. All six pass on today’s main and at the nightly’s commit with the normalizer’s correction alone.
The runtime’s two-node stress test
The same logs held four failures of the runtime’s own two-node test:
| Date | Test and seed | Said |
|---|---|---|
| 2026-10-07 | TestStressTwoNodesPostgres/seed37 |
A parked actor was claimed and consumed a user message without an unpark |
| 2026-10-11 | TestStressTwoNodes/seed13 |
A posted message was consumed 0 times |
| 2026-10-18 | TestStressTwoNodes/seed2 |
The same |
| 2026-11-05 | TestStressTwoNodes/seed98 |
The same |
- “Consumed 0 times” was a product defect. The driver read the open actors and then posted, and an actor terminated from inside a turn could close in between. The store accepted a message for an actor already closed, and a closed actor is never claimed. It reproduced 3 runs in 10 with a 3 ms sleep at the driver’s post; with the fix, 20 of 20 passed. It is fixed in the stores’ insert (
conflict.agent_closed), and for a closing actor on 7 November. The report says the three nightly sightings are tied to it by the identical assertion and their dates, not shown seed by seed, because the logs of that time did not print the recipient’s row. No kept nightly since has failed that test. ParkHoldson seed 37 was a path read from the code. A turn crossing its stop cap parked the actor at a checkpoint commit. The turn’s read-back of its row then failed (a node stopping cancels its context) and was only logged, and the turn’s next commit, of the same epoch, was accepted and un-parked it. Closed on 12 November: a later commit of the cut epoch is refused (conflict.cap_stopped) on every store. The kept log has no line of the failed read, so that seed 37 took this path is not shown. That no other path un-parks an actor without an unpark was read from the claims and commits’ checks, not run.- A 9 November “nodes did not come to rest” failure was the test’s driver, not the disk. At one CPU both memory-store stress tests failed every run, because the driver’s settle returned before the nodes’ goroutines were scheduled. Fixed the same day: on the memory stores a run is now a
synctestbubble. On today’smain, 400 seeds ofTestStressTwoNodespass under-race.
The report’s own summary: nothing of these is open, every failed seed in the kept logs has a cause and each cause is fixed. Two rest on inference and say so.
Defects the wake-path program found
The quality program takes a part of the runtime through three steps: write its states and windows down from the code, make each decision visible in the node’s own record, then drive it through every ordering of its events. The wake path was first. Its defects, in the quality report’s history table:
| Date | Found | Harm | Fix |
|---|---|---|---|
| 2026-11-11 | A message that came due between the wake policy’s read and the idle window’s computation sat behind the lease for the whole window | a late turn | idleFor takes the instant the policy was asked and asks again |
| 2026-11-12 | A node whose store watch ended never opened another | messages for an idle resident waited up to five minutes, with nothing logged | the watch reopens with a backoff |
| 2026-11-12 | With renewals failing for a lease’s time, another node claimed while the first still ran its turn | a tool call with an outside effect could run twice | a node now gives up a resident it cannot prove it holds, before another node may claim it |
| 2026-11-12 | The first rule for that kept its margin only for a lease proven on a renewal tick | a lease taken between ticks could be given up with almost no margin | found by the walk the same day; corrected |
The document that records the wake path also states honestly where its promise does not hold as written. Among three long windows still open, a message that only the poll could find, for an actor this node holds, waits until the resident lets go.
Testing the tests
The nightly of 13 November reported green over failed tests for an unknown time. Its two test steps ran go test ... | tee log with no shell: set, so GitHub ran them without pipefail and the step took tee’s exit status. Fixed the same week. That prompted an audit of every workflow and CI script in four repositories against eight families of hole (pipes, the pipefail trap, swallowed errors, conditions, wrappers, skips, tests that do not run, and reruns, with each required check also pinned to its source), recorded with what was found sound and why in docs/ci-verdicts.md.
The flake log (docs/flakes.md) records every intermittent failure with its cause and the rule it taught: a flake that passes locally under -race is a test that depends on an order or on time, and the fix is the test, never a longer wait and never a skip. One thing it records that matters for the numbers above: on the CI pool’s older Graviton2 machines (c6g, m6g), the same suites ran 1.5 to 1.9 times slower, for example store/sqlite under -race in 440 s against 235 s on an m8g.2xlarge. Those instance types were dropped from the pool by a decision of 14 November, and every figure in docs/ci-verdicts.md from that date is a Graviton3 or 4 figure unless it says otherwise. A time read from a runner is quoted with its type, as this page does.
What has not been measured yet
From the architecture page’s “Not measured yet”, plus what the reports above say they did not do:
- A managed Postgres with synchronous replicas. Every Postgres figure here is a single embedded server on the same machine as the node, so a commit’s network and replication are not in it. With replicas each commit is slower.
- Real Redis under many followers. The Postgres numbers at 200 sessions wait on it. The later runs use a single Valkey 7.2.15 on loopback with persistence off. A cluster, replicas, failover and TLS were not measured under load.
- More than two nodes.
- Ceilings for the turn’s p99 and the followers’ lag. Target (d) is not set, because each report says no ceiling is to be read from it.
- A rerun after the claim-lock change. The lock became a read-write lock on 2026-11-16. The poll’s own pause was measured on a loaded development machine (before and after the 17 November poll change), but the runner matrix has no report after it.
- Real models at load. Every load turn is a one-line answer to a short message on a scripted model, twenty turns a session. The nodes, the driver and the database share one machine. The one run with a hosted model is the ten-person session above, which is one session, not a load test.
- Long-running soak. No report here is an endurance run.
- The effect of the outbox index upkeep under real traffic, as listed above, and a very long session’s store read cost.
How to rerun them
Everything below is from the repository’s own files. Run the load test with nothing else running: it measures the wall clock and waits up to an hour for the machine to be quiet.
# the whole load matrix (Makefile target "load"); needs ARBOR_VALKEY_SERVER set to a Valkey server,
# else the Valkey configurations are left out
make load
make load runs go test -tags load -run '^TestLoad$' -timeout 4h -count=1 -v ./tests/load/ and writes the day’s report to docs/load/<date>.md and .json. Narrow or extend it with comma-separated environment variables:
| Variable | Meaning |
|---|---|
LOAD_CONFIGS |
for example sqlite,postgres-valkey (also two-nodes-valkey) |
LOAD_SESSIONS |
default 10,50,200 |
LOAD_TURNS |
turns per session, default 20 |
LOAD_LATENCIES |
default 0,200ms |
LOAD_RUNS and LOAD_RUN_FROM |
repeats per row (default 2) and the first run’s number |
LOAD_OUT |
where the report goes (default docs/load) |
LOAD_MACHINE |
the machine’s name, printed in the report’s head |
LOAD_PG_POOL |
each node’s Postgres pool size, printed in the head |
LOAD_PG_STATEMENTS=1 |
count Postgres statements per turn |
To drive a node you already run, use arbor soak --load (docs/soak.md, “Load”):
arbor --server http://127.0.0.1:9100 --token "$TOKEN" soak --load --app load-desk \
--sessions 50 --max-turns 20 --followers 10 --turns turns.txt --out /tmp/load1 \
--servers http://127.0.0.1:9101 --label "two nodes, 50 sessions" --note "store: Postgres 17 on this machine"
The other tests:
- The commit test is
TestCommitCeilinginstore/postgres/commit_ceiling_test.go. It carries theintegrationbuild tag and is off unlessARBOR_COMMIT_CEILINGis set.ARBOR_COMMIT_CEILING_SECONDSsets each case’s length (default 20). - The unit and race run is
make test, which isgo test -race -timeout 20m ./.... The integration suite ismake test-integration, which isgo test -tags integration ./.... - The stress runs are what the nightly runs:
go test -race -tags integration -run 'TestStress' -count=1 -timeout 280m ./store/postgres/, withARBOR_STRESS_SEEDS,ARBOR_STRESS_FIRSTandARBOR_STRESS_OPSset. For the runtime’s two nodes it is-run 'TestStressTwoNodes'over./runtime/, with the same tag. - The pump upkeep measures are under the
measuretag and run one at a time, for exampleMEASURE_UP_SEQ=outage MEASURE_UP_HISTORY=1200 go test -tags measure -p 1 -run '^TestMeasurePostgresUpkeepFlip$' -count=1 -v -timeout 60m ./tests/load/. - The real-server Redis tests run when
ARBOR_VALKEY_SERVERnames the server.