Performance, Load, and Memory

🔵 How Agent OS behaves under load: how many concurrent requests it handles, how it limits overload, how much memory it consumes per session, and how to reproduce this. Plus a comparison with another agent server on the same test bench.

The numbers in this article come from one specific test bench (see §34.2) and serve as a guideline for planning, not a guarantee on your hardware.

34.1 What Limits Load

Agent OS is not a load balancer and does not perform inference itself: the LLM is an external provider, and the core schedules calls to it. Therefore the "ceiling" is set by three things: core limits, overload behavior, and the cost of persistence.

One session — one active turn. If a second request arrives for the same session, it cancels the previous one (last-wins) rather than being queued. This protects against races within a single session.

Persistence. Stores write turn data: the JSONL logger, EventDb (events.db), the message store (messages.db), and the access log (access.db). All of them are offloaded to background writer threads with batching (one transaction per batch), so they do not block the async path; but at very high rates their threads begin to compete with each other (see §34.3). JSONL is an optional layer; the DuckDB database (events/transcripts/access) carries the main functionality.

What to watch at runtime (GET /metrics, Prometheus):

• agentos_in_flight_turns, agentos_admission_limit, agentos_admission_rejected_total — overload and 429;
• agentos_active_sessions, agentos_active_processes — working set size;
• agentos_event_db_write_wait_ms, agentos_message_store_queue, agentos_access_log_queue, agentos_event_db_dropped_total — writer congestion;
• agentos_turn_duration_ms, agentos_tokens_prompt_total / _completion_total — latency and token consumption.

34.2 How to Reproduce the Measurements

The test bench from the repository (requires mock_llm + loadgen, build release-fast):

cargo build -p agent_web_bot --profile release-fast \
    --bin agent-os --example mock_llm --example loadgen
./scripts/loadtest.ps1                 # свип 1,2,4,8,16,32,64
./scripts/loadtest.ps1 -Sweep 128,256,1000 -DurationSecs 15

• The mock provider (crates/host/examples/mock_llm.rs) serves an OpenAI-compatible SSE with configurable MOCK_TTFT_MS, MOCK_TPS, MOCK_TOKENS and error injection; at /stats it shows max_concurrent (proof that the core did not exceed the limit).
• The load generator (crates/host/examples/loadgen.rs) measures TTFT and total turn time over SSE, --sessions == --concurrency, each worker — its own session.
• Bench configs: testing/config-scale* (without persistence / with full), max_in_flight_turns: 0, concurrency_limit: 1000.

Bench profile for the numbers below: mock TTFT=2000ms, 200 tok/s, 8 токенов (turn ≈ 2.04 s), 18 logical CPUs, mock + server + load generator on the same machine, 12–15 s per point. It is best to measure on a clean machine: extraneous processes and accumulated databases noticeably distort the result.

34.3 Throughput

Agent OS without persistence (pure scheduler + HTTP) — the core "ceiling":

Latency is practically flat: from c=1 to c=1000, TTFT grew from 2028 to 2090 ms, total turn time — from 2135 to 2215 ms. The core's own overhead is ≈ +30 ms to TTFT and +95 ms to the turn and is almost independent of contention. Throughput grows linearly as long as CPU and permits suffice.

Agent OS with the DuckDB database (EventDb + message store + access log; JSONL off):

Up to ~1000 concurrent turns, the database costs only ~2% of throughput (451 vs 459) and does not move latency. Above ~3000, throughput hits the competing DuckDB writers (events + transcripts + access), the message store queue overflows, and some events are dropped under overload (event_db_dropped) — consider retention/sharding if you keep thousands of concurrent sessions with full history.

34.4 Memory

We measure two quantities of the server process: WorkingSet (resident, including shared code/DLL pages) and Private (its own memory). On Windows — Get-Process; on Linux — VmRSS and Private_Dirty from /proc/<pid>/smaps_rollup.

• About "70 MB at startup." The core's own memory at startup is only ~9 MB; the 30–70 MB WorkingSet is mainly shared, demand-paged pages of the binary (agent-os.exe ~62 MB) and duckdb.dll (~35 MB), which are loaded as code is called and can be reclaimed by the OS. This is a fixed, non-growing item, not a leak.
• The real fixed cost is DuckDB: enabling the database raises private memory from ~9 MB to ~98 MB (buffer manager, even without load). Therefore "70 MB" is the wrong reference point: count the DuckDB base (~100 MB) and the increment per session.
• Growth per session ≈ 0.13 MB (without stores) / 0.19 MB (with the database) and stays linear: one session = a core process + working context.
• Memory is not freed instantly: sessions live up to session_ttl_secs (300 s by default). During long idle periods the cleaner returns RSS to the base.
• Transcripts and events live on disk (DuckDB), not in memory; RSS growth is the state of processes/sessions, not history.

34.5 Behavior under overload (429)

Enable kernel.max_in_flight_turns > 0 (for example, 8 with concurrency_limit: 3) — and overload stops accumulating queues:

• requests over the limit immediately get 429 + Retry-After, cheaply;
• accepted requests are driven through to completion, no 5xx and no SSE breaks;
• a rejected request does not cancel an already started turn in the same session;
• recovery after the load subsides takes seconds.

Recommendation: in production always enable the admission gate, and set concurrency_limit according to the provider's quota (so as not to catch external 429s/timeouts).

34.6 Comparison with opencode

Same bench, same mock (TTFT=2000 ms, 8 tokens), one session per worker. opencode (opencode serve v1.18.32) is pointed at the same mock via a custom provider @ai-sdk/openai-compatible; the full turn time was measured (POST /session/:id/message).

Important — a fair comparison. opencode persists sessions/transcripts to SQLite, so it must be compared with Agent OS with the database (EventDb + message store + access; JSONL off), not with the bare core. At the same time, the opencode server is a server for a single-user coding agent (sessions are tied to a project, there is no admission control), whereas Agent OS is a multi-tenant chat/agent server. The comparison is about the nature of scaling, not about product parity.

Throughput (Agent OS with the database vs opencode)

• Per-turn overhead at c=1: Agent OS ≈ 95 ms, opencode ≈ 190 ms.
• Agent OS keeps latency flat up to 1000; opencode pays for growth in c with latency (c=128 → p50 6.6 s, c=256 → 11.9 s), without an admission gate errors appear at 256, and at 512 — denial of service.
• At comparable latency, the throughput gap is an order of magnitude (68 vs 21 rps at c=128; 451 vs <40 at high c).

Memory

• opencode has a higher fixed base (the runtime itself), the increment per session is comparable. Under real load opencode was observed at 550–680 MB with hundreds of sessions and does not give memory back as readily as Agent OS with its TTL cleaner.
• To estimate Agent OS RAM: `~9 MB (core) + ~100 MB (DuckDB database, if enabled)
  • 0.13–0.19 MB × concurrent sessions`.

Comparison summary

Agent OS is built for concurrent multi-tenant load: flat latency, linear throughput growth, 429 protection, and TTL cleanup. The opencode server degrades in this role: latency grows linearly, and under overload there are errors/hangs without protection. This is expected: it has a different job (an interactive coding agent for a single user), not a server for thousands of concurrent sessions.

34.7 Practical recommendations

• Provider limit: concurrency_limit = how many concurrent streams your API key can actually sustain. This is the main lever.
• Always enable admission: max_in_flight_turns > 0 in production, so that overload yields a fast 429, not queues and timeouts.
• Persistence — as needed: JSONL can be disabled (jsonl_dir: ""); the DuckDB database is needed for transcripts/events, but it costs ~100 MB of fixed memory and is the main bottleneck at high rates.
• Session TTL: keep session_ttl_secs reasonable — this is memory reclamation during idle periods.
• RAM planning: ~ (100 MB database, if enabled) + 0.13–0.19 MB × concurrent sessions, plus headroom for spikes; the "starting 70 MB" is mostly reclaimable code pages, not working memory.
• Monitor /metrics: in_flight_turns, admission_rejected_total, writer queues (message_store_queue, access_log_queue), event_db_dropped_total and turn_duration_ms.
• Regressions: keep baseline numbers and run scripts/loadtest.ps1 in CI (see 16-benchmarking); worry when p95/RSS grows by more than ~20–25%.