Agent health: triage and repair

🔵 As an engineer (admin), how to triage a failing agent: find the failure, understand the cause, make a minimal fix, and prove it with a run. A report without verification is not a result.

1. Find the failing session

Step	Tool	What to look at
Sessions with errors	`trace_sessions(only_errors=true)`	`tool_errors`, `errors`, `warnings`, `agent`, `end_type`
Targeted by agent	`trace_sessions(agent=<template>)`	latest sessions of the template
Text search	`trace_search(query="error")`	the error strings themselves and their sessions
Dialogue	`trace_transcript(session_id)`	request → response, which tools were called
Timeline	`trace_events(session_id)`	order of `tool_call`/`tool_result`, `success:false`

The warnings metric counts both declarative tool warnings and extractor-warning (_warnings: missing/optional response field). A non-zero warnings with tool_errors=0 is a signal of a "soft" failure (search returned empty, field not found).

2. Read the agent configuration

agent_config_show(name) — bundle metadata (list of files), then agent_config_file(name, "tools/x.yaml") — for one needed file at a time.
Protected admin/trace are not readable through the builder (403) — this is expected.
If 404 "unknown agent" — it's either absent from config/agents/, or it's a built-in module bundle (for example, coder from code): it's present in templates_list, but not in agent_configs_list.

3. Classify the cause

Symptom in the trace	Likely cause	Fix
`Error: env var 'X' is not set`	secret not in the environment	`env_set(X, …)` (secret — only in env)
`oauth.token_file '…': expected value at line 1 column 1`	empty/corrupt token JSON	reauthorize: `oauth_connect`/reconnect, auto-refresh in the instruction
`Tool 'X' is not allowed for this process`	tool is called directly, not through `tool_call`	rewrite the call through `tool_call(name,args)` or `tool_attach`
`Network error … /v1/tools/…` (after ~60s)	nested loopback call hung/was not served	do NOT retry; replace with a real file + `tool_call`
`No such file or directory` at `script_tool`	`tool_run_yaml` doesn't pull the sibling script	create `tools/<file>.yaml` + `tool_attach`
The same tool repeats with the same error	no "stop rule" in the instruction	add a ban on repetition and the next step
`no human response within … — approval timed out`	gated tool without a human	`delegate` instead of `agent_run` (approval pops up in chat)
403/404 on `/v1/builder/*`	builder module not active	say so directly, don't invent fields

4. Minimal fix

Change one thing at a time: the instruction, one tool, one hook, an environment variable. Don't rewrite the whole bundle. For agent.yaml — agent_config_file → fix → agent_config_validate → agent_config_upsert(agent_yaml=…); the other files remain untouched.

5. Prove the result

Quick: agent_run(template, message) — one turn; then trace_transcript(session_id) — what actually happened.
Reliable: bench_run(template) (a cheap run) and bench_run_show(run_id) — compare turns/tools/tokens before and after. A fix is "ready" only when the run confirms it.
Show before/after and honestly say what improved and what didn't.

6. Report

Briefly: (1) what was failing and where (session id, exact error), (2) the cause, (3) what was changed (file + the gist), (4) what confirms it (run, metrics), (5) what remains.

Safety

Before anything destructive (agent_kill, session_close, module_disable, agent_config_delete, env_unset, platform_restart) — warn and ask for confirmation.
admin and trace are protected: they can only be tool_attach/tool_detach.
Don't invent endpoints, fields, or config keys — read the real bundle.