Вектор Решений Agent OS
Live demos

Benchmarks and quality control

🔵 agent-os bench runs agents through end-to-end scenarios and writes a report — this way agent quality is checked automatically, not "by eye".

16.1 Running

# против работающего сервера
agent-os bench --url https://HOST --max 5

# поднять собственный локальный сервер (нужен DEEPSEEK_API_KEY в .env)
agent-os bench --port 3001 --max 3

# конкретный агент
agent-os bench --url https://HOST --agent drom --max 5 --parallel 3

# сценарии по имени (подстрока; `a|b` = OR)
agent-os bench --url https://HOST --name "Camry|BMW" --max 10

# с LLM-судьёй качества и кодом возврата для CI
agent-os bench --url https://HOST --agent drom --judge --fail-on-error --out report.json

# 3 полных прогона с усреднением (борьба с шумом LLM/сети)
agent-os bench --url https://HOST --agent drom --repeats 3 --judge

Flags: --url, --port (3001), --agent, --name, --max (0=all), --parallel (4), --timeout (90), --config (config root), --out (report path), --repeats (1), --judge, --judge-model, --fail-on-error. Format — --flag value.

--repeats N runs the entire set of scenarios N times and averages: a turn is considered passed only if it passed in all N runs (pass^N), while tokens/cost/ time/judge are averaged. This eliminates the "swings" of single runs; the report has a field repeats.

The result — bench_report_<timestamp>.json (or --out), with turn counters passed/failed, scenario counters scenarios_*, per-turn failures and the full response text in the field response. --fail-on-error returns exit code 1 if there was at least one failure.

16.2 Scenario format

Scenarios in config/agents/<agent>/bench/scenarios.yaml:

name: "Drom Agent"
model: "drom-agent"

scenarios:
  - name: "Поиск Camry"
    turns:
      - user: "Найди Toyota Camry"
        assert:
          contains: [camry]
          min_links: 2
          valid_urls: true

Multi-turn scenarios

  - name: "Multi-turn: уточнение"
    turns:
      - user: "Первый вопрос"
        assert: { contains: [Слово], min_links: 2 }
      - user: "Уточнение"
        assert: { min_links: 1 }
      - user: "Финальное уточнение"
        assert: { valid_urls: true }

Check types

Check Meaning
contains: [w1, w2] the response must contain substrings (case-insensitive)
not_contains: [w1, w2] these substrings must not be present
regex: ["..."] the response must match each regex
min_links: N at least N markdown links [text](url)
min_urls: N at least N URLs (markdown links + bare http(s)://)
valid_urls: true all URLs start with http:// or https://
min_chars: N minimum response length
max_elapsed_ms: N turn time ceiling (speed)
max_completion_tokens: N response token ceiling (savings)
max_cost_usd: X turn cost ceiling

Set speed and token savings with explicit ceilings (max_elapsed_ms/max_completion_tokens/max_cost_usd), not just the judge.

16.3 Analyzing the result

  1. Look at the console: ✓ (pass) / ✗ (fail).
  2. Read bench_report_*.json: the fields failures and error.
  3. Compare two reports (diff) to find regressions.

16.4 Run storage

Runs are written to logs/bench.db and are visible in the admin API:

Endpoint Description
POST /v1/agents/:name/bench/run run a template benchmark
GET /v1/agents/:name/bench/runs run history for a template
GET /v1/agents/:name/bench/runs/:id a single run + its turns (while running — compact status; ?full=1 — full)
GET /v1/bench/runs latest runs across all templates
GET /v1/bench/catalog templates with configured scenarios

Run body (all fields optional): {"timeout_secs": 90, "parallel": 4, "judge": false, "judge_model": ""}. parallel sets the concurrently executing scenarios, judge enables LLM evaluation of each successful turn (average score — avg_judge_score). In the admin UI these parameters are available on the "Benchmarks" tab.

While a run is in progress (status: running), GET .../runs/:id by default returns a compact response (run, progress {done,total}, hint) without the list of turns — so that an agent waiting for the result doesn't pull the entire list on every poll. The full list is returned when status stops being running, or via ?full=1. The agent doesn't need to poll more often than once every ~30 s — for this there is the tool wait_tool.

If the server was stopped/restarted during a run, such an "orphaned" run is marked failed on the next start (finished_at + error: interrupted: server stopped before the run finished), а не висит в running.

16.5 Practices