New Agent Development Cycle

A step-by-step process for creating a new AI agent on the Agent OS platform — from use cases to deployment and ongoing monitoring. All agent logic is described by configuration (YAML); no Rust code changes are needed. Development follows test-first: first we pin down which questions the agent must answer, then we build the data and tools for those use cases.

1. Use Cases
2. Data
3. Tools
4. Debugging
5. Tests
6. Agent
7. Hooks
8. Bench
9. Deploy
10. Monitoring

↻ Monitoring → new use cases → step 1: the cycle repeats

Step 1. Use Cases and Bench Scenarios (test-first)

Before any code, we pin down what the agent must be able to do: real user questions and expected answers. Agents come in different kinds, and use cases depend on the agent type (more on types — in the "Solutions" section on the Вектор Решений website):

Plus heavy scenarios (combinations of filters, edge cases), multi-step chains (selection → comparison → checkout), and multi-turn (context retention check: "tell me more about the first one").

Use cases are defined in config/agents/<агент>/bench/scenarios.yaml — this same file is later used to run the benchmark (step 8):

name: "NGU Agent"
model: "ngu-agent"

scenarios:
  - name: "Поступление"
    turns:
      - user: "Как поступить в нгу"
        assert:
          contains: [поступлен]
          min_links: 1
          valid_urls: true

Check types: contains (substrings), min_links/min_urls (minimum number of markdown links), valid_urls (all links are http/https). Tips: for rare results use min_links: 1; valid_urls: true catches broken links; Russian queries for Russian-language agents.

Step 2. Data and Sources: APIs, Adapters, Scraping

First we figure out where the agent will get data from and which APIs are available. For each source we pick a ready-made engine (step 3), and if there is no ready-made one — we write an adapter.

2a. Analyzing Sources and APIs

What to find out about each source: response format, request parameters, whether headers/keys are needed (substituted from the environment via env_var()), limits and pagination, data refresh frequency.

An adapter is either a YAML config of an existing engine (for site_tool it is enough to describe fetch + extract + output), or, if non-standard behavior is needed (complex logic, non-HTTP source), a new engine in Rust: implement ToolHandler and register the factory in crates/tools/src/registry.rs. For ready-made dumps/exports there is a universal engine json_tool (declarative search over a JSON file: search + computed fields on MiniJinja). Example — config/agents/drom/tools/drom_find_brands.yaml: searching for brands by a preloaded data/drom_brands.json.

2b. Website Scraping

cargo run --bin agent-os -- scrape https://nsu.ru -w 8 -d 3 -n 500

Parameters: -w workers, -d crawl depth, -n maximum pages, -r delay between requests, -o archive path, -m navigation map path.

Analyzing the collected archive — find repeating blocks (headers, footers, menus) that need to be cleaned out of the text:

cargo run --bin agent-os -- scrape --analyze site.archive

2c. RAG Archive

For full-text search, pages are stored in a binary archive ArchiveBuilder (chunks with overlap, n-grams of order 1–3 as implicit stemming for Russian, BM25). Archive structure:

data/<источник>/
├── meta.json      — source name, document count, abbreviations
├── docs.bin       — chunk documents
└── terms.bin      — dictionary of terms and postings

Abbreviations are extracted automatically from patterns like "Автоматизации Физико-Технических Исследований (АФТИ)" — searching for "афти" finds the full name.

Ready-made examples: data/ngu/ (NSU, assembled from crawled pages at startup), data/hg/ (HobbyGames, ~4300 products).

Step 3. Tools

A tool = a YAML file in config/agents/<агент>/tools/*.yaml. Available engines (kind:):

| kind | What it does | |---| | site_tool | HTTP request → field extraction (JSON pointers, array match, transforms) → formatting | | archive_tool | Archive search: search, search_grouped, lookup, read_chunks, outline + filters by meta-fields | | pdf_tool | PDF download and page-by-page reading | | rss_tool | RSS feed search | | json_tool | Declarative search over preloaded JSON (search.paths, computed) |

Plus built-in core tools: http_fetch, fs_read, job_done.

Example of a minimal site_tool:

kind: site_tool
name: my_scraper
description: Scrapes example.com product pages.
parameters:
  type: object
  properties:
    url: { type: string, description: "Product page URL" }
  required: [url]

fetch:
  url: "{{ input.url }}"        # MiniJinja-шаблон
  timeout_secs: 10

source:
  type: script_json             # JSON внутри <script> на странице
  contains: productData

extract:
  title:
    pointer: /productData/title
  price:
    pointer: /productData/price
    transform: int

Key capabilities of site_tool:

• Fields: simple (pointer, fallback), arrays (array_match, collect_all), computed (compute with conditions when/then);
• Transforms: lookup, truncate, take, skip, len, not_null, bool, int — applied as a chain;
• Output: optional output template (MiniJinja); without it, a map of extracted fields is returned;
• Warnings: unresolved fields go into _warnings, the engine never panics.

Step 4. Local tool debugging

Any YAML tool can be tested without a server and without an LLM:

cargo build --bin agent-os --release
./target/release/agent-os tool config/agents/ngu/tools/ngu_search.yaml '{"query":"поступление"}'

The output shows: the arguments passed, call_template (what the user sees before the call), the LLM response, and result_template. All engines are supported: site_tool, archive_tool, pdf_tool, rss_tool, json_tool.

Step 5. Tool tests

Tests are YAML files in config/agents/<агент>/tests/*.yaml, discovered automatically. Two kinds:

Fixture tests (kind: site_tool_tests) — no network, data is supplied directly. Run: cargo test.

kind: site_tool_tests
tool: tools/my_tool.yaml

tests:
  - name: "extracts correctly"
    input: { url: "https://example.com/p/1" }
    fixture: { productData: { title: "Widget", price: 99 } }
    assert:
      - { path: /title, equals: "Widget" }
      - { path: /price, equals: 99 }

Live tests (kind: site_tool_live_tests, archive_tool_live_tests, rss_tool_live_tests) — real HTTP requests, marked with #[ignore]. Run: cargo test -- --ignored.

kind: site_tool_live_tests
tool: tools/my_tool.yaml

tests:
  - name: "structure smoke test"
    fetch: true
    input: { url: "https://example.com/p/1" }
    assert:
      - { path: /title, not_null: true }

Check types: equals, not_null, len, contains.

Step 6. Agent configuration

config/agents/<агент>/agent.yaml — the entire agent behavior:

name: ngu-agent
welcome: |
  👋 Привет! Я ассистент НГУ...
instructions: |
  You are an assistant for NGU (nsu.ru).
  Use ngu_search to find ANY information...
  Answer in Russian, helpful and informative. Always cite source URLs.
tools:                # встроенные инструменты ядра
  - http_fetch
tool_files:           # YAML-инструменты (пути относительно каталога агента)
  - tools/ngu_search.yaml
hook_files:           # хуки (см. шаг 7)
  - hooks/faq_cache.yaml

Tips for instructions:

• describe which tool to call when and call chains;
• require copying links[] from the tool response into the user response — links are mandatory;
• specify the response format and language;
• for geo-substitution, __GEOIP__ is available (the user's city).

Step 7. Hooks (optional)

A hook intercepts messages in the pipeline. A ready-made example is faq_cache (DuckDB): it remembers question-answer pairs, an LLM judge identifies similar questions, and the cached answer is either inserted as a hint (short_circuit: false) or returned immediately without calling the LLM (short_circuit: true) — saving tokens and speeding up responses.

kind: faq_cache
name: faq_cache
db_path: data/faq_cache_ngu.db
max_candidates: 5
min_confidence: 0.7
short_circuit: true
store: { enabled: true, min_question_chars: 8, min_answer_chars: 8 }
llm:
  base_url: https://api.deepseek.com
  api_key_env: DEEPSEEK_API_KEY
  model: deepseek-chat

Step 8. Running the bench and iterating

We run the scenarios from step 1:

# Продакшн-сервер
cargo run --bin agent-os --release -- bench --url=https://176.108.251.90 --agent=ngu --max=5

# Локальный сервер (нужен DEEPSEEK_API_KEY в .env)
cargo run --bin agent-os --release -- bench --port=3001 --max=3

Flags: --agent= filter by agent, --name= by scenario name (regex), --parallel=, --max= limit, --timeout=.

After the run: look at ✓/✗ in the console, analyze bench_report_*.json (fields failures, error), compare two reports to find regressions. Failures are fixed by editing instructions, tools, or data — if needed, we add new cases to scenarios.yaml — and the cycle repeats.

Step 9. Publishing and deployment

Page on the website

1. Content: config/web/pages/<агент>.md;
2. Registration: a card in cards: or an entry in extra_pages: in config/web/index.md;
3. Demo on live sites — an entry in demos: (opens /demo/overlay?model=<агент>).

The site is read from config/web/ at server startup — content edits require only a restart, without a rebuild.

Deploying to the server

scp -i ~/.ssh/ssh-pk-025d38 -r config/ data/ realityfaker@176.108.251.90:/home/realityfaker/agent_os/
ssh -i ~/.ssh/ssh-pk-025d38 realityfaker@176.108.251.90 "sudo kill \$(pgrep agent-os); cd /home/realityfaker/agent_os && sudo bash -c 'set -a && source .env && set +a && nohup ./target/release/agent-os serve > agent_os.log 2>&1 &'"
curl -sk https://176.108.251.90/health

If Rust code was changed (a new tool engine, a hook) — rebuild cargo build --release -p agent_web_bot before restarting.

Step 10. Monitoring and iterations

After deployment the cycle doesn't end — the agent runs in prod, and this is the main source of data for the next iteration.

What to watch

Iteration

1. Once per period (or after each deployment), run the bench in prod and compare reports — catch regressions;
2. Collect real questions from the FAQ cache and logs — questions the agent answered poorly or didn't answer become new cases in bench/scenarios.yaml (step 1);
3. Edit instructions, tools, or data — and go through the cycle again: tests → bench → deploy;
4. Update the knowledge base/archives as the source site changes.

This is how the cycle closes: each pass makes the agent more accurate and the cases closer to real user queries.

Checklist

• Cases: bench/scenarios.yaml describes the main features and edge cases
• Data: sources and APIs are understood, adapters are ready (or an engine is chosen), the archive is built, junk blocks are cleaned out
• Tools: YAML configs, fields are extracted (site_tool)
• Tests: fixture tests are green, live tests pass
• agent.yaml: welcome, instructions, tool_files
• Hooks: connected if needed
• Bench: run is green, report has no regressions
• Website: page in pages/, entry in index.md
• Deployment: config uploaded, server restarted, /health responds
• Monitoring: logs are clean, bench in prod is green, real questions → new cases