Tools — data collection and retrieval

🔵 Engines for fetching data from outside: HTTP scraping, local archive, JSON, RSS, PDF, geocoding. For a summary table of all engines, see "Tools Reference".

5.4 site_tool

A universal declarative HTTP/API client: request → field extraction → output format. Despite the name, it's not just a scraper — it supports arbitrary REST: method (GET/POST/PUT/PATCH/DELETE), bodies json/form/multipart/body, custom headers, OAuth2, HMAC signing, cookies, pagination. For REST integrations you can write kind: rest_tool — it's an alias of the same engine (clearer in meaning).

kind: site_tool                     # или rest_tool — алиас для REST/API
name: my_scraper
description: Парсит страницу товара.
parameters:
  type: object
  properties:
    url: { type: string }
  required: [url]

fetch:
  url: "{{ input.url }}"          # шаблон MiniJinja
  method: POST                    # GET (по умолч.) | POST | PUT | PATCH | DELETE
  json: { query: "{{ input.q }}" }  # JSON-тело (листья — шаблоны)
  # body: "q={{ input.q | urlencode }}"   # сырое тело (шаблон)
  # form: { search: "{{ input.q }}" }      # x-www-form-urlencoded
  # multipart: { fields: {...}, files: {...} }
  timeout_secs: 15
  cache_ttl_secs: 300
  follow_redirects: true
  cookies: true
  headers:
    User-Agent: "MyBot/1.0"
    Authorization: "Bearer {{ env_var('API_KEY') }}"
  # oauth2:                          # OAuth2 (refresh_token / client_credentials)
  #   token_url: "https://oauth.yandex.ru/token"
  #   grant: refresh_token
  #   client_id_env: X_CLIENT_ID
  #   client_secret_env: X_CLIENT_SECRET
  #   refresh_token_env: X_REFRESH_TOKEN
  #   prefix: "OAuth "               # префикс заголовка (Bearer по умолч.; Yandex — OAuth)

source:
  type: script_json               # JSON внутри <script>
  contains: productData
  # type: json                    # или тело само — JSON

extract: { ... }
output: { ... }

fetch — URL

Two styles:

1. Direct URL — url: "{{ input.url }}".
2. Path builder — base + path_opt + path + query (example — drom_search.yaml):

fetch:
  base: "https://auto.drom.ru/"
  path_opt:
    - when: { has: region }
      value: "region{{ region | region_code }}"
    - when: { equals: [condition, "new"] }
      value: "new"
  path:
    - "{{ brand }}"
    - "{{ model | default('') | slugify }}"
    - "{% if page is defined and page > 1 %}page{{ page }}{% endif %}"
  query:
    unsold: "1"
    minprice: "{{ price_from }}"
  filters:
    slugify: _builtin_
    region_code:
      moscow: "77"
      nsk: "54"
    region_code_ci:
      case_insensitive: true
      default: "77"
      "Москва": "77"

filters maps filter names to _builtin_ (e.g. slugify) or a key→value table. The lookup table is either a plain map, or an object with case_insensitive: true and default (fallback when the value is absent).

env_var('NAME') reads an environment variable — use it for keys in headers.

query is also applied with a direct url (added with URL encoding).

Templates: namespaces and functions

In all config fields (url, base/path/path_opt, query, headers, body, json, form, multipart, output) the same context is available:

• flat names of input arguments — {{ brand }}, {{ title }};
• nested object — {{ input.brand }}, {{ input.title }}.

That is, {{ org }} and {{ input.org }} are interchangeable in any field (previously flat names were only visible to path).

Available MiniJinja functions:

state(...) resolves scope from the calling session, not from the arguments: this way agent settings (e.g. state('agent', 'repo'), written by state_tool) are substituted into the request without passing them as an argument. A missing key renders as empty (undefined); calling state(...) outside a tool invocation or without an enabled state-store is a render error, not a silently empty value.

fetch — shared profile (extends)

To avoid repeating base_url and Authorization, move them into a profile and specify extends (path relative to the tool's YAML):

# tools/_shared.yaml
base_url: "https://api.example.com"
headers:
  Authorization: "Bearer {{ env_var('API_KEY') }}"

# tools/list.yaml
fetch:
  extends: _shared.yaml
  path: ["issues"]
  headers:
    Accept: "text/plain"      # перекрывает Accept из профиля

The tool's own url/base/headers take precedence over the profile.

A profile is not a tool (it has no kind): don't list it in tool_files, otherwise the loader won't recognize the file and the tool won't be registered. You can verify that all tool_files actually loaded in templates_list (fields tools_declared/tools_loaded/tools_failed).

fetch — request body

Exactly one of body / json / form / multipart (an ambiguous config is rejected at load time). json: — JSON body, each string leaf is a template. form: — urlencoded. multipart: { fields, files } — files are local paths.

Booleans/null in MiniJinja render Python-style (True/False/None), which is invalid JSON — in a raw body, pass values through tojson:

body: '{"completed": {{ input.completed | tojson }}}'

For a json body, omit_empty: true recursively drops keys with empty values ("", null, [], {}) — handy for PATCH.

fetch — retries / backoff

Transient failures are retried automatically: HTTP 429, 5xx, network/timeout errors. retries — number of attempts (default 3, 0 disables); backoff_ms — base delay with doubling (default 3000 → 3s, 6s, 12s).

fetch — pagination (token style)

For lists with a next-page token, add pagination:

pagination:
  token_field: /next_page_token
  token_param: page_token
  items_field: /issues
  max_items: 100
  max_pages: 10

The tool follows pages itself (aggregating items) and reports where to continue. Result: { items, page_info: { returned, total, has_more, next_token } }. The LLM continues with a call using page_token: <next_token>.

source — where structured data lives

Other source types:

• json_ld — schema.org JSON-LD; block selection by @type:

  source:
    type: json_ld
    type_selector: Product     # совпадает с "Product" или "https://schema.org/Product"
    graph: true                # искать и во вложенных @graph (по умолчанию true)
  

• file_json — JSON from a local file (no HTTP):

  source: { type: file_json, file: data/input.json }
  

• devalue — Nuxt 3 payload (__NUXT_DATA__): resolves integer references and markers.

  source: { type: devalue, contains: __NUXT_DATA__, min_length: 1000 }
  

• html — extraction via CSS selectors (no JSON needed):

  source:
    type: html
    selector: ".product-card"              # повторяющийся элемент-запись
    fields:
      title: { selector: ".product-card__name", attr: text }
      price: { selector: ".product-card__price-current", attr: text }
      url:   { selector: "a.product-card__name", attr: href }
  

  Returns { "items": [ {field: value, …}, …] }.

fetch — two-step token (auth)

When a token is needed first (anonymous authorization), add auth:

fetch:
  auth:
    token_url: "https://…/api/v1/auth/anonymous"
    method: POST
    json: { app: "mobile" }        # или body / form / headers
    field: token.accessToken       # JSON-указатель /a/b или точка a.b
    header: Authorization
    prefix: "Bearer "

fetch — OAuth2 (oauth2)

For OAuth2 services (Bitrix24, amoCRM, Google) — oauth2. Unlike auth, the engine caches the access token until expiry, rotates the refresh token, and on HTTP 401 refreshes the token itself and retries the request. Secrets are taken from env (never in YAML):

fetch:
  oauth2:
    token_url: "https://…/oauth2/token"
    grant: refresh_token          # refresh_token (по умолч.) | client_credentials
    client_id_env: OAUTH_CLIENT_ID
    client_secret_env: OAUTH_CLIENT_SECRET
    refresh_token_env: OAUTH_REFRESH_TOKEN   # для grant: refresh_token
    # scope: "..."                # для grant: client_credentials
    header: Authorization         # по умолчанию
    prefix: "Bearer "             # по умолчанию

The response is parsed by standard OAuth2 fields: access_token, expires_in (seconds) and optional refresh_token. The returned refresh_token replaces the cached one for subsequent refreshes. oauth2 and auth are mutually exclusive.

fetch — request signing (sign)

HMAC-SHA256 for APIs that require signing (YuKassa, Tinkoff, Sber). The canonical string is a MiniJinja template with the variables method, url, body (canonical body) and input:

fetch:
  sign:
    secret_env: SIGN_SECRET       # ключ подписи из env
    template: "{{ method }}{{ url }}{{ body }}"   # по умолчанию
    header: X-Signature           # либо header, либо query
    # query: sign
    encoding: hex                 # hex (по умолч.) | base64

header and query are mutually exclusive: the signature goes either into a header or into a query parameter.

extract — field types

Three types, distinguished by keys.

Simple field — navigation by JSON pointer:

extract:
  location:
    pointer: /geoInfo/0/text
    value: /sub/path           # доп. навигация вглубь
    fallback: "unknown"
    transform: int

Pointers can be MiniJinja templates from input; the segment * (or ~first) steps over a dynamic key.

Array field — array filtering by discriminator:

  year:
    pointer: /fields
    array_match: { type: year }   # первый элемент с .type == "year"
    value: /payload
    transform: int
  photos:
    pointer: /gallery/images
    collect_all: true             # все элементы → массив
    value: /image/src
    transform: [ { skip: 5 }, { take: 20 } ]

Computed field — conditional logic:

  status:
    compute:
      - when: { op: not_null, pointer: /soldNotification }   then: sold
      - when: { op: equals, pointer: /shouldShowDeleted, value: true }  then: deleted
    default: active

op in deliveries: not_null, equals.

Transformations

Applied in order (can be chained as a list):

output — format

Optional MiniJinja template. Leaves are rendered from input and extracted. {{ extracted._warnings }} is available when fields did not resolve. Without output the extracted map is returned as is.

Error handling

The engine does not panic: errors return success: false with a message (HTTP 403/404/429/5xx, network errors, "structured data not found").

5.5 archive_tool

Search over a local archive.

kind: archive_tool
name: ngu_search
description: Поиск по архиву НГУ.
parameters:
  type: object
  properties:
    query: { type: string }
  required: [query]

mode: search_grouped
archive:
  path: data/ngu
  default_limit: 10

filters:
  - { field: players, op: players_min, param: players_min }
  - { field: price,   op: gte,        param: price_min }

sort:
  field: price
  order: asc

mode

The archive is a directory with meta.json, docs.bin (documents by chunks) and terms.bin (term index); assembled by the agent-os scrape command.

5.6 json_tool

Search over preloaded JSON.

kind: json_tool
name: find_brands
description: Поиск брендов по имени.
parameters:
  type: object
  properties:
    query: { type: string }
    section: { type: string, enum: [auto, moto, truck] }
  required: [query]

data_file: data/drom_brands.json

search:
  paths: ["{{ input.section | default('*') }}.top", "*.all"]
  field: name
  query_param: query
  mode: contains
  limit: 20
  dedupe_by: id
  source_field: section

computed:
  slug: "{{ item.url | path_segment(0) }}"

5.7 rss_tool

Search over RSS feeds.

kind: rss_tool
name: rss_search
description: Поиск новостей по ключевому слову.
parameters:
  type: object
  properties:
    site: { type: string }
    query: { type: string }
  required: [site, query]

feeds:
  lenta: "https://lenta.ru/rss"
  ria: "https://ria.ru/export/rss2/archive/index.xml"
  rbc: "https://rssexport.rbc.ru/rbcnews/news/30/full.rss"

site — key in feeds. The result carries count, titles, snippets, URLs.

5.8 pdf_tool

Reading PDF page by page.

kind: pdf_tool
name: pdf_read
description: Скачать и прочитать PDF.
parameters:
  type: object
  properties:
    url: { type: string }
    page: { type: integer, default: 1 }
  required: [url]

fetch:
  url: "{{ input.url }}"
  timeout_secs: 30
  cache_ttl_secs: 3600

pdf:
  chars_per_page: 3000

output:
  page: "{{ extracted.page }}"
  total_pages: "{{ extracted.total_pages }}"
  text: "{{ extracted.text }}"
  next_page: "{{ extracted.next_page }}"

5.15 geo_tool

Geocoding and places (OpenStreetMap). Free, no keys, geocoding and POI search (Nominatim + Overpass API) — the agent does not make up addresses.

kind: geo_tool
name: geo_places
description: Найти места категории рядом.
operation: places

Categories (RU/EN synonyms): сто/автосервис/car_repair, шиномонтаж, аптека/pharmacy, банк/атм, кафе/ресторан, школа, магазин/ продукты/супермаркет, заправка/азс, парковка. Unknown category → Overpass search shop=<value>.

Optional: nominatim_url / overpass_url, user_agent, ui.

5.15.1 rasp_tool

Transport schedules (trains, commuter trains, planes, buses, motor ships, ferries) via the free Yandex.Raspisaniya API. Authorization — a free key (apikey query parameter), which is issued in the developer dashboard developer.tech.yandex.ru (the "Yandex.Raspisaniya" API).

kind: rasp_tool
name: rasp
description: Расписания транспорта Яндекса.
api_key_env: YANDEX_RASP_API_KEY   # или api_key: "..." (не рекомендуется)
lang: ru_RU                        # опц., язык ответов

Actions (the action argument):

Stations can be specified either by name (resolved to a code automatically) or by a ready code yandex_code (for example s2000006). For search cities are resolved to a settlement code (city→city), for schedule — to a specific station (with priority given to an actual railway station).

{ "action": "station_list", "query": "Москва" }
{ "action": "search", "from": "Москва", "to": "Санкт-Петербург", "date": "2026-09-15", "transport_types": ["train"] }
{ "action": "schedule", "station": "Москва", "date": "2026-09-15", "direction": "departure" }
{ "action": "thread", "uid": "122S_15_2" }
{ "action": "nearest_stations", "lat": 55.75, "lng": 37.61 }

transport_types: plane, train, suburban, bus, water (by default — all). A mandatory condition of the API is to credit Yandex as the data source.

5.23 browser_tool

Headless browser. For sites that render via JavaScript (SPAs, infinite scroll, logins) — a regular HTTP fetch site_tool is not enough. browser_tool loads the page in Chromium, performs interactions and returns the result — either via a JS script executed inside the page ("internal scripting"), or via the rendered text.

The browser is looked for on the system, and if it is absent — it is downloaded automatically (chrome-headless-shell from Chrome-for-Testing; the same logic as OfficeCLI's PDF export).

Browser processes are reused (pool): each call opens a fresh isolated tab (CDP browser context) in a live Chromium and on completion returns the browser to the pool — repeated calls do not pay ~1–2 s for startup. The number processes is limited by the variable AGENT_OS_MAX_BROWSERS (default 4); excess calls wait for a free slot (up to 60 s, then an error).

kind: browser_tool
name: scrape_spa
description: Извлечь товары из JS-листинга.
parameters:
  type: object
  properties:
    url: { type: string }
  required: [url]

steps:                          # необязательные действия, по порядку
  - { type: goto, url: "{{ input.url }}" }
  - { type: wait, selector: ".product-card" }
  - { type: click, selector: "button.load-more" }
  - { type: type, selector: "#q", text: "{{ input.q }}" }
  - { type: check, selector: "input[name=agree]" }
  - { type: eval, script: "return document.title" }
  - { type: screenshot, name: "page.png" }
  - { type: pdf, name: "page.pdf" }

script: |                       # JS внутри страницы → итоговый результат (JSON)
  [...document.querySelectorAll('.product-card')].map(c => ({
    title: c.querySelector('h2')?.innerText,
    price: c.querySelector('.price')?.innerText,
  }))

timeout_secs: 30

Steps

Scripting and result

The string fields of steps (url, selector, text) are MiniJinja templates over input. The JS in script / eval.script runs in the page context; the call's arguments are available as the global variable input (for example input.url).

Result: if script (or the last eval) returned a value, it goes into data and content (a string or pretty-JSON); otherwise the page text is returned. screenshot/pdf form downloadable blocks file in visual; an array of objects is additionally rendered as a table.

5.24 web_search

🔵 Internet search via a provider (FireCrawl by default). The engine is supplied by the module searchers (a linked crate or DLL); the core registers kind: web_search автоматически, когда модуль включён. Демо-агент searchers-demo comes bundled.

Supported providers: firecrawl (default), tavily, brave, serper, serpapi, exa, google (CSE), searxng (self-hosted, no key). Each has its own key/environment variable — the full table is on the page Module searchers.

Call parameters (set by the LLM):

Config:

kind: web_search
name: web_search
description: Поиск в интернете (title, url, snippet).
# provider: firecrawl            # firecrawl|tavily|brave|serper|serpapi|exa|google|searxng
# api_key: fc-...                # явный ключ (не хранить в git)
# api_key_env: FIRECRAWL_API_KEY # по умолчанию — своя для провайдера
# base_url: https://api.firecrawl.dev
# cx: <google-cse-id>            # только для provider: google
# limit: 5

The key is resolved in the order: api_key from the config → the variable api_key_env (by default, its own for each provider: FIRECRAWL_API_KEY, TAVILY_API_KEY, BRAVE_API_KEY, SERPER_API_KEY, SERPAPI_API_KEY, EXA_API_KEY, GOOGLE_API_KEY). searxng requires no key. The key can also be set via the module settings — the value is projected into the corresponding variable without a restart.

The result is JSON: { provider, query, count, results: [ { title, url, description } ] }.

The module, the demo agent, and details are on the page Module searchers.