Module `index` — document index (RAG)

🟢 The module builds a local document index for a persona/organization and exposes it to the agent in two ways: auto-context via the rag hook and explicit search via the index_tool tool. Sources are server folders and cloud drives (Yandex.Disk). The module is dual-transport: the same code works as a linked crate and as a native DLL (lean).

The search engine uses bit signatures of n-grams (char+word 3-grams): fast recall over popcount, then exact re-ranking over the text. No models/embeddings are needed.

What it provides

What	Identifier	Description
Tool	`kind: index_tool`	`sources`, `search`, `open`, `status`, `sync`, `add_folder`, `remove`, `enable/disable`, `connect`
Hook	`kind: rag` (option `index_db`)	The README hook reads the index and injects fragments into every request
Agent	`index-demo`	demo agent + landing page `/ext/index-demo/`
Storage	DuckDB	`data/plugins/index.db` (plugin data_dir)

The module is enabled by default (id index): it is built as a DLL, placed in plugins/ and loaded via config/plugins.yaml (see "Build and deploy").

Sources

Server folder (`type: local`)

An absolute path on the server, necessarily inside security.index.allowed_roots (empty = any paths allowed, limited by the agent_os.Fs sandbox). Root paths (/, C:\) are forbidden.

kind: index_tool
name: index
yandex_tool: yandex_disk_idx   # какой yandex_tool использовать в connect

Adding: index {operation:"add_folder", path:"D:/docs", source:"docs"}.

Yandex.Disk

Log in via OAuth (see "Yandex.Disk"), then index {operation:"connect", ...} and sync. Disk files are downloaded and text is extracted (text — inline, binary — via an external converter), then signatures are built.

Operations `index_tool`

Operation	Arguments	What it does
`sources`	—	list of sources and state (`вкл`/`выкл`) + `allowed_roots`
`add_folder`	`path`, `source`	register a server folder
`remove`	`source`	delete a source with all documents
`enable` / `disable`	`source`	pause/resume without deleting
`connect`	`source`, `code?`	Yandex.Disk OAuth (see below)
`sync` / `build`	`source`	start indexing (in the background)
`status`	—	`Документов: N; единиц: M` + `allowed_roots`
`search`	`query`, `limit?`	two-stage search (owner = user)
`open`	`uri`	full text of a document/unit

Hook `rag` over the index

kind: rag
name: docs_rag
index_db: data/plugins/index.db   # читать индекс, а не архив
top_k: 5
notice: "Найдено {{count}} фрагментов из базы документов."

If index_db is set, the hook performs a two-stage search over the index and strictly restricts the selection by owner (user:<ctx.user_id>). Without index_db — the previous mode over the BM25 archive.

Yandex.Disk: login

Create an OAuth app at https://oauth.yandex.ru (permissions Yandex.Disk REST API: cloud_api:disk.read/write/info), put into .env YANDEX_OAUTH_CLIENT_ID and YANDEX_OAUTH_CLIENT_SECRET.
Register a Redirect URI (https; for localhost http://127.0.0.1:3000/v1/yandex/callback is accepted), set the same one in the tool's oauth.redirect_uri.
index {operation:"connect", source:"my_disk", tool:"yandex_disk_idx"} — a link will be returned; confirm — the server will exchange the code for a token itself (GET /v1/yandex/callback, without copying the code).
index {operation:"sync", source:"my_disk"} → status.

Per-user token: oauth.token_file supports {user} — a sanitized user_id is substituted, each user has their own refresh token and their own session. Example: data/yandex-oauth/{user}/disk_token.json.

Config

The source location is either edited in server.yaml or set dynamically via add_folder/connect (stored in the DB). Server allow-list:

# server.yaml
security:
  index:
    allowed_roots: []     # абсолютные префиксы; пусто = как разрешит Fs sandbox

Build and deploy

Lean DLL: scripts/build-module-dlls.ps1 (or .sh) → target/modules/<profile>/.
Loading: a line in config/plugins.yaml (plugins: [ … agent_os_index.dll ]).
Distribution: scripts/build_distrib.* builds and places the DLL into plugins/; config-starter/plugins.yaml picks it up.
The module is in the dual-transport list (30-dll-ready-modules.md).

Demo agent

index-demo (config/agents/index-demo/) — index_tool + yandex_disk_idx + hook docs_rag + landing page /ext/index-demo/. The agent's instructions guide onboarding: first ask what to index, then connect a source and synchronize.

Module index — document index (RAG)