docs · how it works

Architecture#

Hekma is one Rust library, the engine, with a thin command-line program on top. The engine launches each agent through an adapter, watches the process, reads its token usage, checks budgets in the same step that records usage, and keeps everything in one state directory on disk. Everything it does happens only while some process holds the engine open. This page explains how those parts fit together, then goes one level deeper on supervision, metering, budgets, state and performance.

The big picture#

Diagram source (Mermaid)
flowchart TD
    CLI["hekma command<br/>(one engine per command)"] --> E
    Host["Your program<br/>(embeds hekma-engine)"] --> E
    E["Engine<br/>supervisor, metering,<br/>budgets, event bus"] --> Ad["Adapters<br/>acp, hermes, mock,<br/>adapter.toml"]
    E --> S[("State directory<br/>state.db, secrets.toml,<br/>Agent Homes")]
    Ad --> P["Agent processes<br/>(one process group each)"]
    P -.->|"usage lines, model traffic"| E
  • The caller is either a hekma command, which opens the engine for the length of one command, or your own program, which embeds the engine and keeps it open as long as it likes. Both use the same public API.
  • The engine holds the fleet's records, runs one supervisor for the agents it has started or adopted, meters their usage, enforces budgets and publishes events.
  • Adapters tell the engine how to launch each kind of agent, what it can do on each OS and where its usage comes from. Built-in kinds are compiled in; any other agent is described by an adapter.toml manifest.
  • Agent processes are ordinary child processes, each in its own process group (Unix) or Job Object (Windows).
  • The state directory holds everything durable, so any later process can pick up where the last one stopped.

Supervision happens only while some process holds the engine open. That one fact explains most of Hekma 0.11.0's behaviour; Supervision and detached runs covers what it means in practice.

Crates#

The source is one Cargo workspace at github.com/Ktesio/hekma:

CrateOn crates.ioWhat it is
hekma0.11.0The CLI: the hekma and hkm binaries. It parses arguments, calls the engine's blocking API and prints results. It holds no business rules.
hekma-engine0.7.0The engine: registry, supervisor, process backends, metering, budgets, config, secrets, events and the SQLite store. Its public API is what embedders use.
hekma-adapter-api0.2.0The adapter contract: capability and metering types, the adapter.toml schema and its validation. It depends on nothing else in the workspace.
hekma-adapters-hermes0.2.0The built-in, deprecated hermes kind, compiled into the engine.
hekma-conformancenot publishedThe conformance kit that tests any adapter against the contract. Use it as a path or git dependency.
hekma-hostnot publishedThe long-running service host behind hekma serve (Unreleased: after 0.11.0). See Run Hekma as a service.

The boundaries are enforced, not just intended. The CLI may depend only on the engine and the adapter contract; a CI check fails the build if it reaches anything else. The engine's internals are private Rust modules, so a program embedding the engine can call exactly what the CLI calls and nothing more. The only OS-specific code lives in two backend modules, one for Unix and one for Windows.

Inside the engine#

Engine::open builds three things:

  • The registry, which owns the SQLite connection and every path in the state directory. Nothing outside the engine computes a path; the CLI asks the engine.
  • The supervisor, which holds a handle for every agent process this engine started or adopted, along with each one's metering state.
  • A tokio runtime, private to that engine. A background task on it ticks every 250 ms: it reads new usage, checks each process is alive, and schedules restarts.

The public methods are async and run their database and file work on the runtime's blocking pool. engine.blocking() wraps each one so a synchronous program can call it; the CLI is such a program and never touches the async API.

Lifecycle#

The lifecycle is a fixed table of states and allowed transitions, checked before any adapter or process is touched, so an invalid command fails the same way for every agent. Two kinds of actor move an agent between states:

EdgeDriven by
registered, stopped or failed to startingA start command, or the restart policy after a crash.
starting to runningThe engine, once the process has launched and survived its first moments (for ACP agents, once the protocol handshake completes).
starting to failedThe engine, when the launch fails or the process exits at once.
running to paused and backA pause or resume command, or a budget breach with the pause action.
running or paused to stopping to stoppedA stop command (or a breach with the stop action) starts it; the engine finishes it when the process is gone.
running to failedThe engine's background tick, when the process exits on its own.

Every transition writes the new state to state.db and appends one JSON line to the agent's instance.log, carrying the cause: the command, a launch error, a crash with its exit detail, a restart with its attempt number, or a budget breach with its numbers. The event bus publishes the same record after it is written; Events, resync and diagnostics explains the ordering.

Supervision on each OS#

All process control goes through one internal interface with a Unix and a Windows implementation.

JobUnix (macOS, Linux)Windows
LaunchEach agent starts in its own session and process group, so signals reach the agent and every child it starts.Each agent is assigned to its own Job Object right after launch.
PauseSIGSTOP to the whole group; SIGCONT to resume. A real freeze.Best effort: the state changes and Hekma prints a note, but the process is not suspended.
Graceful stopSIGTERM to the group.No signal: the agent gets the window to exit by itself.
Forced stopSIGKILL to the group once the window runs out.TerminateJobObject, which kills the agent and its descendants.
Start time for adoption/proc/<pid>/stat on Linux, proc_pidinfo on macOS.GetProcessTimes.

Whether pause freezes the process also depends on the adapter's capability declaration: a best-effort declaration behaves like Windows on every OS, and an unsupported one refuses the command.

The stop sequence#

A stop moves the agent to stopping first, then asks the process to exit and waits for the graceful window (30 seconds by default, --timeout on the CLI, the window argument in the API). If the process is still alive when the window ends, Hekma kills the whole group and records stop-forced instead of stop-graceful. It then waits up to 5 seconds to confirm the process is gone. For an ACP agent with a prompt turn in progress, Hekma sends session/cancel before the sequence starts.

A process stuck in the kernel (for example on a full disk) can ignore even SIGKILL for a while. In that case the agent stays stopping and the stop returns an error; running stop again checks once more without waiting.

Crashes and the restart policy#

The background tick checks every held process. A process that exits on its own while running is a crash, even with exit code 0. The engine records running to failed with the exit detail and applies the restart policy:

  • on-failure (the default) restarts the agent after 1 second, then waits 2, 4 and 8 seconds before the next restarts if it keeps crashing. The fifth crash in a row is final: the agent stays failed with a crash-loop cause and needs a start. Only a start command resets the count, so a crash after hours of clean running still counts toward the five.
  • never leaves the agent failed.

A restart is a new process and a new run. Restarts are timed on the engine's runtime, so they happen only while the engine stays open. A detached agent that crashes between commands is found by the next command, which marks it failed with orphan not found on engine restart and does not restart it.

The engine also refuses to treat a broken process table as a fleet of crashes. If checking one process fails ten times in a row while its neighbours check fine, that process is treated as crashed. If several fail in the same tick, the engine blames the environment, keeps every agent, and prints one diagnostic.

Records and adoption#

Before a launched process counts as supervised, the engine writes a spawn record to state.db: the process id, its start time, the restart policy and count, whether it is detached, and the current run id. A process whose start time cannot be read is killed rather than supervised without one.

Every Engine::open reads those records and checks each against the live process table. If a process with that id exists and has the same start time, the engine adopts it: it holds a handle again, resumes the same run, and supervises it like one it started. If the process is gone, or the id now belongs to a different process, the agent is marked failed with the cause orphan not found on engine restart, so no phantom running row survives. Adoption also checks budgets at once, so an agent already over a limit it has not breached in this run gets its breach action straight away.

What happens to the processes when the engine closes depends on how they were started:

  • Attached agents are killed with the engine: the handle kills the process group (Unix) or the Job Object closes with kill-on-close set (Windows). This is why a plain hekma agent start agent stops when the command exits.
  • Detached agents are left running. Their spawn record carries the detached flag, so every engine that adopts them later also leaves them running when it closes. Before letting go, an engine resumes any detached agent you paused, and leaves an agent paused by a budget breach as it is.

Adopted processes differ from launched ones in two ways. The engine is not their parent, so it cannot read their exit code. And on Windows, an adopted process has no Job Object handle (there is no way to reopen one from a process id), so a forced stop reaches only the agent itself, not processes it started.

Detached runs and run ids#

A run id looks like run-1792697966380900000-0: a timestamp plus a counter, minted when a process is launched. It is stored in the spawn record, stamped on every usage row and breach, and resumed by adoption. A detached agent therefore stays in one run across every command that looks at it, and per-run totals and the once-per-run breach rule carry over. A new run starts only when a new process starts: an operator start or a restart after a crash.

Metering#

Usage reaches the ledger through one function, the only code allowed to write it. Two metering sources feed it, and ACP agents add a third figure that is shown but never billed.

Diagram source (Mermaid)
flowchart TD
    A["Self-reported:<br/>KTESIO_USAGE line in agent.log<br/>(stderr for acp)"] --> I
    B["Engine-observed:<br/>loopback proxy reads the<br/>provider's usage object"] --> I
    I["Write one row to the usage ledger<br/>(replays dropped by run + sequence)"] --> C["Check budgets on the new totals"]
    C -->|under| Ev["Publish usage_update"]
    C -->|crossed| Br["Record the breach,<br/>then pause, stop or warn"]
    Ctx["ACP usage_update:<br/>context window only"] -.->|"shown, never billed"| F["show / list"]

Self-reported: the usage line reader#

An agent's stdout goes straight to logs/agent.log through an OS-level redirect, so the file keeps filling even if the engine dies. On each tick the engine reads what is new in that file, picks out KTESIO_USAGE {...} lines, and records each one. For an ACP agent, whose stdout carries the protocol, it reads agent-stderr.log instead. The read position advances only past lines that were written to the ledger, so a database error means a retry, not a loss. A malformed line is skipped with a diagnostic. Meter token usage has the line format.

When an engine adopts a detached agent, it starts reading at the current end of agent.log. Lines the agent wrote while no engine was open are never counted; adoption says so on stderr.

Engine-observed: the loopback proxy#

For an engine-observed agent, the engine starts an HTTP proxy on 127.0.0.1 and an OS-chosen port when the agent starts. It hands the address to the adapter through the reserved metering.base_url key, and the adapter passes it to the agent as its model base URL. The proxy:

  • forwards each request, headers included, to metering.upstream_base_url over HTTP or HTTPS. HTTPS uses a bundled TLS stack and bundled public root certificates, not the operating system's.
  • buffers each response (up to 64 MiB), passes it back unchanged, and reads the token counts from it: an OpenAI-style usage object or an Anthropic one, including cached tokens. Meter token usage has the field mapping.
  • handles streaming: for a "stream": true request to a chat completions path, it adds stream_options.include_usage so the provider sends a final usage frame, and reads that frame.
  • passes the counts to the engine through an in-memory queue. The engine numbers them itself, since the agent sends no sequence. Bodies, headers and keys are never logged or stored.

The proxy is part of the engine that started the agent and closes with it. That is why start --detach is refused for engine-observed agents, and why an engine-observed agent adopted by a later engine is left pointing at a dead port until you restart it.

ACP context usage#

An ACP agent may send usage_update notifications describing its context window (tokens used out of the window size, and optionally a cost). Those describe the session, not billed tokens, so the engine shows the latest figure in show and the acp_context_usage JSON field and never writes it to the ledger. Billed tokens for an ACP agent come only from usage lines on its stderr, or from the proxy when metering.upstream_base_url is set. With neither, Hekma shows — and a note naming what it tried. See Run an ACP agent.

The ledger#

Each usage event is one row in the usage_events table: agent, run id, sequence, input, output and cached tokens, metering source, time, and the rates in force at that moment. A unique index on agent, run and sequence makes a replayed event a no-op, so a duplicate can never be counted twice. Storing the rate on each row means a later rate change never reprices past usage. Money is kept as integer micro-dollars throughout.

Budget enforcement#

Budgets are checked in the same call that writes a usage row, right after the row commits. There is no separate watcher, so there is no moment when a total is over a limit and nobody has acted.

That call re-reads the agent's current config (so a budget change applies to the next event), totals the run and the agent's lifetime from the ledger, and compares them with each limit: tokens first, then dollars if a rate is set. Reaching a limit counts as crossing it. Each budget fires at most once per run.

On a breach, the engine first appends the breach to breaches.log, so the record exists even if the action fails. Then it carries out the breach action through the normal lifecycle: pause uses the pause path (and respects a best-effort or unsupported declaration), stop uses the stop sequence, and warn does nothing more. The resulting transition carries a budget-exceeded cause with the numbers. If the action cannot be carried out, the engine prints a diagnostic and keeps supervising.

Limits are checked against what the ledger has seen. Usage arrives on the 250 ms tick, so requests already in flight when a limit is crossed still reach the provider.

State on disk#

Everything durable is in the state directory:

PathWhat it holds
state.dbSQLite (bundled, WAL mode): agents and their states, spawn records, memory backings and the usage ledger. 0.11.0 uses schema version 9. Opening an older database upgrades it one version at a time, each step in one transaction; a database from a newer Hekma is refused.
secrets.tomlOptional. Values for secret:NAME references. Must be readable only by you on Unix. See secrets.
agents/NAME/The agent's Agent Home, and its working directory when it runs.
agents/NAME/config.tomlThe agent's own config layer, written by config set (replaced in one step, never half-written).
agents/NAME/adapter.jsonThe adapter as resolved at registration, including its launch command. Editing the manifest later has no effect until you register again.
agents/NAME/effective-config.jsonThe config the last start used, with secrets masked. A record only; editing it changes nothing.
agents/NAME/logs/agent.log (raw stdout), agent-stderr.log (raw stderr), output.log (attributed, timestamped lines from both streams plus engine lines, rotated at 10 MB with two older files kept), instance.log (transitions) and breaches.log (breaches). See retention.
agents/NAME/memory/The memory backing folder, when a filesystem backing is attached.

Each state change is one SQLite transaction in WAL mode with synchronous=NORMAL, so a power cut can lose at most the last transaction or so. After a reboot, every process is gone; the next open marks agents that were running as failed and keeps everything else.

Concurrency#

Inside one engine, two locks protect the registry and the supervisor, and every state change, usage write, budget check and event publish happens while the supervisor lock is held. That is what makes events come out in commit order and keeps the budget check atomic with the usage write. The event bus itself takes no lock to subscribe, and ACP protocol readers run on their own threads and queue their notes for the next tick.

Between processes, the shared state is the SQLite file. Each CLI command is its own process with its own engine, and several can run at once against one state directory. Writes that must not interleave, such as attaching memory while another terminal starts the agent, use exclusive transactions; a command that cannot get the database within 5 seconds fails with a retryable error.

One engine per state directory#

The engine does not coordinate ownership of running processes. Every Engine::open adopts every agent with a live spawn record, including agents another engine is still holding. When that second engine closes, it treats an attached agent it adopted like one it started, and kills it.

So a single hekma agent list can kill an agent that a long-running program holds. In a test against 0.11.0, a program held an attached agent and one hekma agent list in another terminal killed it; the program saw a crash and restarted it under its restart policy. The same happens on Hekma main with hekma serve (Unreleased: after 0.11.0): one hekma agent list against the served directory caused a restart of an agent the host had started. Detached agents survive the second engine, but two long-running engines on one directory both supervise them, and both count and enforce.

Keep one process holding the engine per state directory:

  • Give an embedding program or a host a state directory of its own, and point HEKMA_STATE_DIR somewhere else for your own CLI use.
  • Read a busy directory's fleet through the program that holds it (for a host, its Control API), never with hekma agent commands.
  • hekma serve takes a lock file, but it keeps out only other hosts, never ordinary commands.

Performance budgets#

Hekma holds itself to three performance limits, measured by a benchmark program on a real fleet of 25 registered agents with 10 running, on every change in CI:

MeasureLimit
Fleet reads (list, show, usage and their engine calls)under 1 second at the 99th percentile
CPU used by supervisionat most 2% per running agent
Memory used by supervisionat most 50 MiB per running agent

Reads and memory fail the build if exceeded anywhere. CPU is held strictly on a developer machine and with a 1.5x allowance on shared CI runners, taking the median of three measurement windows. One active event subscriber has its own, much smaller allowance on top.