Supervision and detached runs#

Hekma supervises an agent only while some process holds its engine open: a hekma command, or a program that embeds the engine. In 0.11.0 most commands finish at once, so to keep an agent running and governed you start it with --detach and keep one long-running command, such as hekma agent logs NAME --follow, open beside it. This page explains why, shows the patterns that work, and lists what is and is not covered between commands.

What supervision covers#

While a process holds the engine, it does four things for each running agent it holds:

Job	What happens
Crash handling	Notices when the process exits on its own, records the cause, and applies the restart policy.
Metering	Reads the agent's usage as it arrives and writes it to the usage ledger.
Budget enforcement	Checks every new usage event against the agent's budgets and applies the breach action.
Events and logs	Publishes lifecycle, usage and breach events, and attributes output lines to the agent's log.

None of this runs when no process holds the engine. That is the key fact behind every pattern below.

The three ways an agent can run#

Diagram source (Mermaid)

flowchart TD
    A["hekma agent start NAME"] --> A1["Agent stops when<br/>the command exits"]
    D["hekma agent start NAME --detach"] --> D1["Agent keeps running"]
    D1 --> D2["Each later hekma command<br/>supervises it while it runs"]
    D1 --> D3["Between commands:<br/>no metering, no budgets,<br/>no crash handling"]
    W["start --detach, then<br/>hekma agent logs NAME --follow"] --> W1["Supervised for as long<br/>as the follow runs"]

Attached: owned by one command#

hekma agent start my-agent

A plain start launches the agent and returns. The agent belongs to that command, so it is stopped when the command exits, which is almost immediately:

✓ Started Agent Instance my-agent
running
• note: the started process is supervised only for this engine session and stops when this command exits; start it with --detach to keep it running across commands.

The next command finds the process gone and marks the agent failed with the cause orphan not found on engine restart. Attached starts are useful from a program that embeds the engine and stays open; from the CLI, use --detach.

Detached: running between commands#

hekma agent start my-agent --detach

The agent keeps running after the command exits, and Hekma says what that means:

✓ Started Agent Instance my-agent
running
• note: detached: the agent keeps running after this command exits, and the next `hekma` command re-adopts it. Until then it is NOT supervised: no crash detection, no budget enforcement, and no usage/event delivery happen between commands (supervision is command-scoped).

Every later hekma command adopts the agent again (it matches the recorded process id and start time) and supervises it for as long as that command runs. It prints one line on stderr when it does:

[hekma] my-agent: adopted from a previous engine; metering resumes at the current end of the instance logs — any usage the previous engine still held in memory (its park) at its death is not recoverable and goes unaccounted

list and show mark the agent with detached in the Supervision column.

Detached and watched: supervised all the time#

Start the agent detached, then keep one command open that holds the engine:

hekma agent start my-agent --detach
hekma agent logs my-agent --follow

logs --follow streams the agent's output and keeps running until the agent stops or pauses, or you press Ctrl+C. While it runs, the agent is metered and its budgets are enforced live. If a budget with the pause action is crossed, Hekma pauses the agent and the follow ends:

• note: Agent Instance 'my-agent' is paused; no further output until it is resumed.

This is the pattern the Quickstart uses. On a server, run the follow under a process manager (a systemd service, tmux, or similar) so it survives your terminal closing.

What changes between commands#

When no command is running, a detached agent is on its own. In 0.11.0:

	Between commands
The agent's process	Keeps running.
Crash detection and restarts	None. The next command notices the process is gone and marks it `failed`.
Budget enforcement	None.
Usage	Not counted. The next command starts reading from the end of the agent's log, so usage reported in the gap is never added to the ledger.
Output	Still written to the agent's raw log, `agent.log`. A later `logs --follow` streams only lines written after it starts, with a note that their timestamps are when they were read; read `agent.log` for the gap.

Plan around the gap: if an agent's spend matters, keep a supervising command or program running for as long as the agent runs.

Runs, pauses and breaches across commands#

A detached agent stays in one run for its whole life, from start --detach until a real stop or crash. Per-run totals keep adding up across commands, and a breach that already fired in this run is not fired again when the next command adopts the agent. A new start, or a restart after a crash, opens a new run.

Pauses behave differently depending on who paused the agent:

Paused by	What happens when the command exits
You, with `hekma agent pause`	Hekma resumes the agent before letting go of it, so a paused process is never left frozen with nobody supervising it. The pause lasts only as long as the command that holds the engine.
A budget breach with the `pause` action	The agent stays paused between commands, so it cannot spend past its budget while nobody is watching. Resume it with `hekma agent resume NAME`, or stop it.

Limits of detached agents#

Two kinds of agent cannot run detached in 0.11.0:

Engine-observed agents. Their metering proxy lives inside the command that started them, so start --detach is refused with exit code 5 before anything changes. Start them from a program that embeds the engine. See Meter token usage.
Agents that read stdin, including ACP agents. A detached agent starts with stdin closed. A program that waits for input exits at once and the agent ends failed, and hekma agent send has no input channel to write to. ACP agents talk to Hekma over stdin and stdout, so they need an embedding program too. See Run an ACP agent.

Always-on supervision#

For agents that must be supervised around the clock, use a process that holds the engine for as long as the agents run:

Embed the engine in a small Rust program. It can start agents attached, send them input, run ACP turns, subscribe to events and enforce budgets continuously. See Embed the engine.
hekma serve (Unreleased: after 0.11.0) is a long-running host on Hekma's main branch that adopts every agent in a state directory and supervises them until it is told to stop. See Run Hekma as a service.

One supervisor per state directory#

Every hekma command opens its own engine and adopts every running agent it finds, including agents another program is holding. When it exits, it stops any attached agent it adopted, as if it had started it. Detached agents are left running. So if a program that embeds the engine holds attached agents, do not run hekma commands against its state directory: even hekma agent list stops those agents, and the program sees a crash. Give such a program a state directory of its own (HEKMA_STATE_DIR), and read its fleet through the program.