Run Hekma as a service#

Unreleased: after 0.11.0. hekma serve, hekma-host and everything on this page are on Hekma's main branch and ship in the next release; Hekma 0.11.0 does not have them.

hekma serve is one process that holds the engine open for as long as it runs, so every agent in a state directory is supervised all the time: crashes restart, usage is counted and budgets are enforced, with no gaps between commands. Run it under systemd, launchd or a container runtime, stop it with SIGTERM, and it stops each agent cleanly before it exits. This page covers what it does, its flags, the rules it enforces, and recipes for systemd and containers.

What it does#

When it starts, hekma serve:

1. takes a lock on the state directory, so no second host can serve it;
2. opens the engine, which adopts every detached agent still running from earlier;
3. waits, supervising those agents and any that are started while it runs.

On SIGTERM or SIGINT (Ctrl+C) it stops accepting Control API requests, stops every running agent through the normal stop sequence, prints how each one ended, and exits (shutdown has the details, including what happens to paused agents).

While it runs, it prints nothing on stdout. Everything goes to stderr as lines starting with [hekma] :

[hekma] host started; state root /var/lib/hekma — supervising agents until SIGTERM/SIGINT
[hekma] SIGTERM received; stopping running agents (graceful window 5s) ...
[hekma] stopping worker (graceful window 5s) ...
[hekma] worker: stopped gracefully
[hekma] shutdown complete (SIGTERM)

While the host runs, start and stop agents through the Control API. An agent started that way belongs to the host, which supervises it until it stops. The host adopts agents only when it starts, so an agent started from the CLI while the host runs is not supervised by it, and CLI commands can disturb the host's agents (see one host per state directory). This is what changes compared with 0.11.0:

hekma serve and hekma-host#

The same host comes in two forms:

Flags#

hekma serve never prints an update notice. Exit codes: 0 after a shutdown, 1 if it could not start (for example, an unresolvable bearer secret), 2 for a bad flag or a state directory another host already serves.

One host per state directory#

The host holds an operating-system lock on host.lock in the state directory for as long as it runs. A second host on the same directory is refused, naming the first one's process id:

$ hekma serve
✗ the state directory is already served by another host (pid 18918, lock /var/lib/hekma/host.lock) — stop that host first, or serve a different state directory

That exits with code 2. The lock belongs to the running process, not the file: if a host crashes, the next one takes the lock without any cleanup, so never delete host.lock by hand.

The lock keeps out other hosts, but ordinary commands still get in. Every hekma agent command opens its own engine on the same files, and even one hekma agent list stops the agents the host started when it exits; the host then records a crash and restarts them. One engine per state directory explains why.

Commands that change things do more damage. In a test, an agent started with hekma agent start --detach while the host ran was never metered by the host, and the export journal was left unreadable, so the host refused to start again until the journal was renamed. While a host serves a state directory, use the Control API for everything, reads included.

Shutdown#

On SIGTERM or SIGINT, the host:

1. stops the Control API, closing open event streams, so clients see the API go before any agent stops;
2. sends every running agent the stop sequence at once, each with --stop-window;
3. looks again for agents that started during the drain and stops those too;
4. prints one line per agent: stopped gracefully, killed after the window, already gone, or failed to stop (with the reason);
5. exits with code 0.

A second SIGTERM during the drain is ignored with a note; the drain already running finishes. Two things to plan for:

• Exit code 0 does not mean every agent stopped cleanly. The per-agent lines on stderr are the record. A process manager's restart-on-failure setting will not see a failed agent stop.
• Agents that need the full window queue up. The engine runs one forced stop at a time, so in the worst case the drain takes about one window per agent, twice. Give your process manager or container runtime a stop timeout longer than that.
• Paused agents are skipped, then killed. The drain stops only running agents. In a test on main, an agent paused through the API was left out of the drain ("no running agents; nothing to stop") and killed without a graceful stop when the host exited; the next command marked it failed. Resume or stop paused agents before you stop the host.

If the host is killed instead (SIGKILL, out of memory, power loss), no drain runs. On Unix, agents in their own process groups keep running; the next host adopts them and carries on. Usage they reported while no host was running is not counted.

Environment of hosted agents#

Under hekma serve, an agent's process does not inherit the host's environment. It gets:

• the variables its adapter declares: the manifest's [lifecycle.start.env] and config key mappings, and every agent.* key;
• PATH and HOME from the host (on Windows also SystemRoot, SystemDrive, ComSpec, TEMP and TMP), so programs can still start and find other programs.

Nothing else passes: not your shell's exports, credentials or CI variables. In a test, a variable exported in the host's shell did not reach the agent, while a manifest variable and an agent.* key did. This is always on under serve. Give each agent what it needs explicitly, typically an API key as agent.OPENAI_API_KEY secret:OPENAI_API_KEY (see secrets).

Put the state directory on a volume#

Everything the host needs is in the state directory: the lock, state.db, secrets.toml, the Agent Homes and their logs, and the outbox journal (outbox.jsonl and outbox-ack.json). Point --state-dir or HEKMA_STATE_DIR at storage that outlives the host, and a replacement host picks up where the last one stopped, including the export position.

Use a local disk, a bind mount or a block device. Network file shares such as NFS or EFS do not reliably support the file locks and SQLite needs, and two hosts could end up serving one directory.

Log capture#

--log-capture chooses, for every agent the host runs, what is kept of the agent's output:

metadata keeps agent output off the disk once agents stop, not while they run. If the host dies before a clean stop, the raw files remain. If output is sensitive, treat the volume as sensitive, or use off and give up self-reported metering.

Run it with systemd#

A unit for a Linux server, with the state on /var/lib/hekma and the Control API on loopback. Create a user hekma first, and put the bearer token in /etc/hekma/env as CONTROL_BEARER=... (readable only by root).

[Unit]
Description=Hekma agent host
After=network-online.target
Wants=network-online.target

[Service]
User=hekma
Environment=HEKMA_STATE_DIR=/var/lib/hekma
EnvironmentFile=/etc/hekma/env
ExecStart=/usr/local/bin/hekma serve --stop-window 30 --listen 127.0.0.1:8477 --bearer-secret CONTROL_BEARER
KillSignal=SIGTERM
KillMode=mixed
TimeoutStopSec=120
Restart=on-failure
StateDirectory=hekma

[Install]
WantedBy=multi-user.target

KillMode=mixed sends SIGTERM to the host only, so the host stops the agents itself, and systemd cleans up anything left after TimeoutStopSec. Size TimeoutStopSec well past --stop-window. Restart=on-failure restarts a host that crashes; it does not react to an agent that failed to stop, because the host still exits 0. Read the host's output with journalctl -u hekma. The hekma user's PATH and HOME are what agents receive, so install agent programs where that PATH finds them.

Run it in a container#

The static Linux archive, hekma-<tag>-x86_64-unknown-linux-musl.tar.gz, contains hekma, hkm and hekma-host, linked against nothing but the kernel and carrying their own TLS root certificates. It runs on scratch or distroless images. It is x86_64 only, and the installer does not use it; download it from the release page.

This image fetches and checks the archive, then copies the three binaries into an empty image:

# syntax=docker/dockerfile:1
ARG HEKMA_VERSION=vX.Y.Z

FROM alpine:3.22 AS fetch
ARG HEKMA_VERSION
RUN apk add --no-cache curl
WORKDIR /fetch
RUN set -eux; \
    archive="hekma-${HEKMA_VERSION}-x86_64-unknown-linux-musl.tar.gz"; \
    base="https://github.com/Ktesio/hekma/releases/download/${HEKMA_VERSION}"; \
    curl -fsSLO "${base}/${archive}"; \
    curl -fsSLO "${base}/${archive}.sha256"; \
    sha256sum -c "${archive}.sha256"; \
    mkdir /out; \
    tar -xzf "${archive}" -C /out

FROM scratch
COPY --from=fetch /out/hekma /hekma
COPY --from=fetch /out/hkm /hkm
COPY --from=fetch /out/hekma-host /hekma-host
ENV HEKMA_STATE_DIR=/data
VOLUME /data
STOPSIGNAL SIGTERM
ENTRYPOINT ["/hekma-host"]
CMD ["--listen", "0.0.0.0:8477", "--bearer-secret", "CONTROL_BEARER", "--stop-window", "45"]

Set HEKMA_VERSION to the release you deploy, and pin the alpine image by digest if you need reproducible builds. Run it with a stop timeout longer than the window, the token from the environment, and the API published only on the machine's loopback:

docker build -t hekma-host .
docker run -d --name hekma \
  -v hekma-data:/data \
  -e CONTROL_BEARER="use-a-long-random-value" \
  -p 127.0.0.1:8477:8477 \
  --stop-timeout 90 \
  hekma-host
docker stop hekma    # SIGTERM, drain, exit 0
docker logs hekma    # the per-agent stop lines

Things to know in a container:

• Your agents must run in the image too. scratch has no libc, so it runs only static agents (Go, Rust or musl builds). For Node, Python or anything linked against glibc, use a base image with that runtime and copy the three Hekma binaries in the same way.
• hekma-host can be PID 1. It handles SIGTERM and SIGINT itself, even during startup. It does not reap stray grandchildren of agents, so if your agents start subprocesses, run with docker run --init or tini, which also forward SIGTERM.
• Name resolution on scratch relies on the /etc/resolv.conf and /etc/hosts that Docker and Kubernetes normally mount. Without them, hostnames in metering.upstream_base_url do not resolve.
• The Control API has no TLS and no health route. Binding 0.0.0.0 inside the container is fine (the host warns about it); keep the published port on a trusted network. Every route needs the token, so an unauthenticated health probe gets 401.
• --metering-listen is not needed when hekma-host is the entrypoint: agents are children of the container and end with it, so no engine-observed agent survives a host restart.

Metering inside a container#

Engine-observed metering works in a container as it does on a machine: the agent and its metering proxy share the container's network, so the 127.0.0.1 address the agent is given reaches the proxy. An agent in another container or pod (a sidecar) cannot reach it; it either fails every model call or ignores the address and goes around the meter. Run agents in the same container as the host, or use self-reported metering for them.

A container is not a fence. The proxy counts the traffic an agent sends to it, and an agent that calls its provider directly is not counted. If you need enforcement an agent cannot bypass, route its traffic through a gateway you control and treat Hekma's ledger as the record to reconcile against. Metering agents you don't control explains the reasoning.

Read next#

• Control API: drive the host over HTTP.
• Supervision and detached runs: how supervision works without a host, in 0.11.0.
• Architecture: what happens when several processes share a state directory.