Telemetry export (OTLP)#

Unreleased: after 0.11.0. The OTLP exporter is part of hekma serve, which is on Hekma's main branch and ships in the next release; Hekma 0.11.0 does not have it.

hekma serve can export its governance metrics — token usage, budget breaches, derived cost, and lifecycle state — to the observability stack you already run (Grafana, Datadog, or any OpenTelemetry-compatible collector) over OTLP/HTTP+JSON. The exporter lives inside the host process: it subscribes read-only to the engine's committed-event stream and POSTs metric batches to a collector you name.

The privacy default is structural, not a toggle:

With no --otlp-endpoint, the exporter does not exist. No task is spawned, no socket is dialed, nothing leaves the machine. This is asserted by test, not promised by documentation.

The export is metrics-only. No prompts, no responses, no agent content, and no secrets ever ride a payload — the attribute vocabulary is closed: instance and run identifiers, budget words, state names, estimate labels.

Configuration#

hekma serve --otlp-endpoint http://127.0.0.1:4318
hekma serve --otlp-endpoint http://127.0.0.1:4318 \
  --otlp-headers authorization=secret:OTLP_AUTH \
  --otlp-flush-interval 30

Flag	Meaning
`--otlp-endpoint <url>`	The collector's base URL. http:// only — see the https deferral. Every flush POSTs to `<endpoint>/v1/metrics`; pass the base URL, the exporter appends the path. A URL with a query string or fragment, a non-numeric port, or a base already ending in `/v1/metrics` is refused as a usage error before anything starts.
`--otlp-headers <name=value,...>`	Headers attached to every POST (auth and friends). A value may be a `secret:NAME` secret — resolved once at startup from the environment, then the state directory's `secrets.toml`; values are never logged, and a reference that cannot be resolved refuses the startup, naming the secret. Control characters in a name or value, and duplicate names (case-insensitive), are refused at parse time.
`--otlp-flush-interval <secs>`	The flush cadence in whole seconds, 1 to 3600. Default 10. Requires `--otlp-endpoint` — the exporter never half-exists.

The same three flags exist on the hekma-host container binary (see Run Hekma as a service); the last two each require --otlp-endpoint.

The exposition: `otlp-metrics v1`#

The metric surface is versioned — the marker is otlp-metrics v1, which also rides the payload's instrumentation scope. Until the surface is frozen, changes are additive only; a rename or a drop is an announced release-surface change.

The wire format is the standard opentelemetry-proto JSON mapping: lowerCamelCase keys (resourceMetrics, timeUnixNano, asInt), 64-bit integers as decimal strings, doubles as JSON numbers, a numeric aggregationTemporality (1 = cumulative) on sums, and one ExportMetricsServiceRequest body per POST with Content-Type: application/json. Resource attributes: service.name=hekma-host, service.version, and service.instance.id — a stable identifier derived from the state directory, never the raw path.

Metric families#

Name	Type	Unit	Attributes	Notes
`gen_ai.client.token_usage`	histogram	`{token}`	`gen_ai.token.type` (`input`/`output`/`cached`), `hekma.instance`, `hekma.run_id`	One observation per committed usage event per known token type; `input` is inclusive of the cached subset. An unknown cached count yields no `cached` observation — never a fabricated zero. Fixed explicit bounds: 1 000, 10 000, 100 000, 1 000 000, 10 000 000 tokens (six buckets). The run series are bounded to the 64 most-recently-observed runs per instance — older run series leave the live payload (one `[hekma]` diagnostic per flush cycle names the count when evictions happen), but this is a window, not a drop: `hekma.otlp.dropped` never grows from it, and the durable ledger keeps every run.
`hekma.budget.breaches`	sum (cumulative, monotonic)	`{breach}`	`hekma.instance`, `hekma.budget.dimension` (`tokens`/`dollars`), `hekma.budget.scope` (`per-run`/`cumulative`), `hekma.budget.action` (`pause`/`stop`/`warn`)	+1 per recorded breach, regardless of the action taken.
`hekma.usage.cost`	sum (cumulative, monotonic)	`USD`	`hekma.instance`, `hekma.estimate_state` (mandatory)	The derived cost as `asDouble` (integer micro-dollars ÷ 1e6). See the estimate honesty rule and the cost-series limitation.
`hekma.agent.state`	gauge	`{state}`	`hekma.instance`, `hekma.state`	`1` = the instance is in that lifecycle state; `0` = it left it.
`hekma.otlp.dropped`	sum (cumulative, monotonic)	`{event}`	—	The exporter's own drop counter — always emitted, even at zero. Counts engine events lost to bus lag or queue overflow, delivery batches a terminal failure dropped, and points a `200 partialSuccess` rejected.

gen_ai.client.operation_duration is deliberately absent: no per-call latency producer exists yet, and the exposition never fabricates a metric. It joins additively when a producer lands.

What v1 honestly does not see#

Cost points come only from cumulative dollar breaches#

hekma.usage.cost data points come only from cumulative-scope dollar breaches — the moment the budget system records that an instance's lifetime derived cost crossed its cap. A per-run dollar breach trips hekma.budget.breaches but feeds no cost point (its observed total is a run window, not the lifetime cumulative), and a dollar breach without an observed total yields no point either — never a fabricated 0.0 series. The honest consequence: an instance that has never breached its dollar cap exports no cost series at all, even while its token usage streams. Cost points from ordinary usage events (not just breaches) need an engine payload extension; that is a named deferral.

Two windows are unobserved by design#

The exporter observes the committed-event stream from exporter start to exporter stop. Two windows never enter the queue, and hekma.otlp.dropped never counts them — they are not drops:

Adoption-time events before the exporter starts. When the host opens a state directory that already has running agents, the engine's adoption events (and anything committed before serve started the exporter) are not part of the export; the cumulative series open at exporter start.
The shutdown-drain window. After the exporter's bounded final flush, the drain's own stop-ladder events are not exported — the host is already exiting.

The durable ledger remains the lossless record for both windows.

The estimate honesty rule#

Every exported dollar figure carries its hekma.estimate_state attribute — estimated (derived from self-reported tokens at a configured rate) or reconciled (provider-confirmed actuals, once reconciliation lands). The rule is structural: a reconciled figure is a new time series, never a replacement of the estimated one — dashboards can watch both, and no code path can export an unlabeled dollar. Hekma's dollar figures are estimates by default; see Budgets and cost.

Restart-reset semantics#

All sums and histograms are cumulative from process start. When the host restarts, cumulative series restart from zero: counters undercount across a crash window (delivery is at-most-once there — the event bus's documented crash semantics), and the durable ledger remains the lossless record. The export surface is a live dashboard feed, not a billing source; ledger-sourced re-derivation is a named future direction.

Failure behavior: visible, never blocking#

The exporter never delays supervision or enforcement:

Bounded POSTs. Every delivery attempt carries an explicit timeout; the export path cannot hang the host.
Bounded retries. 429/503/504 responses and attempt timeouts are retried within a small fixed budget (three attempts, short backoff). Everything else — connection refused, any other status — is terminal for that batch.
Every loss is visible. A dropped batch, a bus lag, or a queue overflow surfaces a [hekma] diagnostic on the host's stderr and grows hekma.otlp.dropped; a silent drop is a design bug, not a mode. Under a sustained outage the overflow diagnostic is rate-limited to once per flush cycle (the counter stays exact), and a 200 whose partialSuccess reports rejected points is surfaced too — the rejection count and the collector's message are named, and the points join the dropped counter.
The queue is bounded. Under a collector outage the exporter drops the OLDEST events (flat, known memory ceiling) and says so.
Shutdown is bounded. On the shutdown signal the exporter stops before/with the Control API, attempts one bounded final flush, and never holds the host's exit past that bound. In-flight points the final flush could not deliver are named on stderr; cumulative sums reset at restart (see restart-reset semantics).
No durable telemetry. The exporter keeps no files and no cursors; the Usage Ledger's outbox remains the only durable export surface.

The https deferral#

v1 accepts http:// endpoints only. That is a documented deferral, not an oversight: the tested surface must equal the shipped surface, and TLS handshakes are not provably testable in this repository's hermetic suite. The standard bridge — an OpenTelemetry Collector (or any local forwarding collector) accepting http:// on loopback and exporting TLS upstream — works today and is the recommended posture. The deferral reopens when an operator names an https collector as a real need.

Point Hekma at your collector#

Hekma is the client: it POSTs to your collector's OTLP/HTTP receiver, and nothing is received by the host. Set --otlp-endpoint to the collector's OTLP/HTTP receiver URL — for Grafana Alloy or the OpenTelemetry Collector that is typically http://127.0.0.1:4318; the Datadog Agent's OTLP ingest (/v1/metrics) accepts the same payload. The metric families above then appear under their names verbatim.