docs · reference

Adapter contract and conformance kit#

The adapter contract is the agreement between Hekma and every adapter: what an adapter declares, what Hekma guarantees in return, and how the two stay compatible as both change. It is at version 1.0.0. The conformance kit is a Rust library that registers your adapter with a real engine, exercises it, and returns a report you can assert on in your own tests.

The contract#

The contract lives in the hekma-adapter-api crate and has its own version, separate from Hekma's. A manifest adapter writes its terms in adapter.toml; a built-in kind declares the same things in code.

What an adapter declares#

DeclarationManifest sectionWhat Hekma does with it
Kind[adapter] kindShows it in list and stores it with the agent.
Start command[lifecycle.start]Copies it at registration and runs it at every start.
Capabilities per OS[capabilities]Allows, labels or refuses pause, resume and send.
Metering source[metering]Reads usage lines from the agent, or runs a metering proxy.
Config mapping[config]Delivers config keys as environment variables, flags or files at start.
Input channel[interaction]Recorded only; Hekma 0.11.0 always delivers input through stdin.

An adapter must declare at least one supported capability and a metering source, or registration is refused with nothing written.

What Hekma guarantees#

  • One launcher. Hekma starts every agent the same way, in its own process group (a job object on Windows), with the Agent Home as working directory and stdout and stderr captured to files.
  • Stop always ends the process. Stop is SIGTERM to the group, then SIGKILL after the timeout (on Windows, the job is ended after the timeout). No process from the agent's tree is left behind. Stop, pause and resume templates in a manifest are not run in contract 1.0.
  • Capabilities are honoured as declared. A guaranteed pause suspends the process; a best-effort one is labelled as such on the state change; an unsupported one fails before anything changes. An agent without interaction support never gets a stdin pipe, so it cannot hang waiting for input.
  • Usage is never double-counted. Each usage event is keyed by agent, run and sequence; a replay is dropped. Usage the adapter cannot report is left out, never recorded as zero.
  • Config is delivered only where declared. A config key reaches the agent only through a mapping the adapter declared, plus agent.* keys as environment variables. The memory folder path (memory.dir) is offered, never imposed.
  • Secrets stay masked. A secret: value reaches the agent in clear text and appears masked in every Hekma output.

What an adapter promises#

Some duties fall on the adapter author because only the real agent can show whether they hold. The conformance kit cannot test them; review and re-testing at each agent release do.

  • Report usage accurately: one usage line per model call, a sequence that increases within a run, and nothing at all for a call whose usage is unknown.
  • If the agent fills {env:VAR} placeholders in its own config and treats missing variables as empty, make sure each such variable is set before launch.
  • Map the agent's setting that turns off self-update, and say whether the agent has a higher-priority config layer (for example one managed by an organisation) that could override what Hekma passes.
  • Give each agent its own data and config folders inside its Agent Home (for example through HERMES_HOME, or XDG_DATA_HOME and XDG_CONFIG_HOME), and say when a setting you rely on is not documented by the agent's authors.

Versioning#

The rule: a manifest loads only when its contract_version major number matches the contract major this Hekma speaks. Hekma 0.11.0 speaks 1.0.0, so any 1.x.y manifest passes the version check and 0.x or 2.x fails, naming both versions.

Matching majors is required, not sufficient. Within a major the contract only grows: new optional fields, new values. A newer 1.x manifest that uses a field this Hekma does not know still fails, because Hekma rejects unknown fields. Write the oldest 1.x version whose features you use.

The value must be a full X.Y.Z version. Suffixes such as 1.0.0-rc.1 parse, and only the major is compared.

Hekma checks the version at registration. An agent that is already registered starts from the copy it made then, so editing contract_version afterwards has no effect until you remove and register the agent again.

Built-in kinds (acp, hermes, mock) carry no contract version; they ship with the engine and always match it.

Deprecation policy#

Within a contract major, a change that removes or alters something is announced in the release notes and these docs at least one minor release before it takes effect, and the removal happens only in the next major release. Additions need no notice. The same policy covers the CLI's JSON documents and exit codes. The public Rust API of hekma-adapter-api is checked for breaking changes against the 1.0.0 contract on every change to Hekma.

One deprecation is in effect: the hermes kind, in favour of --kind acp with Hermes Agent's hermes-acp command. It keeps working and will not be removed before a major release. See Supported agents.

The conformance kit#

The kit is the hekma-conformance crate. It is not on crates.io, so add it from Git, pinned to the v0.11.0 release commit:

[dev-dependencies]
hekma-conformance = { git = "https://github.com/Ktesio/hekma", rev = "3471f0ef933167c881640bd840ee97321e483dad" }

Pin the commit rather than a branch, so the report format cannot change under you; move the pin when you choose to. The kit pulls in the engine, so the first build compiles Hekma (Rust 1.96.1 or newer).

Run it#

Call the kit from an ordinary test. run_conformance registers your adapter with a fresh engine in a temporary state directory, runs all eight sections, and returns a report:

use std::path::PathBuf;

use hekma_conformance::{run_conformance, TckAdapter};

#[test]
fn my_adapter_is_conformant() {
    let dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("my-adapter");
    let report = run_conformance(&TckAdapter::Manifest(dir));
    assert!(report.is_conformant(), "failures: {:?}", report.failures());
}

TckAdapter::Manifest(dir) takes the directory that holds adapter.toml. TckAdapter::Native("hermes".into()) tests a built-in kind instead. run_mock_conformance(&dir) is a shorthand for the manifest case.

The kit starts your agent for real, several times, so the manifest's start command must work on the machine running the tests. A failed section never stops the run: every section reports.

The kit's own test agent is a small program from the kit crate, fake_agent. The kit looks for it next to your test binary, in target/debug/ (or target/release/). Build it there before running your tests, from a checkout of Hekma at the same commit:

git clone https://github.com/Ktesio/hekma /tmp/hekma-kit
git -C /tmp/hekma-kit checkout 3471f0ef933167c881640bd840ee97321e483dad
CARGO_TARGET_DIR=/tmp/hekma-kit-build cargo build --manifest-path /tmp/hekma-kit/Cargo.toml \
  -p hekma-conformance --bin fake_agent
cp /tmp/hekma-kit-build/debug/fake_agent target/debug/

Add --release to the build and copy from and to release/ if you run your tests with --release. Without the helper in place, the kit tries to build it itself, which fails from outside Hekma's own workspace in 0.11.0, and the sections that need it fail naming the missing binary.

Configure a run#

run_conformance_with takes a TckConfig for CI environments that need a different timeout, their own state directory, or a subset of sections. The defaults match run_conformance.

use std::path::PathBuf;
use std::time::Duration;

use hekma_conformance::{
    run_conformance_with, SectionSelection, TckAdapter, TckConfig, TckSection,
};

#[test]
fn lifecycle_and_config_only() {
    let dir = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("my-adapter");
    let config = TckConfig {
        section_timeout: Duration::from_secs(10),
        sections: SectionSelection::Only(vec![TckSection::Lifecycle, TckSection::ConfigMapping]),
        ..TckConfig::default()
    };
    let report = run_conformance_with(&TckAdapter::Manifest(dir), &config);
    assert!(report.is_conformant(), "failures: {:?}", report.failures());
}
FieldDefaultMeaning
section_timeout30 sHow long each wait inside a section may take. Lower it for a fast CI runner, at the cost of flaky failures on a slow one.
state_roota new temporary directoryUse this directory as the engine's state directory.
sectionsSectionSelection::AllSectionSelection::Only(vec![...]) runs a subset; the report lists only those, in the usual order. An empty list is reported as a failure under the id selection.

A state_root you supply must be empty or not exist yet, and must serve one run only. The kit registers agents under fixed names and writes files at fixed paths, so a reused directory could pass a section on a previous run's files. A directory that is not empty, or cannot be created, fails the run before any section starts, with the reason. The kit leaves the directory's contents for you to inspect.

Sections#

Section idWhat it provesNot applicable when
capability_edgesThe capability declaration Hekma stored matches the one the adapter carries.never
lifecycleYour agent goes from start to running to stopped with the expected state changes. A test agent the kit writes, which exits with code 7, lands failed with a crash cause.never
pausePause and resume behave as declared: a guaranteed pause really freezes a process; a best-effort one carries the best-effort labels.pause is unsupported on this OS, or guaranteed on Windows, where Hekma cannot suspend a process.
config_mappingEvery env mapping in your [config] reaches your agent's environment, and an agent.* key arrives as its own variable. Uses the --dump convention below.The manifest has no [config], or maps only to flags and files; or the adapter is a built-in kind.
metering_self_reportedThree usage batches land as exactly three ledger rows, a replayed batch adds nothing, and fleet totals equal the ledger.The adapter declares engine-observed.
metering_engine_observedThree model calls through the metering proxy land exactly in the ledger.The adapter declares self-reported.
memoryAttach, delivery of the memory folder path through the declared mapping, and detach. With the --dump convention and an env mapping, delivery is proven on your agent itself.The adapter does not map memory.dir.
interactionsend reaches a running agent when interaction is supported, and fails at once when it is declared unsupported.The adapter does not declare interaction for this OS.

The first three sections and config_mapping run your adapter itself. The metering sections, memory and interaction run small test agents the kit writes, declared the same way as yours where it matters, to prove the engine does its part; a failure there never touches your agent. The interaction section always tests stdin, whatever [interaction] channel you declare.

The --dump convention#

To prove config delivery, the kit needs to see your agent's environment. It looks for --dump <path> in your [lifecycle.start] args:

[lifecycle.start]
exec = "/usr/local/bin/my-agent"
args = ["--dump", "config-dump.txt"]

When started with --dump PATH, your agent (or a wrapper script around it) should write its environment to PATH, one env=NAME=VALUE line per variable, soon after start. A relative path is relative to the Agent Home. The kit sets each mapped key to a test value, starts the agent, and waits for the matching lines, such as env=MODEL=tck-model-value and env=TCK_PROBE=verbatim-1. A manifest that has env mappings but no --dump argument fails config_mapping with a reason that says so.

You can keep a test-only copy of the manifest that adds --dump, and point the kit at that copy.

The report#

ConformanceReport is plain data that serializes to JSON:

{
  "schema_version": 1,
  "contract_version": "1.0.0",
  "adapter_kind": "my-adapter",
  "sections": [
    { "section": "capability_edges", "result": { "status": "pass" } },
    { "section": "lifecycle", "result": { "status": "pass" } },
    { "section": "metering_engine_observed", "result": { "status": "not_applicable", "reason": "..." } },
    { "section": "interaction", "result": { "status": "fail", "reason": "..." } }
  ]
}
ItemMeaning
schema_versionThe report format, currently 1. Raised only for a breaking change; new fields can appear without it.
contract_versionThe contract version the engine negotiated.
adapter_kindThe kind under test.
sectionsOne entry per section run, in the order of the table above.
result.statuspass, fail with the first failure's reason, or not_applicable with a reason derived from your declaration.
is_conformant()true when no section failed and at least one ran.
failures()The failed sections as (id, reason) pairs, for assertion messages.
section(id)One section's result, for asserting on a specific section.

A failure before any section can run (an unusable state directory, a manifest that does not register) marks every selected section fail, with a reason that names the stage that failed.

A conformant report shows that the declared, testable parts of the contract hold. It says nothing about the promises only the real agent can keep.