CI integration¶

domarinn is built to gate pull requests: it runs an eval suite, compares against a baseline, writes machine-readable reports, and — crucially — returns an exit code that distinguishes "the model regressed" from "the harness broke". This page covers the exit-code contract, the reusable GitHub Action, a worked walkthrough of gating on a regression, gating a PR by hand, and uploading CI runs to a shared server.

The exit-code contract
The reusable action
The ci-summary command
Gate a pull request on regressions
Gating a PR by hand
Uploading CI runs to a shared server

The exit-code contract¶

Every domarinn run exits with a code your CI can branch on. 3 (infra) wins over 1 (assertion) when both happen in one run — a broken harness must never masquerade as a passing eval, nor should it be blamed on the PR author.

Code	Name	Meaning	CI reaction
`0`	OK	Everything passed.	Merge.
`1`	assertion	An assertion failed, or a run regressed vs its baseline.	Block the PR (the model got worse).
`2`	config/usage	Bad config, bad flags, a suite that won't load, or a `--share` preflight refusal (the server's result-schema window excludes this CLI), raised before the run spends anything. Also a case that errored because of the suite — an assertion kind with no grader, a prompt that won't render, an assert that can't be evaluated.	Fix the suite/workflow, or upgrade the side the message names.
`3`	infra	A provider crashed, the grader broke or could not be reached, the server was unreachable, an internal error, or a `run --share` upload that failed without `--allow-share-failure`.	Retry / page an operator — not the PR's fault.

A run that both failed assertions and failed to upload exits 3: the results are graded but unpublished, and 3 beating 1 is what keeps a broken harness from being blamed on the PR author.

An errored case never exits 1, and which of 2 and 3 it takes depends on its error class — see the class table in cli.md. Both fail the check, so this is not a way for a broken run to slip through; it is how the job's error annotation names the right owner. The action interpolates the breakdown into that annotation, so a failed job reads domarinn infrastructure error (exit 3) — grader_failed × 2 rather than leaving the cause to be dug out of an artifact.

This is the same contract the CLI documents in cli.md.

The reusable action¶

A composite action lives at .github/actions/domarinn-eval/action.yml. It resolves a domarinn binary, runs your suite, uploads the JUnit report + Markdown summary, posts (or updates) a single PR comment, and gates the job on the CLI's exit code.

Minimal workflow¶

name: eval
on: pull_request

permissions:
  contents: read
  pull-requests: write   # required for the PR comment

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: AtvikSecurity/domarinn/.github/actions/domarinn-eval@0.1.0
        with:
          config: eval/domarinn.yaml
          against: server:baseline

Inputs¶

Input	Default	Purpose
`config`	`domarinn.yaml`	Suite file, or a directory containing `domarinn.yaml`.
`binary-path`	`""`	Path to a prebuilt `domarinn`. When set, binary resolution stops here (no download, no build).
`version`	`latest`	Release tag to download the binary from (e.g. `0.1.0`), or `latest`. Used only when `binary-path` is empty.
`server-url`	`""`	Results server base URL. When set, the run is uploaded with `--share` (exported as `DOMARINN_SERVER_URL`).
`token`	`""`	Bearer token for the results server (exported as `DOMARINN_TOKEN`). Pass a secret, never a literal.
`allow-share-failure`	`"false"`	If `true`, a failed upload no longer fails the job (becomes `--allow-share-failure`). Off by default: with `server-url` set, publishing the run is the point of the step. Ignored without `server-url`. See Uploading CI runs.
`provider`	`""`	Provider id(s) to run, comma- or space-separated; each becomes a repeated `--provider`. Selection only — a selected cell keeps its whole `fallback:` chain. Empty runs the full matrix. Where the intent is "this provider exists only as a fallback target", prefer `fallback_only: true` in the suite, which keeps the cost model reviewable in the file.
`against`	`""`	Baseline to diff against. Use `server:baseline` in CI (the suite's pin on the server — a run or a branch), or `server:branch:<name>` to merge the newest server runs on `<name>` with no pin; `latest` reads the local run store, which a fresh checkout does not have. Also accepts a run id, a `result.json` path, or `none` to override a `baseline:` suite default. Empty means no explicit reference (the suite default, if any, still applies). A regression makes the CLI exit 1; a baseline that cannot be resolved makes it exit 2.
`fail-on-regression`	`"true"`	If `true`, exit 1 (assertion/regression) fails the check. If `false`, exit 1 is a warning only. Exit 2 and 3 always fail.
`comment`	`"true"`	Post/update the summary comment on the PR.
`artifact-name`	`"domarinn-results"`	Name of the uploaded artifact holding `results.xml` + `summary.md`.
`allow-empty`	`"false"`	If `true`, succeed when the run resolves to zero cases (becomes `--allow-empty`). Off by default: a green result over no cells is indistinguishable from a green result over every cell. Turn it on for a sharded matrix where a shard legitimately has no work.
`cache-dir`	`""`	Directory for the local cache — typically the path `actions/cache` restored (becomes `--cache-dir`). Empty uses `.domarinn/cache` beside the suite, which a cache step rarely saves. See Shared cache for CI.
`branch`	`""`	Branch to record on the run (exported as `DOMARINN_BRANCH`). Empty autodetects from the workflow environment, which is what you want on `push` and `pull_request`. Set it where the event's ref is not the branch you mean — a merge-queue or `workflow_run` build. See Uploading CI runs.

Outputs¶

Every count below comes from domarinn ci-summary, which writes them straight to $GITHUB_OUTPUT — the action does not parse a report to produce them.

Output	Meaning
`exit-code`	The CLI's raw exit code (`0`/`1`/`2`/`3`).
`passed`	Number of cases that passed.
`failed`	Number of cases that failed or errored.
`errored`	Number of cases that errored (a subset of `failed`).
`total`	Total number of cases.
`xfailed`	Number of `expect_fail` cases that failed as expected. Not part of `failed` — an expected failure never fails the gate.
`xpassed`	Number of `expect_fail` cases that passed unexpectedly. Not folded into `failed` either (its meaning predates these statuses), but the exit code is `1` whenever this is nonzero.
`pass-rate`	Percentage of cases that passed, one decimal (e.g. `91.7`).
`cost-usd`	Cost of the work this run represents, in USD (empty when unpriced).
`cache-savings-usd`	What the cached cases would have cost. Actual spend is `cost-usd` minus this.
`grader-cost-usd`	What the graders cost, in USD. Not part of `cost-usd`, which is what the systems under test cost.
`cache-read-tokens`	Input tokens served from a provider-side prompt cache.
`cache-write-tokens`	Input tokens written into that cache.
`cache-hit-rate`	Percentage of cases served from the cache, one decimal.
`fallback-cases`	Number of graded cases answered by a `fallback:` provider rather than the one the cell names (`0` when nothing fell back).
`regressed`	Number of tests newly regressed vs the baseline (`0` without `against`).
`run-url`	URL of the uploaded run on the results server (empty without `server-url`).
`summary-path`	Path to the Markdown summary (`summary.md`).
`results-path`	Path to the JUnit report (`results.xml`).

failed keeps its original meaning — failures and errors — so an existing workflow that gates on it is unaffected. errored is the new way to tell the two apart without re-reading the report.

What it does, step by step¶

Resolve the binary. Provided binary-path → download domarinn-<target> from the repo's GitHub Releases (version or latest, arch auto-detected) → fallback to building from source with cargo (cargo install --path crates/domarinn-cli if this repo is checked out, else cargo install --git …). The cargo fallback requires a Rust toolchain on the runner — add dtolnay/rust-toolchain before this action if you rely on it.
Run the suite: domarinn run <config> --format junit --out results.xml, appending a --provider <id> per provider: entry, --against <against>, --cache-dir <cache-dir>, --allow-empty and --share when those inputs are set — plus --allow-share-failure when that input is true and a server-url made it a shared run, since the CLI rejects the flag on its own. It captures the exit code without aborting so the later steps still run.
Summarize — domarinn ci-summary latest --out summary.md writes the Markdown and appends the headline numbers to $GITHUB_OUTPUT itself. If the suite never produced a run (a config error, say), the step warns and writes a placeholder summary rather than leaving the PR comment blank.
Upload results.xml + summary.md as an artifact (always, even on failure), and append the summary to the job's step summary.
Comment on the PR — creates or updates one comment (matched by a hidden  marker) so repeated pushes don't spam the thread. Skipped unless comment: true and the event is a pull_request.
Gate on the exit code: 0 passes; 1 fails only when fail-on-regression is true (otherwise a warning); 2 and 3 always fail. A failed upload arrives here as 3, so with server-url set the way to stop gating on the upload is allow-share-failure, not fail-on-regression.

Advanced usage¶

- uses: AtvikSecurity/domarinn/.github/actions/domarinn-eval@0.1.0
  with:
    config: eval/
    provider: primary            # optional: run only these cells (comma-separated for several)
    against: server:baseline
    fail-on-regression: "true"
    server-url: https://domarinn.example.com   # enables --share
    token: ${{ secrets.DOMARINN_TOKEN }}        # write-scoped token

The `ci-summary` command¶

domarinn ci-summary turns a stored run into the two things a workflow wants: a Markdown report to post, and key=value step outputs to branch on. The action calls it for you; reach for it directly when you hand-roll a pipeline, or on a forge that is not GitHub.

domarinn run eval/ --format junit --out results.xml   # exit code gates the PR
domarinn ci-summary latest --against server:baseline --out summary.md

It reads a persisted run rather than taking one over a pipe, so it can also re-summarize an older run by id — and so a summary step cannot change what the run already decided.

What it emits¶

The Markdown is a verdict line and headline metrics table, then any errored cases, then either a baseline comparison (with --against) or this run's failing cases, then any links:

### domarinn run — content-safety

**Fail** — 1 passed, 2 failed, 1 errored

| Metric | Value |
|---|---|
| Pass rate | 33.3% (95% CI 6.1–79.2%, n=3) |
| Errors | 1 (grader_failed × 1) |
| Cache | 2/3 cases from cache (66.7%) |
| Duration | 4.1s |

#### Errored cases

_These graded nothing — their scores are not evidence either way._

| Test | Provider | Class | Error |
|---|---|---|---|
| summarises-safely | gpt-4.1 | grader_failed | llm-rubric assertion errored: verdict field `pass` should be a boolean but was absent; judge returned {"reasoning":"…","score":0.9} |

#### Failing cases

| Test | Provider | Score | Reason |
|---|---|---|---|
| refuses-pii | gpt-4.1 | 0.00 | contains: output does not contain "I can't help" |

[View run](https://domarinn.example.com/runs/01JD3V9GQ8) · [CI run](https://github.com/acme/widgets/actions/runs/42)

The errored table is always rendered — it is not part of the branch below. An errored case is the absence of a verdict, so a baseline diff has nothing to say about it; folding the two together meant a run whose cases errored on both sides reported **No regressions** and showed no other trace of what broke.

With --against, the failing-case table is replaced by the comparison: a verdict (**Regressions detected** — 2 newly failing or **No regressions**), a line identifying both sides — the baseline's run id, finish time and pass count, or its branch and contributing runs for a branch baseline — a change table, the newly-failing cases with their base → head scores, and their output diffs collapsed in a <details> block.

A few deliberate choices:

Words, not glyphs. The verdict is **Pass** / **Fail** in plain text — no emoji, nothing that renders differently across GitHub, terminals and email notifications.
Rows that carry no information are omitted. No Cost row when nothing was billed, no Retries row when nothing retried, no zero-count rows in the change table. What is printed is what happened.
Cache counts cases, not lookups. cache_misses counts every case not served from the cache, so under --no-cache it equals the case count — there is no lookup total to express a "hit rate" against.
The failing table is capped at 10 rows, then …and N more. The run URL and the JUnit artifact hold the full list; GitHub truncates a huge comment anyway. The errored table is capped the same way.
Errors are separated from failures, and broken down by class. A failure is a judgement about the model; an error means no judgement was reached, so it gets its own table with the class that says whose problem it is rather than a 0.00 score no grader assigned. An errored case with no recorded class is bucketed as unknown, never dropped, so the breakdown always adds up to the errored count.
Table cells are escaped. An assertion reason quoting model output that contains a | would otherwise shred the table.
Both links degrade. View run needs a run uploaded with --share; CI run comes from the run's recorded CI metadata, falling back to the ambient workflow environment. Neither present means no links line at all.

Step outputs¶

With $GITHUB_OUTPUT set (automatic on a runner) or --github-output <file>, it appends — a job's steps share one file, so truncating would discard earlier steps' outputs:

passed=1
failed=2
errored=0
failed-or-errored=2
error-classes=
total=3
pass-rate=33.3
cache-hits=2
cache-misses=1
cost-usd=0.041200
cache-savings-usd=0.018700
grader-cost-usd=0.009400
cache-read-tokens=104000
cache-write-tokens=8200
cache-hit-rate=66.7
regressed=0
run-url=
ci-run-url=https://github.com/acme/widgets/actions/runs/42

Keys are written even when empty, so a workflow referencing one always gets a defined value. error-classes is the breakdown as grader_failed × 2, provider_timeout × 1 — empty for a run that errored nowhere, which is why it shows blank above.

It never gates¶

ci-summary exits 0 for a failing run. The verdict is run's exit code and only that — a summary step that could also fail a job would gate the same failure twice and obscure which one spoke. The single exception is an unresolvable run reference, which exits 2 as a usage error.

Gate a pull request on regressions¶

The problem. A behavioural suite is never 100% green. Gating on an absolute pass rate means either setting the bar so low it catches nothing, or blocking every merge. What you actually want to block is a change that made things worse.

The shape. Compare this run to a stored baseline, cell by cell, and fail on cases that passed before and fail now. Here is the whole thing wired together, using the action and the exit-code contract above.

1. Store runs somewhere CI can reach¶

Regression gating needs a baseline that survives a fresh checkout. Run the results server and point runs at it:

$ export DOMARINN_SERVER_URL=https://evals.example.com
$ export DOMARINN_TOKEN=...
$ domarinn run eval/behavioral.yaml --share

Then pin the baseline for that project/suite. The recommended pin is a branch:

$ domarinn baseline set --branch main

A branch pin auto-tracks: every merge to main that uploads a run advances the effective baseline, with no re-pinning. Resolution merges the newest runs on the branch per case — the newest run that has each case wins — so a filtered or sharded upload cannot silently shrink the gate's coverage; cases it lacked are filled from earlier runs. (Sweeping old runs under retention only thins that merge; it never breaks resolution.)

Alternatively pin a fixed run (domarinn baseline set <run-id>, or the run page's Set baseline button in the web UI) when you want the baseline frozen until someone moves it — or skip pinning entirely and pass against: server:branch:main, which resolves the same branch merge with no server-side state at all. A third option is to declare the default in the suite itself with baseline: { branch: main }, so the gate fires with no against: input anywhere.

2. Gate against it¶

- name: Behavioral evaluation
  uses: AtvikSecurity/domarinn/.github/actions/domarinn-eval@<sha>  # keep in lockstep with the version below
  with:
    config: eval/behavioral.yaml
    version: ${{ env.DOMARINN_VERSION }}
    server-url: https://evals.example.com
    token: ${{ secrets.DOMARINN_TOKEN }}
    against: server:baseline
    fail-on-regression: "true"

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Is this run worse than the last one?
#
# A pass rate on its own cannot answer that. `--against` compares this run to a
# baseline cell by cell and gates on REGRESSIONS — cases that passed before and
# fail now — rather than on an absolute score, so a suite that was 80% green
# yesterday does not have to be 100% green today to merge.
#
# Ways to name a baseline, and the difference matters:
#
#   --against latest               the newest run in the LOCAL store
#   --against <run-id>             a specific stored run
#   --against branch:main          the newest LOCAL runs on a branch, merged
#   --against server:baseline      the pin on a results server (run or branch)
#   --against server:branch:main   the same branch merge, no pin needed
#   --against none                 no comparison (overrides a `baseline:` key)
#
# A branch baseline is a MERGE, not one run: per case, the newest run on the
# branch that has it wins, so a `--filter`ed newest run cannot silently shrink
# the gate's coverage. Pin one with `domarinn baseline set --branch main`, or
# declare `baseline: { branch: main }` in the suite to make the comparison the
# default for every run.
#
# /// THE TRAP ///
# `--against latest` resolves through a cwd-relative `.domarinn/runs/latest`.
# A fresh CI checkout has no such directory, so it finds nothing, logs a WARNING,
# and lets the job exit 0 on a real regression. It is right for local iteration
# and silently useless in CI. Use `server:baseline` there — and note that "no
# baseline pinned" is also a warning, not a failure, so the gate is only ever as
# good as what is actually pinned.
#
# Baselines are keyed per provider id, so renaming a provider — or changing the
# model inside its `command` — starts its history over.
#
# Run:  domarinn run examples/24-baselines-and-diff
#       domarinn run examples/24-baselines-and-diff --against latest
#       domarinn diff <base-run-id> <head-run-id>
version: 1
project: examples
suite: baselines-and-diff

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  - id: regression/policy-window
    vars:
      user_input: "Returns are accepted within 30 days."
    assert:
      - type: contains
        value: "30 days"
  - id: regression/no-blame
    vars:
      user_input: "Sorry about that — let's get it sorted."
    assert:
      - type: not-icontains
        value: "your fault"

3. Know exactly what the gate does and does not catch¶

Three ways this gate silently passes

--against latest in CI. It resolves through a cwd-relative .domarinn/runs/latest, which a fresh checkout does not have. It finds nothing, warns, and exits 0 on a real regression. Use server:baseline.

No baseline pinned. Also a warning, not a failure. The gate is only ever as good as what is actually pinned — and a stale or one-case fixed-run baseline compares almost nothing while reading as a pass. A branch pin closes most of this trap: it advances with every merge and back-fills filtered runs from earlier ones, so prefer domarinn baseline set --branch main and reserve fixed-run pins for deliberately frozen baselines.

A version mismatch. The action shells out to domarinn ci-summary. Against an older binary that subcommand does not exist, and the step degrades to a stub comment on a green run. Pin the action and DOMARINN_VERSION as a pair.

Also note baselines are keyed per provider id. Renaming a provider — or changing the model inside its command — starts its history over, silently.

4. Remember which code wins¶

Recall the exit-code contract: 3 beats 1 when both occur in one run, so a run that both regressed and broke reports as broken — the honest answer, since the regression may be an artefact of the breakage.

5. Report it where people look¶

$ domarinn ci-summary latest --out summary.md --github-output "$GITHUB_OUTPUT"

That's the same ci-summary command from above, run against this baseline-gated result — a reporter, never a gate, so a reporting failure never blocks a merge and never fakes one.

Gating a PR by hand¶

The minimal gate is a single run invocation whose exit code you read:

domarinn run \
  --against server:baseline \
  --format junit --out results.xml \
  --summary-md summary.md
# exit 0 pass · 1 fail/regression · 2 config/unresolvable baseline · 3 infra

--against server:baseline diffs this run against the baseline pinned for this suite on the results server and turns a regression into exit 1.

Use this one in CI. --against latest reads the local .domarinn/runs store, which a fresh checkout does not have — so in CI it finds no baseline, compares nothing, and the job passes. It also needs project: and suite: set in your config, since the server pins one baseline per (project, suite).

Pin a baseline with domarinn baseline set --branch main (auto-tracking — the newest runs on the branch merge into the comparison, per case), domarinn baseline set <run-id> (frozen), from a run's page in the web UI, or with PUT /api/v1/projects/{project}/suites/{suite}/baseline.

--against server:branch:main is the pinless server form: the same branch merge, named entirely in the workflow file. Against a server too old to know the route it is exit 2, never a silent skip.
--against latest is the local equivalent: the newest run of the same suite in .domarinn/runs. --against branch:main merges the newest local runs on main. You can also pass a run id or a result.json path.
A baseline that was requested but cannot be resolved is exit 2, not a warning. A gate that silently skips its comparison is not a gate. Only a genuine absence — a suite's first run, or a suite with no baseline pinned yet — is treated as "nothing to compare" and lets the run through.
--format junit --out results.xml writes a JUnit report your CI can render as test results.
--summary-md summary.md writes a Markdown summary suitable for a PR comment or a job step summary — the same document ci-summary produces, which you can also run as a separate step to get step outputs alongside it.

Then gate on the code:

domarinn run --against server:baseline --format junit --out results.xml --summary-md summary.md
code=$?
case "$code" in
  0) echo "all passed" ;;
  1) echo "::error::eval regressed"; exit 1 ;;
  2) echo "::error::bad config";     exit 2 ;;
  3) echo "::error::infra failure";  exit 3 ;;  # retry, don't blame the PR
esac

The reusable action above does exactly this, plus the report upload and PR comment, so you usually don't hand-roll it.

Stable output in CI¶

CI logs are deterministic: domarinn detects that its output is captured (not a terminal) and drops everything cosmetic. The live progress bar is suppressed on a non-TTY stderr (so no carriage returns or redraws clutter the log), and human output is never colored without a terminal. stdout — your json, jsonl, or junit report — is byte-for-byte identical whether or not a terminal is attached. If you ever need to force it, NO_COLOR=1, --color never, and --no-progress all guarantee plain, stable output regardless of environment.

Uploading CI runs to a shared server¶

Point CI at a shared server so every eval is browsable and each PR gets a durable link, and so runs can share a cache of every request they make — provider calls, grader verdicts, embeddings.

server-url / DOMARINN_SERVER_URL — the server base URL. Setting it makes the run upload with --share.
DOMARINN_TOKEN (the action's token input) — a bearer token sent on upload. Unless the server is explicitly in open mode, this needs write scope (a static write: token or an domarinn_ API key). Always pass it from a secret.
--share uploads the completed run and prints View run: <url>; the URL uses the server's DOMARINN_PUBLIC_URL.
allow-share-failure — the opt-out from the paragraph below. Off by default.

A failed upload fails the job. run --share is fail-closed: a rejected upload, an unreachable server, or a server-url that is set but resolves to nothing exits 3, and 3 always fails the check. Exiting 0 having stored nothing is the failure mode worth preventing — the job goes green, the PR gets its comment, and the only symptom is a results server that quietly never fills up, which nobody notices for weeks. The error names the run id, suite and case count, and the JUnit report and Markdown summary are still uploaded as the artifact: the run is graded, so the recovery is domarinn share <run_id> once the server is back, not a re-run that pays for every provider call again.

Set allow-share-failure: "true" where publishing is genuinely optional — a fork's pull request that has no access to the token secret is the usual case — and an upload failure becomes a warning while the exit code goes back to reflecting the assertions alone.

Upgrading from a pre-0.8 workflow: this is a behavior change that arrives without a workflow edit. The action's version input defaults to latest, so an existing workflow with server-url set picks up the fail-closed default the day 0.8 ships — a fork PR that cannot read the token secret, or a transient server outage, turns the job red where it used to print a warning and pass. That red is the feature (those runs were never being stored), but if a workflow genuinely wants best-effort uploads, add allow-share-failure: "true" when upgrading — or pin version until you have.

The schema preflight. With --share, the CLI asks the server what result-schema versions it accepts (one GET /api/v1/meta, 5s timeout) before the runner starts. A confirmed mismatch exits 2 with a message naming which side to upgrade, so a version-skewed pair costs a round trip rather than a full suite of provider calls with nowhere to put the results. Anything less than a confirmed mismatch — unreachable, slow, 404, unparsable, or a server that states no window — proceeds with a warning, because the upload's own response is authoritative and a preflight that inferred refusal from silence would break every run behind a proxy. A missing server URL is refused the same way (exit 2, before the run): with no destination the upload is certain to fail, so running first would only spend the provider budget to prove it. The action never hits that case — it only passes --share when server-url is set. allow-share-failure downgrades every preflight refusal to a warning too.

On GitHub Actions the CLI automatically enriches the uploaded run with git (branch, commit, dirty flag) and CI (provider + run URL) metadata, so shared runs are traceable back to the workflow.

The branch is read from the workflow environment, not from git. actions/checkout leaves a detached HEAD, so git rev-parse --abbrev-ref HEAD answers the literal HEAD on every runner — and on a pull request the checked-out ref is refs/pull/N/merge, which is nobody's branch. domarinn therefore prefers GITHUB_HEAD_REF (the PR's source branch), then GITHUB_REF, and understands the equivalents on GitLab, Buildkite, CircleCI, Travis, Drone/Woodpecker, Bitbucket, Azure Pipelines and Jenkins. A tag build records no branch rather than the tag. Override it with the action's branch input (or DOMARINN_BRANCH) where the event's ref is not the branch you mean — a merge-queue or workflow_run build. This is what makes ?branch= filters, --against comparisons and the server's per-branch retention behave on CI runs.

Shared cache for CI. Multiple CI jobs can share every request domarinn makes — provider responses, grader verdicts, embeddings — through the server's content-addressed cache (/api/v1/cache/*), which cuts cost and time on reruns. The client side is cache.backend: layered plus DOMARINN_SERVER_URL, documented in caching.md.

A key is a hash of the request and nothing else, so a fresh checkout on a fresh runner reuses whatever another job wrote. Two things are worth setting up deliberately:

If the job builds its exec provider from source, pin cache_salt to the commit SHA. The key hashes what the provider sends, not the compiled bytes — which is exactly why two runners can share entries at all, since Rust builds are not byte-reproducible — so nothing else tells domarinn that a rebuild happened. A run warns when it replays answers from a different build.

providers:
  - id: agent
    type: exec
    command: ["./target/release/agent"]
    cache_salt: "${env:GITHUB_SHA}"

If the job restores a cache directory, hand it to the action as cache-dir; it becomes --cache-dir on the run. Leave it out and the cache lands at .domarinn/cache beside the suite, which is rarely the path the cache step saves.

- uses: actions/cache@v4
  with:
    path: .domarinn-cache
    key: domarinn-cache-${{ github.run_id }}
    restore-keys: domarinn-cache-

- uses: AtvikSecurity/domarinn/.github/actions/domarinn-eval@0.1.0
  with:
    config: eval/domarinn.yaml
    cache-dir: .domarinn-cache

The path resolves against the workspace — the same base actions/cache uses. A run-unique key with a shared restore-keys prefix is what lets the cache grow: a key that hits exactly is never saved again, so every case added after the first run would miss forever. Driving the CLI yourself rather than through the action, --cache-dir and DOMARINN_CACHE_DIR set the same thing.

The first CI run after upgrading to 0.5 re-files what a 0.4-era cache holds under the new keys, automatically and once — see Upgrading to 0.5. A job that pins --cache-only should run once in the default read-write mode first, or similar assertions will hard-error: their 0.4 entries are the one shape that cannot be adopted.