Share a cache across a team

The problem. Five engineers and CI all run the same behavioural suite. Every one of them pays separately for identical answers, and a full run is slow enough that people stop doing it before merging.

The shape. A content-addressed cache with a shared tier behind the local disk. The first person to ask a question pays; everyone else replays.

Why this works at all

A key is the SHA-256 of the request and nothing else — one rule, documented once. That is the whole reason a shared cache is possible.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Not paying twice for the same answer.
#
# Every outgoing request is cached, content-addressed, on by default — a
# provider call, the LLM grader, an embedding, an exec grader alike. Run this
# suite twice and the second run does no work at all: it prints the same report
# from the cache, instantly and for free.
#
# The rule the key follows is one sentence:
#
#     Hash what is sent.
#
# The key is the SHA-256 of the redacted request itself, plus the trial index
# (`--repeat`), plus any `cache_salt` in scope. Nothing about your machine, your
# binary, or your credentials is in it — no path, no mtime, no digest of the
# program's own bytes.
#
# That is what makes a cache shareable. A key that varied by machine could not
# be reused by anyone else, which quietly turns a shared cache into an expensive
# local disk.
#
# Three consequences worth knowing:
#   * one entry per key, immutable — first write wins, on every backend
#   * errors are NEVER cached; only successful responses are
#   * `latency` assertions bypass the cache entirely, because a replayed
#     response has no honest latency to report — and under `--cache-only` such
#     a case is refused outright rather than called live
#
# Run it twice and watch the second run:
#   domarinn run examples/21-caching-basics
#   domarinn run examples/21-caching-basics     # every cell a cache hit
#
# To force real work:      --no-cache
# To re-ask only the grader: --no-grader-cache
# To require the cache:    --cache-only   (a miss becomes an infra error, exit 3)
version: 1
project: examples
suite: caching-basics

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    # An exec provider's key hashes what it SENDS — the command, the args, the
    # document on the child's stdin — and says nothing about the program's
    # bytes, so a rebuild does NOT invalidate anything on its own. This salt is
    # the version pin that does. See example 22.
    cache_salt: "dev"

cache:
  # `disk` (the default) stores under `.domarinn/cache` beside this suite.
  # `layered` puts a shared tier behind that local one — S3 when a `cache.s3`
  # block is set, else the results server. Those two are the whole list;
  # `http` and `s3` still parse as deprecated aliases for `layered`. Note the
  # config names only the KIND — the URL and credentials come from the
  # environment, so a suite is safe to commit.
  backend: disk

tests:
  - id: summary/short
    vars:
      user_input: "The meeting moved to Thursday at 10."
    assert:
      - type: icontains
        value: "thursday"
  - id: summary/long
    vars:
      user_input: "Deployment is blocked on the migration; expect Friday."
    assert:
      - type: icontains
        value: "friday"
  - id: summary/empty-ish
    vars:
      user_input: "noted"
    assert:
      - type: length
        min: 1

1. Pick a backend

backend	Shared tier	Use when
disk	none	Solo work. The default.
layered	S3 when cache.s3 is set, else the results server's /api/v1/cache	Anything shared.

Those are the two. http and s3 still parse as deprecated aliases for layered and warn at startup, but they name one tier outright instead of letting cache.s3 choose — so backend: s3 with no cache.s3 block degrades to local disk alone rather than falling back to the server. See caching.md.

A remote always keeps the local tier in front, so a warm local hit never touches the network.

The config names only the kind

cache:
  backend: layered

No URL, no credentials — those come from the environment (DOMARINN_SERVER_URL, DOMARINN_TOKEN, or the AWS credential chain). That is what makes a suite safe to commit, and it means the same file works locally and in CI with no branching.

If the credentials are missing, domarinn falls back to local disk with a warning rather than failing the run. Convenient, and worth knowing about: a misconfigured CI job will look like it is working while paying full price. Check for the warning.

Uploading a run is the other way round. The cache degrades because a cold cache still answers the question, only slower and dearer. domarinn run --share has no such fallback — the results have exactly one destination — so a failed upload fails the run with exit 3, and a server whose result-schema window excludes the CLI — or no server URL configured at all — is refused with exit 2 before the suite spends anything. Pass --allow-share-failure (the action's allow-share-failure input) where publishing is genuinely optional. See Uploading CI runs.

What has to match across environments

For sharing to hold, keep whatever changes the request identical everywhere: a different model, endpoint, params, header value or cache_salt is a different request, and therefore a different entry nobody else hits. Cosmetic differences do not count — a base_url with and without a trailing slash names one endpoint and keys one way.

Nothing about the environment has to match. Different checkout paths, different file timestamps, a binary rebuilt from the same commit, a different working directory, unrelated exported variables, two teammates holding different API keys — none of these move a key, by construction. That is what makes a shared backend worth having, and crates/domarinn-core/tests/cache_portability.rs pins each one so it stays true.

2. Salt at the right granularity

This is the part that decides whether a shared cache stays useful or gets thrown away weekly.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Busting the cache on purpose, at the right granularity.
#
# The cache key is the request, and a request only carries what domarinn can
# SEE. When the system under test loads its own content across a process
# boundary — prompts from a registry, rules from a database, a model baked into
# a binary — domarinn never sees that content, so editing it changes nothing
# about the request and the cache keeps answering with yesterday's results.
#
# `cache_salt` is the lever, and it exists at two levels because it is really
# two different problems. A salt joins the key as its own member, only when it
# is set, and is never sent to the provider.
#
# PROVIDER-LEVEL: a coarse "same build?" pin. Bump it when the program itself
# changes behaviour. Do NOT make it a content digest of everything the program
# reads — that throws the whole cache away on any edit, which is the outcome the
# per-case salt exists to avoid.
#
# PER-CASE: a content digest of just the thing THIS case exercises. Edit one
# prompt and only the cases that use it re-run. This is what keeps a large suite
# affordable while staying honest.
#
# `$digest:` computes that digest for you. The glob is rendered against the
# case's own vars, matched files are hashed in sorted order WITH their relative
# paths, and a glob matching nothing is an error rather than an empty digest —
# because an empty digest would silently mean "never bust".
#
# Run:  domarinn run examples/22-cache-salts
version: 1
project: examples
suite: cache-salts

providers:
  - id: prompt-runner
    type: exec
    # The request carries only a prompt_id; the program reads the prompt file
    # itself. domarinn therefore cannot see a prompt edit — hence the salts.
    command: ["python3", "./run-prompt.py"]
    # Coarse version pin. A commit SHA or a release tag is the usual value.
    # Bump when the program's own logic changes.
    cache_salt: "runner-v3"

tests:
  - id: prompts/greeting
    vars:
      prompt_id: greeting
      user_message: "hello there"
    # Digest of just this case's prompt. Editing prompts/greeting.md busts this
    # case and nothing else.
    cache_salt: "$digest: prompts/{{ prompt_id }}.md"
    assert:
      - type: icontains
        value: "welcome"

  - id: prompts/escalation
    vars:
      prompt_id: escalation
      user_message: "this is unacceptable"
    cache_salt: "$digest: prompts/{{ prompt_id }}.md"
    assert:
      - type: icontains
        value: "specialist"

  # A case may digest more than one file — every prompt the flow touches.
  - id: prompts/whole-flow
    vars:
      prompt_id: greeting
      user_message: "hi"
    cache_salt: "$digest: prompts/*.md"
    assert:
      - type: length
        min: 1

Two levels, two different jobs:

• Provider-level cache_salt — a coarse "same build?" pin. Bump it when the program's own logic changes. A commit SHA or a release tag.
• Per-case cache_salt: "$digest: …" — a content digest of just what this case exercises.

Do not make the provider-level salt a content digest of everything your program reads. It works, and it discards the entire cache on any edit — which is exactly the outcome the per-case salt exists to prevent. With both in place, editing one prompt re-runs the handful of cases that use it and replays the rest.

The theory — why a salt is a version pin rather than an entry ticket, and how $digest: resolves — is in caching.md.

3. Know what is and is not cached

• One entry per key, immutable. First write wins, on every backend — so concurrent writers are race-free by construction.
• Errors are never cached. Only successful responses.
• Grader calls are cached too, as requests like any other: the grader's HTTP call, an embedding, an exec grader's round-trip. A warm run re-pays neither the provider nor the grader. --no-grader-cache re-grades while still replaying provider responses, which is what you want while iterating on a rubric.
• A threshold is not in the key. It is applied on read, so editing a threshold re-scores instantly instead of re-paying the grader.
• Pricing is not in the key either. cost_usd is recomputed on every hit, so correcting a rate re-prices history rather than discarding it.
• latency assertions bypass the cache entirely — a replayed response has no honest latency — and under --cache-only the case is refused rather than called live.

The full list is in caching.md.

4. Verify it is actually shared

$ domarinn run eval/behavioral.yaml
$ domarinn run eval/behavioral.yaml        # second run

The second run should report every cell as a cache hit. If it does not, something in the suite varies between runs — a now() in a var, a $digest: glob matching a file that is being rewritten, or a salt containing a timestamp.

$ domarinn cache stats eval/
$ domarinn cache gc --older-than 30d eval/

See also

• Caching — the full key semantics and backend details.
• Example 21 and 22.
• Server — running the shared tier.