Caching & statistics¶

Four suites about not paying twice for the same answer, and about trusting a pass rate once you do pay for one. They cover the single rule the cache key follows, busting it at the right granularity, turning one run into a confidence interval, and gating on a regression rather than an absolute score. Read these once a suite is calling a real model and cost or confidence starts to matter.

Example 21 — Caching¶

Every outgoing request is cached, content-addressed, on by default. Run this suite twice and the second run does no work at all.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Not paying twice for the same answer.
#
# Every outgoing request is cached, content-addressed, on by default — a
# provider call, the LLM grader, an embedding, an exec grader alike. Run this
# suite twice and the second run does no work at all: it prints the same report
# from the cache, instantly and for free.
#
# The rule the key follows is one sentence:
#
#     Hash what is sent.
#
# The key is the SHA-256 of the redacted request itself, plus the trial index
# (`--repeat`), plus any `cache_salt` in scope. Nothing about your machine, your
# binary, or your credentials is in it — no path, no mtime, no digest of the
# program's own bytes.
#
# That is what makes a cache shareable. A key that varied by machine could not
# be reused by anyone else, which quietly turns a shared cache into an expensive
# local disk.
#
# Three consequences worth knowing:
#   * one entry per key, immutable — first write wins, on every backend
#   * errors are NEVER cached; only successful responses are
#   * `latency` assertions bypass the cache entirely, because a replayed
#     response has no honest latency to report — and under `--cache-only` such
#     a case is refused outright rather than called live
#
# Run it twice and watch the second run:
#   domarinn run examples/21-caching-basics
#   domarinn run examples/21-caching-basics     # every cell a cache hit
#
# To force real work:      --no-cache
# To re-ask only the grader: --no-grader-cache
# To require the cache:    --cache-only   (a miss becomes an infra error, exit 3)
version: 1
project: examples
suite: caching-basics

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    # An exec provider's key hashes what it SENDS — the command, the args, the
    # document on the child's stdin — and says nothing about the program's
    # bytes, so a rebuild does NOT invalidate anything on its own. This salt is
    # the version pin that does. See example 22.
    cache_salt: "dev"

cache:
  # `disk` (the default) stores under `.domarinn/cache` beside this suite.
  # `layered` puts a shared tier behind that local one — S3 when a `cache.s3`
  # block is set, else the results server. Those two are the whole list;
  # `http` and `s3` still parse as deprecated aliases for `layered`. Note the
  # config names only the KIND — the URL and credentials come from the
  # environment, so a suite is safe to commit.
  backend: disk

tests:
  - id: summary/short
    vars:
      user_input: "The meeting moved to Thursday at 10."
    assert:
      - type: icontains
        value: "thursday"
  - id: summary/long
    vars:
      user_input: "Deployment is blocked on the migration; expect Friday."
    assert:
      - type: icontains
        value: "friday"
  - id: summary/empty-ish
    vars:
      user_input: "noted"
    assert:
      - type: length
        min: 1

The rule the key follows is one sentence: hash what is sent. A provider call, the LLM grader, an embedding and an exec grader are all keyed the same way — the SHA-256 of the redacted request, plus the trial index, plus any cache_salt in scope. Nothing about your machine, your binary, or your credentials.

That is what makes a cache shareable. A key that varied by machine could not be reused by anyone else, which quietly turns a shared cache into an expensive local disk.

Three consequences worth knowing:

One entry per key, immutable. The first write wins, on every backend.
Errors are never cached. Only successful responses are stored.
latency assertions bypass the cache entirely, because a replayed response has no honest latency to report — and under --cache-only such a case is refused rather than called live. cost and tokens come from the stored response.

Note the cache: block names only the kind of backend. The URL and credentials come from the environment, so a suite stays safe to commit. See Caching for the full rule and the shared backends.

Example 22 — Cache salts¶

The key is the request, and a request only carries what domarinn can see. When the system under test loads its own content across a process boundary — prompts from a registry, rules from a database — domarinn never sees that content, so editing it changes nothing about the request and the cache keeps answering with yesterday's results.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Busting the cache on purpose, at the right granularity.
#
# The cache key is the request, and a request only carries what domarinn can
# SEE. When the system under test loads its own content across a process
# boundary — prompts from a registry, rules from a database, a model baked into
# a binary — domarinn never sees that content, so editing it changes nothing
# about the request and the cache keeps answering with yesterday's results.
#
# `cache_salt` is the lever, and it exists at two levels because it is really
# two different problems. A salt joins the key as its own member, only when it
# is set, and is never sent to the provider.
#
# PROVIDER-LEVEL: a coarse "same build?" pin. Bump it when the program itself
# changes behaviour. Do NOT make it a content digest of everything the program
# reads — that throws the whole cache away on any edit, which is the outcome the
# per-case salt exists to avoid.
#
# PER-CASE: a content digest of just the thing THIS case exercises. Edit one
# prompt and only the cases that use it re-run. This is what keeps a large suite
# affordable while staying honest.
#
# `$digest:` computes that digest for you. The glob is rendered against the
# case's own vars, matched files are hashed in sorted order WITH their relative
# paths, and a glob matching nothing is an error rather than an empty digest —
# because an empty digest would silently mean "never bust".
#
# Run:  domarinn run examples/22-cache-salts
version: 1
project: examples
suite: cache-salts

providers:
  - id: prompt-runner
    type: exec
    # The request carries only a prompt_id; the program reads the prompt file
    # itself. domarinn therefore cannot see a prompt edit — hence the salts.
    command: ["python3", "./run-prompt.py"]
    # Coarse version pin. A commit SHA or a release tag is the usual value.
    # Bump when the program's own logic changes.
    cache_salt: "runner-v3"

tests:
  - id: prompts/greeting
    vars:
      prompt_id: greeting
      user_message: "hello there"
    # Digest of just this case's prompt. Editing prompts/greeting.md busts this
    # case and nothing else.
    cache_salt: "$digest: prompts/{{ prompt_id }}.md"
    assert:
      - type: icontains
        value: "welcome"

  - id: prompts/escalation
    vars:
      prompt_id: escalation
      user_message: "this is unacceptable"
    cache_salt: "$digest: prompts/{{ prompt_id }}.md"
    assert:
      - type: icontains
        value: "specialist"

  # A case may digest more than one file — every prompt the flow touches.
  - id: prompts/whole-flow
    vars:
      prompt_id: greeting
      user_message: "hi"
    cache_salt: "$digest: prompts/*.md"
    assert:
      - type: length
        min: 1

cache_salt is the lever, and it exists at two levels because it is really two problems:

Level	What it is	Bump it when
Provider	A coarse "same build?" version pin.	The program's own logic changes.
Per case	A content digest of just what this case exercises.	Never by hand — `$digest:` computes it.

Do not make the provider-level salt a content digest of everything the program reads. That throws the whole cache away on any edit, which is precisely the outcome the per-case salt exists to avoid. The two-level arrangement is what keeps a large suite affordable while staying honest: edit one prompt and only the cases that use it re-run.

$digest: renders its glob against the case's own vars, hashes matched files in sorted order with their relative paths, and treats a glob matching nothing as an error — because an empty digest would silently mean "never bust".

Example 23 — Repeat and confidence¶

A pass rate off one run of twenty cases is a number with no error bar. Models are stochastic, and so is anything built on them: "17/20 passed" and "17/20 passed, 95% CI [0.62, 0.96]" are the same measurement, but only one tells you whether yesterday's 15/20 was a regression or noise.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# How sure are you?
#
# A pass rate off one run of twenty cases is a number with no error bar. Models
# are stochastic; so is anything built on them. "17/20 passed" and "17/20 passed,
# 95% CI [0.62, 0.96]" are the same measurement, but only one of them tells you
# whether yesterday's 15/20 was a regression or noise.
#
# `--repeat N` runs every cell N times. domarinn then reports:
#
#   * WILSON confidence intervals on the pass rate — well-behaved at small N and
#     at rates near 0 or 1, where the normal approximation is simply wrong
#   * PASS@K — did at least one of k attempts succeed, which is the right
#     question for anything with a retry loop in front of it
#   * MCNEMAR significance when diffing two runs — a PAIRED test, because the
#     two runs saw the same cases and treating them as independent samples
#     throws away exactly the information that makes the comparison sharp
#
# The trial index is part of the cache key, so repeats genuinely re-sample
# instead of replaying one cached answer N times.
#
# Run:  domarinn run examples/23-repeat-and-confidence --repeat 5
version: 1
project: examples
suite: repeat-and-confidence

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  - id: stability/greeting
    vars:
      user_input: "Happy to help — what can I do for you?"
    assert:
      - type: icontains
        value: "help"
  - id: stability/refusal
    vars:
      user_input: "That's outside what I can access."
    assert:
      - type: icontains
        value: "outside"

--repeat N runs every cell N times, and the report gains three things:

Wilson confidence intervals on the pass rate — well-behaved at small N and at rates near 0 or 1, where the normal approximation is simply wrong.
pass@k — did at least one of k attempts succeed, which is the right question for anything with a retry loop in front of it.
McNemar significance when diffing two runs — a paired test, because both runs saw the same cases, and treating them as independent samples throws away exactly the information that makes the comparison sharp.

The trial index is part of the cache key, so repeats genuinely re-sample instead of replaying one cached answer N times.

Example 24 — Baselines and diff¶

A pass rate on its own cannot tell you whether a change made things worse. --against compares a run to a baseline cell by cell and gates on regressions — cases that passed before and fail now — so a suite that was 80% green yesterday does not have to be 100% green today to merge.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Is this run worse than the last one?
#
# A pass rate on its own cannot answer that. `--against` compares this run to a
# baseline cell by cell and gates on REGRESSIONS — cases that passed before and
# fail now — rather than on an absolute score, so a suite that was 80% green
# yesterday does not have to be 100% green today to merge.
#
# Three ways to name a baseline, and the difference matters:
#
#   --against latest             the newest run in the LOCAL store
#   --against <run-id>           a specific stored run
#   --against server:baseline    the run pinned on a results server
#
# /// THE TRAP ///
# `--against latest` resolves through a cwd-relative `.domarinn/runs/latest`.
# A fresh CI checkout has no such directory, so it finds nothing, logs a WARNING,
# and lets the job exit 0 on a real regression. It is right for local iteration
# and silently useless in CI. Use `server:baseline` there — and note that "no
# baseline pinned" is also a warning, not a failure, so the gate is only ever as
# good as what is actually pinned.
#
# Baselines are keyed per provider id, so renaming a provider — or changing the
# model inside its `command` — starts its history over.
#
# Run:  domarinn run examples/24-baselines-and-diff
#       domarinn run examples/24-baselines-and-diff --against latest
#       domarinn diff <base-run-id> <head-run-id>
version: 1
project: examples
suite: baselines-and-diff

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  - id: regression/policy-window
    vars:
      user_input: "Returns are accepted within 30 days."
    assert:
      - type: contains
        value: "30 days"
  - id: regression/no-blame
    vars:
      user_input: "Sorry about that — let's get it sorted."
    assert:
      - type: not-icontains
        value: "your fault"

--against latest silently never gates in CI

latest resolves through a cwd-relative .domarinn/runs/latest. A fresh CI checkout has no such directory, so it finds nothing, logs a warning, and lets the job exit 0 on a real regression.

It is right for local iteration and useless in CI. Use --against server:baseline there — and note that "no baseline pinned" is also only a warning, so the gate is never better than what is actually pinned. A stale or partial baseline compares almost nothing and reads as a pass.

Baselines are keyed per provider id, so renaming a provider — or changing the model inside its command — starts its history over.