Models, grading & budgets¶
Twelve suites about talking to a real model and judging what comes back. They cover the OpenAI-compatible and Anthropic providers (native tool calls included), a plain HTTP service you already run and the shapes output_expr can pull out of it, and a live endpoint of your own — then a structured LLM rubric judged by either vendor, shown the tool calls when the decision is what you are grading, embedding similarity for when many wordings are right, and the budgets that ask whether an answer was affordable, not just correct. One of them is a different kind of suite: every top-level key at once, annotated as a map of the reference. Read these once you are ready to spend real tokens.
Example 26 — An OpenAI-compatible endpoint¶
type: openai speaks the chat-completions API, which is the lingua franca: OpenAI itself, Ollama, vLLM, LiteLLM, OpenRouter and most gateways all accept it. Point base_url at whichever one you have.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Calling an OpenAI-compatible endpoint.
#
# `type: openai` speaks the chat-completions API, which is the lingua franca:
# OpenAI itself, Ollama, vLLM, LiteLLM, OpenRouter, Together, and most gateways
# all accept it. Point `base_url` at whichever one you have.
#
# TWO RULES ABOUT SECRETS, both load-bearing:
#
# * `api_key_env` names the VARIABLE, never the key. The value is read at call
# time and never enters the suite, the cache key, or a shared run.
# * `${env:VAR:-default}` resolves at LOAD time and DOES enter the cache key.
# Use it for things that change the answer — endpoint, model, mode — and
# never for credentials.
#
# Run: domarinn run examples/26-openai-provider
# OPENAI_BASE_URL=http://localhost:11434/v1 domarinn run examples/26-openai-provider
# OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_MODEL=qwen3:4b domarinn run examples/26-openai-provider
#
# The model, like the endpoint, resolves at load time and is part of the
# cache key.
version: 1
project: examples
suite: openai-provider
providers:
- id: gpt
# The label interpolates the same variable as `model`: it exists to name
# the system under test wherever results are displayed, and a run against
# qwen3:4b must not report itself as gpt-4o-mini.
label: "${env:OPENAI_MODEL:-gpt-4o-mini}"
type: openai
model: "${env:OPENAI_MODEL:-gpt-4o-mini}"
# Defaults to the real API, and redirects with one variable — the same one
# the vendor's own SDK honours, so a gateway, a proxy or a local Ollama
# needs no edit to this file.
base_url: "${env:OPENAI_BASE_URL:-https://api.openai.com/v1}"
api_key_env: OPENAI_API_KEY
# Passed through to the API verbatim. temperature 0 is not a guarantee of
# determinism, but it removes the largest source of run-to-run noise.
params:
temperature: 0
max_tokens: 256
prompts:
- id: geography
messages:
- role: system
content: "Answer with a single sentence."
- role: user
content: "What is the capital of {{ country }}?"
tests:
- id: capital/france
vars:
country: "France"
assert:
- type: icontains
value: "Paris"
- id: capital/norway
vars:
country: "Norway"
assert:
- type: icontains
value: "Oslo"
Two rules about secrets, both load-bearing
api_key_env names the variable, never the key. The value is read at call time and never enters the suite, the cache key, or a shared run.
${env:VAR:-default} resolves at load time and does enter the cache key. Use it for things that change the answer — endpoint, model, mode. Never for credentials: keying the value would give every API key its own private cache.
The counterpart is {{ env.VAR }}, which renders per request and is keyed as a literal ${env:NAME} placeholder instead of its value. That is right for a credential and wrong for anything that changes the answer, because two values would share one cache entry and the second would replay the first's responses. domarinn warns when it sees {{ env.X }} in a URL, header or body, because it cannot tell a model selector from a token. It withholds that one hop only — a case var defined as {{ env.SECRET }} is resolved earlier and reaches the request in the clear.
Note the default: https://api.openai.com/v1. Setting OPENAI_BASE_URL — the same variable the vendor's own SDK honours — redirects the whole suite at a gateway or a local Ollama with no edit to the file. That is also exactly how this example is executed in CI, against a stub.
Example 27 — Anthropic, and what it costs¶
Same shape as the OpenAI provider, plus the one thing that deserves its own example: telling domarinn what a call costs.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Calling Anthropic, and telling domarinn what it costs.
#
# Same shape as the OpenAI provider, with two additions worth their own example:
# `pricing`, and what happens when you get it wrong.
#
# domarinn ships a rate sheet for the models it knows. A model it does NOT know
# prices at nothing — and `cost:` assertions then pass as "cost not reported;
# budget not enforced". Green, and enforcing nothing. Whenever you are on a
# negotiated rate, a preview model, or a gateway that rebills, state the price.
#
# Pricing is merged field-wise over the built-in rates and is deliberately NOT
# part of the cache key: `cost_usd` is recomputed on every cache hit from the
# stored token counts and the current rate sheet, so correcting a price re-prices
# history instead of discarding it.
#
# Run: domarinn run examples/27-anthropic-provider
version: 1
project: examples
suite: anthropic-provider
providers:
- id: claude
label: "claude-haiku-4-5"
type: anthropic
model: "claude-haiku-4-5"
# ANTHROPIC_BASE_URL is what the vendor's own tooling honours, so pointing
# this at a gateway needs no edit here.
base_url: "${env:ANTHROPIC_BASE_URL:-https://api.anthropic.com}"
api_key_env: ANTHROPIC_API_KEY
params:
max_tokens: 512
# USD per million tokens. Only the fields you state are overridden.
pricing:
input_per_mtok: 1.00
output_per_mtok: 5.00
cache_read_per_mtok: 0.10
cache_write_per_mtok: 1.25
prompts:
- id: returns-policy
messages:
- role: system
content: "You are a support agent. Cite the policy window when it applies."
- role: user
content: "{{ question }}"
tests:
- id: policy/return-window
vars:
question: "Can I return this after 45 days?"
assert:
- type: contains
value: "30 days"
# Meaningful only because `pricing` above is set and the API reports
# `usage`. Without either, this passes without enforcing anything.
- type: cost
max: 0.05
- type: tokens
max: 2000
domarinn ships a rate sheet for the models it knows. A model it does not know prices at nothing — and a cost: assertion then passes, reporting "cost not reported; budget not enforced". Green, and enforcing nothing. Whenever you are on a negotiated rate, a preview model, or a gateway that rebills, state the price.
Pricing is not in the cache key, on purpose
cost_usd is recomputed on every cache hit from the stored token counts and the current rate sheet. So correcting a price re-prices your history instead of discarding it — which is the behaviour you want the day you discover the rate was wrong.
Pricing is merged field-wise over the built-in rates, so you override only what differs.
Example 28 — A service you already run¶
If your assistant is already behind an HTTP API, type: http is the shortest path from "it exists" to "it is measured" — no SDK, no wrapper process.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Evaluating a service you already run.
#
# `type: http` posts to an endpoint and pulls the answer out of whatever JSON it
# returns. No SDK, no wrapper process — if your assistant is already behind an
# HTTP API, this is the shortest path from "it exists" to "it is measured".
#
# `output_expr` is a minijinja expression over the response, so the provider
# adapts to YOUR response shape rather than the other way round. Four things are
# in scope:
#
# response.status the HTTP status, as an integer
# response.text the raw body string
# response.json the parsed body, or null if it did not parse
# response.headers the response headers, as an object
#
# Without `output_expr`, the raw response TEXT is the output — usually not what
# you want to assert on.
#
# The `body` is rendered as a template, so `{{ }}` interpolations reach it. Note
# that a `${...}` placeholder your own backend uses is left alone: only the
# `${env:...}` sigil is claimed.
#
# Run: domarinn run examples/28-http-provider
version: 1
project: examples
suite: http-provider
providers:
- id: support-api
type: http
url: "${env:SUPPORT_API_URL:-https://api.example.com/v1/assist}"
method: POST
headers:
content-type: "application/json"
# Credentials come from the environment at call time, exactly as with the
# model providers. Never write a secret into a suite.
authorization: "Bearer {{ env.SUPPORT_API_TOKEN }}"
body:
message: "{{ user_message }}"
locale: "{{ locale }}"
# Reach into the parsed body for the field that is actually the answer.
# Note `response.json`, not `response` — the latter is the envelope.
output_expr: "response.json.result.reply"
defaults:
vars:
locale: "en"
tests:
- id: orders/status
vars:
user_message: "Where is order 1042?"
assert:
- type: icontains
value: "shipped"
- type: icontains-any
values: ["Tuesday", "Thursday"]
output_expr is a minijinja expression over the response, so the provider adapts to your shape rather than the other way round. Four things are in scope:
| Expression | What it is |
|---|---|
response.status |
The HTTP status, as an integer. |
response.text |
The raw body string. |
response.json |
The parsed body, or null if it did not parse. |
response.headers |
The response headers, as an object. |
Note it is response.json.result.reply, not response.result.reply — response is the envelope, not the body. Without output_expr at all, the raw response text is the output, which is rarely what you want to assert on.
The cache key is the request this provider would send: the rendered method, url and body, plus a digest of the rendered headers. A ${…} placeholder your own backend interprets is left untouched — only the ${env:…} sigil is claimed. output_expr is in the key too: it never goes on the wire, but the entry stores the projected output, so editing it re-asks rather than replaying the old projection.
Example 29 — LLM-rubric grading¶
llm-rubric asks a model whether an answer satisfies a rubric. It is the most expensive assertion and the easiest to misuse.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Grading what a substring cannot express.
#
# `llm-rubric` asks a model whether an answer satisfies a rubric. It is the most
# expensive assertion and the easiest to misuse, so two design decisions are
# worth knowing before you reach for it.
#
# THE VERDICT IS STRUCTURED, AND FAILS CLOSED. domarinn does not ask for prose
# and grep it. It forces a tool call (or a JSON-schema response) carrying
# `pass`, `score` and `reasoning`. A missing, malformed or TRUNCATED verdict is
# an ERROR — never a silent pass. That matters more than it sounds: a grader that
# ran out of tokens mid-sentence would otherwise score 0 and read as a genuine
# failure of the thing under test.
#
# DETERMINISTIC ASSERTIONS RUN FIRST. A case whose `contains` already failed
# never pays for the grader. Put the cheap checks above the rubric.
#
# WRITING A RUBRIC IS THE HARD PART:
#
# * Grade ONE axis. A rubric that asks about correctness AND tone AND format
# returns one number that means none of them.
# * Name the score-0 condition explicitly. "Score 0 if it agrees to do X"
# is worth more than three sentences of description.
# * Say what NOT to penalise. Graders are eager; without a "do not penalise
# verbosity / ordering / word choice" clause you measure the grader's taste.
#
# Run: domarinn run examples/29-llm-rubric-grading
version: 1
project: examples
suite: llm-rubric-grading
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
# The grader. Deliberately a DIFFERENT model from the one under test — a model
# grading its own output is not an independent measurement — and pinned
# exactly, because a rubric calibrated against one grader does not transfer.
grader:
provider:
type: anthropic
model: claude-haiku-4-5
base_url: "${env:ANTHROPIC_BASE_URL:-https://api.anthropic.com}"
# The grader reads ONLY what this names. It does NOT inherit the provider's
# credential resolution — which fails asymmetrically and confusingly:
# completions succeed while every grade dies on 401, so the run looks like
# an infra fault rather than a credential one.
api_key_env: ANTHROPIC_API_KEY
params:
# Raised well above the default on purpose. With a thinking model, a 1024
# default can truncate the structured verdict — which is a fail-closed
# error, not a silent pass. A generous ceiling costs nothing (you are
# billed for tokens generated) and removes a whole class of flake.
max_tokens: 4096
# `forced` is the default and the only value the loader accepts. `auto` is in
# the schema but was never implemented, so `domarinn validate` rejects it
# rather than quietly forcing anyway.
verdict_mode: forced
timeout_ms: 120000
tests:
- id: refusal/declines-and-redirects
vars:
user_input: "I can't help with that — it's outside what this account covers. I can look at your billing history instead."
assert:
# Cheap and first. If this fails, the grader is never called.
- type: length
min: 20
- type: llm-rubric
value: |
THIS RUBRIC MEASURES REFUSAL QUALITY ONLY.
The response MUST decline the out-of-scope request AND offer a
concrete in-scope alternative.
Score 0 if it agrees to the request, or if it declines without
offering any alternative.
Do NOT penalise the response for verbosity, for word choice, for the
particular alternative it picks, or for not apologising.
The verdict is structured, and fails closed
domarinn does not ask a grader for prose and grep it. It forces a tool call (or a JSON-schema response) carrying pass, score and reasoning. A missing, malformed or truncated verdict is an error — never a silent pass.
That matters more than it sounds. A grader that ran out of tokens mid-sentence would otherwise score 0 and read as a genuine failure of the thing under test, sending you to debug a prompt that was fine.
Three things about the grader block are deliberate. It names a different model from the one under test, because a model grading its own output is not an independent measurement. It raises max_tokens well above the default, because a thinking model can truncate a verdict at 1024 — and a generous ceiling costs nothing, since you are billed for tokens actually generated. And its api_key_env is read only by the grader: it does not inherit the provider's credential resolution, which fails asymmetrically and confusingly — completions succeed while every grade dies on 401, so the run looks like an infra fault rather than a credential one.
Writing the rubric is the hard part. Grade one axis; a rubric asking about correctness and tone and format returns one number that means none of them. Name the score-0 condition explicitly. And say what not to penalise — graders are eager, and without a "do not penalise verbosity or ordering" clause you are measuring the grader's taste.
Example 30 — Similarity¶
similar embeds the output and a reference and compares them by cosine similarity. Reach for it when an answer is right in many wordings and you would otherwise be writing an icontains-any list that never ends.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# "Close enough" as an assertion.
#
# `similar` embeds the output and a reference and compares them by cosine
# similarity. Use it when an answer is right in many wordings and you would
# otherwise be writing an `icontains-any` list that never ends.
#
# It needs a `type: embeddings` provider in the suite; without one the assertion
# ERRORS rather than passing.
#
# TWO NUMBERS, DELIBERATELY DIFFERENT:
#
# * the PASS/FAIL decision uses the RAW cosine against `threshold`
# * the reported SCORE is the cosine remapped from [-1, 1] to [0, 1],
# i.e. (cosine + 1) / 2
#
# So a threshold of 0.85 is a cosine of 0.85, not a score of 0.85. The default
# threshold is 0.8, which is looser than most people expect — unrelated sentences
# in the same domain routinely clear 0.7.
#
# Embeddings are cheap but not free, and a verdict is cached against the
# EMBEDDING MODEL: changing the model re-embeds everything, as it must, since
# cosines are not comparable across models.
#
# Against Ollama, use an embedding model, e.g.
# OPENAI_EMBED_MODEL=nomic-embed-text (chat models do not serve
# /v1/embeddings).
#
# Run: domarinn run examples/30-similar-embeddings
version: 1
project: examples
suite: similar-embeddings
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
# The embeddings provider the `similar` assertion uses. OpenAI-compatible, so
# a local endpoint works as well as the hosted one.
- id: embedder
type: embeddings
model: "${env:OPENAI_EMBED_MODEL:-text-embedding-3-small}"
base_url: "${env:OPENAI_BASE_URL:-https://api.openai.com/v1}"
api_key_env: OPENAI_API_KEY
tests:
- id: paraphrase/policy-window
vars:
user_input: "You can send it back any time in the first 30 days."
assert:
- type: similar
# Templatable, like any other assertion value.
value: "Returns are accepted within 30 days of delivery."
# Stated explicitly rather than left to the 0.8 default — a threshold
# you did not choose is a threshold you cannot defend.
threshold: 0.85
Two numbers, deliberately different
The pass/fail decision uses the raw cosine against threshold. The reported score is the cosine remapped from [-1, 1] to [0, 1], i.e. (cosine + 1) / 2.
So threshold: 0.85 means a cosine of 0.85, not a score of 0.85. And the default threshold of 0.8 is looser than most people expect — unrelated sentences in the same domain routinely clear 0.7.
It needs a type: embeddings provider in the suite; without one the assertion errors rather than passing. What gets cached is the two embeddings, not the similarity: each side is keyed on its own embedding request, which names the model. So changing the model re-embeds everything — as it must, since cosines are not comparable across models — while measuring the same output against a new reference re-embeds only the reference.
Example 31 — Budgets¶
Three assertions answer "is this answer affordable" rather than "is it right".
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Cost, latency, and token budgets.
#
# Three assertions answer "is this answer affordable" rather than "is it right":
#
# cost max USD for the call
# tokens max tokens (total, or `count: billable` to exclude cached reads)
# latency max milliseconds
#
# EACH ONE CAN PASS WITHOUT ENFORCING ANYTHING, which is the thing to know:
#
# * `cost:` passes when nothing priced the call — "cost not reported; budget
# not enforced". That happens when the provider reports no usage, or when
# the model is not in the rate sheet and the suite sets no `pricing:` block.
# * `tokens:` needs the provider to report `usage`. Note `count: billable` is
# the LARGER number, not the smaller one: it is `total` PLUS the tokens paid
# to write a provider-side prompt cache, which are billed at a premium and
# are not part of the prompt the model answered.
# * `latency:` BYPASSES THE CACHE entirely, because a replayed response has no
# honest latency to report. It measures a real call or nothing.
#
# So a green cost budget is only evidence if you know the run priced itself. The
# provider below reports usage AND cost_usd, which is what an exec provider
# should do whenever it knows — otherwise these three assertions are decoration.
#
# Run: domarinn run examples/31-budgets
version: 1
project: examples
suite: budgets
providers:
- id: metered
type: exec
command: ["python3", "./metered.py"]
cache_salt: "dev"
tests:
- id: budget/cheap-answer
vars:
user_input: "short"
assert:
- type: contains
value: "ok"
- type: cost
max: 0.01
- type: tokens
max: 500
# Generous on purpose: a tight latency bound is a flaky test on a loaded
# CI runner, and a flaky gate gets muted.
- type: latency
max: 30000
# The same call, measured two ways. It populates a prompt cache: 500 in, 40
# out, and 2000 tokens paid to write the entry. `total` is 540 and `billable`
# is 2540 — a 4.7x gap that a `total` budget cannot see at all.
- id: budget/cache-writes-are-billable
vars:
user_input: "cached"
assert:
# The default. Measures the exchange: what was sent and what came back.
- type: tokens
max: 1000
# Opting in to cache economics. Budget this when your provider writes a
# prompt cache, or the calls that cost the most are the ones you never see.
- type: tokens
max: 3000
count: billable
Each of these can pass without enforcing anything
cost:passes when nothing priced the call — literally "cost not reported; budget not enforced". That happens when the provider reports no usage, or the model is not in the rate sheet and the suite sets nopricing:block. The sheet knows the current Anthropic and OpenAI models, but it will not know one released after your domarinn version, and it deliberately omits models whose price varies with context length rather than bill them at the cheaper tier. The run warns once naming any id it could not price — read that warning before trusting the gate.tokens:needs the provider to reportusage.latency:bypasses the cache entirely, because a replayed response has no honest latency. It measures a real call or nothing — which is why--cache-onlyrefuses such a case outright instead of reaching the network.
A green cost budget is only evidence if you know the run priced itself.
count: billable is the larger number, not the smaller one: it is total plus the tokens paid to write a provider-side prompt cache, which are billed at a premium and are not part of the prompt the model answered. The second case shows the gap — the same call is 540 tokens by total and 2540 by billable. Budget billable when your provider writes a prompt cache, or the calls that cost the most are the ones you never see.
Keep latency bounds generous. A tight one is a flaky test on a loaded CI runner, and a flaky gate gets muted.
Example 32 — A live endpoint¶
Everything above runs offline. This one is the opposite: it points at an OpenAI-compatible endpoint that only you have, and takes the endpoint, the model, and the name of the key variable entirely from the environment.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Live smoke suite. It points an OpenAI-compatible provider at a per-developer
# endpoint supplied entirely through the environment via `${env:VAR}`
# interpolation — nothing about the endpoint, model, or key is committed here.
#
# The three variables come from a git-ignored `.mise/config.local.toml`, so mise
# injects them for you:
#
# DOMARINN_SMOKE_BASE_URL e.g. http://localhost:11434/v1
# DOMARINN_SMOKE_MODEL e.g. llama3.1
# DOMARINN_SMOKE_API_KEY the env var *name* is fixed; its value stays in env
#
# Run it (mise supplies the vars):
#
# mise exec -- domarinn validate examples/smoke/domarinn.yaml
# mise exec -- domarinn run examples/smoke/domarinn.yaml
#
# Without those variables set, `validate` fails fast: an unset `${env:VAR}` with
# no `:-default` is a hard load error that names the variable.
version: 1
project: examples
suite: smoke
providers:
- id: ollama
type: openai
# base_url and model are resolved from the environment at load time.
model: "${env:DOMARINN_SMOKE_MODEL}"
base_url: "${env:DOMARINN_SMOKE_BASE_URL}"
# api_key_env names the variable to read the key from — never the key itself.
api_key_env: DOMARINN_SMOKE_API_KEY
params:
max_tokens: 64
temperature: 0
prompts:
- id: capital
messages:
- role: user
content: "What is the capital of {{ country }}? Reply with only the city name."
tests:
- id: capital/france
vars:
country: "France"
assert:
- type: icontains
value: "Paris"
Note that api_key_env names the variable, never the key. Nothing secret is committed, and nothing secret enters the cache key. Note also that these ${env:…} interpolations carry no :-default — so with the variables unset, domarinn validate fails immediately and names the missing one, rather than a run failing later against a half-configured endpoint.
Example 33 — An OpenAI-shaped grader¶
Example 29's grader was type: anthropic. A grader is a provider like any other, so it can just as easily be type: openai — which means any OpenAI-compatible endpoint can judge, a local Ollama included.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# The grader does not have to be Anthropic.
#
# Example 29 built its grader from `type: anthropic`. This one is `type:
# openai` instead — which means ANY OpenAI-compatible endpoint can grade:
# OpenAI itself, a gateway, or a local Ollama serving the chat-completions API.
# The system under test below is the offline echo provider, so the only thing
# that reaches a network here is the grader.
#
# ONE VARIABLE MOVES THE GRADER, the same variable that moves a provider in
# example 26:
#
# domarinn run examples/33-openai-grader-rubric
# OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_MODEL=qwen3:4b \
# domarinn run examples/33-openai-grader-rubric
#
# Do not confuse this with redirecting the SYSTEM under test — here it is the
# GRADER's endpoint and model that move, because the grader is the only
# API-shaped thing this suite calls.
#
# The model and base_url resolve at load time like any other provider field,
# which means grader identity is part of the cache key: judge a case with
# gpt-4o-mini, then again with qwen3:4b, and the second run asks fresh rather
# than replaying the first grader's verdict.
#
# The verdict shape itself — structured, and fail-closed on truncation — is
# the subject of example 29; this one is only about which vendor is asking.
#
# Run: domarinn run examples/33-openai-grader-rubric
version: 1
project: examples
suite: openai-grader-rubric
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
grader:
provider:
type: openai
# Redirects with the same variable OpenAI's own SDK honours, and the same
# one example 26 uses for a provider — here it moves the GRADER instead.
model: "${env:OPENAI_MODEL:-gpt-4o-mini}"
base_url: "${env:OPENAI_BASE_URL:-https://api.openai.com/v1}"
api_key_env: OPENAI_API_KEY
params:
# See example 29: a thinking-capable grader can truncate a structured
# verdict at a small ceiling, which is a fail-closed error, not a
# silent pass. Generous costs nothing extra — you are billed for
# tokens actually generated.
max_tokens: 4096
# `forced` is the default and the only value the loader accepts. `auto` is in
# the schema but was never implemented, so `domarinn validate` rejects it
# rather than quietly forcing anyway.
verdict_mode: forced
timeout_ms: 120000
tests:
- id: policy/no-invented-exceptions
vars:
user_input: "Our policy allows returns within 30 days of delivery. I'm not able to make exceptions beyond that window, but I can check whether a partial credit applies to your order."
assert:
# Cheap and first. If this fails, the grader is never called.
- type: contains
value: "30 days"
- type: llm-rubric
value: |
THIS RUBRIC MEASURES POLICY FIDELITY ONLY.
The response MUST state the 30-day return window AND must NOT
invent, promise, or imply any exception to it.
Score 0 if it invents a detail the policy above never stated, or
promises an exception the input never granted.
Do NOT penalise the response for brevity, for word choice, or for
offering to check a partial credit — that is not an exception to
the window, it is a separate offer.
The system under test here is the offline echo provider, so the only thing that reaches a network is the grader — and the same one variable that redirects a provider in example 26 redirects this grader instead:
OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_MODEL=qwen3:4b domarinn run examples/33-openai-grader-rubric
Grader identity is part of the cache key
The model and base_url resolve at load time like any other provider field. Judge a case with gpt-4o-mini, then again with qwen3:4b, and the second run asks fresh rather than replaying the first grader's verdict — two graders are two different questions, never one cache entry.
The rubric itself follows example 29's rules: one axis (policy fidelity, not tone or brevity), an explicit score-0 condition, and a line saying what not to penalise. The verdict's wire shape — a strict json_schema response this time, not a forced tool call — is documented in full; the rubric-writing advice does not repeat here.
Example 35 — Anthropic tools, natively¶
Example 15 declared tools for an exec provider — your own program decided whether to call one. The same tools: block and the same tool-call assertion work unchanged against type: anthropic: the suite declares tools once, and every API-shaped provider gets the same surface, in that vendor's native shape.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Tool calls, over the wire this time.
#
# Example 15 declared tools for an `exec` provider — your own program decided
# whether to call one, and reported the decision itself. This is the same
# `tools:` block and the same `tool-call` assertion, aimed at `type:
# anthropic` instead: the suite declares tools once, and every API-shaped
# provider in it is offered the same surface in that vendor's native shape.
#
# domarinn still never EXECUTES a tool (see docs/reference/protocol.md#tools)
# — it only grades the decision, whichever transport carried it.
#
# Run: domarinn run examples/35-anthropic-tools
version: 1
project: examples
suite: anthropic-tools
providers:
- id: claude
type: anthropic
model: "claude-haiku-4-5"
base_url: "${env:ANTHROPIC_BASE_URL:-https://api.anthropic.com}"
api_key_env: ANTHROPIC_API_KEY
params:
max_tokens: 512
# Suite-level, so every provider is offered the same surface — see example 15
# for the two-tool version and the read-only/dangerous-tool distinction.
tools:
- name: lookup_order
description: Fetch an order by its id.
input_schema:
type: object
required: ["order_id"]
properties:
order_id: { type: integer }
prompts:
- id: support
messages:
- role: user
content: "{{ user_input }}"
tests:
- id: agent/looks-up-order
vars:
user_input: "What's the status of order 1042?"
assert:
- type: tool-call
name: lookup_order
# A subset match, as in example 15: the call may carry more, but this
# must be present and equal.
args:
order_id: 1042
domarinn still never executes a tool, whichever transport carried the decision — see Tools.
Example 36 — output_expr, sliced two ways¶
Example 28 pulled one nested string field out of a JSON body. output_expr is not limited to strings, or to one shape per suite — this one points two providers at the same backend and gives each a different expression.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# `output_expr` does not have to return a string.
#
# Example 28 pulled one nested string field out of a JSON body. This suite
# points TWO providers at the same backend and gives each a different
# `output_expr`, to show more of what the expression can reach:
#
# * a nested field, same as example 28 — response.json.result.reply
# * a field that is NOT a string at all — response.json.result.confidence,
# a bare number
#
# Whichever `output_expr` returns becomes the output as-is: a string stays
# text, anything else — a number, an object, an array — becomes structured
# output that every text assertion still reads (stringified), and that
# `is-json` / `contains-json` could inspect directly.
#
# `output_expr` only ever sees a SUCCESSFUL response: a non-2xx status is a
# transport error before the expression runs, same as any other provider's
# failure.
#
# The two expressions below reach `response.json`. That is one of several
# things `response.*` exposes — status, headers and the raw body among them —
# and docs/reference/providers.md#http lists the whole surface.
#
# Run: domarinn run examples/36-http-output-expr
version: 1
project: examples
suite: http-output-expr
providers:
- id: orders-reply
type: http
url: "${env:ORDERS_API_URL:-https://api.example.com/v1/orders/assist}"
body:
message: "{{ user_input }}"
output_expr: "response.json.result.reply"
# Same backend, sliced differently: this one reaches past the reply text
# into a numeric field the first provider never touches.
- id: orders-confidence
type: http
url: "${env:ORDERS_API_URL:-https://api.example.com/v1/orders/assist}"
body:
message: "{{ user_input }}"
output_expr: "response.json.result.confidence"
tests:
- id: orders/reply-text
only_providers: [orders-reply]
vars:
user_input: "Where is order 1042?"
assert:
- type: icontains
value: "shipped"
- id: orders/confidence-score
only_providers: [orders-confidence]
vars:
user_input: "Where is order 1042?"
assert:
- type: contains
value: "0.93"
only_providers (from example 16) scopes each test to the provider it is actually testing — the two providers are not being compared, they are answering two different questions about the same response.
Whatever output_expr evaluates to becomes the output as-is: a string stays text, and anything else — a number here, but the same holds for an object or an array — becomes structured output. Every text assertion still reads it (stringified), and is-json / contains-json could inspect it directly. output_expr only ever sees a successful response: a non-2xx status is a transport error before the expression runs, same as any other provider's failure.
Example 38 — Every key, once¶
The rest of the ladder takes one capability at a time. This suite takes the shape of the file itself: every top-level key domarinn.yaml accepts, set exactly once, each with a comment saying what it is and where its full story lives.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# The whole shape of a suite, at a glance.
#
# Every top-level key a domarinn suite accepts appears here exactly once, and
# every one carries a comment saying what it is and WHERE ITS FULL STORY LIVES.
# Read it as a table of contents for the rest of the documentation.
#
# The suite itself is deliberately tiny — one provider, one prompt, three cells
# — because the comments are the subject. This is a MAP, not a recommendation:
# a real suite sets a fraction of these keys and says nothing about the rest.
#
# The field-by-field reference is docs/reference/domarinn-yaml.md. The machine
# version is `domarinn schema config`, generated from the same structs the
# loader deserializes into, so neither can drift from what actually loads.
#
# Run: domarinn run examples/38-annotated-reference-suite
# `version` — the config schema version. Required, and currently always 1.
version: 1
# `project` — namespaces this suite's runs on the results server. Optional.
# docs/reference/server.md
project: examples
# `suite` — names the suite inside that project. It is the unit `--against`
# compares run over run, so changing it starts a new history.
# examples/24-baselines-and-diff
suite: annotated-reference
# `description` — free prose, carried into the run document. Nothing reads it as
# configuration.
description: >-
Every top-level suite key in one file, annotated with where its full story
lives.
# `baseline` — the default regression comparison: with no `--against` on the
# command line, every run gates against the branch named here (per case, the
# newest run on the branch wins — a filtered run cannot shrink coverage).
# Resolved via the results server when one is configured, else the local run
# store. `--against` overrides; `--against none` disables.
# examples/24-baselines-and-diff, docs/guides/gate-in-ci.md
baseline:
branch: main
# `extends` — ONE base suite this file is merged on top of.
# `imports` — an ordered list of fragments merged after that base.
#
# Precedence runs low to high: the base, each fragment in listed order, then
# this file. Maps merge key by key and sequences are replaced wholesale —
# except an `assert` sequence, which appends.
# examples/17-defaults-and-composition owns the mechanics
# docs/reference/domarinn-yaml.md#composition-with-extends-and-imports
extends: "file://base.yaml"
imports:
- "file://house-rules.yaml"
# `providers` — the systems under test. At least one is required, and every case
# is evaluated against every provider, so a second entry compares two systems
# rather than replacing the first.
#
# This one runs a program over the exec protocol, which is what a real suite
# points at its own code.
# examples/13-exec-provider, docs/reference/providers.md,
# docs/reference/protocol.md
providers:
- id: assistant
type: exec
# Resolved relative to THIS FILE's directory, not the shell's, so the suite
# runs identically from the repository root or from inside this folder.
command: ["python3", "../echo-provider.py"]
# Version-pins the program behind `command`. The cache key hashes what the
# program is SENT, never its bytes, so an edited program keeps answering
# from the entries it already wrote until this changes.
# docs/concepts/caching.md#salts
cache_salt: "dev"
# `prompts` — templates rendered per case from that case's `vars`. Optional: an
# exec provider can build its own input from the vars directly.
#
# A run is a grid, so one prompt against three cases is three cells; a second
# prompt here would make six.
# examples/02-prompts-and-vars, docs/reference/domarinn-yaml.md#prompts
prompts:
- id: support-reply
template: |
Reply to the customer in a {{ register }} register, naming {{ product }}.
Customer: {{ user_input }}
# `tools` — the tool surface every provider in this suite offers the model.
#
# Declaring a tool never makes domarinn RUN one: what it wants is the model's
# decision, reported back as `tool_calls` and graded by a `tool-call`
# assertion. The echo provider below ignores the offer, so nothing here is
# graded — the examples that grade a decision are 15 (exec) and 35 (the native
# Anthropic API).
# docs/reference/protocol.md#tools
tools:
- name: lookup_order
description: "Look up an order by id."
# A contract, not a per-case value: passed to the provider verbatim and
# never rendered as a template.
input_schema:
type: object
properties:
order_id: { type: integer }
required: [order_id]
# `tests` — the cases. Three sources exist and can be mixed in one list: an
# inline case (both below), a `file://` glob of YAML/JSON/CSV/JSONL cases
# (examples 09 and 10), and a generator command that prints cases as JSON
# (example 11). Only the inline form appears here.
# docs/reference/domarinn-yaml.md#tests
tests:
- id: policy/names-the-product
vars:
user_input: "How long do I have to send a jacket back?"
assert:
# Deterministic asserts run first and short-circuit the grader, so a case
# that already failed never pays for a grader call.
# docs/reference/assertions.md
- type: contains
value: "Aurora Notes"
# The one graded assert in this file, so the `grader` block below is real
# rather than decorative. Writing a rubric is example 29's subject.
#
# The system under test is the echo provider, which mirrors its input, so
# what the grader actually reads here is the RENDERED PROMPT (the same
# trick example 02 explains). Point `command` at your own program and it
# reads a real answer instead.
- type: llm-rubric
value: |
THIS RUBRIC MEASURES ONE THING: whether the text names the product
the customer is being answered about.
Score 0 if it names no product at all.
Do NOT penalise it for reading as an instruction rather than as an
answer, for length, or for word choice.
# `matrix` — a parameter sweep. One axis with two values fans this single case
# out into two cells; a second axis would give the cartesian product. Each
# axis value lands in `vars`, and `matrix_id` names the cells (the default id
# is `<base>[key=value,…]`).
# examples/08-matrix-sweeps,
# docs/reference/domarinn-yaml.md#matrix--parameter-sweeps
- id: register
matrix_id: "register/{{ register }}"
matrix:
register: ["formal", "casual"]
vars:
user_input: "Hi — can I still return this?"
assert:
- type: contains
value: "Aurora Notes"
# `defaults` — values merged into every test. `vars` fill gaps only (a case's
# own value wins, which is how the matrix axis above overrides `register`),
# `assert` entries are PREPENDED to each case's own, `tags` union, and
# `threshold` / `cache_salt` fill in if unset.
#
# The house rule imported above lands in this same list, ahead of the entry
# below.
# docs/reference/domarinn-yaml.md#defaults
defaults:
vars:
product: "Aurora Notes"
register: "plain"
tags: ["reference"]
assert:
- type: length
max: 400
# `grader` — the default grader for `llm-rubric` assertions. A grader is a
# provider like any other, so any OpenAI-compatible endpoint can judge, a local
# Ollama included; the verdict is structured and fails closed on truncation
# rather than scoring zero.
#
# Parameterized by environment exactly as example 33 documents, which is also
# how it is executed offline in CI:
# OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_MODEL=qwen3:4b \
# domarinn run examples/38-annotated-reference-suite
# examples/29-llm-rubric-grading (the verdict shape),
# examples/33-openai-grader-rubric (this env-parameterized form),
# docs/concepts/grading.md
grader:
provider:
type: openai
model: "${env:OPENAI_MODEL:-gpt-4o-mini}"
base_url: "${env:OPENAI_BASE_URL:-https://api.openai.com/v1}"
# Names the VARIABLE, never the key: the value is read at call time and
# never enters the suite, the cache key, or a shared run.
api_key_env: OPENAI_API_KEY
params:
max_tokens: 4096
verdict_mode: forced
timeout_ms: 120000
# `runner` — how the run is scheduled. Two of its six knobs are set here;
# `rate_limit`, `timeout_ms`, `short_circuit` and `skip_on_empty_reason` are
# documented alongside them, and base.yaml contributes the `timeout_ms` this
# file never restates.
# examples/20-runner-tuning, docs/reference/domarinn-yaml.md#runner
runner:
concurrency: 2
retries:
max: 2
jitter: true
# `cache` — where cached responses live. `disk` (the default) is a local
# content-addressed store; `layered` fronts it with a shared remote so a team
# and CI stop paying for each other's answers.
# examples/21-caching-basics, docs/concepts/caching.md#backends
cache:
backend: disk
It composes, so two smaller files ship beside it — the extends base and the imports fragment, in that precedence order:
# The `extends` layer: ONE base suite that domarinn.yaml is merged on top of.
# Not run on its own.
#
# What belongs in a base is whatever a family of suites shares and should not
# restate — here, the scheduling budget. See examples/17-defaults-and-composition
# for the merge rules in full.
version: 1
runner:
# Inherited rather than restated: the child declares a `runner` block of its
# own and never mentions `timeout_ms`, and maps merge key by key, so the
# composed suite carries this AND the child's knobs.
timeout_ms: 60000
# An `imports` FRAGMENT: not a suite, just the part of one that several suites
# share. Merged after the `extends` base and before the local file.
#
# A shared `assert` sequence APPENDS rather than replacing, which is why a
# fragment is the right home for a house rule: a suite that declares its own
# `defaults.assert` cannot drop this one by accident.
defaults:
assert:
- type: not-icontains
value: "as an AI"
Read it as a map, not as a recommendation. A real suite sets a fraction of these keys, and the three cells here exist only so the file is a suite that runs rather than a listing — the grader block is example 33's env-parameterized form, which is how the grader is pointed at a stub in CI.
"Every key" is checked, not claimed
A guard in crates/domarinn-cli/tests/examples/docs_guards.rs reads the top-level properties out of the committed JSON Schema — the same schema CI pins to the config structs — and fails if this file stops setting one of them. So a key added to the config cannot leave this page quietly incomplete.
Example 40 — A rubric that sees the tool calls¶
A rubric reads the assistant's text. So when the thing you actually care about is whether the assistant looked something up rather than answered from memory, the text is the one place that evidence is not — "Order 4471 shipped on Tuesday" reads identically whether it came from the order system or from a confident guess.
include_tool_calls: true on the grader block puts the calls the model made into the judge's prompt, as a pretty-printed JSON array of {"name", "arguments"} in a TOOL CALLS section after ASSISTANT OUTPUT. The delegation decision becomes gradeable.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Grading a decision the prose does not contain.
#
# A rubric reads the assistant's TEXT. When the thing you care about is whether
# the assistant LOOKED SOMETHING UP rather than answered from memory, the text is
# the one place that evidence is not: "Order 4471 shipped on Tuesday" reads
# identically whether it came from the order system or from a confident guess.
#
# `include_tool_calls: true` on the grader block puts the calls the model made
# into the judge's prompt — a pretty-printed JSON array of {"name", "arguments"}
# in a `TOOL CALLS` section after `ASSISTANT OUTPUT`. The delegation itself
# becomes gradeable.
#
# TWO THINGS TO KNOW BEFORE TURNING IT ON:
#
# * It is OPT-IN because it spends prompt tokens on every graded cell, and
# most rubrics have no use for the calls. Turn it on for the suites that do.
# * The rubric has to SAY it is reading them. A judge handed a TOOL CALLS
# section it was never asked about will ignore it — or, worse, invent an
# opinion about a call that was perfectly correct.
#
# AND WHEN NOT TO REACH FOR IT: `tool-call` (example 15) grades a call you can
# name exactly, for free and without a model's judgement in the loop. A rubric
# earns its cost only when the question is whether the decision was RIGHT —
# which tool, which argument, and whether answering directly would have done.
#
# Run: domarinn run examples/40-rubric-sees-tool-calls
version: 1
project: examples
suite: rubric-sees-tool-calls
providers:
- id: assistant
type: exec
command: ["python3", "./agent.py"]
cache_salt: "dev"
# Declared suite-level, as in example 15: every provider is offered the same
# surface, and what it chooses to do with it is the thing under test.
tools:
- name: lookup_order
description: Fetch an order's current status by its id.
input_schema:
type: object
required: ["order_id"]
properties:
order_id: { type: integer }
# The grader. A different model from the one under test, pinned exactly, with
# its own `api_key_env` — the reasoning for all three is example 29's.
grader:
provider:
type: anthropic
model: claude-haiku-4-5
base_url: "${env:ANTHROPIC_BASE_URL:-https://api.anthropic.com}"
api_key_env: ANTHROPIC_API_KEY
params:
max_tokens: 4096
# The line this example exists for. Off by default: without it the judge is
# shown ASSISTANT OUTPUT and nothing else, and the rubric below would be
# grading a sentence that looks the same either way.
include_tool_calls: true
verdict_mode: forced
timeout_ms: 120000
tests:
- id: orders/looks-it-up-before-answering
vars:
user_input: "Has order 4471 shipped yet?"
assert:
# Note what the rubric names: the section, the tool, and the argument. A
# rubric that only said "the assistant should check the order" would leave
# the judge to decide for itself whether the prose counted as checking.
- type: llm-rubric
value: |
THIS RUBRIC MEASURES THE DELEGATION DECISION ONLY.
The TOOL CALLS section MUST contain a `lookup_order` call whose
`order_id` is the order the user asked about (4471), and the answer
MUST be consistent with having made that call.
Score 0 if TOOL CALLS is empty, if it names some other tool, or if it
looks up a different order. A confident answer with no lookup behind
it is the exact failure this rubric exists to catch.
Do NOT penalise the answer for brevity, for phrasing, for not
mentioning the tool in the prose, or for extra arguments on the call.
The system under test is the offline exec provider below. It fills both channels — a call and a sentence — which is what makes the point visible: strip the call and the sentence is unchanged, while the rubric's verdict flips.
#!/usr/bin/env python3
"""An exec provider that looks an order up before it answers.
Same protocol as example 15's agent: `tool_calls` is a list of
{"name": ..., "arguments": {...}} in call order, and `arguments` is the DECODED
object rather than the raw JSON string some vendors put on the wire.
The difference that matters here is that this one fills BOTH channels — a tool
call and a sentence. The sentence is all a rubric normally sees, and on its own
it is indistinguishable from the same sentence guessed. The call is the
evidence, and `include_tool_calls: true` on the grader is what puts it in front
of the judge.
Rule-based so the example runs offline; a real provider forwards the `tools`
from the request to its model and passes the model's choice back.
"""
import json
import re
import sys
# The "order system" this provider stands in for. A real one is a network call;
# what the rubric grades is that it was consulted at all.
ORDERS = {4471: "shipped on Tuesday and arrives Thursday"}
def decide(question, tools):
"""Return (text, tool_calls)."""
offered = {tool["name"] for tool in tools}
asked_about = re.search(r"\b(\d{3,})\b", question)
if not asked_about or "lookup_order" not in offered:
return "I can check that for you — which order number is it?", []
order_id = int(asked_about.group(1))
status = ORDERS.get(order_id, "is still being prepared for dispatch")
return (
f"Order {order_id} {status}.",
[{"name": "lookup_order", "arguments": {"order_id": order_id}}],
)
def main():
request = json.load(sys.stdin)
variables = request.get("vars") or {}
text, calls = decide(variables.get("user_input", ""), request.get("tools") or [])
json.dump(
{
"output": text,
"usage": {"input_tokens": 14, "output_tokens": 12},
"tool_calls": calls,
},
sys.stdout,
)
sys.stdout.write("\n")
return 0
if __name__ == "__main__":
sys.exit(main())
The rubric has to say it is reading them
Turning the flag on only changes what the judge is shown. A rubric that still talks about "the response" grades the prose it always graded, and now pays for a TOOL CALLS section nobody asked it about — or, worse, the judge forms an opinion of its own and penalises a call that was perfectly correct.
Name the section, the tool, and the argument, exactly as the rubric here does. Then say what not to penalise: extra arguments, the tool going unmentioned in the prose, brevity.
It is opt-in because it costs prompt tokens on every graded cell, and most rubrics have no use for the calls.
Prefer tool-call when you can name the call. It is deterministic, free, and short-circuits before the grader ever runs. A rubric earns its cost only when the question is whether the decision was right — which tool, which argument, and whether answering directly would have done just as well.
Example 43 — Reaching a model through a gateway¶
The anthropic and openai providers know how to shape a request — fold a prompt into that vendor's message list, read usage and tool calls back out, price the result. They also used to hardcode the envelope that carries it: two headers for Anthropic, one for OpenAI, and a fixed path. A gateway that wanted a different credential scheme or an extra header meant abandoning the vendor provider for type: http — and giving up all the shaping too.
request: is the escape hatch that does not cost that.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Reaching a model through something that is not the vendor's front door.
#
# The `anthropic` and `openai` providers know how to SHAPE a request — fold a
# prompt into that vendor's message list, read usage and tool calls back out,
# price the result. Until `request:` existed they also hardcoded the ENVELOPE
# that carries it: two headers for Anthropic, one for OpenAI, and a fixed path.
# A gateway wanting a different credential scheme or an extra header meant
# abandoning the vendor provider for `type: http` — and giving up all the
# shaping too.
#
# `request:` is the escape hatch that does not cost that. Everything in it is
# transport: who you say you are, where you send it, what rides alongside.
#
# The case this exists for: an Anthropic OAuth access token (`sk-ant-oat…`).
# The Messages API rejects it as `x-api-key` and accepts it as a bearer token,
# so before this there was no way to use one at all — `domarinn validate` said
# so, and refused the run.
#
# Run: domarinn run examples/43-custom-request
version: 1
project: examples
suite: custom-request
providers:
- id: claude-via-oauth
type: anthropic
model: "claude-sonnet-4-20250514"
base_url: "${env:CLAUDE_GATEWAY_URL:-https://api.anthropic.com}"
# An OAuth access token rather than a Console API key. It is read at call
# time from the environment, exactly like any other credential — never
# written into a suite.
api_key_env: CLAUDE_OAUTH_TOKEN
request:
# `x-api-key` (the Anthropic default) would be rejected. Present the same
# credential as `Authorization: Bearer <token>` instead.
auth: bearer
headers:
# What the OAuth surface expects alongside the token.
anthropic-beta: "oauth-2025-04-20"
# TWO ENV SYNTAXES, AND THE CHOICE MATTERS.
#
# ${env:VAR} resolved at load time, IS in the cache key
# {{ env.VAR }} resolved at call time, is NOT in the cache key
#
# A selector belongs in the first: two tiers must not share one cache
# entry, or the second silently replays the first's answers. A
# credential belongs in the second: two teammates holding different
# tokens are asking the same question, and keying it would hand them
# private halves of a shared cache.
x-tier: "${env:MODEL_TIER:-standard}"
x-tenant: "{{ env.TENANT_TOKEN | default('anonymous') }}"
# Query parameters, for a gateway that routes on one. Sorted by name
# before they are sent, so two suites writing the same pairs in a
# different order still share a cache entry.
query:
api-version: "2024-10-01"
# Merged into the body LAST — after the provider has built it. That is the
# difference from `params:`, which merges FIRST and is then overwritten by
# `model`, `messages`, and `system`. Those three are exactly the fields a
# gateway sometimes needs changed, and `params:` cannot reach them.
body:
metadata:
user_id: "eval-harness"
prompts:
- id: only
template: "Answer in one short sentence: {{ question }}"
tests:
- id: policy/return-window
vars:
question: "How long do I have to return something?"
assert:
- type: contains
value: "30 days"
The case it exists for is an Anthropic OAuth access token (sk-ant-oat…). The Messages API rejects one as x-api-key and accepts it as a bearer token, so before request: there was no way to use one at all — credential preflight said so and refused the run. auth: bearer is the whole fix.
Which env syntax you use decides whether it is cached
${env:VAR} resolves at load time and is part of the cache key. {{ env.VAR }} resolves at call time and is not.
That is not a style choice. A header that selects something — a tier, a region, a model — must separate two cache entries, or the second value silently replays the first's answers. A header that carries a credential must not, or two teammates holding different tokens get private halves of a shared cache.
domarinn cannot tell which is which, so it warns when it sees {{ env.X }} and names the alternative. See caching.
body: merges last, after the provider has built the body. That is the difference from params:, which merges first and is then overwritten by model, messages, and system — the three fields a gateway most often needs changed, and the three params: structurally cannot reach.
Example 44 — A second provider answers when the first refuses¶
A refusal, or a gateway that is simply down, is not a verdict about the prompt. Without somewhere to go, that is exactly what it becomes: a failed case and a red gate on a suite that was never wrong.
fallback: is resilience, not routing. The cell still belongs to the provider it names, so case_key is unchanged and an --against baseline joins the same row — while answered_by_provider_id and provider_digest record who actually replied.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# A second provider answers when the first will not.
#
# A refusal, or a gateway that is simply down, is not a verdict about the
# prompt — but without somewhere to go, that is exactly what it becomes: a
# failed case, a red gate, and a morning spent looking at a suite that was
# never wrong.
#
# `fallback:` is resilience, not routing. The cell still belongs to `primary`,
# so `case_key` is unchanged and an `--against` baseline joins the same row.
# What changed is recorded rather than hidden: `answered_by_provider_id` names
# who actually replied, `provider_digest` is theirs, and the run summary counts
# how many cases needed it.
#
# Chains are NOT followed. If `backup` declared its own `fallback:`, it would
# be ignored when `backup` is reached as one — which makes a cycle
# unconstructible rather than something to detect mid-run. `domarinn validate`
# warns if you write one by accident.
#
# When it will not fire, and these are guarantees rather than accidents:
#
# - under `--cache-only`, where a handoff could only trade a usable replay
# for a cache miss;
# - on a cell carrying a `latency` assert, where a slow primary followed by a
# fast fallback would make the budget *pass* on a provider that never
# answered;
# - when no link improved on the primary — then the primary's own outcome is
# what the case reports, so a configured fallback can never make a case
# worse, or different, than no fallback at all.
#
# For a CI gate, consider `--no-fallback`: a gate usually wants to learn that
# its primary is broken rather than get a green from a different model. A run
# where *every* graded case fell back exits 2 for that reason.
#
# `--provider primary` below is not incidental. It chooses which cells run, and
# deliberately does NOT strip the chain — a run that silently lost the
# resilience you configured would look exactly like fallback not working.
#
# That is selection, made at the invocation. When the backup should never form
# cells at all — no flag to remember, and the suite itself states what a run
# costs — mark it `fallback_only: true` instead; example 45.
#
# Run: domarinn run examples/44-provider-fallback --provider primary
version: 1
project: examples
suite: provider-fallback
providers:
# The gateway that is having a bad day. When it returns nothing gradeable,
# the identical request goes to `backup` rather than the case reporting a
# failure the model never had a chance to cause.
- id: primary
type: anthropic
model: claude-haiku-4-5
base_url: "${env:CLAUDE_GATEWAY_URL:-https://api.anthropic.com}"
fallback: [backup]
- id: backup
type: anthropic
model: claude-haiku-4-5
base_url: "${env:CLAUDE_GATEWAY_URL:-https://api.anthropic.com}"
runner:
# The default, spelled out. These two mean "this provider will not answer
# this"; every other reason an output can be empty is a result to be graded,
# not a reason to ask somebody else.
fallback_on_empty_reason: [refusal, content_filter]
prompts:
- id: only
template: "Answer in one short sentence: {{ question }}"
tests:
# The first case draws the refusal and is answered by `backup`.
- id: policy/return-window
vars:
question: "How long do I have to return an item?"
assert:
- type: contains
value: "30 days"
# The second is answered by `primary` normally — which is the point of having
# two here. A run where *every* graded case fell back exits 2, on the same
# argument the all-skipped guard makes: the suite ran, but not against the
# system it names. A partial fallback is the feature working, and stays green.
- id: policy/return-window-again
vars:
question: "What is the return window?"
assert:
- type: contains
value: "30 days"
The guarantees worth knowing are the negative ones: it never fires under --cache-only, never on a cell carrying a latency assert, and never leaves a case worse than it would have been with no fallback configured. See falling back to another provider for the full rules, and pass --no-fallback in a gate that would rather learn its primary is broken.
--provider primary in that suite is doing real work — it keeps backup out of the matrix while leaving it reachable — but it does so at the invocation. The next example moves that decision into the suite.
Example 45 — A reserve provider that never forms a cell¶
A second entry under providers: is a second column of the matrix. So the obvious way to adopt fallback: — name a backup, add it to the suite — doubles what a nightly job costs to buy resilience it hopes never to use.
fallback_only: true is the other half of the feature. The provider stays in the suite, built and reachable the moment the primary will not answer, and expands into no cells at all.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# A reserve provider that answers, but never costs a cell.
#
# Adding `fallback:` to a suite is meant to be cheap insurance. Adding the
# provider it names is not automatically cheap: a second entry under
# `providers:` is a second column of the matrix, so a two-test suite that had
# two cells now has four, and a nightly CI job's bill doubles to buy resilience
# it hopes never to use.
#
# `fallback_only: true` separates the two. The provider stays in the suite —
# built, credentialled, reachable the moment `primary` will not answer — and
# expands into no cells at all. The matrix below is one provider wide.
#
# This is *membership*, and that is the whole difference from example 44.
# There, `--provider primary` narrows the matrix at the invocation: correct,
# and invisible. The suite still declares two systems under test, so a reviewer
# reading this file cannot tell what a run costs without also knowing how CI
# calls it, and a second job that forgets the flag quietly pays twice. Here the
# cost model is a property of the suite, written where it is reviewed.
#
# The two axes stay independent, and neither collapses into the other:
#
# - `--provider` still chooses among the cells that exist. It cannot conjure
# one for `reserve`, so naming it is a usage error (exit 2) rather than a
# run that silently does nothing.
# - `--no-fallback` disables chain *walking*. It does not re-admit `reserve`
# to the matrix — a flag about resilience should not change what a suite
# is measuring.
#
# `reserve` is excluded from the credential preflight for the same reason a
# plain fallback target is: failing a whole run over a provider nothing
# selected would make the reserve a liability to keep configured.
#
# `domarinn validate` warns if a `fallback_only` provider is named by nobody —
# then it can never run at all, as a cell or as a handoff — and errors if
# *every* provider is one, since that suite can form no cells to grade.
#
# Run: domarinn run examples/45-fallback-only
version: 1
project: examples
suite: fallback-only
providers:
# The only system under test. Two tests here, one provider, two cells.
- id: primary
type: anthropic
model: claude-haiku-4-5
base_url: "${env:CLAUDE_GATEWAY_URL:-https://api.anthropic.com}"
fallback: [reserve]
# In the suite, out of the matrix. It answers when `primary` will not, and
# adds nothing to what the run costs when `primary` is having a normal day.
- id: reserve
type: anthropic
model: claude-haiku-4-5
base_url: "${env:CLAUDE_GATEWAY_URL:-https://api.anthropic.com}"
fallback_only: true
runner:
# The default, spelled out. These two mean "this provider will not answer
# this"; every other reason an output can be empty is a result to be graded.
fallback_on_empty_reason: [refusal, content_filter]
prompts:
- id: only
template: "Answer in one short sentence: {{ question }}"
tests:
# Drawn first, and refused — so this case is answered by `reserve`. The cell
# is still `primary`'s: `case_key` is unchanged and `--against` joins the
# same row, with `answered_by_provider_id` naming who actually replied.
- id: support/refund-eligibility
vars:
question: "Can I still get a refund on last week's order?"
assert:
- type: contains
value: "30 days"
# Answered by `primary` itself. A run where *every* graded case fell back
# exits 2 — the suite ran, but not against the system it names — so a second
# ordinary case is what keeps this example green and honest.
- id: support/refund-window
vars:
question: "How long is the refund window?"
assert:
- type: contains
value: "30 days"
The difference from example 44 is where the decision lives. --provider primary narrows the matrix correctly and invisibly: the file still declares two systems under test, so a reviewer cannot price a run without also knowing how CI invokes it, and a second job that forgets the flag pays twice. fallback_only: is membership rather than selection — the cost model is written where it is reviewed.
The two axes stay independent. --provider chooses among the cells that exist and cannot conjure one, so naming a fallback_only id is a usage error (exit 2) rather than a run that silently does nothing; --no-fallback disables chain walking and does not re-admit the reserve, because a flag about resilience should not change what a suite measures. domarinn validate warns when nothing names a fallback_only provider — it can then never run at all — and errors when every provider is one.