Models, grading & budgets¶
Eleven suites about talking to a real model and judging what comes back. They cover the OpenAI-compatible and Anthropic providers (native tool calls included), a plain HTTP service you already run and the shapes output_expr can pull out of it, and a live endpoint of your own — then a structured LLM rubric judged by either vendor, embedding similarity for when many wordings are right, and the budgets that ask whether an answer was affordable, not just correct. The last one is a different kind of suite: every top-level key at once, annotated as a map of the reference. Read these once you are ready to spend real tokens.
Example 26 — An OpenAI-compatible endpoint¶
type: openai speaks the chat-completions API, which is the lingua franca: OpenAI itself, Ollama, vLLM, LiteLLM, OpenRouter and most gateways all accept it. Point base_url at whichever one you have.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Calling an OpenAI-compatible endpoint.
#
# `type: openai` speaks the chat-completions API, which is the lingua franca:
# OpenAI itself, Ollama, vLLM, LiteLLM, OpenRouter, Together, and most gateways
# all accept it. Point `base_url` at whichever one you have.
#
# TWO RULES ABOUT SECRETS, both load-bearing:
#
# * `api_key_env` names the VARIABLE, never the key. The value is read at call
# time and never enters the suite, the cache key, or a shared run.
# * `${env:VAR:-default}` resolves at LOAD time and DOES enter the cache key.
# Use it for things that change the answer — endpoint, model, mode — and
# never for credentials.
#
# Run: domarinn run examples/26-openai-provider
# OPENAI_BASE_URL=http://localhost:11434/v1 domarinn run examples/26-openai-provider
# OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_MODEL=qwen3:4b domarinn run examples/26-openai-provider
#
# The model, like the endpoint, resolves at load time and is part of the
# cache key.
version: 1
project: examples
suite: openai-provider
providers:
- id: gpt
# The label interpolates the same variable as `model`: it exists to name
# the system under test wherever results are displayed, and a run against
# qwen3:4b must not report itself as gpt-4o-mini.
label: "${env:OPENAI_MODEL:-gpt-4o-mini}"
type: openai
model: "${env:OPENAI_MODEL:-gpt-4o-mini}"
# Defaults to the real API, and redirects with one variable — the same one
# the vendor's own SDK honours, so a gateway, a proxy or a local Ollama
# needs no edit to this file.
base_url: "${env:OPENAI_BASE_URL:-https://api.openai.com/v1}"
api_key_env: OPENAI_API_KEY
# Passed through to the API verbatim. temperature 0 is not a guarantee of
# determinism, but it removes the largest source of run-to-run noise.
params:
temperature: 0
max_tokens: 256
prompts:
- id: geography
messages:
- role: system
content: "Answer with a single sentence."
- role: user
content: "What is the capital of {{ country }}?"
tests:
- id: capital/france
vars:
country: "France"
assert:
- type: icontains
value: "Paris"
- id: capital/norway
vars:
country: "Norway"
assert:
- type: icontains
value: "Oslo"
Two rules about secrets, both load-bearing
api_key_env names the variable, never the key. The value is read at call time and never enters the suite, the cache key, or a shared run.
${env:VAR:-default} resolves at load time and does enter the cache key. Use it for things that change the answer — endpoint, model, mode. Never for credentials: keying the value would give every API key its own private cache.
The counterpart is {{ env.VAR }}, which renders per request and is keyed as a literal ${env:NAME} placeholder instead of its value. That is right for a credential and wrong for anything that changes the answer, because two values would share one cache entry and the second would replay the first's responses. domarinn warns when it sees {{ env.X }} in a URL, header or body, because it cannot tell a model selector from a token. It withholds that one hop only — a case var defined as {{ env.SECRET }} is resolved earlier and reaches the request in the clear.
Note the default: https://api.openai.com/v1. Setting OPENAI_BASE_URL — the same variable the vendor's own SDK honours — redirects the whole suite at a gateway or a local Ollama with no edit to the file. That is also exactly how this example is executed in CI, against a stub.
Example 27 — Anthropic, and what it costs¶
Same shape as the OpenAI provider, plus the one thing that deserves its own example: telling domarinn what a call costs.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Calling Anthropic, and telling domarinn what it costs.
#
# Same shape as the OpenAI provider, with two additions worth their own example:
# `pricing`, and what happens when you get it wrong.
#
# domarinn ships a rate sheet for the models it knows. A model it does NOT know
# prices at nothing — and `cost:` assertions then pass as "cost not reported;
# budget not enforced". Green, and enforcing nothing. Whenever you are on a
# negotiated rate, a preview model, or a gateway that rebills, state the price.
#
# Pricing is merged field-wise over the built-in rates and is deliberately NOT
# part of the cache key: `cost_usd` is recomputed on every cache hit from the
# stored token counts and the current rate sheet, so correcting a price re-prices
# history instead of discarding it.
#
# Run: domarinn run examples/27-anthropic-provider
version: 1
project: examples
suite: anthropic-provider
providers:
- id: claude
label: "claude-haiku-4-5"
type: anthropic
model: "claude-haiku-4-5"
# ANTHROPIC_BASE_URL is what the vendor's own tooling honours, so pointing
# this at a gateway needs no edit here.
base_url: "${env:ANTHROPIC_BASE_URL:-https://api.anthropic.com}"
api_key_env: ANTHROPIC_API_KEY
params:
max_tokens: 512
# USD per million tokens. Only the fields you state are overridden.
pricing:
input_per_mtok: 1.00
output_per_mtok: 5.00
cache_read_per_mtok: 0.10
cache_write_per_mtok: 1.25
prompts:
- id: returns-policy
messages:
- role: system
content: "You are a support agent. Cite the policy window when it applies."
- role: user
content: "{{ question }}"
tests:
- id: policy/return-window
vars:
question: "Can I return this after 45 days?"
assert:
- type: contains
value: "30 days"
# Meaningful only because `pricing` above is set and the API reports
# `usage`. Without either, this passes without enforcing anything.
- type: cost
max: 0.05
- type: tokens
max: 2000
domarinn ships a rate sheet for the models it knows. A model it does not know prices at nothing — and a cost: assertion then passes, reporting "cost not reported; budget not enforced". Green, and enforcing nothing. Whenever you are on a negotiated rate, a preview model, or a gateway that rebills, state the price.
Pricing is not in the cache key, on purpose
cost_usd is recomputed on every cache hit from the stored token counts and the current rate sheet. So correcting a price re-prices your history instead of discarding it — which is the behaviour you want the day you discover the rate was wrong.
Pricing is merged field-wise over the built-in rates, so you override only what differs.
Example 28 — A service you already run¶
If your assistant is already behind an HTTP API, type: http is the shortest path from "it exists" to "it is measured" — no SDK, no wrapper process.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Evaluating a service you already run.
#
# `type: http` posts to an endpoint and pulls the answer out of whatever JSON it
# returns. No SDK, no wrapper process — if your assistant is already behind an
# HTTP API, this is the shortest path from "it exists" to "it is measured".
#
# `output_expr` is a minijinja expression over the response, so the provider
# adapts to YOUR response shape rather than the other way round. Four things are
# in scope:
#
# response.status the HTTP status, as an integer
# response.text the raw body string
# response.json the parsed body, or null if it did not parse
# response.headers the response headers, as an object
#
# Without `output_expr`, the raw response TEXT is the output — usually not what
# you want to assert on.
#
# The `body` is rendered as a template, so `{{ }}` interpolations reach it. Note
# that a `${...}` placeholder your own backend uses is left alone: only the
# `${env:...}` sigil is claimed.
#
# Run: domarinn run examples/28-http-provider
version: 1
project: examples
suite: http-provider
providers:
- id: support-api
type: http
url: "${env:SUPPORT_API_URL:-https://api.example.com/v1/assist}"
method: POST
headers:
content-type: "application/json"
# Credentials come from the environment at call time, exactly as with the
# model providers. Never write a secret into a suite.
authorization: "Bearer {{ env.SUPPORT_API_TOKEN }}"
body:
message: "{{ user_message }}"
locale: "{{ locale }}"
# Reach into the parsed body for the field that is actually the answer.
# Note `response.json`, not `response` — the latter is the envelope.
output_expr: "response.json.result.reply"
defaults:
vars:
locale: "en"
tests:
- id: orders/status
vars:
user_message: "Where is order 1042?"
assert:
- type: icontains
value: "shipped"
- type: icontains-any
values: ["Tuesday", "Thursday"]
output_expr is a minijinja expression over the response, so the provider adapts to your shape rather than the other way round. Four things are in scope:
| Expression | What it is |
|---|---|
response.status |
The HTTP status, as an integer. |
response.text |
The raw body string. |
response.json |
The parsed body, or null if it did not parse. |
response.headers |
The response headers, as an object. |
Note it is response.json.result.reply, not response.result.reply — response is the envelope, not the body. Without output_expr at all, the raw response text is the output, which is rarely what you want to assert on.
The cache key is the request this provider would send: the rendered method, url and body, plus a digest of the rendered headers. A ${…} placeholder your own backend interprets is left untouched — only the ${env:…} sigil is claimed. output_expr is in the key too: it never goes on the wire, but the entry stores the projected output, so editing it re-asks rather than replaying the old projection.
Example 29 — LLM-rubric grading¶
llm-rubric asks a model whether an answer satisfies a rubric. It is the most expensive assertion and the easiest to misuse.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Grading what a substring cannot express.
#
# `llm-rubric` asks a model whether an answer satisfies a rubric. It is the most
# expensive assertion and the easiest to misuse, so two design decisions are
# worth knowing before you reach for it.
#
# THE VERDICT IS STRUCTURED, AND FAILS CLOSED. domarinn does not ask for prose
# and grep it. It forces a tool call (or a JSON-schema response) carrying
# `pass`, `score` and `reasoning`. A missing, malformed or TRUNCATED verdict is
# an ERROR — never a silent pass. That matters more than it sounds: a grader that
# ran out of tokens mid-sentence would otherwise score 0 and read as a genuine
# failure of the thing under test.
#
# DETERMINISTIC ASSERTIONS RUN FIRST. A case whose `contains` already failed
# never pays for the grader. Put the cheap checks above the rubric.
#
# WRITING A RUBRIC IS THE HARD PART:
#
# * Grade ONE axis. A rubric that asks about correctness AND tone AND format
# returns one number that means none of them.
# * Name the score-0 condition explicitly. "Score 0 if it agrees to do X"
# is worth more than three sentences of description.
# * Say what NOT to penalise. Graders are eager; without a "do not penalise
# verbosity / ordering / word choice" clause you measure the grader's taste.
#
# Run: domarinn run examples/29-llm-rubric-grading
version: 1
project: examples
suite: llm-rubric-grading
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
# The grader. Deliberately a DIFFERENT model from the one under test — a model
# grading its own output is not an independent measurement — and pinned
# exactly, because a rubric calibrated against one grader does not transfer.
grader:
provider:
type: anthropic
model: claude-haiku-4-5
base_url: "${env:ANTHROPIC_BASE_URL:-https://api.anthropic.com}"
# The grader reads ONLY what this names. It does NOT inherit the provider's
# credential resolution — which fails asymmetrically and confusingly:
# completions succeed while every grade dies on 401, so the run looks like
# an infra fault rather than a credential one.
api_key_env: ANTHROPIC_API_KEY
params:
# Raised well above the default on purpose. With a thinking model, a 1024
# default can truncate the structured verdict — which is a fail-closed
# error, not a silent pass. A generous ceiling costs nothing (you are
# billed for tokens generated) and removes a whole class of flake.
max_tokens: 4096
# `forced` is the default and the only value the loader accepts. `auto` is in
# the schema but was never implemented, so `domarinn validate` rejects it
# rather than quietly forcing anyway.
verdict_mode: forced
timeout_ms: 120000
tests:
- id: refusal/declines-and-redirects
vars:
user_input: "I can't help with that — it's outside what this account covers. I can look at your billing history instead."
assert:
# Cheap and first. If this fails, the grader is never called.
- type: length
min: 20
- type: llm-rubric
value: |
THIS RUBRIC MEASURES REFUSAL QUALITY ONLY.
The response MUST decline the out-of-scope request AND offer a
concrete in-scope alternative.
Score 0 if it agrees to the request, or if it declines without
offering any alternative.
Do NOT penalise the response for verbosity, for word choice, for the
particular alternative it picks, or for not apologising.
The verdict is structured, and fails closed
domarinn does not ask a grader for prose and grep it. It forces a tool call (or a JSON-schema response) carrying pass, score and reasoning. A missing, malformed or truncated verdict is an error — never a silent pass.
That matters more than it sounds. A grader that ran out of tokens mid-sentence would otherwise score 0 and read as a genuine failure of the thing under test, sending you to debug a prompt that was fine.
Three things about the grader block are deliberate. It names a different model from the one under test, because a model grading its own output is not an independent measurement. It raises max_tokens well above the default, because a thinking model can truncate a verdict at 1024 — and a generous ceiling costs nothing, since you are billed for tokens actually generated. And its api_key_env is read only by the grader: it does not inherit the provider's credential resolution, which fails asymmetrically and confusingly — completions succeed while every grade dies on 401, so the run looks like an infra fault rather than a credential one.
Writing the rubric is the hard part. Grade one axis; a rubric asking about correctness and tone and format returns one number that means none of them. Name the score-0 condition explicitly. And say what not to penalise — graders are eager, and without a "do not penalise verbosity or ordering" clause you are measuring the grader's taste.
Example 30 — Similarity¶
similar embeds the output and a reference and compares them by cosine similarity. Reach for it when an answer is right in many wordings and you would otherwise be writing an icontains-any list that never ends.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# "Close enough" as an assertion.
#
# `similar` embeds the output and a reference and compares them by cosine
# similarity. Use it when an answer is right in many wordings and you would
# otherwise be writing an `icontains-any` list that never ends.
#
# It needs a `type: embeddings` provider in the suite; without one the assertion
# ERRORS rather than passing.
#
# TWO NUMBERS, DELIBERATELY DIFFERENT:
#
# * the PASS/FAIL decision uses the RAW cosine against `threshold`
# * the reported SCORE is the cosine remapped from [-1, 1] to [0, 1],
# i.e. (cosine + 1) / 2
#
# So a threshold of 0.85 is a cosine of 0.85, not a score of 0.85. The default
# threshold is 0.8, which is looser than most people expect — unrelated sentences
# in the same domain routinely clear 0.7.
#
# Embeddings are cheap but not free, and a verdict is cached against the
# EMBEDDING MODEL: changing the model re-embeds everything, as it must, since
# cosines are not comparable across models.
#
# Against Ollama, use an embedding model, e.g.
# OPENAI_EMBED_MODEL=nomic-embed-text (chat models do not serve
# /v1/embeddings).
#
# Run: domarinn run examples/30-similar-embeddings
version: 1
project: examples
suite: similar-embeddings
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
# The embeddings provider the `similar` assertion uses. OpenAI-compatible, so
# a local endpoint works as well as the hosted one.
- id: embedder
type: embeddings
model: "${env:OPENAI_EMBED_MODEL:-text-embedding-3-small}"
base_url: "${env:OPENAI_BASE_URL:-https://api.openai.com/v1}"
api_key_env: OPENAI_API_KEY
tests:
- id: paraphrase/policy-window
vars:
user_input: "You can send it back any time in the first 30 days."
assert:
- type: similar
# Templatable, like any other assertion value.
value: "Returns are accepted within 30 days of delivery."
# Stated explicitly rather than left to the 0.8 default — a threshold
# you did not choose is a threshold you cannot defend.
threshold: 0.85
Two numbers, deliberately different
The pass/fail decision uses the raw cosine against threshold. The reported score is the cosine remapped from [-1, 1] to [0, 1], i.e. (cosine + 1) / 2.
So threshold: 0.85 means a cosine of 0.85, not a score of 0.85. And the default threshold of 0.8 is looser than most people expect — unrelated sentences in the same domain routinely clear 0.7.
It needs a type: embeddings provider in the suite; without one the assertion errors rather than passing. What gets cached is the two embeddings, not the similarity: each side is keyed on its own embedding request, which names the model. So changing the model re-embeds everything — as it must, since cosines are not comparable across models — while measuring the same output against a new reference re-embeds only the reference.
Example 31 — Budgets¶
Three assertions answer "is this answer affordable" rather than "is it right".
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Cost, latency, and token budgets.
#
# Three assertions answer "is this answer affordable" rather than "is it right":
#
# cost max USD for the call
# tokens max tokens (total, or `count: billable` to exclude cached reads)
# latency max milliseconds
#
# EACH ONE CAN PASS WITHOUT ENFORCING ANYTHING, which is the thing to know:
#
# * `cost:` passes when nothing priced the call — "cost not reported; budget
# not enforced". That happens when the provider reports no usage, or when
# the model is not in the rate sheet and the suite sets no `pricing:` block.
# * `tokens:` needs the provider to report `usage`. Note `count: billable` is
# the LARGER number, not the smaller one: it is `total` PLUS the tokens paid
# to write a provider-side prompt cache, which are billed at a premium and
# are not part of the prompt the model answered.
# * `latency:` BYPASSES THE CACHE entirely, because a replayed response has no
# honest latency to report. It measures a real call or nothing.
#
# So a green cost budget is only evidence if you know the run priced itself. The
# provider below reports usage AND cost_usd, which is what an exec provider
# should do whenever it knows — otherwise these three assertions are decoration.
#
# Run: domarinn run examples/31-budgets
version: 1
project: examples
suite: budgets
providers:
- id: metered
type: exec
command: ["python3", "./metered.py"]
cache_salt: "dev"
tests:
- id: budget/cheap-answer
vars:
user_input: "short"
assert:
- type: contains
value: "ok"
- type: cost
max: 0.01
- type: tokens
max: 500
# Generous on purpose: a tight latency bound is a flaky test on a loaded
# CI runner, and a flaky gate gets muted.
- type: latency
max: 30000
# The same call, measured two ways. It populates a prompt cache: 500 in, 40
# out, and 2000 tokens paid to write the entry. `total` is 540 and `billable`
# is 2540 — a 4.7x gap that a `total` budget cannot see at all.
- id: budget/cache-writes-are-billable
vars:
user_input: "cached"
assert:
# The default. Measures the exchange: what was sent and what came back.
- type: tokens
max: 1000
# Opting in to cache economics. Budget this when your provider writes a
# prompt cache, or the calls that cost the most are the ones you never see.
- type: tokens
max: 3000
count: billable
Each of these can pass without enforcing anything
cost:passes when nothing priced the call — literally "cost not reported; budget not enforced". That happens when the provider reports no usage, or the model is not in the rate sheet and the suite sets nopricing:block.tokens:needs the provider to reportusage.latency:bypasses the cache entirely, because a replayed response has no honest latency. It measures a real call or nothing — which is why--cache-onlyrefuses such a case outright instead of reaching the network.
A green cost budget is only evidence if you know the run priced itself.
count: billable is the larger number, not the smaller one: it is total plus the tokens paid to write a provider-side prompt cache, which are billed at a premium and are not part of the prompt the model answered. The second case shows the gap — the same call is 540 tokens by total and 2540 by billable. Budget billable when your provider writes a prompt cache, or the calls that cost the most are the ones you never see.
Keep latency bounds generous. A tight one is a flaky test on a loaded CI runner, and a flaky gate gets muted.
Example 32 — A live endpoint¶
Everything above runs offline. This one is the opposite: it points at an OpenAI-compatible endpoint that only you have, and takes the endpoint, the model, and the name of the key variable entirely from the environment.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Live smoke suite. It points an OpenAI-compatible provider at a per-developer
# endpoint supplied entirely through the environment via `${env:VAR}`
# interpolation — nothing about the endpoint, model, or key is committed here.
#
# The three variables come from a git-ignored `.mise/config.local.toml`, so mise
# injects them for you:
#
# DOMARINN_SMOKE_BASE_URL e.g. http://localhost:11434/v1
# DOMARINN_SMOKE_MODEL e.g. llama3.1
# DOMARINN_SMOKE_API_KEY the env var *name* is fixed; its value stays in env
#
# Run it (mise supplies the vars):
#
# mise exec -- domarinn validate examples/smoke/domarinn.yaml
# mise exec -- domarinn run examples/smoke/domarinn.yaml
#
# Without those variables set, `validate` fails fast: an unset `${env:VAR}` with
# no `:-default` is a hard load error that names the variable.
version: 1
project: examples
suite: smoke
providers:
- id: ollama
type: openai
# base_url and model are resolved from the environment at load time.
model: "${env:DOMARINN_SMOKE_MODEL}"
base_url: "${env:DOMARINN_SMOKE_BASE_URL}"
# api_key_env names the variable to read the key from — never the key itself.
api_key_env: DOMARINN_SMOKE_API_KEY
params:
max_tokens: 64
temperature: 0
prompts:
- id: capital
messages:
- role: user
content: "What is the capital of {{ country }}? Reply with only the city name."
tests:
- id: capital/france
vars:
country: "France"
assert:
- type: icontains
value: "Paris"
Note that api_key_env names the variable, never the key. Nothing secret is committed, and nothing secret enters the cache key. Note also that these ${env:…} interpolations carry no :-default — so with the variables unset, domarinn validate fails immediately and names the missing one, rather than a run failing later against a half-configured endpoint.
Example 33 — An OpenAI-shaped grader¶
Example 29's grader was type: anthropic. A grader is a provider like any other, so it can just as easily be type: openai — which means any OpenAI-compatible endpoint can judge, a local Ollama included.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# The grader does not have to be Anthropic.
#
# Example 29 built its grader from `type: anthropic`. This one is `type:
# openai` instead — which means ANY OpenAI-compatible endpoint can grade:
# OpenAI itself, a gateway, or a local Ollama serving the chat-completions API.
# The system under test below is the offline echo provider, so the only thing
# that reaches a network here is the grader.
#
# ONE VARIABLE MOVES THE GRADER, the same variable that moves a provider in
# example 26:
#
# domarinn run examples/33-openai-grader-rubric
# OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_MODEL=qwen3:4b \
# domarinn run examples/33-openai-grader-rubric
#
# Do not confuse this with redirecting the SYSTEM under test — here it is the
# GRADER's endpoint and model that move, because the grader is the only
# API-shaped thing this suite calls.
#
# The model and base_url resolve at load time like any other provider field,
# which means grader identity is part of the cache key: judge a case with
# gpt-4o-mini, then again with qwen3:4b, and the second run asks fresh rather
# than replaying the first grader's verdict.
#
# The verdict shape itself — structured, and fail-closed on truncation — is
# the subject of example 29; this one is only about which vendor is asking.
#
# Run: domarinn run examples/33-openai-grader-rubric
version: 1
project: examples
suite: openai-grader-rubric
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
grader:
provider:
type: openai
# Redirects with the same variable OpenAI's own SDK honours, and the same
# one example 26 uses for a provider — here it moves the GRADER instead.
model: "${env:OPENAI_MODEL:-gpt-4o-mini}"
base_url: "${env:OPENAI_BASE_URL:-https://api.openai.com/v1}"
api_key_env: OPENAI_API_KEY
params:
# See example 29: a thinking-capable grader can truncate a structured
# verdict at a small ceiling, which is a fail-closed error, not a
# silent pass. Generous costs nothing extra — you are billed for
# tokens actually generated.
max_tokens: 4096
# `forced` is the default and the only value the loader accepts. `auto` is in
# the schema but was never implemented, so `domarinn validate` rejects it
# rather than quietly forcing anyway.
verdict_mode: forced
timeout_ms: 120000
tests:
- id: policy/no-invented-exceptions
vars:
user_input: "Our policy allows returns within 30 days of delivery. I'm not able to make exceptions beyond that window, but I can check whether a partial credit applies to your order."
assert:
# Cheap and first. If this fails, the grader is never called.
- type: contains
value: "30 days"
- type: llm-rubric
value: |
THIS RUBRIC MEASURES POLICY FIDELITY ONLY.
The response MUST state the 30-day return window AND must NOT
invent, promise, or imply any exception to it.
Score 0 if it invents a detail the policy above never stated, or
promises an exception the input never granted.
Do NOT penalise the response for brevity, for word choice, or for
offering to check a partial credit — that is not an exception to
the window, it is a separate offer.
The system under test here is the offline echo provider, so the only thing that reaches a network is the grader — and the same one variable that redirects a provider in example 26 redirects this grader instead:
OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_MODEL=qwen3:4b domarinn run examples/33-openai-grader-rubric
Grader identity is part of the cache key
The model and base_url resolve at load time like any other provider field. Judge a case with gpt-4o-mini, then again with qwen3:4b, and the second run asks fresh rather than replaying the first grader's verdict — two graders are two different questions, never one cache entry.
The rubric itself follows example 29's rules: one axis (policy fidelity, not tone or brevity), an explicit score-0 condition, and a line saying what not to penalise. The verdict's wire shape — a strict json_schema response this time, not a forced tool call — is documented in full; the rubric-writing advice does not repeat here.
Example 35 — Anthropic tools, natively¶
Example 15 declared tools for an exec provider — your own program decided whether to call one. The same tools: block and the same tool-call assertion work unchanged against type: anthropic: the suite declares tools once, and every API-shaped provider gets the same surface, in that vendor's native shape.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Tool calls, over the wire this time.
#
# Example 15 declared tools for an `exec` provider — your own program decided
# whether to call one, and reported the decision itself. This is the same
# `tools:` block and the same `tool-call` assertion, aimed at `type:
# anthropic` instead: the suite declares tools once, and every API-shaped
# provider in it is offered the same surface in that vendor's native shape.
#
# domarinn still never EXECUTES a tool (see docs/reference/protocol.md#tools)
# — it only grades the decision, whichever transport carried it.
#
# Run: domarinn run examples/35-anthropic-tools
version: 1
project: examples
suite: anthropic-tools
providers:
- id: claude
type: anthropic
model: "claude-haiku-4-5"
base_url: "${env:ANTHROPIC_BASE_URL:-https://api.anthropic.com}"
api_key_env: ANTHROPIC_API_KEY
params:
max_tokens: 512
# Suite-level, so every provider is offered the same surface — see example 15
# for the two-tool version and the read-only/dangerous-tool distinction.
tools:
- name: lookup_order
description: Fetch an order by its id.
input_schema:
type: object
required: ["order_id"]
properties:
order_id: { type: integer }
prompts:
- id: support
messages:
- role: user
content: "{{ user_input }}"
tests:
- id: agent/looks-up-order
vars:
user_input: "What's the status of order 1042?"
assert:
- type: tool-call
name: lookup_order
# A subset match, as in example 15: the call may carry more, but this
# must be present and equal.
args:
order_id: 1042
domarinn still never executes a tool, whichever transport carried the decision — see Tools.
Example 36 — output_expr, sliced two ways¶
Example 28 pulled one nested string field out of a JSON body. output_expr is not limited to strings, or to one shape per suite — this one points two providers at the same backend and gives each a different expression.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# `output_expr` does not have to return a string.
#
# Example 28 pulled one nested string field out of a JSON body. This suite
# points TWO providers at the same backend and gives each a different
# `output_expr`, to show more of what the expression can reach:
#
# * a nested field, same as example 28 — response.json.result.reply
# * a field that is NOT a string at all — response.json.result.confidence,
# a bare number
#
# Whichever `output_expr` returns becomes the output as-is: a string stays
# text, anything else — a number, an object, an array — becomes structured
# output that every text assertion still reads (stringified), and that
# `is-json` / `contains-json` could inspect directly.
#
# `output_expr` only ever sees a SUCCESSFUL response: a non-2xx status is a
# transport error before the expression runs, same as any other provider's
# failure.
#
# The two expressions below reach `response.json`. That is one of several
# things `response.*` exposes — status, headers and the raw body among them —
# and docs/reference/providers.md#http lists the whole surface.
#
# Run: domarinn run examples/36-http-output-expr
version: 1
project: examples
suite: http-output-expr
providers:
- id: orders-reply
type: http
url: "${env:ORDERS_API_URL:-https://api.example.com/v1/orders/assist}"
body:
message: "{{ user_input }}"
output_expr: "response.json.result.reply"
# Same backend, sliced differently: this one reaches past the reply text
# into a numeric field the first provider never touches.
- id: orders-confidence
type: http
url: "${env:ORDERS_API_URL:-https://api.example.com/v1/orders/assist}"
body:
message: "{{ user_input }}"
output_expr: "response.json.result.confidence"
tests:
- id: orders/reply-text
only_providers: [orders-reply]
vars:
user_input: "Where is order 1042?"
assert:
- type: icontains
value: "shipped"
- id: orders/confidence-score
only_providers: [orders-confidence]
vars:
user_input: "Where is order 1042?"
assert:
- type: contains
value: "0.93"
only_providers (from example 16) scopes each test to the provider it is actually testing — the two providers are not being compared, they are answering two different questions about the same response.
Whatever output_expr evaluates to becomes the output as-is: a string stays text, and anything else — a number here, but the same holds for an object or an array — becomes structured output. Every text assertion still reads it (stringified), and is-json / contains-json could inspect it directly. output_expr only ever sees a successful response: a non-2xx status is a transport error before the expression runs, same as any other provider's failure.
Example 38 — Every key, once¶
The rest of the ladder takes one capability at a time. This suite takes the shape of the file itself: every top-level key domarinn.yaml accepts, set exactly once, each with a comment saying what it is and where its full story lives.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# The whole shape of a suite, at a glance.
#
# Every top-level key a domarinn suite accepts appears here exactly once, and
# every one carries a comment saying what it is and WHERE ITS FULL STORY LIVES.
# Read it as a table of contents for the rest of the documentation.
#
# The suite itself is deliberately tiny — one provider, one prompt, three cells
# — because the comments are the subject. This is a MAP, not a recommendation:
# a real suite sets a fraction of these keys and says nothing about the rest.
#
# The field-by-field reference is docs/reference/domarinn-yaml.md. The machine
# version is `domarinn schema config`, generated from the same structs the
# loader deserializes into, so neither can drift from what actually loads.
#
# Run: domarinn run examples/38-annotated-reference-suite
# `version` — the config schema version. Required, and currently always 1.
version: 1
# `project` — namespaces this suite's runs on the results server. Optional.
# docs/reference/server.md
project: examples
# `suite` — names the suite inside that project. It is the unit `--against`
# compares run over run, so changing it starts a new history.
# examples/24-baselines-and-diff
suite: annotated-reference
# `description` — free prose, carried into the run document. Nothing reads it as
# configuration.
description: >-
Every top-level suite key in one file, annotated with where its full story
lives.
# `extends` — ONE base suite this file is merged on top of.
# `imports` — an ordered list of fragments merged after that base.
#
# Precedence runs low to high: the base, each fragment in listed order, then
# this file. Maps merge key by key and sequences are replaced wholesale —
# except an `assert` sequence, which appends.
# examples/17-defaults-and-composition owns the mechanics
# docs/reference/domarinn-yaml.md#composition-with-extends-and-imports
extends: "file://base.yaml"
imports:
- "file://house-rules.yaml"
# `providers` — the systems under test. At least one is required, and every case
# is evaluated against every provider, so a second entry compares two systems
# rather than replacing the first.
#
# This one runs a program over the exec protocol, which is what a real suite
# points at its own code.
# examples/13-exec-provider, docs/reference/providers.md,
# docs/reference/protocol.md
providers:
- id: assistant
type: exec
# Resolved relative to THIS FILE's directory, not the shell's, so the suite
# runs identically from the repository root or from inside this folder.
command: ["python3", "../echo-provider.py"]
# Version-pins the program behind `command`. The cache key hashes what the
# program is SENT, never its bytes, so an edited program keeps answering
# from the entries it already wrote until this changes.
# docs/concepts/caching.md#salts
cache_salt: "dev"
# `prompts` — templates rendered per case from that case's `vars`. Optional: an
# exec provider can build its own input from the vars directly.
#
# A run is a grid, so one prompt against three cases is three cells; a second
# prompt here would make six.
# examples/02-prompts-and-vars, docs/reference/domarinn-yaml.md#prompts
prompts:
- id: support-reply
template: |
Reply to the customer in a {{ register }} register, naming {{ product }}.
Customer: {{ user_input }}
# `tools` — the tool surface every provider in this suite offers the model.
#
# Declaring a tool never makes domarinn RUN one: what it wants is the model's
# decision, reported back as `tool_calls` and graded by a `tool-call`
# assertion. The echo provider below ignores the offer, so nothing here is
# graded — the examples that grade a decision are 15 (exec) and 35 (the native
# Anthropic API).
# docs/reference/protocol.md#tools
tools:
- name: lookup_order
description: "Look up an order by id."
# A contract, not a per-case value: passed to the provider verbatim and
# never rendered as a template.
input_schema:
type: object
properties:
order_id: { type: integer }
required: [order_id]
# `tests` — the cases. Three sources exist and can be mixed in one list: an
# inline case (both below), a `file://` glob of YAML/JSON/CSV/JSONL cases
# (examples 09 and 10), and a generator command that prints cases as JSON
# (example 11). Only the inline form appears here.
# docs/reference/domarinn-yaml.md#tests
tests:
- id: policy/names-the-product
vars:
user_input: "How long do I have to send a jacket back?"
assert:
# Deterministic asserts run first and short-circuit the grader, so a case
# that already failed never pays for a grader call.
# docs/reference/assertions.md
- type: contains
value: "Aurora Notes"
# The one graded assert in this file, so the `grader` block below is real
# rather than decorative. Writing a rubric is example 29's subject.
#
# The system under test is the echo provider, which mirrors its input, so
# what the grader actually reads here is the RENDERED PROMPT (the same
# trick example 02 explains). Point `command` at your own program and it
# reads a real answer instead.
- type: llm-rubric
value: |
THIS RUBRIC MEASURES ONE THING: whether the text names the product
the customer is being answered about.
Score 0 if it names no product at all.
Do NOT penalise it for reading as an instruction rather than as an
answer, for length, or for word choice.
# `matrix` — a parameter sweep. One axis with two values fans this single case
# out into two cells; a second axis would give the cartesian product. Each
# axis value lands in `vars`, and `matrix_id` names the cells (the default id
# is `<base>[key=value,…]`).
# examples/08-matrix-sweeps,
# docs/reference/domarinn-yaml.md#matrix--parameter-sweeps
- id: register
matrix_id: "register/{{ register }}"
matrix:
register: ["formal", "casual"]
vars:
user_input: "Hi — can I still return this?"
assert:
- type: contains
value: "Aurora Notes"
# `defaults` — values merged into every test. `vars` fill gaps only (a case's
# own value wins, which is how the matrix axis above overrides `register`),
# `assert` entries are PREPENDED to each case's own, `tags` union, and
# `threshold` / `cache_salt` fill in if unset.
#
# The house rule imported above lands in this same list, ahead of the entry
# below.
# docs/reference/domarinn-yaml.md#defaults
defaults:
vars:
product: "Aurora Notes"
register: "plain"
tags: ["reference"]
assert:
- type: length
max: 400
# `grader` — the default grader for `llm-rubric` assertions. A grader is a
# provider like any other, so any OpenAI-compatible endpoint can judge, a local
# Ollama included; the verdict is structured and fails closed on truncation
# rather than scoring zero.
#
# Parameterized by environment exactly as example 33 documents, which is also
# how it is executed offline in CI:
# OPENAI_BASE_URL=http://localhost:11434/v1 OPENAI_MODEL=qwen3:4b \
# domarinn run examples/38-annotated-reference-suite
# examples/29-llm-rubric-grading (the verdict shape),
# examples/33-openai-grader-rubric (this env-parameterized form),
# docs/concepts/grading.md
grader:
provider:
type: openai
model: "${env:OPENAI_MODEL:-gpt-4o-mini}"
base_url: "${env:OPENAI_BASE_URL:-https://api.openai.com/v1}"
# Names the VARIABLE, never the key: the value is read at call time and
# never enters the suite, the cache key, or a shared run.
api_key_env: OPENAI_API_KEY
params:
max_tokens: 4096
verdict_mode: forced
timeout_ms: 120000
# `runner` — how the run is scheduled. Two of its six knobs are set here;
# `rate_limit`, `timeout_ms`, `short_circuit` and `skip_on_empty_reason` are
# documented alongside them, and base.yaml contributes the `timeout_ms` this
# file never restates.
# examples/20-runner-tuning, docs/reference/domarinn-yaml.md#runner
runner:
concurrency: 2
retries:
max: 2
jitter: true
# `cache` — where cached responses live. `disk` (the default) is a local
# content-addressed store; `layered` fronts it with a shared remote so a team
# and CI stop paying for each other's answers.
# examples/21-caching-basics, docs/concepts/caching.md#backends
cache:
backend: disk
It composes, so two smaller files ship beside it — the extends base and the imports fragment, in that precedence order:
# The `extends` layer: ONE base suite that domarinn.yaml is merged on top of.
# Not run on its own.
#
# What belongs in a base is whatever a family of suites shares and should not
# restate — here, the scheduling budget. See examples/17-defaults-and-composition
# for the merge rules in full.
version: 1
runner:
# Inherited rather than restated: the child declares a `runner` block of its
# own and never mentions `timeout_ms`, and maps merge key by key, so the
# composed suite carries this AND the child's knobs.
timeout_ms: 60000
# An `imports` FRAGMENT: not a suite, just the part of one that several suites
# share. Merged after the `extends` base and before the local file.
#
# A shared `assert` sequence APPENDS rather than replacing, which is why a
# fragment is the right home for a house rule: a suite that declares its own
# `defaults.assert` cannot drop this one by accident.
defaults:
assert:
- type: not-icontains
value: "as an AI"
Read it as a map, not as a recommendation. A real suite sets a fraction of these keys, and the three cells here exist only so the file is a suite that runs rather than a listing — the grader block is example 33's env-parameterized form, which is how the grader is pointed at a stub in CI.
"Every key" is checked, not claimed
A guard in crates/domarinn-cli/tests/examples/docs_guards.rs reads the top-level properties out of the committed JSON Schema — the same schema CI pins to the config structs — and fails if this file stops setting one of them. So a key added to the config cannot leave this page quietly incomplete.