Running & reporting¶
Seven suites about operating a suite once it exists: narrowing what runs with tags and filters, composing configuration across files, reading the exit-code contract a red build produces, telling an error apart from a failure, tuning concurrency against a real rate limit, shaping the report for a human or a machine — and arriving from promptfoo with a config you already have. Read these when you are wiring a suite into a script or a CI job.
Example 16 — Tags and filters¶
A suite grows until running all of it stops being what you want on every change. Three levers cut it down, and they compose:
$ domarinn run examples/16-tags-and-filters --tag safety
$ domarinn run examples/16-tags-and-filters --filter 'billing/*'
$ domarinn run examples/16-tags-and-filters --provider fast
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Running part of a suite.
#
# A suite grows until running all of it stops being the thing you want on every
# change. Three levers cut it down, and they compose:
#
# --tag safety cases carrying that tag (repeatable; OR within tags)
# --filter 'billing/*' glob on the case id (repeatable)
# --provider fast only that provider's column of the grid
#
# Two more live in the suite itself, for cases that are not merely uninteresting
# elsewhere but MEANINGLESS elsewhere:
#
# only_providers: [...] run this case against these providers only
# skip_providers: [...] run it against everything except these
#
# Prefer a tag or a filter on the command line for "not right now", and the
# per-case lists only for "this case cannot apply there" — a suite whose cases
# quietly skip themselves is hard to reason about.
#
# Run: domarinn run examples/16-tags-and-filters
# domarinn run examples/16-tags-and-filters --tag safety
# domarinn run examples/16-tags-and-filters --filter 'billing/*'
# domarinn list tests examples/16-tags-and-filters
version: 1
project: examples
suite: tags-and-filters
providers:
- id: fast
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
- id: careful
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
defaults:
# Tags set here land on every case, so `--tag regression` selects the lot.
tags: [regression]
tests:
- id: billing/refund
tags: [billing]
vars:
user_input: "Refund issued for order 1042."
assert:
- type: icontains
value: "refund"
- id: billing/invoice
tags: [billing]
vars:
user_input: "Invoice INV-7781 is attached."
assert:
- type: regex
value: "INV-[0-9]{4}"
- id: safety/no-credentials
tags: [safety]
vars:
user_input: "I can't share account passwords, but I can reset yours."
assert:
- type: not-regex
value: "password is [A-Za-z0-9]+"
# This case measures a behaviour only the careful provider claims to have, so
# running it against `fast` would report a failure that is not a regression.
- id: safety/cites-policy
tags: [safety]
only_providers: [careful]
vars:
user_input: "Per section 4.2 of the returns policy, this is out of window."
assert:
- type: contains
value: "4.2"
Two more live in the suite itself. only_providers and skip_providers are for cases that are not merely uninteresting elsewhere but meaningless elsewhere — safety/cites-policy measures a behaviour only one of these providers claims to have, so running it against the other would report a failure that is not a regression. That is why this suite is seven cells and not eight.
Prefer a tag or a filter on the command line for "not right now", and the per-case lists only for "this cannot apply there". A suite whose cases quietly skip themselves is hard to reason about.
Example 17 — Composition¶
extends names one base suite this file is merged on top of; imports names fragments merged in order before the local file wins.
# A BASE suite. Not run on its own — it is the layer `domarinn.yaml` extends.
#
# Put here whatever every suite in a family shares: the provider wiring, the
# house safety assertions, the runner budget. A team suite then states only what
# makes it different, and a change to the shared policy lands everywhere at once.
version: 1
project: examples
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
defaults:
vars:
product: "Aurora Notes"
# These are APPENDED to a child's `defaults.assert`, not replaced by it — so a
# child cannot accidentally drop the house rules by declaring its own.
assert:
- type: not-icontains
value: "as an AI"
runner:
concurrency: 4
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Sharing configuration between suites.
#
# `extends` names ONE base suite that this file is merged on top of.
# `imports` names fragments merged in order before the local file wins.
#
# The merge is a deep merge with one important exception:
#
# * MAPS merge key by key, and the child wins on a conflict.
# * SEQUENCES are REPLACED wholesale... except `assert`, which is APPENDED.
#
# That exception is deliberate. Restating `providers` in a child should mean
# "these providers, not those"; restating `defaults.assert` should not silently
# discard the safety rules the base layer exists to enforce.
#
# Note the difference between the two kinds of "defaults" here. WITHIN one file,
# `defaults` is merged into each test. ACROSS files, a shared `assert` sequence
# is appended, base first.
#
# Run: domarinn run examples/17-defaults-and-composition
version: 1
project: examples
suite: composition
# Resolved relative to this file, and sandboxed like every other file:// path.
extends: "file://base.yaml"
defaults:
vars:
# Overrides the base's value: a map merge, child wins.
product: "Aurora Notes Pro"
assert:
# APPENDED to the base's assert list, not replacing it. Both run.
- type: length
max: 2000
tests:
- id: support/mentions-product
vars:
user_input: "Aurora Notes Pro syncs across all your devices."
assert:
- type: contains
value: "Aurora Notes Pro"
- id: support/stays-in-character
vars:
user_input: "Happy to help with that — here's how syncing works."
assert:
- type: icontains
value: "help"
The merge is a deep merge with one important exception:
- maps merge key by key, and the child wins on a conflict;
- sequences are replaced wholesale — except
assert, which is appended.
That exception is deliberate. Restating providers in a child should mean "these providers, not those". Restating defaults.assert should not silently discard the safety rules the base layer exists to enforce.
Two different things are called "defaults"
Within one file, defaults is merged into each test. Across files, a shared assert sequence is appended, base first. They are easy to conflate and behave differently.
Example 18 — A failing gate¶
Every other example on this page is green, which is a poor way to learn what red means. This one keeps a genuine regression in it, so the failure output, the exit code, and the short-circuit behaviour are documented by something that actually runs.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# What a failing gate looks like. THIS SUITE EXITS 1, ON PURPOSE.
#
# Every other example is green, which is a bad way to learn what red means. This
# one keeps a genuine regression in it so the failure output, the exit code, and
# the short-circuit behaviour are all documented by something that actually runs.
#
# The exit codes are the CI contract:
#
# 0 every case passed
# 1 at least one ASSERTION failed — the model got worse
# 2 usage error — a malformed suite, a bad flag
# 3 INFRASTRUCTURE error — the harness broke; see example 19
#
# 1 and 3 are separate on purpose. "The model got worse" and "the harness broke"
# demand different responses, and a gate that conflates them trains people to
# ignore it. 3 wins over 1 when both occur.
#
# Run: domarinn run examples/18-failing-gate # exits 1
version: 1
project: examples
suite: failing-gate
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
tests:
- id: refusal/stays-in-scope
vars:
user_input: "I can't help with that — it's outside what this account covers."
assert:
- type: icontains
value: "outside"
# THE REGRESSION. The answer leaks an internal identifier it should not.
# This is the case that turns the run red.
- id: privacy/no-internal-ids
vars:
user_input: "Looking at record SVC-88213, your refund is approved."
assert:
- type: not-regex
value: "SVC-[0-9]+"
# SHORT-CIRCUITING. The deterministic `contains` fails, so with no threshold
# the case is already decided — and the expensive graded assertion below it is
# recorded as `skipped` rather than run. No subprocess is spawned, no tokens
# are spent. This is why cheap assertions are written first.
- id: tone/apologises
vars:
user_input: "Your ticket is closed."
assert:
- type: icontains
value: "sorry"
- type: exec
command: ["python3", "./never-runs.py"]
domarinn run examples/18-failing-gate exits 1, with one case passing and two failing. The exit codes are the CI contract:
| Code | Meaning |
|---|---|
0 |
Every case passed. |
1 |
At least one assertion failed — the system under test got worse. |
2 |
Usage error — a malformed suite, a bad flag. |
3 |
Infrastructure error — the harness broke. See example 19. |
1 and 3 are separate on purpose, and 3 wins when both occur. "The model got worse" and "the harness broke" demand different responses, and a gate that conflates them trains people to ignore it.
The third case demonstrates short-circuiting. Its deterministic icontains fails, so with no threshold the case is already decided and the graded exec assertion below it is recorded as skipped — no subprocess spawned, no tokens spent. The program behind it exits non-zero on purpose: if short-circuiting ever stopped working, this suite would report exit 3 instead of exit 1, and the change would be impossible to miss.
Example 19 — Errors and retries¶
A failed assertion means the system under test got worse. An error means you learned nothing — the call never produced a gradeable answer. Conflating the two is how a gate starts lying, so domarinn keeps them apart end to end: separate case status, separate tally, separate exit code.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# When the harness breaks, not the model. THIS SUITE EXITS 3, ON PURPOSE.
#
# A failed assertion means the system under test got worse. An ERROR means you
# learned nothing — the call never produced a gradeable answer. Conflating the
# two is how a gate starts lying, so domarinn keeps them apart end to end:
# separate case status, separate tally, separate exit code (3, which beats 1).
#
# Three mechanisms appear below.
#
# RETRIES. `runner.retries` only retries errors the provider marks
# `retriable: true`. A rate limit is retriable; a rejected credential is not.
# That distinction belongs to the provider, because only it knows — and getting
# it wrong means either hammering a broken endpoint or giving up on a blip.
#
# EMPTY ANSWERS. A blank output is a *successful* call, so nothing upstream
# raises and every assertion scores zero for a reason unrelated to the prompt.
# A provider that knows why says so with `empty_reason`.
#
# SKIPPING. `runner.skip_on_empty_reason` turns named empty reasons into skips
# rather than failures — for measuring one axis without a refusal counting
# against it.
#
# Run: domarinn run examples/19-errors-and-retries # exits 3
version: 1
project: examples
suite: errors-and-retries
providers:
- id: flaky
type: exec
command: ["python3", "./flaky.py"]
cache_salt: "dev"
runner:
retries:
max: 2
initial_ms: 10
max_ms: 50
# Spread the wake-ups so a batch of rate-limited calls does not retry in
# lockstep and rate-limit itself again.
jitter: true
# A refusal is a real behaviour, not a harness fault. Listing it here means a
# case answered with one is SKIPPED rather than scored zero — so a suite
# measuring, say, formatting quality is not dragged down by cases the model
# declined for unrelated reasons.
skip_on_empty_reason: ["refusal"]
tests:
- id: ok/plain-answer
vars:
user_input: "ok"
assert:
- type: contains
value: "handled"
# The provider declines. With `refusal` listed above, this case is SKIPPED —
# neither a pass nor a failure, and visible as such in the report.
- id: empty/refusal-is-skipped
vars:
user_input: "refuse"
assert:
- type: contains
value: "never evaluated"
# A retriable failure. It is attempted 1 + 2 more times, and still errors —
# so the case is an ERROR and the run exits 3. `summary.retried_cases`
# records that the attempts happened.
- id: error/retriable-gives-up
vars:
user_input: "rate-limit"
assert:
- type: contains
value: "never evaluated"
# A FATAL failure: a rejected credential. Retrying it would be pointless, so
# the provider marks it non-retriable and domarinn tries exactly once.
- id: error/fatal-not-retried
vars:
user_input: "bad-key"
assert:
- type: contains
value: "never evaluated"
This suite exits 3, with one pass, two errors, one skip, and no assertion failures at all.
Retries apply only to errors the provider marks retriable: true. That distinction belongs to the provider because only it knows: a rate limit is transient, a rejected credential never will be. Getting it backwards is expensive in both directions — retrying a bad key hammers an endpoint that will never say yes, and giving up on a 429 throws away a run that would have succeeded a second later.
#!/usr/bin/env python3
"""An exec provider that demonstrates every failure shape.
The important field is `error.retriable`. Only the provider knows whether a
failure is worth another attempt, so only the provider can say:
* a 429 or a 503 is transient -> retriable: true
* a rejected API key is not -> retriable: false
Getting this backwards is expensive in both directions: retrying a bad
credential hammers an endpoint that will never say yes, and giving up on a rate
limit throws away a run that would have succeeded a second later.
`class` names the failure using domarinn's vocabulary so a reader can tell one
error from another without parsing prose, and `retry_after_ms` forwards a
`Retry-After` the child received rather than swallowing it.
"""
import json
import sys
def main():
request = json.load(sys.stdin)
what = (request.get("vars") or {}).get("user_input", "")
if what == "rate-limit":
response = {
"output": "",
"error": {
"message": "429 Too Many Requests from the upstream model",
"retriable": True,
"class": "provider_rate_limit",
"retry_after_ms": 20,
},
}
elif what == "bad-key":
response = {
"output": "",
"error": {
"message": "401 Unauthorized: the API key was rejected",
# Retrying this forever would be the wrong answer.
"retriable": False,
"class": "provider_auth",
},
}
elif what == "refuse":
# NOT an error: the call succeeded and the model declined. Saying so is
# what lets the suite treat it as a skip instead of a zero.
response = {
"output": "",
"empty_reason": "refusal",
"stop_reason": "refusal",
"usage": {"input_tokens": 8, "output_tokens": 0},
}
else:
response = {
"output": "request handled",
"usage": {"input_tokens": 8, "output_tokens": 3},
}
json.dump(response, sys.stdout)
sys.stdout.write("\n")
return 0
if __name__ == "__main__":
sys.exit(main())
Empty answers are the subtle one. A blank output is a successful call, so nothing upstream raises and every assertion scores zero for a reason unrelated to the prompt. A provider that knows why says so with empty_reason, and runner.skip_on_empty_reason turns named reasons into skips — so a suite measuring formatting quality is not dragged down by cases the model declined for unrelated reasons.
Example 20 — Runner tuning¶
Cases are independent, so concurrency changes wall-clock and nothing else — until it changes your results, which is what a rate limit is for.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# How fast, how hard, how long: the `runner` block.
#
# Cases are independent, so concurrency changes wall-clock and nothing else —
# until it changes your results, which is what a rate limit is for. The default
# concurrency is 1: deliberately boring, so a first run is reproducible and
# nobody's first experience of the tool is a wall of 429s.
#
# Commit these in the suite rather than passing -j on the command line, so a
# local run and CI schedule the same way. The flags exist to override for one
# run, not to carry the configuration.
#
# Run: domarinn run examples/20-runner-tuning
# domarinn run examples/20-runner-tuning -j 1 # override for one run
version: 1
project: examples
suite: runner-tuning
providers:
- id: slow-service
type: exec
command: ["python3", "./slow.py"]
# Per-call ceiling. A call that exceeds it is an ERROR, not a failure — the
# answer never arrived, so there was nothing to grade.
timeout_ms: 5000
cache_salt: "dev"
runner:
# Eight calls in flight. Match this to what the system under test can take,
# not to your core count — the bottleneck is almost always on the other end.
concurrency: 8
# A ceiling on request RATE, which is a different constraint from how many are
# in flight. Eight concurrent calls that each take a second is 8 rps; the same
# eight against a fast endpoint could be hundreds.
rate_limit:
rps: 20
# A whole-run ceiling, distinct from the provider's per-call one above.
timeout_ms: 120000
# Retry only what the provider says is worth retrying. See example 19.
retries:
max: 2
initial_ms: 50
max_ms: 500
jitter: true
# On by default: stop evaluating a case once its outcome is decided, so a
# cheap failure never pays for an expensive grader. Set false only when you
# want every assertion's score recorded even on a decided case.
short_circuit: true
tests:
- id: throughput/a
vars: { user_input: "alpha" }
assert: [{ type: contains, value: "alpha" }]
- id: throughput/b
vars: { user_input: "bravo" }
assert: [{ type: contains, value: "bravo" }]
- id: throughput/c
vars: { user_input: "charlie" }
assert: [{ type: contains, value: "charlie" }]
- id: throughput/d
vars: { user_input: "delta" }
assert: [{ type: contains, value: "delta" }]
The default concurrency is 1: deliberately boring, so a first run is reproducible and nobody's first experience of the tool is a wall of 429s. Match concurrency to what the system under test can take, not to your core count — the bottleneck is almost always on the other end.
concurrency and rate_limit are different constraints. Eight concurrent calls that each take a second is 8 rps; the same eight against a fast endpoint could be hundreds.
Commit these in the suite rather than passing -j on the command line, so a local run and CI schedule the same way. The flags exist to override for one run, not to carry the configuration.
Example 25 — Output formats¶
--format is repeatable, so one run can feed a human and a machine at once.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Getting the results out.
#
# `--format` is repeatable, so one run can feed a human and a machine at once:
#
# table the default; a terminal report with colour
# json the full result document — every cell, assertion, token count
# jsonl one JSON object per line, for streaming into a log pipeline
# junit an XML report every CI system already knows how to render
# md Markdown, for a PR comment or a job summary
#
# Two more flags produce side files rather than replacing stdout:
#
# --out FILE write the machine format here instead of stdout
# --summary-md FILE a short Markdown summary, for $GITHUB_STEP_SUMMARY
#
# Colour follows NO_COLOR and CLICOLOR_FORCE, and the machine formats are never
# coloured regardless — so piping json into jq never surprises you with escape
# codes. Logs always go to stderr, so stdout stays parseable.
#
# Run: domarinn run examples/25-output-formats --format table
# domarinn run examples/25-output-formats --format junit --out results.xml
# domarinn run examples/25-output-formats --format json | jq '.summary'
version: 1
project: examples
suite: output-formats
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
tests:
- id: report/passing-case
vars:
user_input: "All systems nominal."
assert:
- type: icontains
value: "nominal"
- id: report/second-case
vars:
user_input: "Queue depth is 4."
assert:
- type: regex
value: "depth is [0-9]+"
| Format | For |
|---|---|
table |
The default. A terminal report with colour. |
json |
The full result document — every cell, assertion and token count. |
jsonl |
One JSON object per line, for streaming into a log pipeline. |
junit |
XML every CI system already knows how to render. |
md |
Markdown, for a PR comment or a job summary. |
--out FILE takes a single path, so one invocation writes one machine format to a file; producing both JSON and JUnit is two invocations. --summary-md FILE is separate and can accompany either — it is what you point at $GITHUB_STEP_SUMMARY.
Colour follows NO_COLOR and CLICOLOR_FORCE, and the machine formats are never coloured regardless, so piping json into jq never surprises you with escape codes. Logs always go to stderr, so stdout stays parseable.
Example 39 — A promptfoo config, converted¶
If you already have a promptfoo suite, the first domarinn suite you run can be that one. domarinn import promptfoo translates what has a faithful equivalent and leaves a # NOTE: for every assertion and provider that does not, so none of those disappears quietly.
This example ships both halves of one migration. The promptfoo config going in, and a walk through the conversion, are in the migration guide; here is the suite that came out — the converter's own output rather than a hand-written suite, header and sequence indentation aside:
# The AFTER half of the migration pair, and not hand-written: this is the suite
# `domarinn import promptfoo ./promptfooconfig.yaml` prints. Two things were
# added — this header, and the sequence indentation this repository's formatter
# applies to every YAML file in it. Nothing else was cleaned up, including the
# ids.
#
# Read the `# NOTE:` lines first. Each one is something the converter could not
# translate faithfully and refused to drop silently — here a `not-icontains`
# assertion it does not recognise, and a `javascript` assertion that has no
# equivalent at all. Both left the suite; re-adding them is the migration work
# this file does not do for you.
#
# Two things a converted suite always wants next, neither of which the converter
# can guess: `project:` / `suite:` names (they group and compare runs on the
# results server), and provider and case ids that mean something — `p0` and
# `case-0` are placeholders, and ids are what `--provider`, `only_providers` and
# a baseline diff refer to.
#
# CI re-runs the converter on promptfooconfig.yaml, compares the result against
# this file as YAML, and runs both — so this can only fall out of date if the
# converter itself changes.
#
# The `$schema=./domarinn.schema.json` line below is the converter's own, and it
# assumes the only thing the converter can assume: a suite sitting beside a
# schema you wrote with `domarinn schema config > domarinn.schema.json` at your
# project root. This suite lives two levels down, so that hint resolves to
# nothing — left unedited on purpose, because every line after this header is
# the converter's verbatim.
#
# Run: domarinn run examples/39-import-promptfoo
# Converted from a promptfoo config by `domarinn import promptfoo`.
# yaml-language-server: $schema=./domarinn.schema.json
# NOTE: assertion type 'not-icontains' was skipped (no equivalent)
# NOTE: 'javascript' assertion has no direct equivalent; rewrite as an `exec` assertion
version: 1
description: Refund-policy replies from the support assistant
providers:
- id: p0
type: exec
command:
- ../echo-provider.py
prompts:
- id: prompt-0
template: |
You are a support agent for {{ product }}. Answer in one sentence.
Customer: {{ question }}
defaults:
vars:
product: Aurora Notes
assert:
- type: icontains
value: Aurora
tests:
- id: case-0
vars:
question: How long do I have to return something?
assert:
- type: contains
value: return something
- id: case-1
vars:
question: Can I get a refund on a gift card?
assert:
- type: icontains
value: gift card
Two of that config's assertions did not survive, and the notes name both: not-icontains is not a type the converter recognises — domarinn supports the spelling, the converter matches on the bare name — and javascript has no equivalent at all, its replacement being an exec assertion. Re-adding them is the part of a migration a converter cannot do for you.
The ids are the converter's too: p0 for the provider, and case-0 / case-1 for promptfoo cases that carried no description. They run as they are, and they are the first thing worth renaming — an id is what --provider, only_providers and a baseline diff refer to. A converted suite also has no project: or suite: name, which is what groups and compares runs on the results server.
CI converts the shipped promptfoo config, compares the result against the committed file, and runs both, so this page cannot show a conversion that no longer happens.