First steps¶
The five suites below are the fastest way to get a working mental model: what a run actually is, how a prompt gets filled in per case, which assertions cost nothing, and how a case turns a set of scores into a single pass or fail. None of them need a model, an API key, or even a network connection. Start here if this is your first domarinn suite.
Example 01 — Hello, eval¶
The smallest thing that works. One provider, one case, one assertion.
The system under test is a shell one-liner that prints a fixed answer, so this suite needs no model, no API key, and not even Python — just a POSIX shell. Read it as three answers: providers is what is being tested, tests is which inputs it gets, and assert is what must be true of the answer.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# The smallest suite that runs.
#
# One provider, one test, one assertion. The "system under test" here is a shell
# one-liner that prints a fixed answer — no model, no API key, and deliberately
# no Python either, so this suite runs anywhere a POSIX shell does.
#
# Read it as three answers:
# providers: WHAT is being tested
# tests: WHICH inputs it gets
# assert: WHAT must be true of the answer
#
# Run: domarinn run examples/01-hello-eval
version: 1
project: examples
suite: hello
providers:
# An `exec` provider is any program that reads one JSON request on stdin and
# writes one JSON response on stdout. That is the whole contract — see
# docs/protocol.md. This one ignores the request (`cat >/dev/null`) and always
# answers the same thing.
- id: assistant
type: exec
command:
- "sh"
- "-c"
- 'cat >/dev/null; printf ''{"output":"Hello! How can I help you today?"}'''
tests:
- id: greeting/basic
assert:
- type: contains
value: "Hello"
An exec provider is any program that reads one JSON request on stdin and writes one JSON response on stdout. That is the entire contract — see the exec protocol. Swap the command for your own program and this suite is already testing your system.
Example 02 — Prompts and variables¶
A prompt is a template; a case's vars fill it in. domarinn renders the prompt once per case with real Jinja, then hands the result to the provider.
A run is a grid, not a list
Every prompt is evaluated against every test. Two prompts and three cases is six cells, not three. That is the point — comparing two phrasings of the same instruction on identical inputs is what a prompt suite is for — but it means every assertion has to hold for both prompts. An assertion that only makes sense for one of them will fail half the grid.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Prompts and variables.
#
# A `prompt` is a template; a test's `vars` fill it in. domarinn renders the
# prompt once per case with real Jinja (minijinja), then hands the result to the
# provider.
#
# A run is a GRID: every prompt is evaluated against every test. Two prompts and
# three cases below is therefore SIX cells, not three — which is the point, since
# comparing two phrasings of the same instruction on identical inputs is exactly
# what a prompt suite is for. Every assertion here has to hold for both prompts.
#
# The system under test echoes whatever prompt it was given, which makes the
# rendering visible to the assertions. Point `command` at your own program and
# the same suite grades your system instead.
#
# Run: domarinn run examples/02-prompts-and-vars
version: 1
project: examples
suite: prompts-and-vars
providers:
- id: echo
type: exec
# Relative to THIS FILE's directory, not the shell's working directory — so
# the suite runs the same from the repo root or from inside this folder.
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
prompts:
# A single-string template. Use `messages:` instead when you need roles.
- id: support-reply
template: |
You are a support agent for {{ product }}.
Answer in at most {{ max_sentences }} sentences.
Customer: {{ question }}
# The same information as a chat transcript. A provider that talks to a chat
# API wants this shape; one that takes a single string wants the one above.
- id: support-chat
messages:
- role: system
content: "You are a support agent for {{ product }}. Be brief."
- role: user
content: "{{ question }}"
defaults:
# Every case inherits these, so a case only states what makes it different.
vars:
product: "Aurora Notes"
max_sentences: 2
tests:
- id: refund/policy
vars:
question: "How do I get a refund?"
assert:
- type: contains
value: "Aurora Notes"
- type: contains
value: "refund"
- id: export/format
vars:
question: "Can I export my notes as Markdown?"
assert:
- type: icontains
value: "markdown"
# A case may override an inherited var. `product` appears in both prompts, so
# an assertion about it holds across the whole row.
- id: refund/other-product
vars:
product: "Aurora Notes Pro"
question: "How do I get a refund?"
assert:
- type: contains
value: "Aurora Notes Pro"
Two prompt shapes appear here. template: is a single string, which suits a provider that takes one blob of text. messages: is a chat transcript with roles, which suits a provider that talks to a chat API. A prompt sets exactly one of the two.
defaults.vars supplies values every case inherits, so a case states only what makes it different — and may override an inherited value, as refund/other-product does.
Example 03 — Deterministic assertions¶
These run locally, cost nothing, and need no model. They also run first: a case whose contains fails never pays for an LLM grader. Reach for a rubric only for what these cannot express.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Every deterministic assertion, on one page.
#
# These run locally, cost nothing, and need no model. They are also the ones
# that run FIRST — a case whose `contains` fails never pays for an LLM grader.
# Reach for a rubric only for what these cannot express.
#
# Run: domarinn run examples/03-deterministic-asserts
version: 1
project: examples
suite: deterministic-asserts
providers:
- id: echo
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
tests:
# --- substring matching -------------------------------------------------
- id: substring/exact-case
vars:
user_input: "Your order #1042 shipped on Tuesday."
assert:
# `contains` is case-SENSITIVE.
- type: contains
value: "shipped"
# `icontains` is not.
- type: icontains
value: "TUESDAY"
# `icontains-any` passes when at least one candidate appears — useful when
# several phrasings are equally acceptable.
- type: icontains-any
values: ["shipped", "dispatched", "on its way"]
# --- shape ---------------------------------------------------------------
- id: shape/prefix-and-pattern
vars:
user_input: "ERROR: disk quota exceeded (code 507)"
assert:
- type: starts-with
value: "ERROR:"
# A Rust regex (no backtracking). Anchors and character classes work as
# you expect; look-around does not exist.
- type: regex
value: "code [0-9]{3}"
# --- exact equality ------------------------------------------------------
- id: equality/verbatim
vars:
user_input: "OK"
assert:
- type: equals
value: "OK"
# --- size ----------------------------------------------------------------
- id: size/within-budget
vars:
user_input: "A short, well-scoped answer."
assert:
# Either bound may be omitted. This one catches both an empty answer and
# a runaway one.
- type: length
min: 10
max: 500
# --- a computed condition ------------------------------------------------
- id: expression/jinja
vars:
user_input: "score=88"
threshold: 80
assert:
# `jinja` evaluates a boolean expression. `output` is the provider's
# answer; the case's vars are in scope too. Note this is minijinja, not
# Python: strings have no `.startswith()`, so use the `in` operator or a
# filter (`output | trim`, `output | length`).
- type: jinja
value: "'score=' in output and threshold >= 80"
jinja is minijinja, not Python
The jinja assertion evaluates a boolean expression with output and the case's vars in scope. It is Jinja2 semantics, so strings have no .startswith() method — use the in operator, or a filter like output | length. An expression that errors is recorded as an assertion error, not a silent failure.
Example 04 — Structured output¶
When a system is asked for JSON there are two separate questions, and it pays to assert them separately: is it parseable at all (is-json), and does it have the right shape (contains-json with a schema).
The difference matters because a model that wraps its JSON in prose fails the first and passes the second. That is the most common failure mode of an under-instructed model, so the example asserts it explicitly rather than hiding it.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Grading structured output.
#
# When a system is asked for JSON, there are two separate questions, and it pays
# to assert them separately:
#
# 1. Is it parseable at all? -> `is-json`
# 2. Does it have the right shape? -> `contains-json` with a `schema`
#
# A model that returns prose around its JSON fails (1) but can still pass (2),
# because `contains-json` looks for an embedded object. That difference is the
# whole reason both exist.
#
# Run: domarinn run examples/04-json-output
version: 1
project: examples
suite: json-output
providers:
- id: extractor
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
tests:
# A clean response: the entire output is one JSON object.
- id: extract/clean
vars:
user_input: '{"customer":"Ada Lovelace","order_id":1042,"refund_requested":true}'
assert:
- type: is-json
# Schema-checked. The schema is a normal JSON Schema; `required` is what
# turns "it parsed" into "it has the fields we depend on".
- type: contains-json
schema:
type: object
required: ["customer", "order_id", "refund_requested"]
properties:
customer: { type: string }
order_id: { type: integer }
refund_requested: { type: boolean }
# A chatty response: the object is real but wrapped in prose. This is the
# common failure mode of an under-instructed model, and the reason to reach
# for `contains-json` rather than `is-json` when you cannot control that.
- id: extract/wrapped-in-prose
vars:
user_input: |
Sure! Here is the extracted record:
{"customer":"Grace Hopper","order_id":77,"refund_requested":false}
Let me know if you need anything else.
assert:
# The whole output is NOT JSON — assert that, so the example documents the
# distinction rather than hiding it.
- type: not-is-json
- type: contains-json
schema:
type: object
required: ["customer", "order_id"]
# A schema is optional: bare `contains-json` only asks "is there JSON in here".
- id: extract/any-json
vars:
user_input: "result: [1, 2, 3]"
assert:
- type: contains-json
The schema is ordinary JSON Schema. required is what turns "it parsed" into "it has the fields we depend on" — without it, a model that returns {} passes.
Example 05 — Weights and thresholds¶
Each assertion scores 1.0 or 0.0 (a graded assertion may score in between), and the case score is their weighted mean:
What that score means depends on whether the case sets a threshold:
- no threshold — all-must-pass. One failed assertion fails the case.
- a threshold — the case passes when
score >= threshold.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Weights, thresholds, and negation — how a case decides pass or fail.
#
# Each assertion scores 1.0 (pass) or 0.0 (fail); a graded assertion may score
# in between. The CASE score is the weighted mean:
#
# score = sum(score_i * weight_i) / sum(weight_i)
#
# What that score means depends on whether the case sets a `threshold`:
#
# * no threshold -> ALL-MUST-PASS. One failed assertion fails the case.
# * a threshold -> the case passes when score >= threshold.
#
# Use a threshold when an answer can be partly right and you want to say how
# right is right enough. Use the default when every assertion is a hard
# requirement.
#
# Run: domarinn run examples/05-weights-and-thresholds
version: 1
project: examples
suite: weights-and-thresholds
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
tests:
# --- the default: every assertion is a hard requirement -------------------
- id: gate/all-must-pass
vars:
user_input: "Your refund of $42.00 has been approved and will arrive in 3-5 days."
assert:
- type: icontains
value: "refund"
- type: icontains
value: "approved"
# --- partial credit -------------------------------------------------------
# Three assertions, one of which fails: the answer never mentions a timeframe.
# score = (1 + 1 + 0) / 3 = 0.667, which clears the 0.6 threshold, so the case
# PASSES while still recording that something was missing.
- id: gate/partial-credit
threshold: 0.6
vars:
user_input: "Your refund of $42.00 has been approved."
assert:
- type: icontains
value: "refund"
- type: icontains
value: "approved"
- type: icontains
value: "days"
# --- weights: not every requirement matters equally ------------------------
# Naming the amount is what makes the answer correct; the polite closing is
# nice to have. Weighting the first 3x means missing the pleasantry costs
# 1/4 of the score instead of half of it.
# score = (1*3 + 0*1) / 4 = 0.75 -> passes at threshold 0.7
- id: gate/weighted
threshold: 0.7
vars:
user_input: "Refund issued: $42.00."
assert:
- type: contains
value: "$42.00"
weight: 3
- type: icontains
value: "let me know"
weight: 1
# --- negation -------------------------------------------------------------
# Any assertion type can be negated by prefixing `not-`. This is the readable
# form of "the answer must NOT do X", and it works in every test source —
# inline YAML, a file:// dataset, a CSV column, or a generator's output.
- id: gate/must-not-leak
vars:
user_input: "I can help with your order. Please confirm your email address."
assert:
- type: not-icontains
value: "password"
- type: not-regex
value: "[0-9]{16}"
Two of these cases pass with a failing assertion inside, which is the whole subject: gate/partial-credit scores (1 + 1 + 0) / 3 = 0.667 and clears its 0.6 threshold, and gate/weighted scores (1×3 + 0×1) / 4 = 0.75 and clears 0.7. Weighting the amount three times the pleasantry is what makes a missing pleasantry cost a quarter of the score instead of half.
Use a threshold when an answer can be partly right and you want to say how right is right enough. Use the default when every assertion is a hard requirement.
Any assertion type can be negated by prefixing not-, as gate/must-not-leak shows. This works in every test source — inline YAML, a file:// dataset, a CSV column, or a generator's output.