Examples¶

A ladder of complete, runnable suites, each one demonstrating a single capability. Copy any of them, point the provider at your own system, and domarinn run.

Every suite in these pages is executed in CI

The YAML on the example pages is not a transcription. It is pulled at build time from the real directories under examples/, and every one of those directories is run end to end by crates/domarinn-cli/tests/examples.rs — which asserts the exit code, the pass/fail tally, and the exact case ids each suite produces. A page cannot document a suite that no longer works, because the page and the test read the same bytes.

The mental model

A run is a grid. domarinn takes the providers (the systems under test), the prompts (optional — a provider may build its own input), and the tests (the units you author), and evaluates one cell per combination. Each cell calls its provider once and grades the answer against that test's assertions.

Deterministic assertions run first and can short-circuit the expensive ones, so a case that fails contains never pays for an LLM grader. Each evaluated cell is a case, and a case ends pass, fail, error (the harness broke — never counted as an assertion failure), or skip.

The examples below are grouped into six pages, roughly in the order you would reach for them:

Group	Covers
First steps	The smallest suite that runs, up through weights and thresholds.
Templates & test data	The `!raw` escape hatch, file vars, matrix sweeps, datasets, generators, and multi-turn prompts.
Your own system	The `exec` provider, a custom assertion, tool-call grading, and the protocol itself in bash.
Running & reporting	Tags and filters, composition, exit codes, errors and retries, runner tuning, output formats, and importing a promptfoo config.
Caching & statistics	The one cache-key rule, salts, repeat and confidence, baselines and diff.
Models, grading & budgets	OpenAI-compatible and Anthropic providers (native tool calls included), HTTP and `output_expr`, LLM-rubric grading judged by either vendor, a rubric that sees the tool calls, similarity, budgets — and every top-level key at once, annotated.

#	Example	Demonstrates
01	Hello, eval	The smallest suite that runs. No model, no key, no toolchain.
02	Prompts and variables	Prompt templates filled per case — and why a run is a grid.
03	Deterministic assertions	Every zero-cost assertion type, on one page.
04	Structured output	`is-json` versus `contains-json` with a schema.
05	Weights and thresholds	How a case decides pass or fail, and how to give partial credit.
06	The `!raw` escape hatch	Test input that must reach the system byte-for-byte.
07	File-content vars	Pull a var's value from a file beside the suite — parsed, raw, or sandboxed.
08	Matrix sweeps	Fan one case out over the cartesian product of its axes.
09	Datasets from files	Cases in `file://` globs, owned and reviewed separately.
10	A CSV dataset	The format a non-engineer will hand you, read directly.
11	Test generators	Cases computed by a program, so coverage cannot drift.
12	Render health	Grade an external system with zero-LLM assertions.
13	Your own system	The `exec` provider: test what actually ships.
14	A custom assertion	A correctness rule only you can express.
15	Tool-call assertions	Grade the decision, not the prose.
16	Tags and filters	Running part of a suite.
17	Composition	`extends`, `imports`, and how the merge actually works.
18	A failing gate	What red looks like. Exits 1 on purpose.
19	Errors and retries	Errors are not failures. Exits 3 on purpose.
20	Runner tuning	Concurrency, rate limits, timeouts.
21	Caching	Not paying twice for the same answer.
22	Cache salts	Busting the cache at the right granularity.
23	Repeat and confidence	A pass rate with an error bar.
24	Baselines and diff	Gate on regressions, not on an absolute score.
25	Output formats	One run feeding a human and a machine.
26	An OpenAI-compatible endpoint	The lingua franca: OpenAI, Ollama, vLLM, LiteLLM, a gateway.
27	Anthropic, and what it costs	`pricing`, and why a `cost:` budget can be green and enforce nothing.
28	A service you already run	The `http` provider and `output_expr`.
29	LLM-rubric grading	A structured, fail-closed grader — and how to write its rubric.
30	Similarity	Cosine distance, for when many wordings are right.
31	Budgets	Cost, tokens, latency — and how each can enforce nothing.
32	A live endpoint	Point a suite at your own OpenAI-compatible endpoint.
33	An OpenAI-shaped grader	An `llm-rubric` grader that is `type: openai` — any compatible endpoint, including a local Ollama.
34	A multi-turn conversation	A `messages:` prompt carrying real history, not just the newest line.
35	Anthropic tools, natively	Tool-call grading over the native API — the sibling of example 15's `exec` version.
36	`output_expr`, sliced two ways	Pulling more than one shape — including a non-string one — out of the same response.
37	The exec protocol, in bash	The same provider contract, spoken in bash and `jq` instead of Python.
38	Every key, once	One runnable suite setting every top-level key, annotated as a map of the reference.
39	A promptfoo config, converted	A promptfoo config and the suite `domarinn import promptfoo` turns it into, both shipped.
40	A rubric that sees the tool calls	`include_tool_calls`, for grading the delegation decision instead of the prose.

Examples¶

See also¶