domarinn

A declarative LLM eval harness: one static Rust binary that is a CLI, an evaluation engine, and a self-hostable results server with an embedded web UI. It treats your own system — a command, an HTTP endpoint, or a model — as the thing under test.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# The smallest suite that runs.
#
# One provider, one test, one assertion. The "system under test" here is a shell
# one-liner that prints a fixed answer — no model, no API key, and deliberately
# no Python either, so this suite runs anywhere a POSIX shell does.
#
# Read it as three answers:
#   providers: WHAT is being tested
#   tests:     WHICH inputs it gets
#   assert:    WHAT must be true of the answer
#
# Run:  domarinn run examples/01-hello-eval
version: 1
project: examples
suite: hello

providers:
  # An `exec` provider is any program that reads one JSON request on stdin and
  # writes one JSON response on stdout. That is the whole contract — see
  # docs/protocol.md. This one ignores the request (`cat >/dev/null`) and always
  # answers the same thing.
  - id: assistant
    type: exec
    command:
      - "sh"
      - "-c"
      - 'cat >/dev/null; printf ''{"output":"Hello! How can I help you today?"}'''

tests:
  - id: greeting/basic
    assert:
      - type: contains
        value: "Hello"

$ domarinn run examples/01-hello-eval

That suite is not a sketch. It is a real directory in this repository, it needs no API key and no model, and CI runs it end to end — as it does every other example on this site.

Start here

One path, in order. It starts offline, with no API key, and spends nothing until you choose to point a suite at a real model.

1. Install — a prebuilt static binary, cargo, source, or Docker.
2. Your first eval — write and run the suite above offline, then add a real model and an LLM grader, and share the result.
3. How a run works — four ideas that explain most of the behaviour. Read it once and the rest of these docs stop being surprising.

Everything below is the map you come back to afterwards.

Concepts

How domarinn thinks. Reach for these when a behaviour surprises you.

Page	What it covers
How a run works	The grid, short-circuiting, the fail-closed grader, and the cache key.
Architecture — one binary	The crates, the run document, and the versioned boundaries between them.
Providers & the exec boundary	What a provider is, and what crosses the process boundary into yours.
Grading	The llm-rubric grader: structured verdicts, fail-closed, and writing a rubric.
The one-rule cache	The one key rule, every knob, salts, backends, and sharing a cache.
Statistics	--repeat, Wilson intervals, McNemar significance, pass@k, and baselines.
Why domarinn is built this way	The reasoning behind each of the decisions above.

Guides

End-to-end walkthroughs for a real situation, each assembled from suites that run in CI. Six of the twelve, to give the shape; the Guides page lists them all with what each one costs to run.

Guide	When you reach for it
Test a model API	Compare two models, or catch a snapshot changing under you.
Test your app via exec	Test what actually ships, not the model underneath it.
A zero-cost gate on every PR	Catch template and formatting regressions with no key and no spend.
Gate a PR in CI	Block a merge that makes things worse, without demanding perfection.
Migrate from promptfoo	domarinn import promptfoo, and exactly what changes.
Troubleshooting	Symptom, cause, fix — including the ways a gate can be green and check nothing.

Reference

Every field, flag, route and status code, verified against the code that reads it.

Page	What it covers
domarinn.yaml	Every top-level key of the suite file, and the !raw escape hatch.
Providers	exec, http, anthropic, openai, and embeddings — behavior and pricing.
Assertions	Every assertion type, weights and thresholds, short-circuiting, fail-closed semantics.
CLI	Every command and flag, and the CI exit-code contract.
Server	Running the results server, plus accounts: logins, roles, API keys, auth modes.
REST API	Every route, its shape, and the run-ingest contract.
MCP endpoint	Read-only eval history for an agent, in the same binary.
Exec protocol	The JSON protocol for writing providers, assertions, and generators in any language.
The web UI	Every view of the embedded UI, with a screenshot of each.

Examples

A ladder of 39 complete suites, one capability each, grouped into six pages. Every one is a real directory under examples/, transcluded into the page you read it on and executed by CI — so a page cannot document a suite that stopped working. The Examples index maps each number to its page.

Group	Covers
First steps	The smallest suite that runs, up through weights and thresholds.
Templates & test data	!raw, file vars, matrix sweeps, datasets, generators, multi-turn prompts.
Your own system	The exec provider, a custom assertion, tool-call grading, the protocol in bash.
Running & reporting	Filters, composition, exit codes, retries, runner tuning, output formats, import.
Caching & statistics	The cache-key rule, salts, repeat and confidence, baselines and diff.
Models, grading & budgets	Model providers, HTTP and output_expr, rubric grading, similarity, budgets.

Why domarinn

• Your system is the system under test. An external command or HTTP endpoint is a first-class provider; prompts are optional.
• Real Jinja with a per-value !raw escape hatch, so literal template syntax in a test input is never interpolated.
• Deterministic assertions run first and short-circuit the expensive grader.
• The grader is structured and fails closed — no verdicts parsed out of prose, no silent passes.
• Errors are not failures. Separate status, separate tally, separate exit code, so "the harness broke" never reads as "the model got worse".
• Statistics built in — confidence intervals and paired significance, not bare pass rates.
• Caching you can share over a server URL or an S3 bucket, because every request — provider, grader, embedding — is keyed on the request itself and nothing about your machine.
• One binary — CLI, engine, and server, with the web UI embedded.

The reasoning behind each of these is in Why domarinn is built this way.