Your own system¶
Five suites about testing what you actually ship, not a model that merely resembles it. They cover the exec provider that runs your own program end to end, writing an assertion for a correctness rule only you can express, grading a tool call instead of a sentence of prose, and the exec protocol itself in a language other than Python. Read these once you are ready to point domarinn at your own code.
Example 12 — Render health¶
The cheapest useful suite there is: run an external system, grade its output with deterministic assertions, spend nothing. No API key, no model, no network — which makes it safe to run on every pull request.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# A render-health suite: it exercises an external "system under test" (here
# ../echo-provider.py) and grades its output with deterministic, zero-LLM
# assertions. No API key and no toolchain required — just python3.
version: 1
project: examples
suite: render-health
providers:
- id: renderer
type: exec
command: ["python3", "../echo-provider.py"]
timeout_ms: 30000
# A rebuilt binary should not serve stale cache; bump this to a git SHA / hash.
cache_salt: "dev"
tests:
# A literal SSTI payload that must NEVER be interpolated: !raw keeps it verbatim.
- id: adversarial/ssti-literal
tags: [adversarial]
vars:
user_input: !raw "{{7*7}} {% for x in range(9) %}x{% endfor %}"
assert:
- type: not-contains
value: "49"
- id: greeting/basic
tags: [smoke]
vars:
user_input: "hello world"
# A deterministic template filter in action: slugify normalizes free text
# into a URL/id-safe token ("Hello, World!" -> "hello-world").
slug: "{{ 'Hello, World!' | slugify }}"
assert:
- type: contains
value: "hello"
- type: length
max: 200000
defaults:
# A deterministic guard applied to every case: the rendered output must never
# be unreasonably large (a cheap catch for a runaway template).
assert:
- type: length
max: 100000
Two things are worth reading closely. The !raw tag on the SSTI payload keeps {{7*7}} verbatim, so the not-contains: "49" assertion is actually testing something. And defaults.assert applies a length ceiling to every case in the suite — a cheap catch for a runaway template — without repeating it per case.
Example 13 — Your own system¶
This is the provider domarinn is built around. An exec provider runs your program, so the eval exercises the same code path production does — the same rendering, the same client, the same guardrails — instead of a raw model endpoint that resembles it.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Your own system as the system under test.
#
# This is the provider domarinn is built around. An `exec` provider runs YOUR
# program, so the eval exercises the same code path production does — the same
# rendering, the same client, the same guardrails — instead of a raw model
# endpoint that resembles it.
#
# Three things are worth knowing before you point this at your binary:
#
# 1. `command` paths resolve relative to THIS FILE's directory, not the shell's
# working directory. `domarinn run examples/13-exec-provider` from the repo
# root and `domarinn run .` from inside it behave identically.
#
# 2. The cache key hashes what the program is SENT — the command, its args,
# the document on stdin — never the program's own bytes. Rebuild your
# binary and the cache still answers with the old results, which is
# usually what you want during a refactor and never what you want after a
# behaviour change. `cache_salt` is the lever; see example 22.
#
# 3. Because the model is part of argv, it is part of that request. Two
# models cannot collide in the cache, and there is deliberately no flag
# that swaps a provider's model — you write a second provider.
#
# Run: domarinn run examples/13-exec-provider
version: 1
project: examples
suite: exec-provider
providers:
- id: assistant-fast
label: "assistant (fast model)"
type: exec
command: ["python3", "./assistant.py", "--model", "fast"]
# Environment for the child process only. Values interpolate from the
# ambient environment, so a per-developer endpoint stays out of git:
# ASSISTANT_ENDPOINT: "${env:ASSISTANT_ENDPOINT:-http://localhost:8080}"
env:
ASSISTANT_STYLE: "concise"
# A slow system under test needs a longer leash than the default.
timeout_ms: 30000
# Bump this when you edit assistant.py — the cache key hashes what the
# program is SENT, never its bytes, so an edited script otherwise keeps
# answering from stale cached results. See example 22 for the salt levels.
cache_salt: "dev"
# A second provider, not a flag. Both run against every case, so the report
# compares them cell by cell.
- id: assistant-careful
label: "assistant (careful model)"
type: exec
command: ["python3", "./assistant.py", "--model", "careful"]
env:
ASSISTANT_STYLE: "thorough"
timeout_ms: 30000
cache_salt: "dev"
tests:
- id: billing/refund-window
vars:
user_input: "Can I still return this after 45 days?"
assert:
# True of both models: the answer states the policy window.
- type: contains
value: "30"
# `not-` keeps the safety requirement readable.
- type: not-icontains
value: "as an AI"
- id: billing/escalation
vars:
user_input: "This is the third time I've asked."
assert:
- type: icontains-any
values: ["escalate", "supervisor", "specialist"]
Three things that surprise people
commandpaths resolve relative to the suite file's directory, not the shell's working directory.domarinn run examples/13-exec-providerfrom the repo root anddomarinn run .from inside it behave identically.- The cache key hashes what the program is sent, not the program's bytes. Rebuild your binary and the cache still answers with the old results. That is what makes an entry reusable on another machine, and it is why example 22 exists.
- There is deliberately no flag that swaps a provider's model. The model is part of argv, therefore part of the request. To compare two models you write two providers — which is what the example above does, and why the report has two columns.
The program itself is ordinary. Read a request, do your thing, write a response:
#!/usr/bin/env python3
"""A stand-in for YOUR system under test.
Real ones call a model; this one is deterministic so the example runs offline.
What matters is the shape, which is identical either way:
* read one JSON request on stdin
* do whatever your system does
* write one JSON response on stdout, then exit 0
The request carries `prompt` (rendered, when the suite defines prompts), `vars`,
`params`, `test` and `tools`. The response needs only `output`; `usage` and
`cost_usd` are what make the `tokens:` and `cost:` assertions meaningful, so
report them if you can.
Note the model arrives as an argv flag rather than being read from the
environment. That is deliberate: the cache key hashes the request, and argv is
part of it, so two models can never share a cache entry. A variable this program
read from the ambient environment, without the suite declaring it, would be
invisible to the key.
"""
import argparse
import json
import os
import sys
def answer(model, style, question):
"""Whatever your system actually does. Deterministic here."""
lowered = question.lower()
if "third time" in lowered:
body = "I'm sorry for the runaround — I'm escalating this to a specialist now."
elif "return" in lowered or "refund" in lowered:
body = "Returns are accepted within 30 days of delivery."
else:
body = "Happy to help — could you tell me a little more?"
if model == "careful" and style == "thorough":
body += " I'll follow up in writing once it's confirmed."
return body
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--model", required=True)
args = parser.parse_args()
request = json.load(sys.stdin)
variables = request.get("vars") or {}
question = variables.get("user_input", "")
style = os.environ.get("ASSISTANT_STYLE", "concise")
text = answer(args.model, style, question)
json.dump(
{
"output": text,
# Reporting usage is what lets `tokens:` and `cost:` assertions do
# anything. Omit it and a `cost:` budget passes as "not reported".
"usage": {"input_tokens": len(question.split()), "output_tokens": len(text.split())},
# Response metadata, never part of the cache key.
"model": args.model,
"stop_reason": "end_turn",
},
sys.stdout,
)
sys.stdout.write("\n")
return 0
if __name__ == "__main__":
sys.exit(main())
Reporting usage is what makes tokens: and cost: assertions meaningful. Omit it and a cost: budget passes as "cost not reported" — green, and enforcing nothing.
Example 14 — A custom assertion¶
The built-in assertions cover substrings, shapes, schemas and rubrics. When a correctness rule is genuinely yours — a checksum, a units conversion, a lookup against a table only you have — an exec assertion runs your program and takes its verdict.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# An assertion you write yourself.
#
# The built-in assertions cover substrings, shapes, schemas and rubrics. When a
# correctness rule is genuinely YOUR domain — a checksum, a units conversion, a
# lookup against a table only you have — an `exec` assertion runs your program
# and takes its verdict.
#
# The contract mirrors the provider one: read a JSON request on stdin, write
# {"pass": bool, "score"?: float, "reason"?: string} on stdout. The request
# carries the output, the case's vars, and whatever `config` you set here.
#
# Unlike the deterministic assertions, an exec assertion is GRADED: it runs
# after them, and is skipped entirely when the case is already decided — see
# example 18 for what that looks like and why it saves money.
#
# Run: domarinn run examples/14-custom-exec-assert
version: 1
project: examples
suite: custom-exec-assert
providers:
- id: assistant
type: exec
command: ["python3", "../echo-provider.py"]
cache_salt: "dev"
tests:
# The invoice total must equal the sum of its line items — arithmetic no
# substring match can express.
- id: invoice/adds-up
vars:
user_input: '{"lines":[19.99,5.00,0.51],"total":25.50}'
assert:
- type: exec
command: ["python3", "check-total.py"]
config:
tolerance: 0.005
- id: invoice/rounding-slack
vars:
user_input: '{"lines":[10.00,10.00],"total":20.004}'
assert:
- type: exec
command: ["python3", "check-total.py"]
config:
tolerance: 0.01
The contract mirrors the provider one, and the request carries the output, the case's vars, and whatever config the suite set:
#!/usr/bin/env python3
"""A domarinn `exec` assertion: does the invoice total match its line items?
Reads one JSON request on stdin:
{"domarinn": {...}, "output": <the provider's answer>,
"vars": {...}, "config": {...}, "test": {...}, "provider": {...}}
and writes one verdict on stdout:
{"pass": true, "score": 1.0, "reason": "..."}
`reason` is what shows up beside the case in the report, so write it for the
person reading a red build, not for a log.
Exit 0 whenever you produced a verdict — including a failing one. A non-zero
exit means the checker itself broke, which domarinn reports as an
infrastructure error rather than a test failure.
"""
import json
import sys
def verdict(passed, reason, score=None):
return {
"pass": passed,
"score": (1.0 if passed else 0.0) if score is None else score,
"reason": reason,
}
def main():
request = json.load(sys.stdin)
tolerance = (request.get("config") or {}).get("tolerance", 0.005)
output = request.get("output")
if not isinstance(output, str):
output = json.dumps(output)
try:
invoice = json.loads(output)
except json.JSONDecodeError as exc:
json.dump(verdict(False, f"output is not JSON: {exc}"), sys.stdout)
return 0
try:
expected = sum(invoice["lines"])
actual = invoice["total"]
except (KeyError, TypeError) as exc:
json.dump(verdict(False, f"missing lines/total: {exc}"), sys.stdout)
return 0
drift = abs(expected - actual)
json.dump(
verdict(
drift <= tolerance,
f"line items sum to {expected:.4f}, total says {actual:.4f} "
f"(drift {drift:.4f}, tolerance {tolerance})",
),
sys.stdout,
)
return 0
if __name__ == "__main__":
sys.exit(main())
Write reason for the person reading a red build, not for a log — it is what appears beside the case in the report. And exit 0 whenever you produced a verdict, including a failing one: a non-zero exit means the checker itself broke, which domarinn reports as an infrastructure error rather than a test failure.
Example 15 — Tool-call assertions¶
When an agent's right answer is "call this tool with these arguments", there is no sentence to match. domarinn declares the tools, your provider offers them to the model, and the tool-call assertion grades which one it chose.
domarinn never executes a tool. It is an evaluation harness, not an agent runtime — what it wants back is the decision, which is the thing under test.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# Grading the DECISION, not the prose.
#
# When an agent's right answer is "call this tool with these arguments", there is
# no sentence to match. domarinn declares the tools, your provider offers them to
# the model, and the `tool-call` assertion grades which one it chose.
#
# domarinn never EXECUTES a tool. It is an evaluation harness, not an agent
# runtime — what it wants back is the decision, which is the thing under test.
#
# /// A caution worth reading before you add `tools:` to an existing suite ///
# Declaring tools changes measured behaviour, often a lot. A model that would
# have written "I'll look that up for you" now emits a tool call and no prose,
# so every rubric watching the text channel suddenly sees nothing. Add tools when
# you intend to assert on tool calls — not as background context.
#
# Run: domarinn run examples/15-tool-call-asserts
version: 1
project: examples
suite: tool-call-asserts
providers:
- id: agent
type: exec
command: ["python3", "./agent.py"]
cache_salt: "dev"
# Suite-level, so every provider is offered the same surface. `input_schema` is
# passed through verbatim and never rendered as a template.
tools:
- name: lookup_order
description: Fetch an order by its id.
input_schema:
type: object
required: ["order_id"]
properties:
order_id: { type: integer }
- name: issue_refund
description: Refund an order. Irreversible.
input_schema:
type: object
required: ["order_id", "amount_usd"]
properties:
order_id: { type: integer }
amount_usd: { type: number }
tests:
# The model should look the order up before saying anything about it.
- id: agent/looks-up-first
vars:
user_input: "What's the status of order 1042?"
assert:
- type: tool-call
name: lookup_order
# `args` is a SUBSET match: the call may carry more, but these must be
# present and equal. Asserting the whole object would make the test
# brittle against a harmless extra field.
args:
order_id: 1042
# The dangerous tool must not be touched for a read-only question.
- type: not-tool-call
name: issue_refund
# Arguments can be shape-checked instead of pinned, when the exact value is
# not what you are testing.
- id: agent/refund-shape
vars:
user_input: "Please refund order 77 for $19.99."
assert:
- type: tool-call
name: issue_refund
schema:
type: object
required: ["order_id", "amount_usd"]
properties:
order_id: { type: integer }
amount_usd: { type: number }
# A question needing no tool at all: the agent should just answer. This is the
# case that catches tool-eagerness, and it is the one most often missing.
- id: agent/no-tool-needed
vars:
user_input: "What are your opening hours?"
assert:
- type: not-tool-call
name: lookup_order
- type: not-tool-call
name: issue_refund
- type: icontains
value: "9"
args is a subset match: the call may carry more fields, but the ones named must be present and equal. Asserting the whole object would make the test brittle against a harmless extra field. When the exact value is not what you are testing, schema shape-checks the arguments instead.
Declaring tools changes what you measure
A model offered tools stops writing "I'll look that up for you" and starts emitting a tool call with no prose — so every rubric watching the text channel suddenly sees nothing, and a suite that was green goes red for reasons that have nothing to do with the change you were testing. Add tools: when you intend to assert on tool calls, not as background context.
The case most often missing is the one where no tool should be called. agent/no-tool-needed is that case, and it is what catches tool-eagerness.
Example 37 — The exec protocol, in bash¶
Every other exec provider in this ladder is Python — bar example 01's inline shell one-liner — because Python ships with the fewest assumptions about your machine. The protocol itself does not care: anything that can read one JSON document from stdin and write one to stdout qualifies. Here it's bash and jq.
# yaml-language-server: $schema=../../domarinn.schema.json
#
# The exec protocol needs no SDK. Anything that can read one JSON document
# from stdin and write one JSON document to stdout can be a provider — here
# it's bash and jq, not Python or Rust.
#
# The contract is exactly the one every other exec example in this ladder
# uses (see example 13, and docs/reference/protocol.md): one request in, one
# response out, exit 0 for "I produced a response" — even a failing one — and
# a non-zero exit only for your own program breaking.
#
# Run: domarinn run examples/37-exec-provider-bash
version: 1
project: examples
suite: exec-provider-bash
providers:
- id: assistant
type: exec
command: ["bash", "./provider.sh"]
# Bump this when you edit provider.sh, like every hand-written file-backed
# exec example in this ladder — the cache key hashes what the program is
# SENT, never its bytes, so an edited script otherwise keeps answering
# from stale cached results. (Example 01 needs no salt: its whole program
# is inline argv, so editing it changes the request. Example 39 is
# file-backed and has none only because it is converter output, verbatim.)
cache_salt: "dev"
tests:
- id: greeting/echoes-input
vars:
user_input: "hello from the suite"
assert:
# Proves `.vars.user_input` reached the script and came back.
- type: contains
value: "hello from the suite"
# Proves `.test.id` reached it too — the request carries more than
# just the vars, and jq can pull out whichever field it needs.
- type: contains
value: "greeting/echoes-input"
#!/usr/bin/env bash
# The exec protocol needs no SDK: read one JSON request from stdin, write one
# JSON response to stdout. This is the same contract ../echo-provider.py
# speaks, in bash + jq instead of Python.
#
# Exit 0 means "I produced a response" — even one describing a failure; a
# non-zero exit means THIS SCRIPT broke, which domarinn reports as an
# infrastructure error rather than a graded result. `set -e` below is what
# makes an unhandled jq/bash error surface as that non-zero exit instead of
# limping on with a half-built reply.
set -euo pipefail
# domarinn sets this to the wire version it is speaking, so a child that
# supports more than one can branch on it. Today there is only "1".
[[ "${DOMARINN_PROTOCOL:-}" == "1" ]] || {
echo "provider.sh: unsupported protocol '${DOMARINN_PROTOCOL:-<unset>}'" >&2
exit 1
}
# Exactly one JSON document arrives on stdin, then domarinn closes it — `cat`
# sees a clean EOF rather than blocking for more input.
request="$(cat)"
# `-r` unwraps the JSON string to raw text so it can be interpolated below
# without its quotes. `// ""` is the fallback for a suite that never sets
# `user_input` — an exec provider is not guaranteed any particular var.
user_input="$(jq -r '.vars.user_input // ""' <<<"$request")"
test_id="$(jq -r '.test.id' <<<"$request")"
# Exactly one JSON object on stdout; `output` is the only field domarinn
# requires. `-n` builds it from `null` rather than re-reading `request`, so
# nothing already consumed leaks into the reply by accident.
jq -cn --arg out "case ${test_id}: ${user_input}" \
'{output: $out, usage: {input_tokens: 1, output_tokens: 1}}'
Three things worth reading in that script. It reads DOMARINN_PROTOCOL before doing anything else — the version domarinn is speaking, so a program that supports more than one can branch on it. jq -r '.vars.user_input // ""' unwraps the JSON string and supplies a fallback, because an exec provider is never guaranteed any particular var. And the reply is built with jq -cn rather than piping request back through a filter, so nothing already consumed can leak into the response by accident.