Evaluate your own application¶

The problem. You are shipping an application, not a model. It has its own prompt registry, its own client, its own retry logic and its own guardrails. Evaluating the model underneath it measures something adjacent to the product — and passes cleanly while the thing customers touch is broken.

The shape. Make your application runnable as a subprocess that speaks a small JSON protocol, and point domarinn at that.

1. Add an eval entry point to your app¶

One JSON request on stdin, one JSON response on stdout. That is the whole contract.

#!/usr/bin/env python3
"""A stand-in for YOUR system under test.

Real ones call a model; this one is deterministic so the example runs offline.
What matters is the shape, which is identical either way:

  * read one JSON request on stdin
  * do whatever your system does
  * write one JSON response on stdout, then exit 0

The request carries `prompt` (rendered, when the suite defines prompts), `vars`,
`params`, `test` and `tools`. The response needs only `output`; `usage` and
`cost_usd` are what make the `tokens:` and `cost:` assertions meaningful, so
report them if you can.

Note the model arrives as an argv flag rather than being read from the
environment. That is deliberate: the cache key hashes the request, and argv is
part of it, so two models can never share a cache entry. A variable this program
read from the ambient environment, without the suite declaring it, would be
invisible to the key.
"""

import argparse
import json
import os
import sys


def answer(model, style, question):
    """Whatever your system actually does. Deterministic here."""
    lowered = question.lower()
    if "third time" in lowered:
        body = "I'm sorry for the runaround — I'm escalating this to a specialist now."
    elif "return" in lowered or "refund" in lowered:
        body = "Returns are accepted within 30 days of delivery."
    else:
        body = "Happy to help — could you tell me a little more?"
    if model == "careful" and style == "thorough":
        body += " I'll follow up in writing once it's confirmed."
    return body


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", required=True)
    args = parser.parse_args()

    request = json.load(sys.stdin)
    variables = request.get("vars") or {}
    question = variables.get("user_input", "")
    style = os.environ.get("ASSISTANT_STYLE", "concise")

    text = answer(args.model, style, question)
    json.dump(
        {
            "output": text,
            # Reporting usage is what lets `tokens:` and `cost:` assertions do
            # anything. Omit it and a `cost:` budget passes as "not reported".
            "usage": {"input_tokens": len(question.split()), "output_tokens": len(text.split())},
            # Response metadata, never part of the cache key.
            "model": args.model,
            "stop_reason": "end_turn",
        },
        sys.stdout,
    )
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Reuse your real code path. The value of this scenario comes entirely from the eval exercising the same rendering and client logic production does — a wrapper that reimplements them tests the wrapper.

2. Point a suite at it¶

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Your own system as the system under test.
#
# This is the provider domarinn is built around. An `exec` provider runs YOUR
# program, so the eval exercises the same code path production does — the same
# rendering, the same client, the same guardrails — instead of a raw model
# endpoint that resembles it.
#
# Three things are worth knowing before you point this at your binary:
#
#   1. `command` paths resolve relative to THIS FILE's directory, not the shell's
#      working directory. `domarinn run examples/13-exec-provider` from the repo
#      root and `domarinn run .` from inside it behave identically.
#
#   2. The cache key hashes what the program is SENT — the command, its args,
#      the document on stdin — never the program's own bytes. Rebuild your
#      binary and the cache still answers with the old results, which is
#      usually what you want during a refactor and never what you want after a
#      behaviour change. `cache_salt` is the lever; see example 22.
#
#   3. Because the model is part of argv, it is part of that request. Two
#      models cannot collide in the cache, and there is deliberately no flag
#      that swaps a provider's model — you write a second provider.
#
# Run:  domarinn run examples/13-exec-provider
version: 1
project: examples
suite: exec-provider

providers:
  - id: assistant-fast
    label: "assistant (fast model)"
    type: exec
    command: ["python3", "./assistant.py", "--model", "fast"]
    # Environment for the child process only. Values interpolate from the
    # ambient environment, so a per-developer endpoint stays out of git:
    #   ASSISTANT_ENDPOINT: "${env:ASSISTANT_ENDPOINT:-http://localhost:8080}"
    env:
      ASSISTANT_STYLE: "concise"
    # A slow system under test needs a longer leash than the default.
    timeout_ms: 30000
    # Bump this when you edit assistant.py — the cache key hashes what the
    # program is SENT, never its bytes, so an edited script otherwise keeps
    # answering from stale cached results. See example 22 for the salt levels.
    cache_salt: "dev"

  # A second provider, not a flag. Both run against every case, so the report
  # compares them cell by cell.
  - id: assistant-careful
    label: "assistant (careful model)"
    type: exec
    command: ["python3", "./assistant.py", "--model", "careful"]
    env:
      ASSISTANT_STYLE: "thorough"
    timeout_ms: 30000
    cache_salt: "dev"

tests:
  - id: billing/refund-window
    vars:
      user_input: "Can I still return this after 45 days?"
    assert:
      # True of both models: the answer states the policy window.
      - type: contains
        value: "30"
      # `not-` keeps the safety requirement readable.
      - type: not-icontains
        value: "as an AI"

  - id: billing/escalation
    vars:
      user_input: "This is the third time I've asked."
    assert:
      - type: icontains-any
        values: ["escalate", "supervisor", "specialist"]

Three things that surprise people

command paths resolve relative to the suite file's directory, not the shell's working directory. That is what lets domarinn run eval/ from the repo root and domarinn run . from inside eval/ behave identically.

The cache key hashes what your program is sent, not the program's bytes. Rebuild your binary and the cache still replays the old answers. Set a provider cache_salt when a rebuild should invalidate — see guide 09.

There is no flag that swaps the model. The model is part of argv, therefore part of the request — which is why two models can never collide in the cache. To compare two, declare two providers and scope a run with --provider.

3. Report what you know¶

Three response fields do disproportionate work:

Field	Enables
`usage`	`tokens:` assertions, and `cost_usd` existing at all
`cost_usd`	a `cost:` budget that actually enforces something
`empty_reason`	a refusal or a tool-only turn being distinguished from a blank failure
`error.retriable`	retries that fire on rate limits and not on rejected credentials

Omit them and the corresponding assertions pass while enforcing nothing. See example 31 and example 19.

4. Generate the cases from your registry¶

If your app owns a catalogue of prompts, tools or policies, enumerate it rather than listing cases by hand — otherwise the thing added last Tuesday has no test:

tests:
  - generator:
      command: ["./target/release/my-eval", "generate-tests"]

See example 11. Back it with a test in your own codebase asserting the registry and the manifest agree.

5. Guard the thing that silently invalidates everything¶

Pin the suite's model to the one production actually serves, and assert it in your own test suite:

#[test]
fn the_eval_suite_uses_the_model_production_serves() {
    assert_eq!(eval_suite_model(), production_default_model());
}

Without a guard like this, a suite can sit on a model nobody serves — and every pass rate you published from it was measuring something you do not ship. It is a cheap test and it catches a failure that is otherwise invisible for months.