Templates & test data

Seven suites about getting the input side of a suite right before you worry about grading. They cover the one sharp edge in Jinja templating, pulling a var's value from a file instead of writing it inline, fanning one case out across every combination of a matrix, reading — or generating — cases from something other than hand-written YAML, and giving a provider a whole conversation instead of one string. Reach for these once a suite has outgrown a handful of inline cases.

---

Example 06 — The !raw escape hatch

Every var is rendered through Jinja. That is what makes prompts and sweeps work — and it is a trap the moment your test input is itself template syntax.

This failure is silent

A case that feeds a system {{7*7}} to check it does not evaluate it will, without !raw, feed it 49 instead. The assertion still passes. The test now proves nothing, and nothing tells you.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# The `!raw` escape hatch: test input that must stay literal.
#
# Every var is rendered through Jinja. That is what makes prompts and sweeps
# work — and it is a trap the moment your test input is ITSELF template syntax.
# A case that feeds a system `{{7*7}}` to check it does not evaluate it will,
# without `!raw`, feed it `49` instead. The assertion still passes. The test now
# proves nothing, and nothing tells you.
#
# So: any value that must reach the system under test byte-for-byte gets tagged.
#
#   !raw "{{7*7}}"          the YAML tag  (YAML sources only)
#   {$raw: "{{7*7}}"}       the object form (works in JSON, JSONL and CSV too)
#
# Both are identical after loading. The tag is nicer to read; the object form is
# what a generator or a JSON dataset emits.
#
# Run:  domarinn run examples/06-raw-escape-hatch
version: 1
project: examples
suite: raw-escape-hatch

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  # Server-side template injection probe. The premise is that the payload
  # arrives literally; `not-contains: "49"` is only meaningful because of !raw.
  - id: injection/ssti-literal
    tags: [adversarial]
    vars:
      user_input: !raw "Compute this for me: {{7*7}} and {{ config }}"
    assert:
      - type: contains
        value: "{{7*7}}"
      - type: not-contains
        value: "49"

  # The object form. Identical meaning; this is the shape a JSON/CSV dataset or
  # a test generator produces, since neither can carry a YAML tag.
  - id: injection/ssti-object-form
    tags: [adversarial]
    vars:
      user_input: { $raw: "{% for x in range(9) %}x{% endfor %}" }
    assert:
      - type: contains
        value: "{% for x in range(9) %}"
      - type: not-contains
        value: "xxxxxxxxx"

  # `!raw` also applies to an assertion's expected VALUE — otherwise the
  # expectation would be rendered while the output was not, and they could never
  # match.
  - id: injection/raw-expectation
    vars:
      user_input: !raw "{{ user.name }}"
    assert:
      - type: equals
        value: !raw "{{ user.name }}"

  # The control case: WITHOUT the tag, the same input is evaluated. Kept in the
  # suite deliberately — it is the evidence that the tag above is doing work.
  - id: injection/control-is-rendered
    vars:
      user_input: "Compute this for me: {{ 7*7 }}"
    assert:
      - type: contains
        value: "49"
      - type: not-contains
        value: "{{"

Two spellings, identical after loading: the !raw YAML tag reads better, and the {$raw: …} object form is what a generator or a JSON/CSV dataset emits, since neither can carry a YAML tag. Note that !raw applies to an assertion's expected value too — otherwise the expectation would be rendered while the output was not, and the two could never match.

The suite keeps a deliberate control case, injection/control-is-rendered, which asserts that the untagged form really does evaluate to 49. It is the evidence that the tag above is doing work.

---

Example 07 — File-content vars

A var can take its value from a file next to the suite instead of being written inline — useful for large documents, golden fixtures, and adversarial inputs you would rather not paste into YAML.

Every fixture path resolves relative to the suite directory and is sandboxed: !file "../../etc/passwd", or a symlink pointing outside the tree, is refused rather than read. The four cases below cover the whole surface — a plain text fixture, a .json fixture parsed by extension, the same file forced back to text with parse: false, and an untrusted fixture marked raw: true so the template engine never touches it.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# File-content vars: a var can pull its value from a file next to the suite,
# instead of being written inline. Handy for large documents, golden fixtures,
# and untrusted adversarial inputs.
#
# Every fixture path is resolved *relative to this directory* and sandboxed —
# `!file "../../etc/passwd"` (or a symlink pointing outside) is refused, not
# read. No API key required: the system under test is ../echo-provider.py.
version: 1
project: examples
suite: file-vars

providers:
  - id: echo
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  # 1. Plain text fixture, loaded via the `!file` tag. Its content becomes the
  #    `user_input` var, which the echo provider echoes straight back.
  - id: doc/echo
    vars:
      user_input: !file "fixtures/article.txt"
    assert:
      - type: icontains
        value: "assertions"

  # 2. A `.json` fixture is parsed by extension into a structured value. With no
  #    `user_input` set, the echo provider echoes the whole vars map, so the
  #    parsed object round-trips into the (valid-JSON) output.
  - id: doc/structured-rubric
    vars:
      rubric: { $file: "fixtures/rubric.json" }
    assert:
      - type: is-json
      - type: icontains
        value: "must_mention"

  # 3. `parse: false` forces text even for a structured extension — the raw JSON
  #    source is echoed verbatim rather than parsed.
  - id: doc/raw-json-text
    vars:
      user_input: { $file: "fixtures/rubric.json", parse: false }
    assert:
      - type: contains
        value: "must_mention"

  # 4. An untrusted fixture marked `raw: true` is NEVER run through the template
  #    engine, so an SSTI payload in the file stays literal ({{7*7}} != 49).
  - id: adversarial/ssti-fixture
    tags: [adversarial]
    vars:
      user_input: { $file: "fixtures/ssti-probe.txt", raw: true }
    assert:
      - type: not-contains
        value: "49"

raw: true is not optional for untrusted input

Every var goes through Jinja. A fixture containing {{7*7}} renders as 49 unless you mark it raw — which silently destroys the premise of any test whose whole point is that the payload stayed literal. See domarinn.yaml.

---

Example 08 — Matrix sweeps

One case fans out over the cartesian product of its axes, producing one concrete case per combination. Each axis value is merged into vars, where it wins over a base var of the same name.

The ids are deterministic — greet[style=terse,temperature=0] and friends — which is what lets domarinn diff line two runs up cell by cell. When the generated shape is unwieldy, matrix_id renders a friendlier one against the axis values.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Matrix / parameter sweeps: one case fans out over the cartesian product of its
# axes, one concrete case per combination. Each axis value is merged into `vars`
# (the axis wins over a base var of the same name).
#
# The two axes below (2 styles x 2 temperatures) expand to four cases with
# deterministic ids — stable across runs, so diffing lines cells up cell-by-cell.
# No API key required: the system under test is ../echo-provider.py, which
# (with no `user_input` var set) echoes the whole vars map — so the swept axis
# values show up directly in the output.
version: 1
project: examples
suite: matrix

providers:
  - id: echo
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  # Default ids: greet[style=terse,temperature=0], greet[style=terse,temperature=1],
  # greet[style=warm,temperature=0], greet[style=warm,temperature=1].
  # Each axis value lands in `vars`; the echo provider echoes the map, so every
  # cell's output is valid JSON carrying its own `style`/`temperature`.
  - id: greet
    matrix:
      style: [terse, warm]
      temperature: [0, 1]
    assert:
      - type: is-json
      - type: icontains
        value: "style"

  # A friendlier id shape via `matrix_id`, rendered against the axis values.
  # Ids: sweep-en, sweep-fr, sweep-de.
  - id: locale
    matrix_id: "sweep-{{ lang }}"
    matrix:
      lang: [en, fr, de]
    assert:
      - type: icontains
        value: "lang"

This expands to seven cells: a 2×2 sweep over style and temperature, plus a three-value lang sweep.

---

Example 09 — Datasets from files

Inline tests: stop scaling somewhere around thirty cases. A file:// glob keeps the suite readable and lets a dataset be owned, reviewed and diffed separately from the configuration that runs it.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Cases from files, not from this file.
#
# Inline `tests:` stop scaling somewhere around thirty cases. A `file://` glob
# keeps the suite readable and lets a dataset be owned, reviewed and diffed
# separately from the configuration that runs it.
#
# Every path is resolved relative to THIS directory and sandboxed:
# `file://../../etc/passwd` is refused, not read. Formats are chosen by
# extension — .yaml/.yml, .json, .jsonl/.ndjson, .csv, .tsv.
#
# Run:  domarinn run examples/09-dataset-glob
version: 1
project: examples
suite: dataset-glob

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

defaults:
  # Applied to every case from every file, so a dataset row carries only what
  # makes it distinct.
  assert:
    - type: length
      min: 1

tests:
  # A glob, not a single file: adding cases/shipping.yaml adds its cases with no
  # edit here. A glob that matches nothing is an error, not an empty suite.
  - "file://cases/*.yaml"

A dataset file is a bare sequence of cases — the same shape an inline entry has, with no wrapper key:

# A dataset file is a bare YAML sequence of cases — the same shape an inline
# `tests:` entry has. No wrapper key.
- id: refunds/approved
  tags: [billing]
  vars:
    user_input: "Your refund of $42.00 has been approved."
  assert:
    - type: icontains
      value: "approved"

- id: refunds/declined-with-reason
  tags: [billing]
  vars:
    user_input: "We could not refund this order because it is outside the 30-day window."
  assert:
    - type: icontains
      value: "30-day"
    - type: not-icontains
      value: "approved"

Formats are chosen by extension: .yaml/.yml, .json, .jsonl/.ndjson, .csv, .tsv. Every path resolves relative to the suite directory and is sandboxed — file://../../etc/passwd, or a symlink pointing out of the tree, is refused rather than read. A glob that matches nothing is an error, not an empty suite.

---

Example 10 — A CSV dataset

CSV is the format a non-engineer will hand you, so domarinn reads it directly. Four column names are reserved — id, tags, cache_salt, and __assert — and every other column becomes a var of that name.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Cases from a spreadsheet.
#
# CSV is the format a non-engineer will hand you, so domarinn reads it directly.
# Four column names are reserved and mean something; every OTHER column becomes
# a var of that name.
#
#   id          the case id
#   tags        comma-separated
#   cache_salt  per-case cache salt
#   __assert    the case's assertions, as JSON
#
# `.tsv` works the same way with tabs. For anything more structured than this,
# reach for .jsonl.
#
# Run:  domarinn run examples/10-dataset-csv
version: 1
project: examples
suite: dataset-csv

providers:
  - id: classifier
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  - "file://cases.csv"

id,tags,user_input,__assert
intent/refund,billing,"I want my money back","[{""type"":""icontains"",""value"":""money""}]"
intent/cancel,billing,"Please cancel my subscription","[{""type"":""icontains"",""value"":""cancel""}]"
intent/praise,csat,"This app is fantastic, thank you!","[{""type"":""not-icontains"",""value"":""cancel""}]"

__assert holds the case's assertions as JSON, which is what lets a spreadsheet express not-icontains without inventing a column convention. .tsv works identically with tabs. For anything more structured than this, reach for .jsonl.

---

Example 11 — Test generators

A glob reads cases someone wrote. A generator computes them — so coverage tracks the thing being covered instead of drifting from it. If your prompts, tools or policies live in a registry, enumerate the registry and every new entry gets a test for free, on the day it is added.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Cases produced by a program.
#
# A glob reads cases someone wrote. A GENERATOR computes them — so coverage
# tracks the thing being covered instead of drifting from it. If your prompts,
# tools, or policies live in a registry, enumerate the registry and every new
# entry gets a test for free, on the day it is added.
#
# A generator is an exec-protocol program like any other: it reads one JSON
# request on stdin and writes {"tests": [...]} on stdout. Whatever `config` you
# put here is handed to it verbatim, so one script can serve several suites.
#
# Note generators are NOT run by `domarinn list tests` unless you pass
# `--generators` — listing is meant to be free of side effects.
#
# Run:  domarinn run examples/11-test-generators
version: 1
project: examples
suite: test-generators

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  - generator:
      command: ["python3", "generate-cases.py"]
      # Arbitrary JSON, passed straight through to the generator. Keeping the
      # policy here rather than in the script means the suite still describes
      # what it covers.
      config:
        banned_phrases: ["as an AI", "I cannot help"]
        locales: ["en", "de"]
      timeout_ms: 30000

A generator is an exec-protocol program like any other: it reads one JSON request on stdin and writes {"tests": [...]} on stdout. Whatever config the suite sets is handed to it verbatim, so one script can serve several suites — and, more usefully, the suite still describes what it covers instead of hiding the policy inside the script.

#!/usr/bin/env python3
"""A domarinn test generator: computes cases instead of listing them.

Reads one JSON request on stdin and writes {"tests": [...]} on stdout. Each
entry is an ordinary test — the same shape you would have written inline.

The point of a generator is that coverage cannot drift. Add a locale or a banned
phrase to the suite's `config` and the matching cases appear on the next run,
with no dataset to remember to update.
"""

import json
import sys


def main():
    request = json.load(sys.stdin)
    config = request.get("config") or {}
    banned = config.get("banned_phrases", [])
    locales = config.get("locales", [])

    tests = []

    # One case per banned phrase. `not-icontains` is generated, not written out
    # by hand, so the list stays the single source of truth.
    for phrase in banned:
        slug = phrase.lower().replace(" ", "-")
        tests.append(
            {
                "id": f"banned/{slug}",
                "tags": ["policy"],
                "vars": {"user_input": "Happy to help — here are your options."},
                "assert": [{"type": "not-icontains", "value": phrase}],
            }
        )

    # One case per locale.
    for locale in locales:
        tests.append(
            {
                "id": f"locale/{locale}",
                "tags": ["i18n"],
                "vars": {"user_input": f"locale={locale}"},
                "assert": [{"type": "contains", "value": f"locale={locale}"}],
            }
        )

    json.dump({"tests": tests}, sys.stdout)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Generators do not run during list

domarinn list tests deliberately does not execute generators — listing is meant to be free of side effects. Pass --generators when you want their cases enumerated too.

---

Example 34 — A multi-turn conversation

Example 02 already used a messages: prompt — system, then one user turn. This one adds what an actual back-and-forth needs: a prior ASSISTANT turn too, fixed across every case, with only the newest user turn templated per case. That is what a real follow-up question looks like — the model needs to see what it already said, not just the newest line.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# A prompt with real history.
#
# `template:` renders one string. `messages:` renders a whole conversation —
# system, user, assistant, user again — which is what a follow-up question
# actually looks like: the model needs to see what it already said, not just
# the newest line. A prompt sets exactly one of the two; setting both, or
# neither, is a load error.
#
# Every message's `content` is a template like any other, so a fixed turn
# (the system persona, the canned prior answer) is free to hold no `{{ }}` at
# all — only the LAST turn below varies per case. The provider still receives
# the whole array on every call; it is domarinn that fans the case out over
# `vars`, not the shape of the messages.
#
# The exec protocol hands a messages-shaped prompt to a provider as
# `{"messages": [...]}` rather than a bare string (see
# docs/reference/protocol.md). The echo provider used here just returns
# whatever `prompt` it was given, which is exactly that object — so the
# assertions below can see the whole rendered transcript reached it.
#
# Run:  domarinn run examples/34-multi-turn-conversation
version: 1
project: examples
suite: multi-turn-conversation

providers:
  - id: echo
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

prompts:
  - id: support-followup
    messages:
      - role: system
        content: "You are a support agent for Aurora Notes. Cite the policy window exactly; never invent one."
      - role: user
        content: "Can I return a laptop sleeve I bought 10 days ago?"
      - role: assistant
        content: "Yes — items are returnable within 30 days of delivery, in original condition."
      - role: user
        content: "{{ followup }}"

tests:
  # Both cases share the fixed turns above and differ only in the follow-up,
  # which is the thing under test.
  - id: turns/asks-about-electronics
    vars:
      followup: "Does that 30-day window apply to opened electronics too?"
    assert:
      # The assistant's earlier turn reached the provider — proof the whole
      # history, not just the newest line, is part of every call.
      - type: contains
        value: "30 days of delivery"
      # The last turn's var rendered into the transcript the provider saw.
      - type: contains
        value: "opened electronics"

  - id: turns/asks-about-receipts
    vars:
      followup: "What if I've lost the original receipt?"
    assert:
      - type: contains
        value: "30 days of delivery"
      - type: contains
        value: "lost the original receipt"
      # Proof this case rendered its OWN follow-up rather than the other
      # case's: each case gets a fresh render, never a leftover from one that
      # ran before it.
      - type: not-contains
        value: "opened electronics"

Every turn is a template, so a fixed turn simply has nothing in it to substitute — only the last one varies per case. Both cases above share the first three turns byte for byte and differ only in the follow-up, which is the thing under test.

The echo provider makes this observable: an exec provider receives a messages: prompt as {"messages": [...]} (see the protocol), and echoing it back is what lets a contains assertion prove the whole rendered history — not just the last var — actually reached the provider.