Templates & test data

Seven suites about getting the input side of a suite right before you worry about grading. They cover the one sharp edge in Jinja templating, pulling a var's value from a file instead of writing it inline, fanning one case out across every combination of a matrix, reading — or generating — cases from something other than hand-written YAML, and giving a provider a whole conversation instead of one string. Reach for these once a suite has outgrown a handful of inline cases.

---

Example 06 — The !raw escape hatch

Every var is rendered through Jinja. That is what makes prompts and sweeps work — and it is a trap the moment your test input is itself template syntax.

This failure is silent

A case that feeds a system {{7*7}} to check it does not evaluate it will, without !raw, feed it 49 instead. The assertion still passes. The test now proves nothing, and nothing tells you.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# The `!raw` escape hatch: test input that must stay literal.
#
# Every var is rendered through Jinja. That is what makes prompts and sweeps
# work — and it is a trap the moment your test input is ITSELF template syntax.
# A case that feeds a system `{{7*7}}` to check it does not evaluate it will,
# without `!raw`, feed it `49` instead. The assertion still passes. The test now
# proves nothing, and nothing tells you.
#
# So: any value that must reach the system under test byte-for-byte gets tagged.
#
#   !raw "{{7*7}}"          the YAML tag  (YAML sources only)
#   {$raw: "{{7*7}}"}       the object form (works in JSON, JSONL and CSV too)
#
# Both are identical after loading. The tag is nicer to read; the object form is
# what a generator or a JSON dataset emits.
#
# Run:  domarinn run examples/06-raw-escape-hatch
version: 1
project: examples
suite: raw-escape-hatch

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  # Server-side template injection probe. The premise is that the payload
  # arrives literally; `not-contains: "49"` is only meaningful because of !raw.
  - id: injection/ssti-literal
    tags: [adversarial]
    vars:
      user_input: !raw "Compute this for me: {{7*7}} and {{ config }}"
    assert:
      - type: contains
        value: "{{7*7}}"
      - type: not-contains
        value: "49"

  # The object form. Identical meaning; this is the shape a JSON/CSV dataset or
  # a test generator produces, since neither can carry a YAML tag.
  - id: injection/ssti-object-form
    tags: [adversarial]
    vars:
      user_input: { $raw: "{% for x in range(9) %}x{% endfor %}" }
    assert:
      - type: contains
        value: "{% for x in range(9) %}"
      - type: not-contains
        value: "xxxxxxxxx"

  # `!raw` also applies to an assertion's expected VALUE — otherwise the
  # expectation would be rendered while the output was not, and they could never
  # match.
  - id: injection/raw-expectation
    vars:
      user_input: !raw "{{ user.name }}"
    assert:
      - type: equals
        value: !raw "{{ user.name }}"

  # The control case: WITHOUT the tag, the same input is evaluated. Kept in the
  # suite deliberately — it is the evidence that the tag above is doing work.
  - id: injection/control-is-rendered
    vars:
      user_input: "Compute this for me: {{ 7*7 }}"
    assert:
      - type: contains
        value: "49"
      - type: not-contains
        value: "{{"

Two spellings, identical after loading: the !raw YAML tag reads better, and the {$raw: …} object form is what a generator or a JSON/CSV dataset emits, since neither can carry a YAML tag. Note that !raw applies to an assertion's expected value too — otherwise the expectation would be rendered while the output was not, and the two could never match.

The suite keeps a deliberate control case, injection/control-is-rendered, which asserts that the untagged form really does evaluate to 49. It is the evidence that the tag above is doing work.

---

Example 07 — File-content vars

A var can take its value from a file next to the suite instead of being written inline — useful for large documents, golden fixtures, and adversarial inputs you would rather not paste into YAML.

Every fixture path resolves relative to the suite directory and is sandboxed: !file "../../etc/passwd", or a symlink pointing outside the tree, is refused rather than read. The four cases below cover the whole surface — a plain text fixture, a .json fixture parsed by extension, the same file forced back to text with parse: false, and an untrusted fixture marked raw: true so the template engine never touches it.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# File-content vars: a var can pull its value from a file next to the suite,
# instead of being written inline. Handy for large documents, golden fixtures,
# and untrusted adversarial inputs.
#
# Every fixture path is resolved *relative to this directory* and sandboxed —
# `!file "../../etc/passwd"` (or a symlink pointing outside) is refused, not
# read. No API key required: the system under test is ../echo-provider.py.
version: 1
project: examples
suite: file-vars

providers:
  - id: echo
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  # 1. Plain text fixture, loaded via the `!file` tag. Its content becomes the
  #    `user_input` var, which the echo provider echoes straight back.
  - id: doc/echo
    vars:
      user_input: !file "fixtures/article.txt"
    assert:
      - type: icontains
        value: "assertions"

  # 2. A `.json` fixture is parsed by extension into a structured value. With no
  #    `user_input` set, the echo provider echoes the whole vars map, so the
  #    parsed object round-trips into the (valid-JSON) output.
  - id: doc/structured-rubric
    vars:
      rubric: { $file: "fixtures/rubric.json" }
    assert:
      - type: is-json
      - type: icontains
        value: "must_mention"

  # 3. `parse: false` forces text even for a structured extension — the raw JSON
  #    source is echoed verbatim rather than parsed.
  - id: doc/raw-json-text
    vars:
      user_input: { $file: "fixtures/rubric.json", parse: false }
    assert:
      - type: contains
        value: "must_mention"

  # 4. An untrusted fixture marked `raw: true` is NEVER run through the template
  #    engine, so an SSTI payload in the file stays literal ({{7*7}} != 49).
  - id: adversarial/ssti-fixture
    tags: [adversarial]
    vars:
      user_input: { $file: "fixtures/ssti-probe.txt", raw: true }
    assert:
      - type: not-contains
        value: "49"

raw: true is not optional for untrusted input

Every var goes through Jinja. A fixture containing {{7*7}} renders as 49 unless you mark it raw — which silently destroys the premise of any test whose whole point is that the payload stayed literal. See domarinn.yaml.

---

Example 08 — Matrix sweeps

One case fans out over the cartesian product of its axes, producing one concrete case per combination. Each axis value is merged into vars, where it wins over a base var of the same name.

The ids are deterministic — greet[style=terse,temperature=0] and friends — which is what lets domarinn diff line two runs up cell by cell. When the generated shape is unwieldy, matrix_id renders a friendlier one against the axis values.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Matrix / parameter sweeps: one case fans out over the cartesian product of its
# axes, one concrete case per combination. Each axis value is merged into `vars`
# (the axis wins over a base var of the same name).
#
# The two axes below (2 styles x 2 temperatures) expand to four cases with
# deterministic ids — stable across runs, so diffing lines cells up cell-by-cell.
# No API key required: the system under test is ../echo-provider.py, which
# (with no `user_input` var set) echoes the whole vars map — so the swept axis
# values show up directly in the output.
version: 1
project: examples
suite: matrix

providers:
  - id: echo
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  # Default ids: greet[style=terse,temperature=0], greet[style=terse,temperature=1],
  # greet[style=warm,temperature=0], greet[style=warm,temperature=1].
  # Each axis value lands in `vars`; the echo provider echoes the map, so every
  # cell's output is valid JSON carrying its own `style`/`temperature`.
  - id: greet
    matrix:
      style: [terse, warm]
      temperature: [0, 1]
    assert:
      - type: is-json
      - type: icontains
        value: "style"

  # A friendlier id shape via `matrix_id`, rendered against the axis values.
  # Ids: sweep-en, sweep-fr, sweep-de.
  - id: locale
    matrix_id: "sweep-{{ lang }}"
    matrix:
      lang: [en, fr, de]
    assert:
      - type: icontains
        value: "lang"

This expands to seven cells: a 2×2 sweep over style and temperature, plus a three-value lang sweep.

---

Example 09 — Datasets from files

Inline tests: stop scaling somewhere around thirty cases. A file:// glob keeps the suite readable and lets a dataset be owned, reviewed and diffed separately from the configuration that runs it.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Cases from files, not from this file.
#
# Inline `tests:` stop scaling somewhere around thirty cases. A `file://` glob
# keeps the suite readable and lets a dataset be owned, reviewed and diffed
# separately from the configuration that runs it.
#
# Every path is resolved relative to THIS directory and sandboxed:
# `file://../../etc/passwd` is refused, not read. Formats are chosen by
# extension — .yaml/.yml, .json, .jsonl/.ndjson, .csv, .tsv.
#
# Run:  domarinn run examples/09-dataset-glob
version: 1
project: examples
suite: dataset-glob

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

defaults:
  # Applied to every case from every file, so a dataset row carries only what
  # makes it distinct.
  assert:
    - type: length
      min: 1

tests:
  # A glob, not a single file: adding cases/shipping.yaml adds its cases with no
  # edit here. A glob that matches nothing is an error, not an empty suite.
  - "file://cases/*.yaml"

A dataset file is a bare sequence of cases — the same shape an inline entry has, with no wrapper key:

# A dataset file is a bare YAML sequence of cases — the same shape an inline
# `tests:` entry has. No wrapper key.
- id: refunds/approved
  tags: [billing]
  vars:
    user_input: "Your refund of $42.00 has been approved."
  assert:
    - type: icontains
      value: "approved"

- id: refunds/declined-with-reason
  tags: [billing]
  vars:
    user_input: "We could not refund this order because it is outside the 30-day window."
  assert:
    - type: icontains
      value: "30-day"
    - type: not-icontains
      value: "approved"

Formats are chosen by extension: .yaml/.yml, .json, .jsonl/.ndjson, .csv, .tsv. Every path resolves relative to the suite directory and is sandboxed — file://../../etc/passwd, or a symlink pointing out of the tree, is refused rather than read. A glob that matches nothing is an error, not an empty suite.

---

Example 10 — A CSV dataset

CSV is the format a non-engineer will hand you, so domarinn reads it directly. Four column names are reserved — id, tags, cache_salt, and __assert — and every other column becomes a var of that name.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Cases from a spreadsheet.
#
# CSV is the format a non-engineer will hand you, so domarinn reads it directly.
# Four column names are reserved and mean something; every OTHER column becomes
# a var of that name.
#
#   id          the case id
#   tags        comma-separated
#   cache_salt  per-case cache salt
#   __assert    the case's assertions, as JSON
#
# `.tsv` works the same way with tabs. For anything more structured than this,
# reach for .jsonl.
#
# Run:  domarinn run examples/10-dataset-csv
version: 1
project: examples
suite: dataset-csv

providers:
  - id: classifier
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  - "file://cases.csv"

id,tags,user_input,__assert
intent/refund,billing,"I want my money back","[{""type"":""icontains"",""value"":""money""}]"
intent/cancel,billing,"Please cancel my subscription","[{""type"":""icontains"",""value"":""cancel""}]"
intent/praise,csat,"This app is fantastic, thank you!","[{""type"":""not-icontains"",""value"":""cancel""}]"

__assert holds the case's assertions as JSON, which is what lets a spreadsheet express not-icontains without inventing a column convention. .tsv works identically with tabs. For anything more structured than this, reach for .jsonl.

---

Example 11 — Test generators

A glob reads cases someone wrote. A generator computes them — so coverage tracks the thing being covered instead of drifting from it. If your prompts, tools or policies live in a registry, enumerate the registry and every new entry gets a test for free, on the day it is added.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Cases produced by a program.
#
# A glob reads cases someone wrote. A GENERATOR computes them — so coverage
# tracks the thing being covered instead of drifting from it. If your prompts,
# tools, or policies live in a registry, enumerate the registry and every new
# entry gets a test for free, on the day it is added.
#
# A generator is an exec-protocol program like any other: it reads one JSON
# request on stdin and writes {"tests": [...]} on stdout. Whatever `config` you
# put here is handed to it verbatim, so one script can serve several suites.
#
# Note generators are NOT run by `domarinn list tests` unless you pass
# `--generators` — listing is meant to be free of side effects.
#
# Run:  domarinn run examples/11-test-generators
version: 1
project: examples
suite: test-generators

providers:
  - id: assistant
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  - generator:
      command: ["python3", "generate-cases.py"]
      # Arbitrary JSON, passed straight through to the generator. Keeping the
      # policy here rather than in the script means the suite still describes
      # what it covers.
      config:
        banned_phrases: ["as an AI", "I cannot help"]
        locales: ["en", "de"]
      timeout_ms: 30000

A generator is an exec-protocol program like any other: it reads one JSON request on stdin and writes {"tests": [...]} on stdout. Whatever config the suite sets is handed to it verbatim, so one script can serve several suites — and, more usefully, the suite still describes what it covers instead of hiding the policy inside the script.

#!/usr/bin/env python3
"""A domarinn test generator: computes cases instead of listing them.

Reads one JSON request on stdin and writes {"tests": [...]} on stdout. Each
entry is an ordinary test — the same shape you would have written inline.

The point of a generator is that coverage cannot drift. Add a locale or a banned
phrase to the suite's `config` and the matching cases appear on the next run,
with no dataset to remember to update.
"""

import json
import sys


def main():
    request = json.load(sys.stdin)
    config = request.get("config") or {}
    banned = config.get("banned_phrases", [])
    locales = config.get("locales", [])

    tests = []

    # One case per banned phrase. `not-icontains` is generated, not written out
    # by hand, so the list stays the single source of truth.
    for phrase in banned:
        slug = phrase.lower().replace(" ", "-")
        tests.append(
            {
                "id": f"banned/{slug}",
                "tags": ["policy"],
                "vars": {"user_input": "Happy to help — here are your options."},
                "assert": [{"type": "not-icontains", "value": phrase}],
            }
        )

    # One case per locale.
    for locale in locales:
        tests.append(
            {
                "id": f"locale/{locale}",
                "tags": ["i18n"],
                "vars": {"user_input": f"locale={locale}"},
                "assert": [{"type": "contains", "value": f"locale={locale}"}],
            }
        )

    json.dump({"tests": tests}, sys.stdout)
    sys.stdout.write("\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Generators do not run during list

domarinn list tests deliberately does not execute generators — listing is meant to be free of side effects. Pass --generators when you want their cases enumerated too.

---

Example 34 — A multi-turn conversation

Example 02 already used a messages: prompt — system, then one user turn. This one adds what an actual back-and-forth needs: a prior ASSISTANT turn too, fixed across every case, with only the newest user turn templated per case. That is what a real follow-up question looks like — the model needs to see what it already said, not just the newest line.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# A prompt with real history.
#
# `template:` renders one string. `messages:` renders a whole conversation —
# system, user, assistant, user again — which is what a follow-up question
# actually looks like: the model needs to see what it already said, not just
# the newest line. A prompt sets exactly one of the two; setting both, or
# neither, is a load error.
#
# Every message's `content` is a template like any other, so a fixed turn
# (the system persona, the canned prior answer) is free to hold no `{{ }}` at
# all — only the LAST turn below varies per case. The provider still receives
# the whole array on every call; it is domarinn that fans the case out over
# `vars`, not the shape of the messages.
#
# The exec protocol hands a messages-shaped prompt to a provider as
# `{"messages": [...]}` rather than a bare string (see
# docs/reference/protocol.md). The echo provider used here just returns
# whatever `prompt` it was given, which is exactly that object — so the
# assertions below can see the whole rendered transcript reached it.
#
# Run:  domarinn run examples/34-multi-turn-conversation
version: 1
project: examples
suite: multi-turn-conversation

providers:
  - id: echo
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

prompts:
  - id: support-followup
    messages:
      - role: system
        content: "You are a support agent for Aurora Notes. Cite the policy window exactly; never invent one."
      - role: user
        content: "Can I return a laptop sleeve I bought 10 days ago?"
      - role: assistant
        content: "Yes — items are returnable within 30 days of delivery, in original condition."
      - role: user
        content: "{{ followup }}"

tests:
  # Both cases share the fixed turns above and differ only in the follow-up,
  # which is the thing under test.
  - id: turns/asks-about-electronics
    vars:
      followup: "Does that 30-day window apply to opened electronics too?"
    assert:
      # The assistant's earlier turn reached the provider — proof the whole
      # history, not just the newest line, is part of every call.
      - type: contains
        value: "30 days of delivery"
      # The last turn's var rendered into the transcript the provider saw.
      - type: contains
        value: "opened electronics"

  - id: turns/asks-about-receipts
    vars:
      followup: "What if I've lost the original receipt?"
    assert:
      - type: contains
        value: "30 days of delivery"
      - type: contains
        value: "lost the original receipt"
      # Proof this case rendered its OWN follow-up rather than the other
      # case's: each case gets a fresh render, never a leftover from one that
      # ran before it.
      - type: not-contains
        value: "opened electronics"

Every turn is a template, so a fixed turn simply has nothing in it to substitute — only the last one varies per case. Both cases above share the first three turns byte for byte and differ only in the follow-up, which is the thing under test.

The echo provider makes this observable: an exec provider receives a messages: prompt as {"messages": [...]} (see the protocol), and echoing it back is what lets a contains assertion prove the whole rendered history — not just the last var — actually reached the provider.

---

Example 41 — Per-case conversation history

Example 34 fixes the transcript in the prompt: every case shares the same prior turns. This one moves the history to the cases — each brings its own prior turns, different lengths and roles included, and the prompt names where they splice in with the bare history marker entry:

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Per-case conversation history.
#
# Example 34 fixes the transcript in the prompt: every case shares the same
# prior turns and varies only the newest line. This example moves the history
# to the CASES: each one brings its own prior turns — different lengths,
# different roles — and the prompt names where they splice in with the bare
# `history` marker entry. A prompt may hold at most one marker; a prompt
# without one gets each case's history right after its leading system
# turn(s), and a `template:` prompt becomes the transcript's newest user
# turn.
#
# A case writes its history inline, or points at a whole transcript with
# `history: file://…` (a YAML/JSON list of `{role, content}`). A CSV test
# file carries it in the reserved `__history` column — a JSON list of turns
# or a `file://` path, with an empty cell meaning none. Turn contents are
# templates like any other message content, and may be `file://` themselves.
#
# History is part of each case's request identity: two cases that differ only
# in their prior turns key — and cache — separately.
#
# The echo provider returns the prompt it was handed, so the assertions below
# can see exactly which transcript reached the provider — and, just as
# important, which other case's transcript did NOT.
#
# Run:  domarinn run examples/41-per-case-history
version: 1
project: examples
suite: per-case-history

providers:
  - id: echo
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

prompts:
  - id: support
    messages:
      - role: system
        content: "You are a support agent for Aurora Notes. Answer from the conversation so far."
      - history
      - role: user
        content: "{{ followup }}"

tests:
  # No history: the marker simply disappears and the case is a first contact.
  - id: history/first-contact
    vars:
      followup: "What is your return window?"
    assert:
      - type: contains
        value: "What is your return window?"
      # Nothing from the other cases' conversations leaked in.
      - type: not-contains
        value: "carrying case"

  # Two prior turns, inline. This case's transcript is system → user →
  # assistant → user, and only this case sees these turns.
  - id: history/follow-up
    history:
      - role: user
        content: "Can I return a carrying case bought 10 days ago?"
      - role: assistant
        content: "Yes — within 30 days of delivery, in original condition."
    vars:
      followup: "And if the box is already open?"
    assert:
      # The assistant's earlier turn reached the provider…
      - type: contains
        value: "30 days of delivery"
      # …together with this case's newest line.
      - type: contains
        value: "already open"

  # A whole transcript from a file. Longer conversations live next to the
  # suite instead of bloating it; the file is a YAML list of turns.
  - id: history/escalation
    history: file://convos/escalation.yaml
    vars:
      followup: "So who exactly is going to call me back?"
    assert:
      - type: contains
        value: "escalate this to a supervisor"
      - type: not-contains
        value: "carrying case"

  # CSV rows: `__history` is reserved, like `__assert`.
  - file://cases.csv

The history/escalation case points at a transcript file instead of inlining it — the same shape, a YAML list of turns:

# A transcript a case points at with `history: file://convos/escalation.yaml`.
# The same shape as inline history: a list of {role, content} turns.
- role: user
  content: "My order arrived damaged and the support chat disconnected twice."
- role: assistant
  content: "I'm sorry — I can escalate this to a supervisor right away."
- role: user
  content: "Please do."
- role: assistant
  content: "Done. You'll be contacted within one business day."

And the CSV rows carry theirs in the reserved __history column, JSON-encoded (or a file:// path). An empty cell means unset — with a defaults.history in play it would inherit the default; a literal [] cell is the opt-out:

id,__history,followup,__assert
csv/with-history,"[{""role"": ""user"", ""content"": ""Do you ship to Iceland?""}, {""role"": ""assistant"", ""content"": ""We do — flat rate, 5 business days.""}]",How much is the flat rate?,"[{""type"": ""contains"", ""value"": ""5 business days""}, {""type"": ""contains"", ""value"": ""How much is the flat rate?""}]"
csv/no-history,,Do you ship to Iceland?,"[{""type"": ""contains"", ""value"": ""Do you ship to Iceland?""}, {""type"": ""not-contains"", ""value"": ""flat rate""}]"

Three details worth knowing. A prompt may hold at most one marker, and a prompt without one still works: each case's history lands right after the prompt's leading system turn(s), and a template: prompt becomes the transcript's newest user turn. Second, history joins each case's request identity — two cases differing only in their prior turns key, and cache, separately. Third, the not-contains assertions above are the point: each case's transcript is its own, and nothing from another case's conversation leaks in.

See per-case history in the reference for the full splice rules.

---

Example 42 — Replaying a tool-using transcript

Example 41 gives each case its own prior turns, but only as prose — and that is the wrong shape for the case history is most useful for. An agent told to "call lookup_order first" will, in a single-turn eval, call it and stop: nothing ever feeds a result back, so the decision you actually wanted to grade never happens. History is what supplies turn one's result. But turn one is a tool call, and turn two's input is a tool result:

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Replaying a tool-using transcript.
#
# Example 41 gives each case its own prior turns, but only as prose. That is
# the wrong shape for the case `history` is most useful for: an agent told to
# "call `lookup_order` first" will, in a single-turn eval, call it and stop —
# nothing ever feeds a result back, so the decision you actually want to grade
# never happens. History is what supplies turn one's result. But turn one IS a
# tool call, and turn two's input IS a tool result.
#
# So a turn carries two more things:
#
#   * an `assistant` turn may have `tool_calls` — the same {id, name,
#     arguments} shape a provider REPORTS on the way out, so the `tool_calls`
#     block of a stored case pastes straight into a suite;
#   * a `tool` turn carries the result, naming the call it answers with
#     `tool_call_id` (optional — position pairs them when it is omitted).
#
# Each provider maps these to its own shape, and they disagree in exactly
# mirrored ways: `anthropic` coalesces a round of parallel results into ONE
# user message of `tool_result` blocks, while `openai` expands the same round
# into separate `role: "tool"` messages. `exec` receives the JSON verbatim.
#
# A turn's `content` may also be a list of typed blocks rather than a string,
# which is how a model's `thinking` is replayed. Thinking is NEVER templated:
# its `signature` is a vendor integrity token over those exact bytes, so
# rendering a variable inside one would invalidate the replay.
#
# domarinn still never EXECUTES a tool — it replays what happened and grades
# what the model does next.
#
# The echo provider returns the prompt it was handed, so the assertions below
# can see exactly which structured turn reached the provider — and that it
# arrived as structure rather than as a paraphrase of one.
#
# Run:  domarinn run examples/42-tool-call-history
version: 1
project: examples
suite: tool-call-history

providers:
  - id: echo
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

# The declared surface, so a reader sees the tool the transcript replays a
# call into. Names must match: a call is not templated, for the same reason
# this declaration is not.
tools:
  - name: lookup_order
    description: Fetch an order by its id.
    input_schema:
      type: object
      required: ["order_id"]
      properties:
        order_id: { type: integer }
  # Declared because the file-based transcript below replays a call into it —
  # the same "names must match" rule, applied to a transcript loaded from disk.
  - name: refund_policy
    description: Look up the return window for a product category.
    input_schema:
      type: object
      required: ["category"]
      properties:
        category: { type: string }

prompts:
  - id: support
    messages:
      - role: system
        content: "You are a support agent. Use the tools you are given."
      - history
      - role: user
        content: "{{ followup }}"

tests:
  # One round trip: the model called a tool, the tool answered, and the case
  # grades what it says next. `tool_call_id` names which call the result
  # answers — omit it and position pairs them instead.
  - id: tools/replays-a-call
    history:
      - role: user
        content: "Where is order 1042?"
      - role: assistant
        tool_calls:
          - id: call_1
            name: lookup_order
            arguments: { order_id: 1042 }
      - role: tool
        tool_call_id: call_1
        content: '{"status": "shipped", "eta": "Tuesday"}'
    vars:
      followup: "So when does it arrive?"
    assert:
      # The call reached the provider as a structured turn…
      - type: contains
        value: "lookup_order"
      # …and so did the result it answers.
      - type: contains
        value: "shipped"

  # A parallel round: one assistant turn, two calls, two results. This is the
  # case that makes the vendor split concrete — `anthropic` folds both results
  # into one user message, `openai` sends two `role: "tool"` messages, and the
  # suite says neither.
  - id: tools/parallel-round
    history:
      - role: user
        content: "Status of 1042 and 1043?"
      - role: assistant
        content: "Let me look up both."
        tool_calls:
          - name: lookup_order
            arguments: { order_id: 1042 }
          - name: lookup_order
            arguments: { order_id: 1043 }
      - role: tool
        content: '{"status": "shipped"}'
      - role: tool
        content: '{"status": "pending"}'
    vars:
      followup: "Which one is late?"
    assert:
      - type: contains
        value: "pending"
      # No id was written anywhere; domarinn derived one from position so both
      # vendors still get the correlation they require.
      - type: contains
        value: "Let me look up both."

  # Arguments are templated leaf by leaf, and every leaf keeps its own type:
  # an untemplated `1042` stays the integer the tool's schema wants, while the
  # string leaf beside it is rendered against the case vars.
  #
  # The corollary is worth stating, because it is easy to trip over: a leaf you
  # template is a STRING on the way out. `{order_id: "{{ order }}"}` sends
  # `"7"`, not `7` — and so does a whole-value `arguments: "{{ some_var }}"`,
  # since templating is string-in/string-out at every leaf. A non-string
  # argument therefore has to be written literally, as `order_id` is below.
  - id: tools/templated-arguments
    history:
      - role: user
        content: "Where is order {{ order }} for {{ who }}?"
      - role: assistant
        tool_calls:
          - name: lookup_order
            arguments: { order_id: 1042, note: "for {{ who }}" }
      - role: tool
        content: '{"status": "delivered"}'
    vars:
      order: 7
      who: "Ada"
      followup: "Thanks — was it signed for?"
    assert:
      # The string leaf rendered against the case vars…
      - type: contains
        value: "for Ada"
      # …while the untemplated number stayed a number: JSON writes an integer
      # bare, so a quoted "1042" here would mean the type was lost.
      - type: contains
        value: '"order_id":1042'
      - type: not-contains
        value: '"order_id":"1042"'

  # A `content` list instead of a string: this is how a model's reasoning is
  # replayed. The `signature` is an integrity token over the thinking bytes, so
  # neither it nor the text it covers is ever templated.
  - id: tools/replays-thinking
    history:
      - role: user
        content: "Where is order 1042?"
      - role: assistant
        content:
          - type: thinking
            thinking: "The customer gave an order id, so lookup_order applies."
            signature: "sig-abc123"
          - type: text
            text: "Checking that now."
        tool_calls:
          - name: lookup_order
            arguments: { order_id: 1042 }
      - role: tool
        content: '{"status": "shipped"}'
    vars:
      followup: "And the carrier?"
    assert:
      # The signature survived verbatim — proof the block was not re-rendered.
      - type: contains
        value: "sig-abc123"
      - type: contains
        value: "Checking that now."

  # A whole tool-using transcript from a file, so a long agent trace lives next
  # to the suite instead of inside it.
  - id: tools/from-a-file
    history: file://convos/lookup.yaml
    vars:
      followup: "Can you refund it?"
    assert:
      - type: contains
        value: "refund_policy"
      - type: not-contains
        value: "1042"

An assistant turn's tool_calls use the same {id, name, arguments} shape a provider reports on the way out, so the tool_calls block of a stored case pastes straight into a suite. A tool turn carries the result, naming the call it answers with tool_call_id — optional, because position pairs them when a transcript omits it, which is what makes the tools/parallel-round case above work without a single id written anywhere.

A whole tool-using trace can live in a file, which is usually where one captured from a real agent run already is:

# A tool-using transcript a case points at with
# `history: file://convos/lookup.yaml`.
#
# The same shape as inline history, tool turns included — which is the point:
# a trace captured from a real agent run is a file, not something you want
# pasted into a suite.
- role: user
  content: "This jacket does not fit. What are my options?"
- role: assistant
  content: "Let me check what applies to your order."
  tool_calls:
    - id: call_policy
      name: refund_policy
      arguments:
        category: apparel
- role: tool
  tool_call_id: call_policy
  content: '{"window_days": 30, "condition": "unworn, tags attached"}'
- role: assistant
  content: "You have 30 days from delivery, provided the tags are still attached."

Three details. Arguments are templated leaf by leaf, so each leaf keeps its own type: an untemplated {order_id: 1042} stays an integer, which is what a tool-call assertion comparing against the decoded object needs. The corollary is easy to trip over — a leaf you do template comes out a string, including a whole-value "{{ var }}" reference — so a non-string argument has to be written literally, and varying one per case is not expressible today. A turn's content may be a list of typed blocks instead of a string, which is how thinking is replayed; that one is never templated, because its signature is a vendor integrity token over those exact bytes. And the two vendors disagree in mirrored ways about a round of parallel results — anthropic folds them into one user message, openai sends one message each — which domarinn handles so the suite does not have to say.