A zero-cost gate on every PR¶

The problem. Prompts and templates rot silently. A variable gets renamed and stops substituting; a section separator leaks into the output; a refactor drops the authorization framing. None of it is caught by a type checker, and none of it is worth an LLM bill to catch.

The shape. A suite that renders every template your system owns and grades the result with deterministic assertions only. No API key, no network, seconds to run — so it can be a required check on every pull request rather than a nightly job people learn to ignore.

1. Make your system renderable¶

The system under test runs in "render" mode: given a template id, produce the rendered text and print it. That is an exec provider — one JSON request in, one JSON response out.

# yaml-language-server: $schema=../../domarinn.schema.json
#
# A render-health suite: it exercises an external "system under test" (here
# ../echo-provider.py) and grades its output with deterministic, zero-LLM
# assertions. No API key and no toolchain required — just python3.
version: 1
project: examples
suite: render-health

providers:
  - id: renderer
    type: exec
    command: ["python3", "../echo-provider.py"]
    timeout_ms: 30000
    # A rebuilt binary should not serve stale cache; bump this to a git SHA / hash.
    cache_salt: "dev"

tests:
  # A literal SSTI payload that must NEVER be interpolated: !raw keeps it verbatim.
  - id: adversarial/ssti-literal
    tags: [adversarial]
    vars:
      user_input: !raw "{{7*7}} {% for x in range(9) %}x{% endfor %}"
    assert:
      - type: not-contains
        value: "49"

  - id: greeting/basic
    tags: [smoke]
    vars:
      user_input: "hello world"
      # A deterministic template filter in action: slugify normalizes free text
      # into a URL/id-safe token ("Hello, World!" -> "hello-world").
      slug: "{{ 'Hello, World!' | slugify }}"
    assert:
      - type: contains
        value: "hello"
      - type: length
        max: 200000

defaults:
  # A deterministic guard applied to every case: the rendered output must never
  # be unreasonably large (a cheap catch for a runaway template).
  assert:
    - type: length
      max: 100000

Make the salt move with the build

Rendering is cheap, and a stale render is worse than a re-render — but omitting cache_salt does not get you that. The key hashes what the provider sends, never the bytes of the program sending it, so a rebuilt renderer replays yesterday's output and the gate passes on a template it never rendered.

Pin the salt to something that moves with the build — cache_salt: "${env:GITHUB_SHA}" in CI — or run this one gate with --no-cache. It costs seconds. See caching.md.

2. Assert the structural invariants¶

These are the ones worth having, in rough order of how often they catch something:

Assertion	Catches
`not-regex: "\\{\\{.*\\}\\}"`	An unsubstituted variable — a render hole.
`not-contains` on your separator	Internal metadata or a user/system delimiter leaking through.
`length: {min: 1}`	An empty render, usually a missing template.
`length: {max: N}`	A runaway loop, before it becomes a token bill.
`contains` on required framing	A refactor dropping a policy or authorization line.

Put the size ceiling in defaults.assert so it applies to every case without repetition.

3. Make coverage automatic¶

Listing templates by hand guarantees that the one added last Tuesday has no test. A generator enumerates your registry instead, so a new template gets a render test the day it lands:

tests:
  - generator:
      command: ["./target/release/my-eval", "generate-tests"]
      timeout_ms: 60000

Back it with a unit test in your own codebase asserting the registry and the manifest agree. Otherwise a template can escape the enumeration and the coverage gap is invisible.

4. Wire it into CI¶

- name: Prompt render health
  run: domarinn run eval/render-health.yaml

That is the whole step. Exit 0 passes, 1 means an assertion failed, 3 means the harness broke. No secrets, so it runs on fork pull requests too — which is exactly where you want a cheap check.

Verify it actually gates¶

A gate nobody has seen fail is a gate nobody should trust. Break something on purpose:

$ # temporarily remove a variable from a template, then:
$ domarinn run eval/render-health.yaml
$ echo $?
1

If that prints 0, your assertions are not asserting. Example 18 is a suite that is red by design, kept green-in-CI precisely so the failure path stays exercised.