Guides¶
The Examples page is a ladder of per-capability suites — one feature per file. Guides are the layer above: end-to-end walkthroughs for a real situation, tying several capabilities, the right commands, and the verification steps together.
Each one is built from suites on the Examples page, so everything here is runnable and everything here is tested.
Read this first if you are new
domarinn evaluates a grid: every provider against every prompt against every test, one call per cell. Assertions grade the answer; cheap ones run first and can short-circuit the expensive ones; the case that cell produces ends pass, fail, error or skip.
The load-bearing distinction in most of these guides is fail versus error. A failure means the system under test got worse. An error means you learned nothing. They gate differently — exit 1 and exit 3 — and a pipeline that conflates them is a pipeline people stop reading.
See How a run works.
The first four answer "what am I pointing domarinn at"; the rest are what you do with it once it runs.
| # | Guide | When you reach for it | Cost |
|---|---|---|---|
| 01 | Test a model API | Compare two models, or catch a snapshot changing under you. | Per run |
| 02 | Evaluate your own application | Test what actually ships, not the model underneath it. | Depends |
| 03 | Test an HTTP endpoint | The assistant is already behind your own JSON API. | Depends |
| 04 | Local LLMs with Ollama | Iterate on prompts and rubrics with no key and no spend. | Free |
| 05 | A zero-cost gate on every PR | Catch template and formatting regressions with no key and no spend. | Free |
| 06 | Gate a PR in CI | Block a merge that makes things worse, without demanding perfection. | Per run |
| 07 | Grade an assistant against a policy | Scope enforcement, refusals, and rubrics that grade one thing. | Grader only |
| 08 | Evaluate structured output | An agent that must emit a parseable object with correct field semantics. | Grader only |
| 09 | Share a cache across a team | Stop everyone paying separately for the same answers. | Saves money |
| 10 | Self-host the server | Run the results server yourself: Docker, compose, Kubernetes, backups. | Self-hosted |
| 11 | Migrate from promptfoo | Convert an existing promptfoo config and see exactly what changes. | Free |
| 12 | Troubleshooting | Symptom, cause, fix — the ways a gate can be green and check nothing. | Free |
The two-layer pattern¶
Most teams that get value out of this end up with the same shape, and it is worth stating plainly because it is not obvious from the reference docs:
- A deterministic layer that runs on every pull request. No API key, no spend, seconds to run. It catches render holes, missing sections, format drift, leaked internals. Guide 05.
- A graded layer that runs on a schedule or on demand, costs real money, and is gated against a stored baseline. It catches behavioural regressions the first layer cannot see. Guide 06.
Trying to do both in one suite means either paying for the cheap checks or skipping the expensive ones. Splitting them means the fast gate can be mandatory.
See also¶
- Examples — the per-capability ladder these are built from.
- Install — get the binary, then write and run your first suite.
- Troubleshooting — when a step does not go green.