Providers & the exec boundary¶

Your system is the system under test¶

Most eval tools assume the thing being tested is a model behind a prompt. domarinn inverts that: an exec command or an http endpoint is a normal provider, on equal footing with a native anthropic or openai client, and prompts are optional — a system that resolves its own prompts from its own registry does not want yours.

The consequence people notice first is the one that matters: what you test is what ships. The eval drives the same code path production does — the same retrieval, the same guardrails, the same client-side rendering — rather than measuring the model underneath the product and calling that close enough. Existing tools that bolt this on as an afterthought pay for it in shims: a dummy config field, a provider adapter that just shells out to a binary, tests generated by pointing a config at a function. Here it is the first-class case, not a workaround.

The consequence people notice second is the price, and it is the reason the rest of this page exists: domarinn cannot see inside your program, only what it sends and receives, so it has no way to know when your logic changed rather than the question you asked it. That price is paid explicitly, once, as cache_salt — see below — rather than silently, on every run, as a stale answer nobody notices.

Five provider types, one boundary¶

`type`	System under test?	Crosses the boundary as
`exec`	yes	a spawned process, one JSON document each way over stdio
`anthropic`	yes	one HTTP call to the Messages API
`openai`	yes	one HTTP call to a chat-completions endpoint
`http`	yes	one HTTP call, templated to whatever your service expects
`embeddings`	no — powers `similar`	one HTTP call, filtered out of the graded matrix

Four of the five are systems under test, and all four share the same shape from domarinn's side regardless of transport: a request goes out, a response comes back, and nothing else is observed. exec is the general case — any language, over stdin/stdout — while anthropic, openai and http are native clients for the boundary's most common shapes. Full field-by-field behavior for each is reference/providers.md; this page is about the boundary itself, not the five ways to cross it.

What crosses the boundary¶

For an exec provider, transport is deliberately minimal: domarinn writes one JSON request to the child's stdin, closes it, and reads one JSON response from stdout. No streaming, no back-and-forth, no second turn — see the protocol reference for the exact envelope. An HTTP-shaped provider (anthropic, openai, http) crosses the same kind of boundary over a socket instead of a pipe: one request, one response.

That is not just an implementation detail — it is exactly what the cache can key on. The key is a hash of the request: for exec, the command, its args, and the stdin document (minus the test block, which is correlation metadata, not part of the question); for an HTTP-shaped call, the resolved URL, method and body. Nothing about the program's own bytes, its dependencies, or what it does internally is observable from the outside, so none of it can be — or needs to be — in the key.

Why a process, not a plugin¶

The alternative to a process boundary is an embedded interpreter or a dynamic plugin: load your code into domarinn's own address space and call it directly. domarinn does not do this, for the reason the promptfoo migration guide states plainly: a process boundary "costs a spawn and buys you a checker you can run and test on its own." Your provider or custom assertion is a normal program in whatever language you already use, runnable and testable without domarinn in the loop at all, immune to a crash taking the harness down with it, and requiring no shared ABI, runtime, or SDK version to line up with domarinn's own.

The consequence: `cache_salt`¶

The one-shot boundary is also the reason cache_salt exists. domarinn cannot see inside your program — it observes only what crosses the boundary — so it has no way to know when the program's own logic changed, only when the request it was asked did. A rebuild of your exec provider, a new prompt template it loaded from disk by id, a different model baked into an http endpoint: none of these change the request, so none of them change the cache key, and a stale answer replays convincingly. cache_salt is the explicit, deliberate lever for exactly this — a version pin on an exec provider (the only type that takes one), and a content digest or plain string per case or in defaults for everything else, an http endpoint's swapped model included — because the alternative is a heuristic that guesses wrong in both directions. See The one-rule cache § Salts for the two granularities and $digest:.