Skip to content

Migrating from promptfoo

domarinn ships a converter:

$ domarinn import promptfoo promptfooconfig.yaml > domarinn.yaml

It prints YAML to stdout and leaves a # NOTE: comment wherever an assertion or a provider did not map. Read those before running anything — they are the parts that need a decision, not a translation.

Convert, then run validate

$ domarinn import promptfoo promptfooconfig.yaml > domarinn.yaml
$ domarinn validate .
$ domarinn list tests .

validate catches shape problems without calling anything, and list tests shows you what the suite actually resolves to — worth checking before a first run spends money.

A worked conversion

Both halves of one conversion ship in the repository as example 39, and both are run in CI. The promptfoo config going in:

# The BEFORE half of a migration, in promptfoo's own format.
#
# `domarinn import promptfoo` converts this file into the domarinn.yaml beside
# it. Anything with a faithful equivalent is translated; anything without one
# becomes a `# NOTE:` line in the output, and two of the assertions below are
# there precisely to produce those notes — a migration's real work is the parts
# that need a decision rather than a translation.
#
# The provider is the ladder's shared echo program rather than a model, so both
# halves of the pair run offline. Everything else is shaped like a small, real
# promptfoo config.
#
# Convert:  domarinn import promptfoo examples/39-import-promptfoo/promptfooconfig.yaml
description: Refund-policy replies from the support assistant

providers:
  # promptfoo's `exec:` provider form: one program, invoked per case. The
  # converter maps it to a domarinn `type: exec` provider whose `command` is
  # that program.
  - "exec:../echo-provider.py"

prompts:
  - |
    You are a support agent for {{ product }}. Answer in one sentence.

    Customer: {{ question }}

defaultTest:
  vars:
    product: Aurora Notes
  assert:
    - type: icontains
      value: Aurora
    # Carried across unchanged: domarinn has the same `not-<type>` sugar,
    # documented in docs/reference/assertions.md and used by
    # examples/13-exec-provider.
    - type: not-icontains
      value: as an AI

tests:
  - vars:
      question: How long do I have to return something?
    assert:
      - type: contains
        value: "return something"
      # No equivalent, and deliberately so: the replacement is an `exec`
      # assertion (examples/14-custom-exec-assert), a program of your own over a
      # small JSON protocol rather than an embedded interpreter.
      - type: javascript
        value: output.length < 400

  - vars:
      question: Can I get a refund on a gift card?
    assert:
      - type: icontains
        value: gift card
      # No equivalent either: domarinn ships no moderation classifier. The
      # usual replacement is an `llm-rubric` that states the policy outright.
      - type: moderation

And the suite coming out — the converter's own output rather than a hand-written suite, with a header on top naming the two cosmetic things that were added:

# The AFTER half of the migration pair, and not hand-written: this is the suite
# `domarinn import promptfoo ./promptfooconfig.yaml` prints. Two things were
# added — this header, and the sequence indentation this repository's formatter
# applies to every YAML file in it. Nothing else was cleaned up, including the
# ids.
#
# Read the `# NOTE:` lines first. Each one is something the converter could not
# translate faithfully and refused to drop silently — here a `javascript`
# assertion and a `moderation` assertion, neither of which has an equivalent.
# Both left the suite; replacing them is the migration work this file does not
# do for you.
#
# Two things a converted suite always wants next, neither of which the converter
# can guess: `project:` / `suite:` names (they group and compare runs on the
# results server), and provider and case ids that mean something — `p0` and
# `case-0` are placeholders, and ids are what `--provider`, `only_providers` and
# a baseline diff refer to.
#
# CI re-runs the converter on promptfooconfig.yaml, compares the result against
# this file as YAML, and runs both — so this can only fall out of date if the
# converter itself changes.
#
# The `$schema=` line below is the converter's own: it names the published
# config schema, so an editor with the YAML language server validates and
# completes this file wherever it lives.
#
# Run:  domarinn run examples/39-import-promptfoo
# Converted from a promptfoo config by `domarinn import promptfoo`.
# yaml-language-server: $schema=https://raw.githubusercontent.com/AtvikSecurity/domarinn/main/domarinn.schema.json
# NOTE: assertion type 'javascript' was skipped (no direct equivalent; rewrite it as an `exec` assertion)
# NOTE: assertion type 'moderation' was skipped (no equivalent)
version: 1
description: Refund-policy replies from the support assistant
providers:
  - id: p0
    type: exec
    command:
      - ../echo-provider.py
prompts:
  - id: prompt-0
    template: |
      You are a support agent for {{ product }}. Answer in one sentence.

      Customer: {{ question }}
defaults:
  vars:
    product: Aurora Notes
  assert:
    - type: icontains
      value: Aurora
    - type: not-icontains
      value: as an AI
tests:
  - id: case-0
    vars:
      question: How long do I have to return something?
    assert:
      - type: contains
        value: return something
  - id: case-1
    vars:
      question: Can I get a refund on a gift card?
    assert:
      - type: icontains
        value: gift card

Read the two # NOTE: lines first, because they are the whole reason to look at a conversion before running it. One assertion was a javascript check, which has no equivalent — the replacement is an exec assertion. The other was moderation, for which domarinn ships no classifier; an llm-rubric that states the policy is the usual stand-in. Both have to be replaced by hand; neither was dropped in silence. The not-icontains house rule beside them came across unchanged.

What did convert is worth reading for what it does not decide for you. exec:../echo-provider.py became a provider block with the id p0, the two promptfoo cases became case-0 and case-1, and the suite carries no project: or suite: name at all — the converter cannot invent names, and those are the strings that group runs on a results server and that --provider, only_providers and a baseline diff refer to. Rename them before the ids reach a stored baseline.

CI re-runs the converter on that config, compares the output against the committed suite, then runs both, so the pair above can only fall out of step if the converter itself changes.

What maps cleanly

promptfoo domarinn
providers: [openai:gpt-4o] (also openai:chat:…) a type: openai block with model: gpt-4o
providers: [anthropic:messages:claude-…] a type: anthropic block
providers: [ollama:llama3.1:8b] (also ollama:chat:…) a type: openai block with base_url: http://localhost:11434/v1 and request: {auth: none} — Ollama's OpenAI-compatible endpoint, as in Local LLMs
providers: [exec:python provider.py] a type: exec block, the command split on whitespace
a provider's label label
a provider's config: temperature, max_tokens, top_p, top_k, stop, seed, … params:, verbatim (Ollama's num_predict becomes max_tokens)
a provider's config: apiBaseUrl / apiKeyEnvar base_url / api_key_env
prompts: (a plain string, inline or file://, or a {raw} / {id: file://…} mapping) prompts: with a single-string template:
tests: with vars tests: with vars
a test var whose value is a list a matrix axis — promptfoo's one-case-per-value expansion
tests: file://cases.csv (or any file:// test file) the same file:// test source, loaded as-is — see below
defaultTest defaults
contains, icontains, regex, equals, starts-with the same, kebab-case
contains-any (case-sensitive) a regex alternation of the escaped values, which matches case-sensitively as promptfoo does
icontains-any icontains-any with values:
contains-all / icontains-all one contains / icontains per value, sharing the weight
not-<type> not-<type> — the same sugar
is-json, contains-json (its value schema becomes schema:) the same
llm-rubric llm-rubric, with the grader configured in a grader: block
similar similar, needing a type: embeddings provider in the suite
cost / latency with threshold: cost / latency with max:
threshold, weight the same

What changes, and why

Provider id strings become blocks. There is no openai:gpt-4o shorthand. Every provider is a config block with an id and a type. It is more to write, and it is what lets two providers differ by base_url, params or pricing without inventing more string syntax — and what makes --provider and only_providers refer to something you named.

Templating is Jinja, not Nunjucks. {{ var }} is the same. Filters mostly are not: domarinn ships json_encode, b64encode, sha256, slugify, regex_replace, truncate and friends, plus now(), uuid() and randint(). Check any filter you rely on.

A converted prompt is always a template:, never messages:. The converter reads a promptfoo prompt that is a plain string, or a mapping carrying one in raw or a file:// id, and emits one single-string template. Anything else — an inline list of chat messages, say — is skipped with a # NOTE:. domarinn's chat-transcript form — a system turn, prior turns, then the newest one — is a messages: prompt, and it has to be written by hand. If your promptfoo suite threaded a _conversation variable to vary the history per test, the equivalent here is per-case history — each case carries its own prior turns (inline or file://), spliced at the prompt's history marker; that too is a by-hand rewrite, not something the converter emits. A file:// string is carried across as a file:// template, which domarinn does load from disk — but as a template, and relative to the suite directory, refusing anything outside it: a prompt file that lived above your promptfoo config has to move in beside the suite, and a promptfoo prompt function (file://prompt.py:fn) arrives as a path with nothing in domarinn to call it.

Test files are referenced, not converted. A tests: file://… entry is carried across as the same file:// test source, with a # NOTE:, because domarinn reads YAML, JSON, JSONL, CSV and TSV case files itself. What it does not do is translate their contents: the path now resolves against the suite directory, an assertion type in a YAML/JSON file must be one domarinn knows, and a promptfoo CSV's __expected columns arrive as ordinary vars — rewrite them as a JSON __assert column. A file://….py or .js test generator is skipped with a note; the equivalent is a generator: test source.

A negated "all" has no equivalent. not-contains-any converts (to not-regex), but not-contains-all — "at least one value is missing" — does not, and is reported.

JavaScript and Python assertions do not exist. There is no javascript: or python: assertion type. The replacement is an exec assertion — your program, any language, over a small JSON protocol. It is a process boundary rather than an embedded interpreter, which costs a spawn and buys you a checker you can run and test on its own.

Some graded assertion types have no equivalent. moderation, factuality, answer-relevance, select-best, perplexity, rouge, levenshtein and classifier are not implemented. Most are expressible as an llm-rubric with an explicit rubric, which is more work to write and considerably easier to reason about when it disagrees with you.

There is no per-assertion transform or metric. Shape the output in your provider, where you can test it. The converter reports any assertion, test, provider or top-level key it did not carry across — transform, metric, options, outputPath and the like — in a # NOTE: rather than dropping it.

Output format is a flag, not config. --format json|jsonl|junit|md|table, --out, --summary-md. There is no outputPath key.

--no-cache is rarely what you want. promptfoo's cache keyed on things that made sharing impractical, so --no-cache became habit. domarinn's key is portable by construction — it is a hash of the request and nothing else — so the cache is shareable across a team and CI. Reach for cache_salt instead of disabling caching, or --no-grader-cache when it is only the grader you want re-asked.

What you gain

Worth knowing about, because they have no promptfoo equivalent and change how a suite is used:

  • Regression gating against a stored baseline, with a defined exit-code contract — 1 for a failed assertion, 3 for a broken harness.
  • Statistics — Wilson confidence intervals, McNemar paired significance, pass@k. A pass rate with an error bar.
  • Errors that are not failures — a separate status, tally and exit code, so "the harness broke" never reads as "the model got worse".
  • A fail-closed grader — structured verdicts, with truncation an error rather than a zero.
  • A self-hostable results server with accounts, run comparison and a shared cache, in the same single binary.

See also