Skip to content

Evaluate structured output

The problem. An agent must emit a structured object — a verdict per catalogue item, a set of extracted fields, a report with required sections. "Did it produce valid JSON" and "did it produce the right JSON" are different questions, and only the first one is easy.

The shape. Stack three layers, cheapest first: parseability, then shape, then semantics.

1. Parseability and shape, for free

# yaml-language-server: $schema=../../domarinn.schema.json
#
# Grading structured output.
#
# When a system is asked for JSON, there are two separate questions, and it pays
# to assert them separately:
#
#   1. Is it parseable at all?        -> `is-json`
#   2. Does it have the right shape?  -> `contains-json` with a `schema`
#
# A model that returns prose around its JSON fails (1) but can still pass (2),
# because `contains-json` looks for an embedded object. That difference is the
# whole reason both exist.
#
# Run:  domarinn run examples/04-json-output
version: 1
project: examples
suite: json-output

providers:
  - id: extractor
    type: exec
    command: ["python3", "../echo-provider.py"]
    cache_salt: "dev"

tests:
  # A clean response: the entire output is one JSON object.
  - id: extract/clean
    vars:
      user_input: '{"customer":"Ada Lovelace","order_id":1042,"refund_requested":true}'
    assert:
      - type: is-json
      # Schema-checked. The schema is a normal JSON Schema; `required` is what
      # turns "it parsed" into "it has the fields we depend on".
      - type: contains-json
        schema:
          type: object
          required: ["customer", "order_id", "refund_requested"]
          properties:
            customer: { type: string }
            order_id: { type: integer }
            refund_requested: { type: boolean }

  # A chatty response: the object is real but wrapped in prose. This is the
  # common failure mode of an under-instructed model, and the reason to reach
  # for `contains-json` rather than `is-json` when you cannot control that.
  - id: extract/wrapped-in-prose
    vars:
      user_input: |
        Sure! Here is the extracted record:

        {"customer":"Grace Hopper","order_id":77,"refund_requested":false}

        Let me know if you need anything else.
    assert:
      # The whole output is NOT JSON — assert that, so the example documents the
      # distinction rather than hiding it.
      - type: not-is-json
      - type: contains-json
        schema:
          type: object
          required: ["customer", "order_id"]

  # A schema is optional: bare `contains-json` only asks "is there JSON in here".
  - id: extract/any-json
    vars:
      user_input: "result: [1, 2, 3]"
    assert:
      - type: contains-json

is-json asks whether the whole output parses. contains-json looks for an object embedded anywhere in it — which is what you want when the model wraps its answer in prose you cannot fully suppress.

Give contains-json a schema, and give the schema a required list. Without one, a model that returns {} passes:

- type: contains-json
  schema:
    type: object
    required: ["verdicts", "summary"]
    properties:
      verdicts: { type: array }
      summary: { type: string }

That is a real gate and it costs nothing. Most structured-output bugs die here.

2. Semantics, with a rubric over specific fields

Schema validation cannot tell you that an item was marked TESTED when the evidence says otherwise. A rubric can — provided you point it at named fields rather than at "the JSON":

- type: contains-json
  schema: { type: object, required: ["verdicts"] }
- type: llm-rubric
  value: |
    The JSON `verdicts` array MUST contain an entry for every id in the input
    catalogue, and no others.

    An item with no recorded activity MUST be marked `MISSED`, never `TESTED`.

    Score 0 if any item with zero recorded activity is marked `TESTED`, or if
    an id appears that was not in the catalogue.

    Do NOT penalise the wording of `summary`, the ordering of `verdicts`, or
    any additional field the object carries.

Note the ordering: contains-json runs first and short-circuits, so a model that emitted nothing parseable never reaches the grader.

3. Include the negative case

The case most suites are missing is the one where the correct answer is not the confident one — an item that must be marked "not tested" despite a misleading upstream signal. Those are the cases that catch a model optimising for looking complete.

4. Pass JSON inputs as string vars

When a case's input is itself structured, pass it as a JSON string var:

vars:
  catalog: '[{"id":"a","applies_when":"HTTP endpoints present"}]'
  activity: '{"a":{"row_count":0}}'

Watch for template syntax in the payload

A JSON fixture containing {{ will be rendered before it reaches the system. If the payload must arrive byte-for-byte, tag it !raw — or load it from a file with {$file: "…", raw: true}, as example 07 does.

See also