Evaluation Guide¶

When to use this: You want to measure whether something works — a feature you just shipped, a skill, an agent, or the project as a whole — and get a verdict you can trust rather than a vibe. This guide covers the three evaluation instruments little-loops ships, generating an eval harness from an issue, writing criteria that don't rubber-stamp, and reading the resulting signal. For per-loop context variables and FSM state tables, see the Built-in Loops Reference. For FSM fundamentals (states, evaluators, routing), start with the Loops Guide.

What Evaluation Owns¶

little-loops has four guides in this neighborhood and they are easy to confuse. The line between them is simple: evaluation measures; everything else changes.

If you want to…	Read
Measure whether a feature, skill, or agent works, and how well	This guide
Run a skill over many work items with quality gates, so the work gets done	Automatic Harnessing Guide
Automatically rewrite a prompt file until it scores better on a labeled corpus	Prompt Optimization Guide
Automatically rewrite a skill, command, agent, or loop YAML against a benchmark	Harness Optimization Guide
Prove an external API behaves the way you assume, before writing code against it	Learning Tests Guide

The relationship is one-directional and worth stating plainly: every optimization loop needs an evaluator, and this guide is where the evaluator comes from. apo-textgrad hill-climbs a prompt against a score; harness-optimize keeps an edit only when the score improves. Neither can do anything useful until you have a measurement that means something. Build the instrument first.

The corollary is the failure mode to watch for: an optimizer pointed at a weak evaluator does not fail loudly — it converges confidently on the wrong thing.

The Three Instruments¶

Instrument	Shape	Grades on	Use when
`ll-harness <runner>`	One shot, one command	Exit code and/or an LLM criterion	You want a single yes/no on one skill, command, MCP tool, or prompt
FSM eval harness	A loop YAML, retries on failure	`llm_structured` verdict on a user-perspective run	You want to evaluate a feature the way a user experiences it
`ll-harness dsl`	A directory of task files, batch	Pass rate with a 95% Wilson interval	You want a rate across many small tasks, comparable across models

`ll-harness` — the runner evaluation CLI¶

Four runners, all with the same evaluation flags (scripts/little_loops/cli/harness.py):

Runner	Invokes	Subject
`ll-harness skill <name>`	A little-loops skill via the active host CLI	stochastic
`ll-harness cmd "<shell>"`	A shell command	deterministic
`ll-harness mcp <server>:<tool>`	An MCP tool call	deterministic
`ll-harness prompt "<text>"`	A raw prompt to the host model	stochastic

Stochastic vs. deterministic (ENH-3415): skill/prompt drive an LLM host CLI, so the same target can behave differently run to run; cmd/mcp are a subprocess call / arbitrary tool call with no LLM in the loop, so one run is as good as another. A one-shot verdict on a stochastic subject can't distinguish a real capability from a lucky sample, so ll-harness grades the stochastic runners over DEFAULT_STOCHASTIC_SAMPLES (3) runs by default instead of one — see N-sample redundancy for stochastic subjects below for the pass-rate verdict surface, --samples override, and the preflight-vs-promotion boundary. cmd/mcp stay one-shot by default.

Exit codes are the contract: 0 PASS, 1 FAIL, 2 internal error or timeout, 3 ABSTAIN or (ENH-3415) INCONCLUSIVE.

Two independent gates, both optional:

--exit-code N — mechanical. The runner's exit code must equal N.
--semantic "<criterion>" — an llm_structured judgment over the runner's stdout. Only a yes verdict passes.

--retry-of ID (ENH-3407) marks a run as an infra retry of attempt ID rather than a fresh sample: it's gated before the run (refused, exit 1, if the prior attempt isn't a genuine timeout for this same cell), and on success supersedes the prior row instead of counting as an independent repetition. --retry-of without an explicit --samples pins the effective sample count to 1 even on a stochastic runner (a retry supersedes exactly one attempt); an explicit --samples > 1 alongside --retry-of is refused, exit 1 (ENH-3415). See CLI Reference → ll-harness for the full refusal-rule list.

Pass both or you are not measuring anything. With neither flag, passed initializes to True and no check ever flips it (_grade(), harness.py:~865) — the command reports PASS unconditionally (on every sample, if graded over N>1). See Gotchas.

The FSM eval harness¶

A generated loop YAML that runs the feature as a user would, then judges the experience. Generated by /ll:create-eval-from-issues (see below), or hand-written. The canonical hand-written example in this repo is loop-specialist-eval.yaml: an execute state that drives the loop-specialist agent against a seeded broken-loop fixture, and a check_skill state whose llm_structured prompt enumerates three must-all-be-true conditions.

`ll-harness dsl` — the batch task set¶

Runs every *.yaml task file in a directory as a prompt, grades each task against its own expected: mapping (a structured answer contract appended to the prompt), and reports a pass rate with a Wilson 95% confidence interval:

DSL pass-rate: 12/14  [0.57, 0.94] (95% CI)
  graded 14 of 16 tasks — 2 ungradable (no `expected:` and no --semantic)
  failed: task-03.yaml (on_yes: expected 'done' got 'finish')
          task-09.yaml (unparseable answer — no JSON object in response)
  admissions: 1 (timeout×1)

The admissions: line (ENH-3408) appears only when the invocation admitted at least one --retry-of retry; it is omitted otherwise.

The interval is the point. A 17/20 run and a 170/200 run have the same ratio and very different meaning; the CI is what stops you from acting on the first one. --model lets you run the same task set across models for comparison.

Task files use the Option B schema (prompt, blanks, expected, source_dsl, task_type). Generate them with /ll:create-eval-from-issues --dsl <loop-yaml-or-issue>. A task that declares expected is graded deterministically, key-by-key, against a fenced json answer object the model is asked to produce — no extra LLM call. A task that declares no expected needs --semantic to be graded at all; without either, it is ungraded: excluded from the pass-rate denominator, reported on its own line, and the run exits nonzero rather than silently reporting a perfect score. See exit codes below.

Exit codes: 0 all graded tasks passed; 1 a graded task failed, or ≥1 task was ungraded; 2 every task was ungraded, or ≥1 task hit a per-task infra error (host timeout or crash); 3 ≥1 task abstained (the --semantic judge could not decide) and nothing failed or was ungraded.

Of that infra-error bucket, only a genuine timeout is retriable today: --retry-of's admissibility gate (ENH-3407) reads the persisted timed_out column, and a host crash or other runner error has no persisted signal to gate on yet (a deliberate fail-closed scope decision, not an oversight). --retry-of on dsl additionally requires path to name a single task file, not a directory.

Quick Start¶

Measure one skill, right now:

ll-harness skill check-code --exit-code 0 --semantic "reports lint and type results without crashing"

Measure a shipped issue as a user would experience it:

/ll:create-eval-from-issues FEAT-919       # writes .loops/eval-harness-feat-919.yaml
ll-loop validate eval-harness-feat-919     # fix the two warnings it will emit
ll-loop run eval-harness-feat-919

Measure the project's overall health:

ll-loop run evaluation-quality             # writes .loops/quality-report-YYYY-MM-DD.md

Generating an Eval Harness From an Issue¶

/ll:create-eval-from-issues reads an issue's Expected Behavior, Use Case, and Acceptance Criteria sections, synthesizes an execute prompt and evaluation criteria from them, and fills those into a scaffold produced by ll-loop scaffold-eval. The scaffold is schema-validated in-process before the skill sees it.

You can call the mechanical layer directly to see what you are getting:

ll-loop scaffold-eval --issues FEAT-919 --stdout

Variant A — one issue¶

name: eval-harness-feat-919
initial: execute
states:
  execute:
    action: <EXECUTE_PROMPT>
    action_type: prompt
    next: check_skill
    timeout: 300
  check_skill:
    action: Evaluate the feature from a real user's perspective.
    action_type: prompt
    evaluate:
      type: llm_structured
      prompt: |-
        <EVALUATION_CRITERIA_PROMPT>
    on_yes: done
    on_no: execute
    on_partial: execute
    timeout: 180
  done:
    terminal: true
max_steps: 5
timeout: 1800
category: harness

Note on_no: execute — a failed evaluation sends the harness back to re-run the feature, not to a failure terminal. With max_steps: 5 and two counted states per cycle, that is roughly two attempts before the step budget ends the run. A harness that stops at max_steps has not passed; check the terminal state, not just the absence of an error.

Variant B — two or more issues¶

Two or more issue IDs produce a discover → execute → check_skill → advance → discover work-list loop, with processed IDs appended to ${context.run_dir}/processed.txt. Its max_steps is 50 (the schema default, so it is omitted from the emitted YAML). At roughly four counted states per item, budget about ten issues per run before the step ceiling becomes the binding constraint.

Proof-First Gates¶

When learning_tests.enabled is true in .ll/ll-config.json and an issue declares learning_tests_required: targets, the scaffold splices one check_proof_<slug> state per target between execute and check_skill, chained on on_yes (scripts/little_loops/cli/loop/scaffold_eval.py:45-101). The eval will not reach its quality judgment until each declared external-API assumption is proven. See the Learning Tests Guide.

Two warnings you must resolve¶

ll-loop scaffold-eval emits a valid loop that is not yet a good loop. Both warnings are expected, and both are yours to fix:

Warning	Fix
`Loop declares no 'scope:'`	Add `scope: ["path/"]` naming what the eval writes, or `scope: ["."]` as an explicit repo-wide opt-in. Without it, the run takes a repo-root lock that false-conflicts with every other concurrent loop.
`prompt does not include evidence-contract keywords` (MR-8)	Add a verbatim-quote requirement to the criteria prompt — see the next section.

Variant B adds two artifact_versioning warnings for processed.txt. That file is a work-list cursor, not an output artifact; artifact_versioning_ok: true is the honest suppression.

Writing Criteria That Don't Rubber-Stamp¶

An llm_structured evaluator with a vague prompt is a rubber stamp with extra steps. Three rules, all enforced or recommended somewhere in the codebase:

1. The execute prompt is a user instruction, not a test assertion. Write what a person does, not what should be true afterward. "Use the sprint dashboard as a real user would — open it, filter to P1 issues, and try to reassign one. Note any errors, delays, or surprises." The evaluation comes later; mixing them lets the executing agent optimize for the grade instead of doing the task.

2. Criteria are numbered, observable conditions with an explicit failure signal. Each condition must be checkable from the interaction output alone. State what NO looks like, drawn from any "should not" conditions in the issue — an evaluator given only success conditions has nothing to say no with.

3. Demand evidence. End every criteria prompt with a verbatim-quote clause. This is not style: verdicts issued without cited evidence default to optimism, and ll-loop validate enforces the keyword check as MR-8.

Did the sprint dashboard (FEAT-919) deliver a satisfying user experience? Assess all of:
(1) The dashboard rendered within 3 seconds of navigation.
(2) Filtering to P1 returned only P1 issues, with the count matching the header.
(3) Reassignment persisted after a page refresh.
Answer YES only if all conditions were clearly met. Answer NO and specify which
condition(s) failed and what was observed — in particular if any action produced a
stack trace or a silent no-op.
Provide a VERBATIM quote from the observed output that supports your verdict. Do not
assert a verdict without evidence.

The MR-8 lint scans FSM YAML evaluate.prompt text only — it will not catch a missing evidence contract in a skill markdown body.

Verification vs. Evaluation¶

Two skills generate harnesses from an issue and they answer different questions:

Skill	Question	Output
`/ll:create-eval-from-issues`	Is this a good experience?	`.loops/eval-harness-<slug>.yaml` — placeholders to fill
`/ll:verify-issue-loop`	Does it meet each acceptance criterion?	`.loops/verify-<ID>-<slug>.yaml` — immediately runnable
`/ll:verify-issue-loop --mode adversarial`	Can I break it?	`.loops/adversarial-<ID>-<slug>.yaml`

--mode criteria walks acceptance criteria in order and fails fast on the first one that fails. --mode adversarial runs three probe classes — boundary values, malformed/hostile inputs, failure modes — and fails when fewer than three probe classes are genuinely attempted, even if every attempted probe passed. An adversarial run that finds nothing because it barely tried is a failed run, which is the correct reading.

Rough sequencing: verify (criteria) before you close the issue, evaluate before you claim the feature is good, adversarial before you depend on it.

Reading the Signal¶

A single run¶

Field	Meaning
`Exit`	The runner's exit code, and the expectation if `--exit-code` was passed
`Semantic`	The `llm_structured` verdict, or `[not checked]`
`Result`	`PASS` only if every requested check passed

[not checked] in either row means that gate contributed nothing. Two [not checked] rows and a PASS is a null result.

--output json adds stdout/stderr, a read-only prepatch_evidence block when --issue-id is passed and a bundle exists, and (ENH-3223) history_pass_rate / history_abstention_rate — the run's target's historical rates over the last 30 days, once at least 3 prior authoritative runs exist (ENH-3408: a --retry-of chain counts once, as its surviving attempt, not once per attempt). These are target-scoped: pooled across every --semantic string ever run against that target, not attributable to the specific criterion just evaluated (semantic_prompt, the column that would allow that, is unwritten). When any admitted retry belongs to the counted population, a history_admissions map ({reason: count}) is folded in too. See CLI Reference → ll-harness for the full field table.

N-sample redundancy for stochastic subjects (ENH-3415)¶

A subject is stochastic here if an LLM host CLI drives it — the skill and prompt runners — so its behavior varies run to run even with an unchanged target. cmd and mcp are deterministic (a subprocess call / arbitrary tool call, no LLM in the loop) and stay one-shot. A one-shot verdict on a stochastic subject can't tell a real capability from a lucky sample, so ll-harness grades skill/prompt over DEFAULT_STOCHASTIC_SAMPLES (3) runs by default and reports a pass-rate instead of a bare pass/fail; --samples N overrides in either direction on any runner. The sample loop never stops early on a pass — all N samples always run, so a flaky subject can't get lucky on run 1 and skip the rest.

The verdict bands on the raw tally, not a confidence-interval threshold: PASS requires every requested sample to be graded and pass (a pass alongside any abstention or timeout is INCONCLUSIVE, exit 3, not a softened pass); FAIL requires every graded sample to fail; anything else — including all-abstained (ABSTAIN, exit 3) or all-errored (ERROR, exit 2) — falls to INCONCLUSIVE/exit 3. Operating characteristic to keep in mind when choosing N: a runner with a true per-run pass probability p still slips through a PASS at pⁿ (0.7³ ≈ 0.34, 0.7⁵ ≈ 0.17) — raise --samples for tighter certification, at the cost of --timeout-multiplied wall time and host-CLI/judge-call cost.

Preflight vs. promotion is a load-bearing boundary, not a detail. A capability preflight probe (e.g. checking a host CLI can do X at all) may stop on the first confirming sample — that's a budget decision, not a correctness one. A verdict that gates a promotion decision (closing an issue, merging a change, selecting a candidate) may not stop early: this is exactly the n-sample-redundancy guard evolutionary-search harnesses treat as mandatory structure, not optional rigor, and it is why ll-harness's sample loop always runs to N. Never let whichever code path happens to run last silently decide this for you.

Each sample is persisted as its own attempt_kind='repetition' row (same write path as a one-shot run), so this invocation's own sample_pass_rate and the cross-invocation history_pass_rate below are the same statistic over different windows. dsl already resamples across its own task set and refuses an explicit --samples > 1 rather than resampling each task on top of that.

Measuring a delta (ENH-3435)¶

A "this change improved things" verdict is a claim against a before-number, and that number must be measured, not remembered. ll-harness is a one-shot CLI with no proposal phase of its own, so the baseline arm is a separate invocation the loop sequences around the mutation — measure on the clean tree, mutate, then compare:

# 1. before the first proposal: measure the unmutated incumbent (n = the candidate's n)
ll-harness skill check-code --measure-baseline --semantic "<criterion>" --samples 3

# ...the loop mutates the skill file (uncommitted is fine — the baseline is keyed
#    on the incumbent's *content*, resolved from the HEAD blob, not on HEAD itself)...

# 2. after the mutation: compare against the measured baseline
ll-harness skill check-code --compare-baseline --semantic "<criterion>" --samples 3

Rules that make the delta honest:

Baselines are keyed on content + input + conditions, never on head_sha. The match key is (runner, target, input_hash, target_content_hash, conditions_fp); a conditions fingerprint (sha256 over every condition-relevant argument — judge model, subject model, timeout, host CLI, --exit-code, trace flags, --hosts) means a baseline measured under different grading cannot silently serve. head_sha is provenance: the report flags "measured at a different HEAD" rather than refusing, because the meta-loop flow commits accepted candidates and a HEAD-keyed lookup would re-pay the measurement it just made.
The compare gate runs before the candidate arm. No baseline / partial baseline (< n authoritative rows) / condition or input mismatch / effective n = 1 / unmutated subject all refuse with exit 2 and zero subject invocations. Exit 2 (not 1) so a loop routing on the exit code can tell "no baseline measured" from "candidate genuinely failed".
The store is bidirectional. The candidate rows a --compare-baseline run writes become the baseline for that content once the candidate is accepted — the next --measure-baseline reuses them with zero new runs. A baseline doubles first-run spend and is amortized from there.
A delta is a rate difference with both Wilson intervals, never a banded verdict. The run's exit code still comes from the candidate tally's banding alone; at the default n=3 the intervals will usually overlap and the report shows that rather than hiding it.
prompt/cmd/mcp need --baseline-of <attempt-id> — their target is the content, so there is no HEAD blob to resolve an incumbent from.

Known limitation — ambient subject model on the skill runner: only prompt takes --model; a skill baseline's subject_model is NULL, so a host default-model change between the two arms (they can be days apart) passes every condition check. The delta provenance carries an explicit subject model not pinned note rather than a false claim of parity. Pin the host model (LL_HOST_CLI-level or config) when a loop sequences measure/compare across days.

No loop in this repo invokes this sequence yet (harness-optimize.yaml scores through ${context.scorer} with its own baseline_score state); the two-invocation sequence above is the documented contract a follow-up can wire as that scorer. The JSON baseline.delta field is the loop-consumable output.

Across runs¶

Every ll-harness invocation writes a row to the harness_events table in .ll/history.db (schema v31), including semantic_verdict, duration_ms, head_sha, branch, and the content hash of the evaluated target. The content pin is what lets you compare a skill's pass rate across commits without re-diffing by hand.

ll-harness itself is now a consumer (ENH-3223, see above) — its per-run report folds in the target's historical rate. For anything beyond a single target's own report, query the table through the Python API:

from little_loops.history_reader import recent_harness_events, harness_eval_pass_rate

recent_harness_events(runner="skill", target="check-code", limit=20)
harness_eval_pass_rate("check-code", since="2026-08-01")   # None if nothing scored

harness_eval_pass_rate counts every authoritative (non-superseded) row with a non-NULL semantic_passed — ll-harness sets this on every non-abstained run, including ones gated solely on --exit-code with no --semantic at all, so its denominator is all non-abstained authoritative runs for the target, not only the semantically-judged ones. A --retry-of chain contributes exactly one row (ENH-3408): the surviving attempt, not one row per attempt, and its superseded predecessor never counts as a failure. A target with zero scored runs returns None, not 1.0.

Loops That Consume an Eval¶

Four built-in loops sit in the evaluation category, and only one of them evaluates a feature. Read this table before picking one by name.

Loop	What it actually evaluates	Terminals
`eval-driven-development`	Your project, via a harness you name — implement, run harness, capture failures as issues, refine, repeat	`done`, `failed`
`agent-eval-improve`	An AI agent against a task suite, refining its config until quality converges	`done`, `failed`
`evaluation-quality`	Project health — issue quality, code health, backlog health. Not a feature eval.	`done`, `failed`, `summarize_max_steps`
`outer-loop-eval`	Another loop — its YAML structure and one execution of it	`done`, plus four failure terminals

`eval-driven-development`¶

The measure-fix-remeasure cycle: ll-auto implements P1/P2 issues → commit → run your named harness → capture failures as new issues → refine → back to implement.

ll-loop run eval-driven-development --context harness_name=eval-harness-feat-919

harness_name is required and has no usable default. Left empty, run_harness executes ll-loop run --no-lock with no argument, fails, and routes to diagnose — whose prompt explicitly checks for exactly that. The --no-lock flag is load-bearing: the parent already holds the project lock, so a locking child would deadlock on acquire().

readiness_threshold / outcome_threshold are deliberately not declared in the loop's context: block — declaring them would shadow commands.confidence_gate.* in .ll/ll-config.json (BUG-2767). Override per-run with --context readiness_threshold=NN.

`agent-eval-improve`¶

ll-loop run agent-eval-improve \
  --context agent_config=.loops/my-agent.yaml \
  --context task_suite=evals/tasks.json \
  --context quality_target=0.70

Variable	Default
`agent_config`	`agent.yaml`
`task_suite`	`evals/`
`quality_target`	`0.85`

The convergence gate compares the last line of the scoring state's output against quality_target with a tolerance of 0.03, routing target → done, progress → refine_config, stall → done.

A done terminal does not mean the target was reached. Two other paths land there: analyze_failures answering NO (failures look random, no actionable pattern) routes straight to done, and a stall verdict from the convergence gate does too. Read the captured score, not the terminal name.

For deterministic scoring — unit tests, exact-match — install the loop (ll-loop install agent-eval-improve) and set use_benchmark: true with a benchmark_scorer command to swap the LLM-graded score_results step for the lib/benchmark.yaml numeric path.

`evaluation-quality`¶

Despite the name, this is a periodic project checkup, not a feature evaluator. It samples ll-issues refine-status, runs your configured test_cmd plus lint_cmd, scores three dimensions (issue quality 40%, code health 40%, backlog health 20%, re-weighted across the measured dimensions when a command is not configured), routes the single worst dimension to a targeted remediation loop, and writes .loops/quality-report-YYYY-MM-DD.md with a trend comparison against prior reports.

Variable	Default
`issue_quality_threshold`	`70`
`code_health_threshold`	`80`
`backlog_health_threshold`	`75`

It remediates one dimension per run — the one furthest below its threshold — then reports and exits. It is designed to be run repeatedly (before sprint planning, or weekly), not once.

`outer-loop-eval`¶

Evaluates a loop: loads the target's YAML, analyzes it via /ll:debug-loop-run, runs it as a sub-loop, re-analyzes the execution, then generates an improvement report and refuses to call it done if every section says "None identified."

ll-loop run outer-loop-eval '{"input": "issue-refinement", "loop_input": "..."}'

run_sub_loop inherits a 1800s default timeout. For a slow target loop, pass --timeout to the outer invocation.

Gotchas¶

ll-harness with no --exit-code and no --semantic always passes. passed starts True and only a requested check can flip it (_grade(), harness.py:~865). ll-harness skill check-code alone reports PASS whatever the skill did — and, since skill defaults to 3 samples (ENH-3415), it reports PASS 3/3 whatever the skill did on all 3 runs.
A generated eval harness declares no scope:. It will validate, run, and take a repo-root lock that false-conflicts with every other concurrent loop. Add scope: before the first run.
Variant A's on_no re-runs the feature, it does not fail. max_steps: 5 gives about two attempts, then the run ends on the step budget. Distinguish "reached done" from "ran out of steps."
Variant B tops out near ten issues. max_steps: 50, ~4 counted states per item, plus max_retries: 3 on execute. Split larger sets across runs.
evaluation-quality and outer-loop-eval are not feature evaluators, despite living in the evaluation category. The first grades your project, the second grades a loop.
agent-eval-improve reaches done on three different conditions, only one of which is convergence at quality_target.
Nothing reads harness_events from the CLI. Use little_loops.history_reader, and remember harness_eval_pass_rate ignores exit-code-only runs.
MR-8's evidence-contract lint only inspects FSM YAML evaluate.prompt text. A skill body that asks for a verdict without evidence is unchecked.