> ## Documentation Index
> Fetch the complete documentation index at: https://docs.platform.nora.my/llms.txt
> Use this file to discover all available pages before exploring further.

# Annotate examples

> Add expected outputs, rubrics, or verdicts so simulation can grade objectively.

An example without a target is just a question. To measure quality, each example needs some way to grade the Agent's answer. Nora supports several annotation styles — pick the one that fits your domain.

## Annotation types

### Exact match

Grade passes iff the Agent's answer exactly equals the expected output. Only useful for structured outputs (JSON, enum, IDs).

### Fuzzy match

Grade passes iff similarity above threshold (edit distance, semantic similarity, or a mix). Good for short factual answers where wording flexibility is OK.

### Rubric

A checklist the Agent's answer is graded against:

```
- Mentions the 30-day refund policy: yes/no
- Includes a link to the refund form: yes/no
- Uses a friendly, apologetic tone: yes/no
- Does NOT commit to a refund without approval: must-not-fail
```

Rubrics are graded by a judge model. Cheap and reliable for text answers.

### Reference answer + judge

Provide a "good" example answer. The judge model compares the Agent's answer to the reference and scores it on faithfulness, completeness, and correctness.

Good middle-ground when exact match is too strict and a full rubric is too much work.

### Custom judge (code)

Write a small JS/Python function that takes the Agent's answer and returns a score + verdict. Full control — useful when grading involves domain logic (e.g., "answer's SQL must produce the same rows as expected").

## When to use which

* **Structured output** (function-calling, JSON) — Exact or Fuzzy match on the structured fields.
* **Short factual** ("what's the return policy") — Fuzzy match or reference + judge.
* **Long text** (explanations, summaries) — Rubric.
* **Domain-specific** (SQL, code, calculations) — Custom judge.

You can mix within one dataset — different examples with different annotation types.

## Multi-annotator agreement

For datasets where annotation is subjective, add multiple annotators. The dataset viewer shows agreement stats — examples with high disagreement are worth revisiting.

Nora doesn't force consensus. It shows all annotations and lets simulation pick which annotator's verdict to use.

## Adding annotations

Per-example annotation editor. Or bulk-annotate:

* Filter to a subset (by tag, by score, by "unannotated").
* Apply the same rubric or reference to all.
* Review individually if needed.

## Annotation from feedback

If your Signals include text corrections, the corrections auto-populate the expected output when saving to a dataset. Saves the annotation step for anything with existing feedback.

## Reviewing annotations

Datasets in review show:

* **Unannotated** — examples without a target. Won't be scored in simulation.
* **Needs review** — auto-populated from feedback, human should verify.
* **Ready** — human-annotated and confirmed.

Simulation only scores `ready` examples by default. Toggle to include `needs review` if you want a broader test.

## Editing later

Annotations are versioned. Editing one creates a new version; simulation results are pinned to the annotation version they used. So old simulation results stay valid even as annotations evolve.