Skip to main content
A Dataset is a collection of examples you can run agents against. Each example is an input (a question, a task, a payload) paired with an expected output (the correct answer, or “any of these are acceptable”, or just a rubric). Datasets are how you measure agent quality objectively. Without a dataset, “did the change help” is a vibe. With one, it’s a number.

​
When to use datasets

  • Regression testing — every deploy runs the dataset, catches regressions before users do.
  • Simulation — proposed improvements are tested on the dataset, not on production.
  • Benchmarking — compare Flow versions, model choices, prompt variants.
  • Coverage — check that your Flow handles the range of inputs it’s supposed to.

​
Types of datasets

Golden set

Hand-curated correct examples. Small, high-quality.

Holdout

Set-aside test data — not used for training/improvement, only for evaluation.

Regression suite

Examples of previously-fixed failures. Guards against regressions.

Production sample

Real traces from production — check quality on real traffic.
Each is just a Dataset with a different purpose. Nora doesn’t force separate types — but tagging your datasets by role makes them easy to use appropriately.

​
How to build one

Create a dataset

From production traces, hand-authored, imports, or clusters.

Annotate examples

Add expected outputs, rubrics, or verdict labels.

​
Coverage and versioning

Coverage

Are your examples representative of production?

Versioning

Datasets change over time — track versions.

​
The dataset lifecycle

  1. Seed — small manual set of 10-30 examples.
  2. Grow — add every fixed cluster’s failing traces as new examples.
  3. Prune — retire examples that are no longer relevant (dead features, outdated cases).
  4. Split — as it grows, keep a holdout that never trains anything, only evaluates.
A healthy Flow has 100-1000 examples per major intent. Growth is organic — the Improvement flow adds examples every time it fixes a cluster.

​
Where datasets live

Under Reliable → Datasets in the workspace. Each Dataset shows:
  • Example count.
  • Last update.
  • Which Flows / Improvements have used it.
  • Coverage stats (see Coverage).