Skip to main content
If you evaluate a fix on the same data you used to design it, your quality number is inflated. That’s the “overfitting” problem in one sentence. A holdout is a subset of your data that the Improvement flow is not allowed to touch. Simulation runs against it. Any signal there passes the honesty bar.

​
Creating a holdout

From any dataset, click Split → Create holdout. Options:
  • Random N% — take a random 20% (default).
  • By tag — hold out everything with certain tags (e.g., hold out hard examples).
  • By date range — hold out examples added after a cutoff (evaluates future-proofness).
  • Manual pick — check individual examples.
The holdout becomes a separate dataset. It’s cross-referenced back to the source dataset.

​
Holdout enforcement

Once marked, the Improvement flow’s proposal generation cannot read the holdout examples. Diagnosis, root-cause analysis, and lever selection use only the non-holdout portion. Simulation, however, runs proposals against both portions:
  • Train portion — where the improvement was designed. Shows the “in-distribution” improvement.
  • Holdout portion — the honest signal. Shows generalization.
Big gap between the two = overfitting.

​
The overfitting warning

If a proposal massively improves train performance but doesn’t budge holdout performance, Nora warns:
This proposal moved train score +12% but holdout +0.5%. Likely overfits to the failing examples used in the improvement flow. Consider broadening the improvement pattern.
Warnings don’t block — you can still ship — but they’re a strong hint.

​
Rotation

Holdouts should rotate. If the same examples are always the holdout, they become stale — the Improvement flow can’t cover the patterns they test. Recommended:
  • Rotate the holdout monthly (or per quarter).
  • Add new holdout examples as new failures are fixed.
  • Retire holdout examples when they become trivial (always passing).
Rotation is manual (you decide when). Nora shows a “holdout age” indicator when it’s been > 60 days without changes.

​
Holdout ≠ regression suite

Both are test-only. Difference:
  • Holdout — general representative sample of your traffic. Broad coverage.
  • Regression suite — specific known failures you want to guard against forever. Narrow, defensive.
Keep them separate. A change might pass one and fail the other, and the interpretation is different.

​
Enterprise: production holdout

For high-stakes Flows, you can configure Nora to sample a percentage of production traffic as an ongoing holdout. Runs land in the dataset automatically, get human-annotated (or judge-annotated) offline, and expand the holdout organically. Turn on in Flow settings → Sampling → Production holdout.

​
Splits and simulation

Simulation reports separately for each portion:
  • Train: 47/50 pass.
  • Holdout: 18/20 pass.
  • Overall: 65/70 pass.
The overall number is convenient. The two portions individually are what matters — the delta tells you where you stand on generalization.