The comparison view
For every dataset example:- Left: baseline output.
- Center: target output.
- Right: expected output.
- Verdict:
improvement/regression/same/both failed.
- Improvements first — where the target beat the baseline.
- Regressions first — where the target broke something the baseline handled.
- Diff score — biggest change in either direction.
Summary metrics
At the top:- Pass rate delta — target’s pass rate minus baseline’s.
- Improvements — count of “target beat baseline”.
- Regressions — count of “target worse than baseline”.
- Wash — count of “same verdict”.
- Cost delta — is the target more or less expensive?
- Latency delta — is the target slower or faster?
The regression drill-down
Regressions are the deal-breakers. Click View regressions to see just those examples. For each regression:- What the baseline said.
- What the target now says (worse).
- Why it changed — Nora’s automated diff of the trace shows what step diverged.
- New prompt is more terse and drops a required disclaimer.
- New retrieval preset drops a source the baseline relied on.
- Model swap changes an edge behavior.
The improvement drill-down
Similar view for improvements. Understand why the target won so you can generalize:- Was it a specific pattern that the change targeted?
- Or did it broadly lift many examples?
Statistical significance
For datasets with < 50 examples, Nora shows a “high variance — proceed with caution” flag. Small samples make it easy to see noise as signal. For > 50 examples, a proper significance test is shown: p-value for pass-rate delta, confidence interval. Only trust deltas outside the noise floor.Attributed changes
The improvement flow tags every simulation with the specific proposal that generated the target. Before/after inherits that tag, so you can trace:“Improvement #42 (retrieval preset change): +8% pass rate, +1 regression, +0.02s latency”Great for building an audit trail of what changed and why.