Skip to main content
Signals are the entry point. Improvements are the exit. Here’s what happens in between.

​
The path

Every step has a checkpoint where a human can decide, override, or step back.

​
Step-by-step

​
1. Signal fires

A user clicks 👎 on an answer. Or verification blocks a response. Or a tool errors repeatedly. Any source creates a Signal in the queue.

​
2. Triage

You confirm the signal is real (see Confirm or mute). Muting stops here — false alarms don’t proceed.

​
3. Clustering

Signals join clusters as they arrive (see Clusters). One signal is a data point; a cluster is a pattern worth fixing. You can promote a single signal directly, but clusters give a better base — you fix the class, not the instance.

​
4. Promotion to Improvement

Click Create Improvement on a cluster (or signal). The Improvement flow opens with:
  • The cluster context (signals, root cause hypothesis, sample failures).
  • Diagnosis (see Diagnose a failure).
  • Suggested levers to change (see Levers).

​
5. Proposals

Nora proposes 1-3 candidate fixes. Each proposal:
  • Names the change (prompt tweak, tool description clarify, retrieval preset change, new memory rule, etc.).
  • Shows the diff — exactly what would change.
  • Predicts the impact (based on similar past improvements).
You pick which proposal(s) to test. See Proposals.

​
6. Simulation

Every proposal runs against the cluster’s signals + a broader dataset. Metrics: how many previously-failing cases now pass, how many previously-passing cases now fail (regressions), cost/latency delta. See Simulation and Before/after.

​
7. Approval

The best proposal is highlighted. You (or another reviewer) approve — the change is applied to Draft. Approval is human-gated by default. Auto-approval available for low-risk changes with strong simulation results.

​
8. Deploy

Approved changes ride the normal publish flow — apply to Draft, then Publish (see Publish).

​
9. Watch

After deploy, the cluster is monitored for recurrence. Fresh signals matching the pattern re-open it (regressions). Sustained absence for 30 days → cluster resolves.

​
Time-to-fix

Typical:
  • Simple prompt fix: minutes from signal to deploy.
  • New rule from feedback: hours (test → approve → publish).
  • Retrieval preset change: hours to days (need broader simulation).
  • Complex fix requiring new data or graph edits: days.
The goal is compounding: each fix informs the next, cycles get shorter, fewer signals fire.

​
What if the fix doesn’t work?

Two safety valves:
  • Simulation caught it — the proposal never deploys. Try a different lever.
  • Deploy caught it — the cluster re-opens as a regression. Roll back the Improvement (single click) and try again.
Rollback is instant because the change is versioned (see Rollback).

​
Where to go next

Datasets

Curate the test cases that keep fixes honest.

Simulation

Run changes against data before they hit production.

Optimization

The Improvement flow, in depth.