Simulation overview

Simulation is how you know if a change works — without pointing production traffic at it first. Give it a Dataset and a proposed change, and it runs the change against every example, scores each, and shows you the delta.

​

Two big use cases

1. Guarding deploys — every Flow change runs against a regression dataset. If simulation fails, the deploy is blocked.
2. Comparing Improvements — two candidate fixes, one dataset, side-by-side. Pick the winner.

​

What simulation runs

Any of:

• A Flow version — the whole workflow, end-to-end.
• A Draft state — your unpublished changes.
• An Improvement candidate — a proposal from the Improvement flow.
• An arbitrary variant — e.g., “same Flow but with model X”.

Everything runs against the chosen Dataset. Results are stored, comparable, and versioned.

​

What you get

​

Speed and cost

Simulation runs one dataset example per Agent invocation, in parallel. Speed:

• 50 examples on a fast model: ~30-60 seconds.
• 500 examples: 5-10 minutes.
• 5000 examples: 30-60 minutes.

Cost is real — every simulation run costs what the Agent would cost in production. Budget accordingly, and prefer smaller dev datasets during iteration and larger production datasets for final validation. The simulation view shows total cost upfront so you can decide whether to proceed.

​

When simulation isn’t enough

Simulation runs against known examples with known expected outputs. It can’t:

• Predict how new traffic (unseen intents) will land.
• Catch quality issues that require deep human judgment.
• Model interactions between multiple Agents talking to each other.

For those, use the smaller-population canary mechanism (see Versions): deploy to a small % of real traffic first, watch signals, then roll out to 100%.