Skip to main content
Simulations are cheap to start and fast to iterate. The workflow is intentionally lightweight.

​
From the app

  1. Reliable → Simulation → New run.
  2. Pick a target:
    • Current Draft.
    • A specific published version.
    • An Improvement candidate.
  3. Pick a dataset (or dataset subset via tag filter).
  4. Optionally: pick a baseline to compare against (defaults to current published).
  5. Preview — shows the number of examples, estimated cost, estimated time.
  6. Run.
Progress streams in real time. You can leave the page and come back — completion notification lands in your workspace notification bar.

​
From the CLI

Same options: --target accepts draft, published, version:<id>, or improvement:<id>. --baseline accepts the same or none. Add --tags customer-facing,hard to filter the dataset. Add --parallelism 20 to control concurrency (default 10).

​
From CI

Simulations block-or-pass a deploy pipeline. Standard pattern:
The --dry-run --gate-simulation combo runs the target Draft against the dataset. If pass rate is under the min, the command fails and CI stops. Common: run gate simulation on every PR, block merge if it fails.

​
What runs where

Every simulation runs in an isolated environment:
  • Tools that require credentials use test credentials (a separate connector per Flow, marked test-only). Prevents accidental production side-effects during a run.
  • Actions marked “must not run in simulation” are stubbed. Their outputs are simulated to success.
  • Memory writes go to a per-simulation scratch space that’s discarded after.
You can override the environment (Run → Advanced → Environment) to run against production credentials — rare, only when you specifically want end-to-end validation. Add safeguards.

​
Watching results

The results table shows one row per example:
  • Input — the dataset example’s input.
  • Target output — what the target produced.
  • Baseline output — what the baseline produced (if comparing).
  • Expected — the dataset’s expected output.
  • Verdict — pass / fail / regression / improvement.
  • Cost and latency.
Sort, filter, click any row for the full trace of that example.

​
Stopping

Long simulations can be stopped at any point. Partial results are kept and marked “partial”. Useful when it’s clear the change isn’t working after the first 20 examples.

​
Re-running

Re-running the same simulation is possible but usually pointless — same target + same dataset = same result (with model randomness noise). What you probably want:
  • Re-run with new dataset version — pick up new examples.
  • Re-run with new target — after adjusting your Draft based on this run’s insights.
Every re-run creates a new simulation record, so history is preserved.