Bootstrap a Demo Project with the CLI

One command fills a SignalFlag project with real test results — two comparable batches, an A/B comparison between them, and a trends dashboard. The demos replay data captured from evaluations that actually ran, so what you get is a working project to explore rather than a screenshot tour. There is nothing to configure, no Docker, and no build to register.

Two demos ship today. Each lands in its own project, so you can run both and leave them side by side. This page walks through the command, then details exactly what each demo puts in your account.

Time: about 5 minutes per demo, plus a few minutes of waiting for metrics.

Before you start:

• You have a SignalFlag account. Sign up at app.resim.ai if you don't.
• You have Python 3.10 or newer.
• You can open a browser to authenticate.

Looking for the short version?

See SignalFlag Working in Five Minutes is install, run, look — the fastest path to something on screen. This page is the same command documented in full, plus a section on each demo's contents.

---

What you're building

Whichever demo you pick, you end up with the same shape of results:

• Two batches of tests, run against two versions of the same system. Both batches run the same experiences, which is what pairs every test with its counterpart in the other batch.
• Metrics on every test, computed from the replayed data by a metrics config that ships inside the package.
• The files the runs produced, attached to the tests that made them, so the raw material sits next to the charts drawn from it — on every MuJoCo test, and on the navigation tests that recorded camera footage.
• An A/B comparison between the two batches, showing which experiences regressed and which improved.
• A trends dashboard that groups results by build version and takes in every run of the demo, not just this one.

---

Step 1: Install the CLI

Shell

pip install signalflag

The wheel installs the demo as a console script under two names, signalflag-demo and resim-demo. They are the same command — resim-demo is kept so anything already scripted against the old package name keeps working. python -m signalflag.demo runs it too.

The SDK used to be published as resim-sdk. That name still installs and works; signalflag is the same package and the one to use from here on.

---

Step 2: See which demos ship

--demo has no default. The demos are peers, so running one you didn't ask for would be a surprise. Run the command with no arguments and it lists what is on offer instead:

Shell

signalflag-demo

Output

signalflag-demo replays real test data into your own ReSim project.

Pick one with --demo:

  mujoco       An ALOHA bimanual manipulation policy in MuJoCo: one test per
               cube placement, compared across two policy builds.
  navigation   A hospital navigation suite: dense telemetry across 34
               scenarios, covering every chart type ReSim ships.

For example:
  signalflag-demo --demo navigation

Each demo has its own project, branch, metrics config and dashboard, so running both leaves you with two independent sets of results, and neither goes near your real projects:

--demo	Project	Branch	Batches	Data to download
navigation	SignalFlag SDK Demo	sdk-demo	2 × 34 tests	~7 MB
mujoco	SignalFlag SDK Demo (MuJoCo)	sdk-demo-mujoco	2 × 20 tests	~110 MB

The navigation demo and the MuJoCo demo below go through what each one contains, metric by metric.

---

Step 3: Bootstrap the demo

Shell

signalflag-demo --demo navigation

Four things happen, in this order.

It fetches the data it replays. Each demo's data is a single tarball in a public bucket, pinned by checksum and cached after the first run. The download comes before the login on purpose: there is no point sending you through a browser only to fail on a download afterwards.

Output

Downloading demo data (7 MB)
  7 / 7 MB (100%)
Checking it downloaded intact

It authenticates you. The first run opens a device-code login. Later runs reuse the cached token.

Output

Authenticating by Device Code

Please navigate to: https://resim.us.auth0.com/activate?user_code=XXXX-XXXX

Visit that URL, sign in, and the demo continues on its own.

It creates the project, the branch and the dashboard. The project is created only if it doesn't already exist. Syncing the demo's metrics config is what creates the branch and the dashboard along with it.

Output

Creating project 'SignalFlag SDK Demo'

It replays the two batches. Every test is created, its emissions and events are pushed through the same typed SDK calls you would use for your own data, its files are attached, and it is closed.

Output

Running batch A: Nav stack v2 (baseline) (34 tests)
  Hospital East Hallway Navigation From North Hallway - Bright with Pedestrians
  Hospital East Hallway Navigation From North Hallway - Bright without Pedestrians
  ...
Running batch B: Nav stack v3 (candidate) (34 tests)
  ...

Both batches land on one branch. A dashboard is scoped to a single branch, so sharing one is what lets a single dashboard trend across both; the batches are told apart by their build version instead.

---

Step 4: Follow the four links

When it finishes, the demo prints where to go:

Output

SignalFlag SDK demo complete.

  Batch A (baseline, nav-v2.0.0)    https://app.signalflag.ai/projects/.../batches/...
  Batch B (candidate, nav-v3.0.0)   https://app.signalflag.ai/projects/.../batches/...
  A/B comparison                    https://app.signalflag.ai/projects/.../batches/.../compare-batch/batch/...
  Trends dashboard                  https://app.signalflag.ai/projects/.../dashboards/...

Metrics are computed after each batch closes, which takes a few minutes.
Reload these pages when it finishes.

Metrics are computed once each batch closes, so the pages look empty at first. Give it a couple of minutes and reload.

• Batch A and Batch B list their tests with a pass, warning or blocker status. Open any test for its metrics, plus an Events tab and a Logs tab wherever that run recorded something for them.
• The A/B comparison matches metrics by name between the two runs, so each chart from Batch A sits next to the same chart from Batch B. The Tests tab groups experiences by how their outcome differs, which puts the regressions and the fixes in their own lists. See the A/B comparison guide.
• The dashboard aggregates across runs rather than showing one, grouped by build version. Run the demo again and the same bars take in the new runs too.

---

Command reference

Flags

Flag	Default	What it does
--demo {mujoco,navigation}	none	Which demo to run. Required.
--project-name	the demo's own	Project to run in, created if it does not exist.
--branch	the demo's own	Branch to create both batches on.
--data-dir	none	Replay an already-extracted bundle from this directory instead of downloading one.
--quiet	off	Only report failures.

Passing any of the other flags without --demo is an error rather than a listing, because dropping your argument and exiting successfully would read as though the run had happened:

Output

signalflag-demo: error: --demo is required when passing --project-name

To keep a demo out of the default project, name your own:

Shell

signalflag-demo --demo navigation --project-name "my project"

Where the data is cached

Each demo's bundle is downloaded once and cached under its own key, so switching demos does not evict the other's data. The cache root honours XDG_CACHE_HOME and otherwise sits under ~/.cache:

• ~/.cache/resim/sdk-demo/navigation-v1/
• ~/.cache/resim/sdk-demo/mujoco-v4/

Every bundle is pinned by sha256. A download that does not match its expected digest is refused rather than replayed, and the archive is checked for unsafe member paths before anything is extracted. Delete a cache directory to force a fresh download.

Running without a download

--data-dir points the demo at an already-extracted bundle — useful on a restricted network, or to re-run repeatedly without touching the cache:

Shell

signalflag-demo --demo mujoco --data-dir ./extracted-bundle

The directory must contain the bundle's manifest.json at its root.

Exit codes

Code	Meaning
0	The demo ran, or a bare invocation listed the demos.
1	The replay data could not be fetched or used, or contact with SignalFlag was lost. Batches already created stay in the app; re-run to start a fresh pair.
2	Bad arguments — an unknown --demo, or another flag passed without --demo.

Calling it from Python

Useful when you want the IDs back rather than reading them off the terminal:

Python

from signalflag.demo import DEMOS, run

result = run(demo="navigation")

result.project_id     # the project it used or created
result.batch_ids      # {"a": ..., "b": ...}
result.dashboard_id   # None if the dashboard could not be resolved
result.urls           # {"batch_a": ..., "batch_b": ..., "compare": ..., "dashboard": ...}

DEMOS.keys()          # dict_keys(['navigation', 'mujoco'])

run() takes the same options as the command line — project_name positionally, then demo, branch, data_dir and quiet as keywords, plus a client if you would rather authenticate yourself than be sent to a browser. It raises signalflag.demo.DemoDataError when the replay data cannot be fetched.

---

The navigation demo

Shell

signalflag-demo --demo navigation

A mobile robot running errands around a hospital. This is the fuller tour of the two: the source data is dense telemetry, and the config that reads it draws nine of the ten system templates — everything but artifact — plus two custom ones. Start here if you want to see what the platform renders.

What lands in your project

Project	SignalFlag SDK Demo
Branch	sdk-demo
Batch A	Nav stack v2 (baseline), build version nav-v2.0.0, 34 tests
Batch B	Nav stack v3 (candidate), build version nav-v3.0.0, 34 tests
Metrics set	Demo Metrics — 13 test metrics and 5 batch metrics
Dashboard	SDK Demo Trends, from the Demo Trends set, over a rolling 30 days

Each test is one scenario: a route through the hospital, at one of three lighting levels, with or without pedestrians — Hospital East Hallway Navigation From North Hallway - Bright with Pedestrians, and so on. The same 34 run in both batches, so every test pairs with its counterpart in the A/B view. Seventeen of them have pedestrians in them, and eleven carry camera footage — the same eleven on both sides, so the media metrics appear on both halves of those pairs.

Pass and fail are not copied from the source runs. They are recomputed from the config's status checks against the replayed data, which works out as 16 passed, 9 warned and 9 blocked in Batch A against 15, 8 and 11 in Batch B. Seven scenarios change outcome between the two — four get worse, three get better — and those are the ones that land in the comparison's Failing on just one batch group. The rest divide between the groups either side of it, so none of the four comes up empty.

The data it replays

Thirteen topics, declared in the demo's config and emitted per test:

Topic	Fields	What it carries
odom_linear_velocity	x, y, z	Robot velocity from the drive stack.
goal_distance	goal_name, distance_m	Distance to each named goal, over the run.
time_to_goal	goal_name, time_s	Time taken to reach each goal. One row per goal actually reached.
goal_count	count	How many goals the run was asked to reach.
goal_status	state	What the navigation stack was doing, as a discrete state.
nearest_human_distance	person_name, distance_m	Distance to the nearest tracked person. Only the 17 scenarios with pedestrians emit it.
pose_difference	position_diff_m	Gap between wheel odometry and the localizer's pose estimate.
localization_uncertainty	cov_x, cov_xy, cov_y, cov_yaw, position_uncertainty_m, yaw_uncertainty_rad	The localizer's own confidence in that estimate.
covariance_accuracy	mahalanobis_distance, normalized_innovation_squared, normalized_x, normalized_y, normalized_yaw, position_error_m, within_1_sigma, within_2_sigma, within_3_sigma, yaw_error_rad	Whether that confidence is honest — true error against reported covariance.
robot_trajectory	raw_metric	A Plotly figure the test rendered itself.
camera_video	camera_name, filename (video)	The onboard camera clip.
camera_frame	camera_name, filename (image)	A still pulled from that clip. Separate from camera_video because image and video columns cannot share a topic.
goal_reached	event: name, description, status, tags	One event each time the robot reaches a goal.

What you see on a test

Metric	Template	What it shows
Robot Speed	line	Ground speed in m/s over the run.
Speed Distribution	histogram	How much of the run was spent at each speed.
Time in Speed Band	pie	Share of samples stopped (under 0.05 m/s), creeping (under 0.3 m/s) or cruising.
Distance to Goal	line	Distance to each goal over time, one line per goal.
Goal Status	state timeline	The navigation stack's state, as bands along the run.
Localization Error	line	Odometry against the localizer's estimate, in metres.
Uncertainty vs Actual Error	line	Predicted uncertainty, observed error and Mahalanobis distance on one set of axes. A localizer whose uncertainty tracks its error is one you can trust.
Covariance Calibration	table	Sample count, the share of samples inside 1σ, 2σ and 3σ against their targets, and median NIS.
Nearest-Human Distance	line	Distance to the nearest person. Warns inside 1 m, blocks inside 0.5 m. Skipped on the scenarios with no pedestrians.
Time to Reach Final Goal	scalar (seconds)	Time taken to reach the run's goals, added up. Blocks when the run does not reach every goal it was given, and charges 75 s instead.
Camera Feed	video	The onboard clip, on the 11 tests that carry one.
Camera Frame	image	The still from that clip.
Robot Trajectory	custom (raw.liquid)	Top-down view of the path the robot took, from the figure the test emitted.

Those two status checks are the whole of pass and fail here: everything else is there to be read. The config is written for runs with any number of goals, but every scenario in this data sets exactly one, so the per-goal charts draw a single line and a run either reaches its goal in 35 to 85 seconds or blocks.

The Events tab carries a Goal 1 Reached event pinned to the moment the robot got there, with a navigation tag — on 27 of the 34 tests in Batch A and 24 in Batch B. The Logs tab holds the camera clip and the still frame on the eleven tests that recorded them.

What you see on the batch

Metric	Template	What it shows
Mean Localization Error by Experience	bar	Average localizer error in each scenario, across the batch.
Localization Error Spread	custom (strip.liquid)	Worst error per scenario, as a strip plot.
Covariance Calibration Summary	table	Sample count, the 1σ and 2σ shares and median NIS, across every test in the batch.
Pedestrian Proximity Incidents	bar	Per tracked person, how many scenarios came within 0.5 m (a hit) or 0.5–1 m (too close). At most one count per scenario.
Total Time to Reach Final Goal	scalar (seconds)	Every scenario's time added up, with the same 75 s charged for a missed goal.

What you see on the dashboard

SDK Demo Trends looks at every test on the branch inside a rolling 30-day window and groups by build version, so it compares versions of the stack rather than snapshotting one run:

Metric	Template	What it shows
Localization Error by Build Version	bar	Mean localizer error per version. A taller bar than the version before it is a regression.
Test Outcomes by Build Version	bar	How each version's tests turned out, one bar per outcome. Click a bar to open a batch that version ran in.
Tests Run	scalar	Tests on this branch inside the window.

Run the demo again and those bars take in the new runs rather than multiplying into new points, which is what you want from a dashboard pinned to a branch.

---

The MuJoCo demo

Shell

signalflag-demo --demo mujoco

Two ALOHA robot arms passing a cube between them, evaluated in MuJoCo. Worth a look once you have toured the navigation demo, because the shape of the data is different: a staged task with a reward ladder rather than dense telemetry, and a question — did this policy configuration get better or worse — rather than a survey of chart types.

What lands in your project

Project	SignalFlag SDK Demo (MuJoCo)
Branch	sdk-demo-mujoco
Batch A	n_heads=16, attention re-partitioned, build version aloha-n16, 20 tests
Batch B	n_heads=8, checkpoint default, build version aloha-n8, 20 tests
Metrics set	ALOHA Metrics — 9 test metrics and 5 batch metrics
Dashboard	ALOHA Policy Trends, from the ALOHA Trends set, over a rolling 30 days

Each test is one starting placement of the cube on the table. That placement is the only thing that differs between tests, so each is named for it: Cube start (0.099, 0.446) seed 50. The same 20 placements run on both sides, which is what pairs every test with its counterpart.

The two batches are the same policy checkpoint built two ways, differing in one hyperparameter: the number of attention heads. One matches the value the policy was trained with and the other does not, which is enough to change how often the handover completes — 8 of 20 placements solved on the re-partitioned build against 14 on the checkpoint default, with 8 placements changing outcome between them. That is the question the A/B page answers here.

The task has stages rather than a single outcome. The cube gets touched, lifted, carried across, then released, and most of the metrics follow from that ladder: no contact, right gripper contact, lifted, transfer attempted, handover complete.

The data it replays

Eight topics:

Topic	Fields	What it carries
summary_metrics	overall_success_rate, overall_average_sum_reward, overall_average_max_reward, total_evaluation_time, total_regressions	One row per test, summarising the episode.
step_reward	reward	The staged reward, one row per simulation step.
task_stage	state	Which rung of the ladder the episode is on, per step.
cube_pose	x, y, z	World position of the cube. The handover counts as complete once x crosses -0.10.
stage_timing	stage, seconds_from_start, seconds_from_previous, regressions	One row per stage the episode reached.
cube_trajectory	raw_metric	A Plotly figure the evaluation drew itself.
episode_gif	filename (image)	The whole episode as a looping animation.
stage_reached	event: name, description, status, tags, metrics	Raised the first time each stage is reached, carrying the timings that explain the transition.

What you see on a test

Metric	Template	What it shows
Task Stage	state timeline	Which stage the episode was in, so a run that grasps and then drops the cube looks different from one that never grasps it.
Reward Over Time	line	The staged reward: 1 on contact, 2 lifted, 3 transfer attempted, 4 handover complete.
Handover Progress	line	How far the cube is past the handover line. Zero is the line; the handover completes once this goes positive.
Cube Height	line	The cube sits at 0.05 m on the table, so a rise is the lift and a drop back is a lost grasp.
Episode Summary	table	Handover completed, furthest stage, episode length, sum reward, ladder regressions. Warns when the handover did not complete.
Episode Replay	image	The looping animation of the attempt.
Cube Path	custom (raw.liquid)	Top-down view of the cube's path, coloured by stage. The only chart that shows the shape of the motion rather than one axis of it against time.
Stage Latency	bar	Simulated seconds from the start of the episode to each stage it reached. A bar rather than a table column so an A/B comparison can overlay two runs on one set of axes.
Stage Timeline	table	Each stage reached, when, how long the stage before it took, and how many times the episode had fallen back by then.

Completing the handover passes and anything short of it warns. Nothing blocks: a cube placement the policy cannot solve is not a broken build.

Every test carries two files on its Logs tab. transfer_cube.mcap goes up as robotics data, and uploading it as the kind of file it is rather than as bytes is what lets it open in the viewer inline instead of only offering a download. episode.gif goes up as a generic log, and reaches the page by a different route: the Episode Replay metric finds it by the filename the episode_gif topic emits. Your own tests attach files the same way; see Attaching files.

The Events tab carries one event per stage the episode first reached — Cube Contacted, Cube Lifted, Transfer Attempted, Handover Complete — each one carrying the time to that stage, the time spent in the previous one, and the regressions so far.

What you see on the batch

Metric	Template	What it shows
Mean Time Per Stage	bar	Average simulated seconds each stage took, across the placements that reached it.
Seeds Reaching Each Stage	bar	How many placements got as far as each stage.
Furthest Stage Distribution	pie	How far the policy got, as a share of the batch.
Run Summary	table	Placements evaluated and solved, mean sum reward, mean episode length, mean ladder regressions.
Reward Distribution	histogram	Spread of total episode reward across the batch.

What you see on the dashboard

ALOHA Policy Trends groups by build version, so each policy configuration becomes one category. Every bar here is click-through: select one to open a batch that build ran in.

Metric	Template	What it shows
Success Rate by Build Version	bar	Share of placements where the handover completed. The headline number a policy change is trying to move.
Test Outcomes by Build Version	bar	How each build's tests turned out, one bar per outcome.
Furthest Stage by Build Version	bar	Where each build's placements ended up on the ladder. A build can hold its pass count while its failures move between stages, and that shift is the more informative signal.
Time to Handover by Build Version	bar	Mean seconds to complete the handover, counting only the placements that completed it. Separates getting there from getting there quickly.
Reward by Build Version	bar	Mean total episode reward. A build that drops here has regressed, whatever its pass count says.
Trend Summary	table	Builds compared, tests run, batches, inside the window.

---

How the two compare

	navigation	mujoco
System under test	A hospital delivery robot's navigation stack	An ALOHA bimanual manipulation policy
What varies between batches	Two versions of the stack	One hyperparameter of the same checkpoint
Tests per batch	34 scenarios	20 cube placements
Topics	13	8
Metrics	21 — 13 test, 5 batch, 3 dashboard	20 — 9 test, 5 batch, 6 dashboard
System templates covered	Nine of ten — all but artifact — plus two custom Liquid templates	Seven of ten — no scalar, video or artifact — plus one custom template
Status checks	Two, both of which can block	One, warn only
Files per test	Camera clip and still, on 11 of 34	MCAP and episode animation, on all 20
Data to download	~7 MB	~110 MB

Neither is a subset of the other. The navigation demo is the wider tour of what SignalFlag renders; the MuJoCo demo is the closer look at comparing two builds of one policy.

---

Read the config that drives it

Everything either demo displays comes from a metrics config and its templates, shipped inside the package. Nothing is hardcoded into the platform, and nothing about these files is special to the demo — this is the same config you would write for your own system:

Python

from signalflag.demo import config_path, templates_path

print(config_path("navigation").read_text())
print(templates_path("navigation"))   # directory of the .liquid templates it references

config_path("mujoco") gives you the other one. Three pieces matter, in this order.

Topics describe the shape of the data your tests emit:

config.resim.yml

topics:
  odom_linear_velocity:
    schema:
      x: float
      y: float
      z: float

Metrics are SQL queries over that data, each rendered by a template. A metric's type decides what it runs against: test per test, batch across a batch, dashboard across every batch in the dashboard's window.

config.resim.yml

metrics:
  Robot Speed:
    type: test
    description: Ground speed of the robot over the course of the run.
    query_string: |
      SELECT
        'Speed' AS group_name,
        timestamp / 1E9 AS "Time (s)",
        SQRT(POWER(x, 2) + POWER(y, 2)) AS "Speed (m/s)"
      FROM odom_linear_velocity;
    template_type: system
    template: line

Batch and dashboard metrics can join against the built-in metadata table, which is where the build version, experience name and status of each test live:

config.resim.yml

metrics:
  Localization Error by Build Version:
    type: dashboard
    query_string: |
      SELECT
        'Mean' AS group_name,
        m.build_version AS "Build Version",
        AVG(pd.position_diff_m) AS "Mean Localization Error (m)"
      FROM pose_difference pd
      JOIN metadata m ON pd.job_id = m.job_id
      WHERE m.build_version IS NOT NULL
      GROUP BY m.build_version
      ORDER BY m.build_version;
    template_type: system
    template: bar

Metrics sets group metrics so that a batch or a dashboard can run them. Note the key is metrics sets, with a space:

config.resim.yml

metrics sets:
  Demo Metrics:            # run against each batch
    metrics:
      - Robot Speed
      - Speed Distribution
      # ...
  Demo Trends:             # powers the dashboard, which spans batches
    metrics:
      - Localization Error by Build Version
      # ...

dashboards:
  SDK Demo Trends:
    metrics_set: Demo Trends
    refresh: auto
    day_range: 30

A metric with template_type: custom names a .liquid file instead of a system template, and that file has to be synced alongside the config — which is why templates_path() exists next to config_path(). The metrics guide covers topics, every template, status checks and custom templates in full.

---

Next steps

• Emit your own data. Get Your First Metrics in SignalFlag walks through the same API the demo uses, starting from an empty script.
• Write your own metrics. Start with the metrics guide, and develop them against a debug dashboard before running a batch.
• Run tests in SignalFlag's infrastructure. Run Your First Test Batch sets up builds and experiences so SignalFlag executes your system rather than you replaying results into it.
• Reference. The SignalFlag SDK guide covers the SDK's API surface in full.