# Docs > Documentation for SignalFlag, the test infrastructure platform for physical AI SignalFlag is the test infrastructure platform for physical AI. These docs cover setting up the platform, running test batches, integrating with CI, building metrics, and consuming results via the CLI, SDK, MCP server, and web app. They also include reference material for the open-core libraries and conceptual explanations of simulation-based evaluation. # Tutorials # Tutorials Hands-on walkthroughs that take you from zero to working results. ~5 min Step 1 See SignalFlag Working in Five Minutes Run one command to fill a project with real test results, an A/B comparison, and a dashboard. Nothing to configure. [Start tutorial →](./sdk-demo/) ~15 min Deep dive Bootstrap a Demo Project with the CLI The same command in full: every flag, and a section on each demo detailing its topics, metrics, status checks and dashboard. [Read the reference →](./demo-projects/) ~10 min Step 2 Get Your First Metrics in SignalFlag Use the Python SDK to push data from your own machine and see metrics in the dashboard. No Docker or cloud infrastructure needed. [Start tutorial →](./first-external-batch/) ______________________________________________________________________ Once you've completed the tutorials, the [How-to Guides](../setup/) cover bringing your own system, experiences, and metrics into SignalFlag. # See SignalFlag Working in Five Minutes This tutorial runs a demo that fills a SignalFlag project with real test results, so you can see what the platform does before connecting anything of your own. No Docker, no container registry, no config to write. One command. **Time:** about 5 minutes, plus a few minutes of waiting for metrics. **Before you start:** - You have a SignalFlag account. Sign up at [app.signalflag.ai/signup](https://app.signalflag.ai/signup) if you don't. - You have Python 3.10 or newer. - You can open a browser to authenticate. ______________________________________________________________________ ## What you're building The demo replays test data from a real navigation project into your own SignalFlag project. When it finishes you'll have: - **Two batches of tests**, one per version of the navigation stack. The same scenarios ran in both, so every test has a counterpart in the other batch. - **Metrics on every test**: speed and localization error over time, goal state timelines, pass/fail checks, and more, with camera footage on the tests that recorded it. - **The files each run produced**, attached to the test they came from, so the raw material sits next to the charts drawn from it. - **An A/B comparison** between the two batches, showing which scenarios regressed and which improved. - **A dashboard** comparing the two versions across every run of them. That's the navigation demo. A second one replays MuJoCo policy evaluations — see [Choosing a demo](#choosing-a-demo) below. ______________________________________________________________________ ## Step 1: Install the SDK Shell ``` pip install signalflag ``` The SDK used to be published as `resim-sdk`. That name still installs and works; `signalflag` is the same package and the one to use from here on. ______________________________________________________________________ ## Step 2: Run the demo Shell ``` signalflag-demo --demo navigation ``` The package also installs `resim-demo`, an alias for the same command, so anything already scripted against the old name keeps working. First it downloads the test data it replays, reporting progress as it goes. That happens once — later runs reuse the cached copy. Then it asks you to authenticate in a browser: Output ``` Authenticating by Device Code Please navigate to: https://resim.us.auth0.com/activate?user_code=XXXX-XXXX ``` Visit that URL, sign in, and the demo continues on its own. Subsequent runs reuse the cached token. Each demo works in a project of its own, creating it if it doesn't exist, so it won't touch your real projects. The navigation demo uses one called `SignalFlag SDK Demo`. To use a different one: Shell ``` signalflag-demo --demo navigation --project-name "my project" ``` ### Choosing a demo `--demo` picks which demo to run. There's no default — running `signalflag-demo` on its own lists the ones that ship: Shell ``` signalflag-demo ``` | `--demo` | What it replays | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | `navigation` | A robot running errands around a hospital. Dense telemetry across 34 scenarios, covering almost every chart type SignalFlag renders. | | `mujoco` | An ALOHA bimanual manipulation policy evaluated in MuJoCo, one test per cube placement, compared across two policy builds. | Each lands in its own project and on its own branch, so running both leaves you with two independent sets of results to look at. [Bootstrap a Demo Project with the CLI](../demo-projects/) details what each one contains, metric by metric. ______________________________________________________________________ ## Step 3: Follow the four links When the demo finishes it prints where to go: Output ``` SignalFlag SDK demo complete. Batch A (baseline, nav-v2.0.0) https://app.signalflag.ai/projects/.../batches/... Batch B (candidate, nav-v3.0.0) https://app.signalflag.ai/projects/.../batches/... A/B comparison https://app.signalflag.ai/projects/.../batches/.../compare-batch/batch/... Trends dashboard https://app.signalflag.ai/projects/.../dashboards/... Metrics are computed after each batch closes, which takes a few minutes. Reload these pages when it finishes. ``` Metrics are computed once each batch closes, so the pages look empty at first. Give it a couple of minutes and reload. ### Batch A and Batch B Each batch page lists its tests with a pass, warning, or blocker status. Open any test to see its metrics: speed over time, distance to each goal, what the navigation stack was doing as a state timeline, a table judging whether the localizer's confidence was honest, and on some tests the onboard camera footage. The **Events** tab shows the moment the robot reached its goal, and the **Logs** tab holds the files that run produced. Batch-level metrics sit on the batch page itself, aggregating across every test. ### The A/B comparison This is the page that answers "did this version make things better or worse". Metrics are matched by name between the two runs, so each chart from Batch A sits next to the same chart from Batch B. **Merged Metrics** goes further and draws both runs on the same axes. The **Tests** tab is the fastest read: it groups the scenarios by how their outcome differs between the two batches, so the regressions and the fixes are each their own list. See the [A/B comparison guide](../../guides/ab-comparison/) for the full picture. ### The dashboard The batches show you two runs. The dashboard aggregates across them, grouped by build version: how each version's tests turned out, its mean localization error, and how many tests have run. Run the demo again and those bars take in the new runs too. ______________________________________________________________________ ## Next steps - **See what else the demos contain.** [Bootstrap a Demo Project with the CLI](../demo-projects/) documents every flag of the command and details both demos: their topics, every metric, their status checks and dashboards. - **Emit your own data.** [Get Your First Metrics in SignalFlag](../first-external-batch/) walks through the same API the demo uses, starting from an empty script. - **Write your own metrics.** Start with the [metrics guide](../../guides/metrics/), and develop them against a debug dashboard before running a batch. # Bootstrap a Demo Project with the CLI One command fills a SignalFlag project with real test results — batches of tests, metrics computed on them, and a trends dashboard. The demos replay data captured from evaluations that actually ran, so what you get is a working project to explore rather than a screenshot tour. There is nothing to configure, no Docker, and no build to register. Three demos ship today. Two compare two versions of the same system head to head — an A/B pair, with a comparison page between the batches. The third, `session`, is a different shape: real field sessions from one system, replayed one batch per session and trended over time rather than paired off. Each demo lands in its own project, so you can run all three and leave them side by side. This page walks through the command, then details exactly what each demo puts in your account. **Time:** about 5 minutes per demo, plus a few minutes of waiting for metrics. **Before you start:** - You have a SignalFlag account. Sign up at [app.signalflag.ai/signup](https://app.signalflag.ai/signup) if you don't. - You have Python 3.10 or newer. - You can open a browser to authenticate. Looking for the short version? [See SignalFlag Working in Five Minutes](../sdk-demo/) is install, run, look — the fastest path to something on screen. This page is the same command documented in full, plus a section on each demo's contents. ______________________________________________________________________ ## What you're building The navigation and MuJoCo demos give you the same shape of results: - **Two batches of tests**, run against two versions of the same system. Both batches run the same experiences, which is what pairs every test with its counterpart in the other batch. - **Metrics on every test**, computed from the replayed data by a metrics config that ships inside the package. - **The files the runs produced**, attached to the tests that made them, so the raw material sits next to the charts drawn from it — on every MuJoCo test, and on the navigation tests that recorded camera footage. - **An A/B comparison** between the two batches, showing which experiences regressed and which improved. - **A trends dashboard** that groups results by build version and takes in every run of the demo, not just this one. The `session` demo gives you a different shape, because it is answering a different question — not "did build A beat build B" but "how did today's session compare to the last several": - **Four batches, one per field session**, each holding exactly one test — the whole session, not one scenario cut out of it. - **Every session's raw recording attached to its test** — the mcap snippets and clip GIFs a real robotics session produces, not synthetic stand-ins. - **A system, and every session's experience tagged**, so they show up together in the project's [Sessions](../../guides/sessions/) library — this demo's real trend view, so there is no metrics dashboard. ______________________________________________________________________ ## Step 1: Install the CLI Shell ``` pip install signalflag ``` The wheel installs the demo as a console script under two names, `signalflag-demo` and `resim-demo`. They are the same command — `resim-demo` is kept so anything already scripted against the old package name keeps working. `python -m signalflag.demo` runs it too. The SDK used to be published as `resim-sdk`. That name still installs and works; `signalflag` is the same package and the one to use from here on. ______________________________________________________________________ ## Step 2: See which demos ship `--demo` has no default. The demos are peers, so running one you didn't ask for would be a surprise. Run the command with no arguments and it lists what is on offer instead: Shell ``` signalflag-demo ``` Output ``` signalflag-demo replays real test data into your own ReSim project. Pick one with --demo: mujoco An ALOHA bimanual manipulation policy in MuJoCo: one test per cube placement, compared across two policy builds. navigation A hospital navigation suite: dense telemetry across 34 scenarios, covering every chart type ReSim ships. session Four real field sessions from a legged robot: GNSS, IMU and gait telemetry, one batch per session on a shared branch. For example: signalflag-demo --demo navigation ``` Each demo has its own project, branch and metrics config, so running all three leaves you with three independent sets of results, and none goes near your real projects: | `--demo` | Project | Branch | Batches | Data to download | | ------------ | ---------------------------------------- | --------------------------- | ------------ | ---------------- | | `navigation` | `SignalFlag SDK Demo` | `sdk-demo` | 2 × 34 tests | ~7 MB | | `mujoco` | `SignalFlag SDK Demo (MuJoCo)` | `sdk-demo-mujoco` | 2 × 20 tests | ~110 MB | | `session` | `SignalFlag SDK Demo (Logs to Insights)` | `sdk-demo-logs-to-insights` | 4 × 1 test | ~950 MB | [The navigation demo](#the-navigation-demo), [the MuJoCo demo](#the-mujoco-demo) and [the session demo](#the-session-demo) below go through what each one contains, metric by metric. ______________________________________________________________________ ## Step 3: Bootstrap the demo Shell ``` signalflag-demo --demo navigation ``` Four things happen, in this order. **It fetches the data it replays.** Each demo's data is a single tarball in a public bucket, pinned by checksum and cached after the first run. The download comes before the login on purpose: there is no point sending you through a browser only to fail on a download afterwards. Output ``` Downloading demo data (7 MB) 7 / 7 MB (100%) Checking it downloaded intact ``` **It authenticates you.** The first run opens a device-code login. Later runs reuse the cached token. Output ``` Authenticating by Device Code Please navigate to: https://resim.us.auth0.com/activate?user_code=XXXX-XXXX ``` Visit that URL, sign in, and the demo continues on its own. **It creates the project, the branch and the dashboard.** The project is created only if it doesn't already exist. Syncing the demo's metrics config is what creates the branch and the dashboard along with it. Output ``` Creating project 'SignalFlag SDK Demo' ``` **It replays the two batches.** Every test is created, its emissions and events are pushed through the same typed SDK calls you would use for your own data, its files are attached, and it is closed. Output ``` Running batch A: Nav stack v2 (baseline) (34 tests) Hospital East Hallway Navigation From North Hallway - Bright with Pedestrians Hospital East Hallway Navigation From North Hallway - Bright without Pedestrians ... Running batch B: Nav stack v3 (candidate) (34 tests) ... ``` Both batches land on one branch. A dashboard is scoped to a single branch, so sharing one is what lets a single dashboard trend across both; the batches are told apart by their build version instead. ______________________________________________________________________ ## Step 4: Follow the four links When it finishes, the demo prints where to go: Output ``` SignalFlag SDK demo complete. Batch A (baseline, nav-v2.0.0) https://app.signalflag.ai/projects/.../batches/... Batch B (candidate, nav-v3.0.0) https://app.signalflag.ai/projects/.../batches/... A/B comparison https://app.signalflag.ai/projects/.../batches/.../compare-batch/batch/... Trends dashboard https://app.signalflag.ai/projects/.../dashboards/... Metrics are computed after each batch closes, which takes a few minutes. Reload these pages when it finishes. ``` Metrics are computed once each batch closes, so the pages look empty at first. Give it a couple of minutes and reload. - **Batch A and Batch B** list their tests with a pass, warning or blocker status. Open any test for its metrics, plus an **Events** tab and a **Logs** tab wherever that run recorded something for them. - **The A/B comparison** matches metrics by name between the two runs, so each chart from Batch A sits next to the same chart from Batch B. The **Tests** tab groups experiences by how their outcome differs, which puts the regressions and the fixes in their own lists. See the [A/B comparison guide](../../guides/ab-comparison/). - **The dashboard** aggregates across runs rather than showing one, grouped by build version. Run the demo again and the same bars take in the new runs too. ______________________________________________________________________ ## Command reference ### Flags | Flag | Default | What it does | | ------------------------------------ | -------------- | ---------------------------------------------------------------------------------- | | `--demo {mujoco,navigation,session}` | none | Which demo to run. Required. | | `--project-name` | the demo's own | Project to run in, created if it does not exist. | | `--branch` | the demo's own | Branch to create every batch on. | | `--data-dir` | none | Replay an already-extracted bundle from this directory instead of downloading one. | | `--quiet` | off | Only report failures. | Passing any of the other flags without `--demo` is an error rather than a listing, because dropping your argument and exiting successfully would read as though the run had happened: Output ``` signalflag-demo: error: --demo is required when passing --project-name ``` To keep a demo out of the default project, name your own: Shell ``` signalflag-demo --demo navigation --project-name "my project" ``` ### Where the data is cached Each demo's bundle is downloaded once and cached under its own key, so switching demos does not evict the other's data. The cache root honours `XDG_CACHE_HOME` and otherwise sits under `~/.cache`: - `~/.cache/resim/sdk-demo/navigation-v1/` - `~/.cache/resim/sdk-demo/mujoco-v4/` - `~/.cache/resim/sdk-demo/session-v1/` Every bundle is pinned by sha256. A download that does not match its expected digest is refused rather than replayed, and the archive is checked for unsafe member paths before anything is extracted. Delete a cache directory to force a fresh download. ### Running without a download `--data-dir` points the demo at an already-extracted bundle — useful on a restricted network, or to re-run repeatedly without touching the cache: Shell ``` signalflag-demo --demo mujoco --data-dir ./extracted-bundle ``` The directory must contain the bundle's `manifest.json` at its root. ### Exit codes | Code | Meaning | | ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `0` | The demo ran, or a bare invocation listed the demos. | | `1` | The replay data could not be fetched or used, or contact with SignalFlag was lost. Batches already created stay in the app; re-run to start a fresh pair. | | `2` | Bad arguments — an unknown `--demo`, or another flag passed without `--demo`. | ### Calling it from Python Useful when you want the IDs back rather than reading them off the terminal: Python ``` from signalflag.demo import DEMOS, run result = run(demo="navigation") result.project_id # the project it used or created result.batch_ids # {"a": ..., "b": ...} result.dashboard_id # None if the dashboard could not be resolved result.urls # {"batch_a": ..., "batch_b": ..., "compare": ..., "dashboard": ...} DEMOS.keys() # dict_keys(['navigation', 'mujoco', 'session']) ``` `session` has four batches rather than two, so its `result.batch_ids` has four keys — the four session dates. It has no dashboard, so `result.dashboard_id` is always `None` and `result.urls` has no `"compare"` or `"dashboard"` key; it has a `"sessions"` entry instead, pointing at the project's Sessions library. `run()` takes the same options as the command line — `project_name` positionally, then `demo`, `branch`, `data_dir` and `quiet` as keywords, plus a `client` if you would rather authenticate yourself than be sent to a browser. It raises `signalflag.demo.DemoDataError` when the replay data cannot be fetched. ______________________________________________________________________ ## The navigation demo Shell ``` signalflag-demo --demo navigation ``` A mobile robot running errands around a hospital. This is the fuller tour of the two: the source data is dense telemetry, and the config that reads it draws nine of the ten system templates — everything but `artifact` — plus two custom ones. Start here if you want to see what the platform renders. ### What lands in your project | | | | ----------- | --------------------------------------------------------------------- | | Project | `SignalFlag SDK Demo` | | Branch | `sdk-demo` | | Batch A | **Nav stack v2 (baseline)**, build version `nav-v2.0.0`, 34 tests | | Batch B | **Nav stack v3 (candidate)**, build version `nav-v3.0.0`, 34 tests | | Metrics set | `Demo Metrics` — 13 test metrics and 5 batch metrics | | Dashboard | `SDK Demo Trends`, from the `Demo Trends` set, over a rolling 30 days | Each test is one scenario: a route through the hospital, at one of three lighting levels, with or without pedestrians — `Hospital East Hallway Navigation From North Hallway - Bright with Pedestrians`, and so on. The same 34 run in both batches, so every test pairs with its counterpart in the A/B view. Seventeen of them have pedestrians in them, and eleven carry camera footage — the same eleven on both sides, so the media metrics appear on both halves of those pairs. Pass and fail are not copied from the source runs. They are recomputed from the config's status checks against the replayed data, which works out as 16 passed, 9 warned and 9 blocked in Batch A against 15, 8 and 11 in Batch B. Seven scenarios change outcome between the two — four get worse, three get better — and those are the ones that land in the comparison's **Failing on just one batch** group. The rest divide between the groups either side of it, so none of the four comes up empty. ### The data it replays Thirteen topics, declared in the demo's config and emitted per test: | Topic | Fields | What it carries | | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `odom_linear_velocity` | `x`, `y`, `z` | Robot velocity from the drive stack. | | `goal_distance` | `goal_name`, `distance_m` | Distance to each named goal, over the run. | | `time_to_goal` | `goal_name`, `time_s` | Time taken to reach each goal. One row per goal actually reached. | | `goal_count` | `count` | How many goals the run was asked to reach. | | `goal_status` | `state` | What the navigation stack was doing, as a discrete state. | | `nearest_human_distance` | `person_name`, `distance_m` | Distance to the nearest tracked person. Only the 17 scenarios with pedestrians emit it. | | `pose_difference` | `position_diff_m` | Gap between wheel odometry and the localizer's pose estimate. | | `localization_uncertainty` | `cov_x`, `cov_xy`, `cov_y`, `cov_yaw`, `position_uncertainty_m`, `yaw_uncertainty_rad` | The localizer's own confidence in that estimate. | | `covariance_accuracy` | `mahalanobis_distance`, `normalized_innovation_squared`, `normalized_x`, `normalized_y`, `normalized_yaw`, `position_error_m`, `within_1_sigma`, `within_2_sigma`, `within_3_sigma`, `yaw_error_rad` | Whether that confidence is honest — true error against reported covariance. | | `robot_trajectory` | `raw_metric` | A Plotly figure the test rendered itself. | | `camera_video` | `camera_name`, `filename` (video) | The onboard camera clip. | | `camera_frame` | `camera_name`, `filename` (image) | A still pulled from that clip. Separate from `camera_video` because image and video columns cannot share a topic. | | `goal_reached` | event: `name`, `description`, `status`, `tags` | One event each time the robot reaches a goal. | ### What you see on a test | Metric | Template | What it shows | | --------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | Robot Speed | line | Ground speed in m/s over the run. | | Speed Distribution | histogram | How much of the run was spent at each speed. | | Time in Speed Band | pie | Share of samples stopped (under 0.05 m/s), creeping (under 0.3 m/s) or cruising. | | Distance to Goal | line | Distance to each goal over time, one line per goal. | | Goal Status | state timeline | The navigation stack's state, as bands along the run. | | Localization Error | line | Odometry against the localizer's estimate, in metres. | | Uncertainty vs Actual Error | line | Predicted uncertainty, observed error and Mahalanobis distance on one set of axes. A localizer whose uncertainty tracks its error is one you can trust. | | Covariance Calibration | table | Sample count, the share of samples inside 1σ, 2σ and 3σ against their targets, and median NIS. | | Nearest-Human Distance | line | Distance to the nearest person. **Warns inside 1 m, blocks inside 0.5 m.** Skipped on the scenarios with no pedestrians. | | Time to Reach Final Goal | scalar (seconds) | Time taken to reach the run's goals, added up. **Blocks when the run does not reach every goal it was given**, and charges 75 s instead. | | Camera Feed | video | The onboard clip, on the 11 tests that carry one. | | Camera Frame | image | The still from that clip. | | Robot Trajectory | custom (`raw.liquid`) | Top-down view of the path the robot took, from the figure the test emitted. | Those two status checks are the whole of pass and fail here: everything else is there to be read. The config is written for runs with any number of goals, but every scenario in this data sets exactly one, so the per-goal charts draw a single line and a run either reaches its goal in 35 to 85 seconds or blocks. The **Events** tab carries a `Goal 1 Reached` event pinned to the moment the robot got there, with a `navigation` tag — on 27 of the 34 tests in Batch A and 24 in Batch B. The **Logs** tab holds the camera clip and the still frame on the eleven tests that recorded them. ### What you see on the batch | Metric | Template | What it shows | | ------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------ | | Mean Localization Error by Experience | bar | Average localizer error in each scenario, across the batch. | | Localization Error Spread | custom (`strip.liquid`) | Worst error per scenario, as a strip plot. | | Covariance Calibration Summary | table | Sample count, the 1σ and 2σ shares and median NIS, across every test in the batch. | | Pedestrian Proximity Incidents | bar | Per tracked person, how many scenarios came within 0.5 m (a hit) or 0.5–1 m (too close). At most one count per scenario. | | Total Time to Reach Final Goal | scalar (seconds) | Every scenario's time added up, with the same 75 s charged for a missed goal. | ### What you see on the dashboard `SDK Demo Trends` looks at every test on the branch inside a rolling 30-day window and groups by build version, so it compares versions of the stack rather than snapshotting one run: | Metric | Template | What it shows | | ----------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------- | | Localization Error by Build Version | bar | Mean localizer error per version. A taller bar than the version before it is a regression. | | Test Outcomes by Build Version | bar | How each version's tests turned out, one bar per outcome. Click a bar to open a batch that version ran in. | | Tests Run | scalar | Tests on this branch inside the window. | Run the demo again and those bars take in the new runs rather than multiplying into new points, which is what you want from a dashboard pinned to a branch. ______________________________________________________________________ ## The MuJoCo demo Shell ``` signalflag-demo --demo mujoco ``` Two ALOHA robot arms passing a cube between them, evaluated in MuJoCo. Worth a look once you have toured the navigation demo, because the shape of the data is different: a staged task with a reward ladder rather than dense telemetry, and a question — did this policy configuration get better or worse — rather than a survey of chart types. ### What lands in your project | | | | ----------- | ----------------------------------------------------------------------------- | | Project | `SignalFlag SDK Demo (MuJoCo)` | | Branch | `sdk-demo-mujoco` | | Batch A | **n_heads=16, attention re-partitioned**, build version `aloha-n16`, 20 tests | | Batch B | **n_heads=8, checkpoint default**, build version `aloha-n8`, 20 tests | | Metrics set | `ALOHA Metrics` — 9 test metrics and 5 batch metrics | | Dashboard | `ALOHA Policy Trends`, from the `ALOHA Trends` set, over a rolling 30 days | Each test is one starting placement of the cube on the table. That placement is the only thing that differs between tests, so each is named for it: `Cube start (0.099, 0.446) seed 50`. The same 20 placements run on both sides, which is what pairs every test with its counterpart. The two batches are the same policy checkpoint built two ways, differing in one hyperparameter: the number of attention heads. One matches the value the policy was trained with and the other does not, which is enough to change how often the handover completes — 8 of 20 placements solved on the re-partitioned build against 14 on the checkpoint default, with 8 placements changing outcome between them. That is the question the A/B page answers here. The task has stages rather than a single outcome. The cube gets touched, lifted, carried across, then released, and most of the metrics follow from that ladder: **no contact**, **right gripper contact**, **lifted**, **transfer attempted**, **handover complete**. ### The data it replays Eight topics: | Topic | Fields | What it carries | | ----------------- | -------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | `summary_metrics` | `overall_success_rate`, `overall_average_sum_reward`, `overall_average_max_reward`, `total_evaluation_time`, `total_regressions` | One row per test, summarising the episode. | | `step_reward` | `reward` | The staged reward, one row per simulation step. | | `task_stage` | `state` | Which rung of the ladder the episode is on, per step. | | `cube_pose` | `x`, `y`, `z` | World position of the cube. The handover counts as complete once `x` crosses -0.10. | | `stage_timing` | `stage`, `seconds_from_start`, `seconds_from_previous`, `regressions` | One row per stage the episode reached. | | `cube_trajectory` | `raw_metric` | A Plotly figure the evaluation drew itself. | | `episode_gif` | `filename` (image) | The whole episode as a looping animation. | | `stage_reached` | event: `name`, `description`, `status`, `tags`, `metrics` | Raised the first time each stage is reached, carrying the timings that explain the transition. | ### What you see on a test | Metric | Template | What it shows | | ----------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Task Stage | state timeline | Which stage the episode was in, so a run that grasps and then drops the cube looks different from one that never grasps it. | | Reward Over Time | line | The staged reward: 1 on contact, 2 lifted, 3 transfer attempted, 4 handover complete. | | Handover Progress | line | How far the cube is past the handover line. Zero is the line; the handover completes once this goes positive. | | Cube Height | line | The cube sits at 0.05 m on the table, so a rise is the lift and a drop back is a lost grasp. | | Episode Summary | table | Handover completed, furthest stage, episode length, sum reward, ladder regressions. **Warns when the handover did not complete.** | | Episode Replay | image | The looping animation of the attempt. | | Cube Path | custom (`raw.liquid`) | Top-down view of the cube's path, coloured by stage. The only chart that shows the shape of the motion rather than one axis of it against time. | | Stage Latency | bar | Simulated seconds from the start of the episode to each stage it reached. A bar rather than a table column so an A/B comparison can overlay two runs on one set of axes. | | Stage Timeline | table | Each stage reached, when, how long the stage before it took, and how many times the episode had fallen back by then. | Completing the handover passes and anything short of it warns. Nothing blocks: a cube placement the policy cannot solve is not a broken build. Every test carries two files on its **Logs** tab. `transfer_cube.mcap` goes up as robotics data, and uploading it as the *kind* of file it is rather than as bytes is what lets it open in the viewer inline instead of only offering a download. `episode.gif` goes up as a generic log, and reaches the page by a different route: the **Episode Replay** metric finds it by the filename the `episode_gif` topic emits. Your own tests attach files the same way; see [Attaching files](../../guides/resim-sdk/#attaching-files). The **Events** tab carries one event per stage the episode first reached — `Cube Contacted`, `Cube Lifted`, `Transfer Attempted`, `Handover Complete` — each one carrying the time to that stage, the time spent in the previous one, and the regressions so far. ### What you see on the batch | Metric | Template | What it shows | | --------------------------- | --------- | ----------------------------------------------------------------------------------------------- | | Mean Time Per Stage | bar | Average simulated seconds each stage took, across the placements that reached it. | | Seeds Reaching Each Stage | bar | How many placements got as far as each stage. | | Furthest Stage Distribution | pie | How far the policy got, as a share of the batch. | | Run Summary | table | Placements evaluated and solved, mean sum reward, mean episode length, mean ladder regressions. | | Reward Distribution | histogram | Spread of total episode reward across the batch. | ### What you see on the dashboard `ALOHA Policy Trends` groups by build version, so each policy configuration becomes one category. Every bar here is click-through: select one to open a batch that build ran in. | Metric | Template | What it shows | | --------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Success Rate by Build Version | bar | Share of placements where the handover completed. The headline number a policy change is trying to move. | | Test Outcomes by Build Version | bar | How each build's tests turned out, one bar per outcome. | | Furthest Stage by Build Version | bar | Where each build's placements ended up on the ladder. A build can hold its pass count while its failures move between stages, and that shift is the more informative signal. | | Time to Handover by Build Version | bar | Mean seconds to complete the handover, counting only the placements that completed it. Separates getting there from getting there quickly. | | Reward by Build Version | bar | Mean total episode reward. A build that drops here has regressed, whatever its pass count says. | | Trend Summary | table | Builds compared, tests run, batches, inside the window. | ______________________________________________________________________ ## The session demo Shell ``` signalflag-demo --demo session ``` Four real field sessions from a legged robot walking outdoors, captured on four different days. This demo is not an A/B pair — there is no build to compare against another — so it answers a different question than the other two: not "did this change help," but "how did today's session compare to the last several." Each session becomes its own batch, all four land on one branch, and the demo tags every session's experience so it shows up in the project's [Sessions](../../guides/sessions/) library alongside its own evaluation. ### What lands in your project | | | | -------------- | ---------------------------------------------------------------------------------------------------------------- | | Project | `SignalFlag SDK Demo (Logs to Insights)` | | Branch | `sdk-demo-logs-to-insights` | | System | `Session Evaluations`, created on first run | | Experience tag | `resim-session`, applied to every session's experience | | Batches | Four, one per session date — `2024-11-04`, `2024-11-14`, `2024-11-15`, `2024-11-18` — each with exactly one test | | Metrics set | `Session Metrics` — 6 test metrics and 3 batch metrics | There is no dashboard here — the [Sessions](../../guides/sessions/) library is this demo's trend view, grouping by session rather than by build version, so a metrics dashboard alongside it would be empty by construction. Every batch's one test is the whole session, not a scenario cut out of it — the same mcap snippets, gif clips and telemetry a real ingested recording carries. Because each batch has exactly one test, the demo prints a direct link to that test's page rather than to the batch as a whole, opened on its log viewer tab. ### The data it replays Thirteen topics per session: | Topic | Fields | What it carries | | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | | `mission` | `mission_idx`, `start_s`, `end_s`, `duration_s`, `distance_m`, `net_displacement_m`, `mean_speed_mps`, `max_speed_mps`, `moving_share`, `dlio_distance_m`, `odom_dlio_gap_m`, `gnss_distance_m`, `gnss_odom_gap_m`, `has_gnss`, `gnss_coverage`, `gnss_fix_count`, `altitude_min_m`, `altitude_max_m`, `altitude_gain_m`, `start_lat`, `start_lon`, `foot_slip_share`, `odom_gnss_fit_rmse_m` | One row per mission ReSim's own boundary detector found in the session. | | `motion_phase` | `mission_idx`, `phase_idx`, `kind`, `start_s`, `end_s`, `duration_s`, `distance_m`, `mean_speed_mps`, `max_speed_mps` | Contiguous stopped/moving segments within a mission. | | `movement_summary` | `seq`, `mission_idx`, `t_s`, `kind`, `label`, `clip` (image), `snippet`, `phase_duration_s`, `phase_distance_m`, `phase_mean_speed_mps` | One row per start/stop transition, with the clip that shows it and the raw mcap snippet it was cut from. | | `robot_pose` | `mission_idx`, `series`, `t_s`, `x`, `y`, `z`, `east_m`, `north_m`, `has_enu` | Robot position, in both the local and the ENU frame where available. | | `robot_state` | `mission_idx`, `series`, `t_s`, `feet_in_contact`, `feet_slipping`, `max_foot_force_n`, `max_joint_effort_nm`, `max_joint_velocity_rps`, `speed_mps` | Gait state — how many feet are down, and whether any are slipping. | | `speed_sample` | `mission_idx`, `series`, `t_s`, `speed_mps` | Ground speed. | | `imu_sample` | `mission_idx`, `series`, `t_s`, `accel_mag_mps2`, `gyro_mag_rps` | IMU acceleration and gyro magnitude. | | `gnss_track` | `mission_idx`, `series`, `t_s`, `latitude`, `longitude`, `altitude_m`, `east_m`, `north_m` | Raw GNSS fixes. | | `command_tracking` | `mission_idx`, `series`, `t_s`, `cmd_speed_mps`, `act_speed_mps`, `speed_error_mps`, `cmd_yaw_rps` | Commanded against actual speed. | | `battery_sample` | `mission_idx`, `series`, `t_s`, `state_of_charge`, `cell_temperature_c` | Battery state over the session. | | `session_events` | event: `name`, `description`, `status`, `tags`, `recording_key`, `mission_idx`, `t_s`, `metrics` | Mission boundaries and clip references, one per moment worth marking. | | `session_summary` | `n_missions_detected`, `n_missions_ground_truth`, `detection_correct`, `boundary_error_s`, `total_duration_s`, `active_duration_s`, `total_distance_m`, `total_gnss_distance_m`, `mean_speed_mps` | Headline numbers for the whole session. | | `session_inventory` | `start_ns`, `end_ns`, `files`, `sizes_bytes`, `formats` | What the session's mcap snippets were cut from. | ### What you see on a test | Metric | Template | What it shows | | --------------------- | ----------------------------- | ------------------------------------------------------------------------ | | Speed Over Time | custom (`custom_line.liquid`) | Ground speed over the session. | | IMU Acceleration | line | IMU acceleration magnitude over the session. | | Foot Contact and Slip | table | Gait state over the session — feet down, feet slipping, peak foot force. | | Motion Phases | table | Every stopped/moving segment the session was cut into. | | Movement Snapshots | image | A frame from every start/stop transition. | | GNSS Track | table | Raw GNSS fixes over the session. | The **Events** tab carries one event per mission boundary and transition the session was cut on. The **Logs** tab holds every mcap snippet and gif clip the session produced. ### What you see on the batch | Metric | Template | What it shows | | -------------------- | -------- | ----------------------------------------------------------------------------------- | | Session Summary | table | Missions detected against ground truth, total distance, total duration, mean speed. | | GNSS Coverage | scalar | Mean share of the session with a GNSS fix, across missions. **Warns below 0.5.** | | Raw Source Inventory | table | The source recording this session's snippets were cut from. | ______________________________________________________________________ ## How the three compare | | `navigation` | `mujoco` | `session` | | --------------------------- | ------------------------------------------------------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------- | | Shape | A/B pair | A/B pair | Trended over time, no pair | | System under test | A hospital delivery robot's navigation stack | An ALOHA bimanual manipulation policy | A legged robot's field sessions | | What varies between batches | Two versions of the stack | One hyperparameter of the same checkpoint | The day the session was recorded | | Batches | 2 | 2 | 4 | | Tests per batch | 34 scenarios | 20 cube placements | 1 — the whole session | | Topics | 13 | 8 | 13 | | Metrics | 21 — 13 test, 5 batch, 3 dashboard | 20 — 9 test, 5 batch, 6 dashboard | 9 — 6 test, 3 batch | | Dashboard | Yes, by build version | Yes, by build version | None — the Sessions library is the trend view instead | | System templates covered | Nine of ten — all but `artifact` — plus two custom Liquid templates | Seven of ten — no scalar, video or artifact — plus one custom template | Four of ten, plus one custom Liquid template | | Status checks | Two, both of which can block | One, warn only | One, warn only | | Files per test | Camera clip and still, on 11 of 34 | MCAP and episode animation, on all 20 | Every mcap snippet and gif clip the session produced | | Data to download | ~7 MB | ~110 MB | ~950 MB | None is a subset of the others. The navigation demo is the wider tour of what SignalFlag renders; the MuJoCo demo is the closer look at comparing two builds of one policy; the session demo is the one to run if what you actually have is field recordings rather than a build to compare against another. ______________________________________________________________________ ## Read the config that drives it Everything either demo displays comes from a metrics config and its templates, shipped inside the package. Nothing is hardcoded into the platform, and nothing about these files is special to the demo — this is the same config you would write for your own system: Python ``` from signalflag.demo import config_path, templates_path print(config_path("navigation").read_text()) print(templates_path("navigation")) # directory of the .liquid templates it references ``` `config_path("mujoco")` and `config_path("session")` give you the other two. Three pieces matter, in this order. **Topics** describe the shape of the data your tests emit: config.resim.yml ``` topics: odom_linear_velocity: schema: x: float y: float z: float ``` **Metrics** are SQL queries over that data, each rendered by a template. A metric's `type` decides what it runs against: `test` per test, `batch` across a batch, `dashboard` across every batch in the dashboard's window. config.resim.yml ``` metrics: Robot Speed: type: test description: Ground speed of the robot over the course of the run. query_string: | SELECT 'Speed' AS group_name, timestamp / 1E9 AS "Time (s)", SQRT(POWER(x, 2) + POWER(y, 2)) AS "Speed (m/s)" FROM odom_linear_velocity; template_type: system template: line ``` Batch and dashboard metrics can join against the built-in `metadata` table, which is where the build version, experience name and status of each test live: config.resim.yml ``` metrics: Localization Error by Build Version: type: dashboard query_string: | SELECT 'Mean' AS group_name, m.build_version AS "Build Version", AVG(pd.position_diff_m) AS "Mean Localization Error (m)" FROM pose_difference pd JOIN metadata m ON pd.job_id = m.job_id WHERE m.build_version IS NOT NULL GROUP BY m.build_version ORDER BY m.build_version; template_type: system template: bar ``` **Metrics sets** group metrics so that a batch or a dashboard can run them. Note the key is `metrics sets`, with a space: config.resim.yml ``` metrics sets: Demo Metrics: # run against each batch metrics: - Robot Speed - Speed Distribution # ... Demo Trends: # powers the dashboard, which spans batches metrics: - Localization Error by Build Version # ... dashboards: SDK Demo Trends: metrics_set: Demo Trends refresh: auto day_range: 30 ``` A metric with `template_type: custom` names a `.liquid` file instead of a system template, and that file has to be synced alongside the config — which is why `templates_path()` exists next to `config_path()`. The [metrics guide](../../guides/metrics/) covers topics, every template, status checks and [custom templates](../../guides/metrics/custom-templates/) in full. ______________________________________________________________________ ## Next steps - **Emit your own data.** [Get Your First Metrics in SignalFlag](../first-external-batch/) walks through the same API the demo uses, starting from an empty script. - **Write your own metrics.** Start with the [metrics guide](../../guides/metrics/), and develop them against a debug dashboard before running a batch. - **Reference.** The [SignalFlag SDK guide](../../guides/resim-sdk/) covers the SDK's API surface in full. # Get Your First Metrics in SignalFlag In this tutorial you'll get data into SignalFlag and see real metrics in the dashboard — without Docker, without a container registry, and without running anything in SignalFlag's infrastructure. You'll use the SignalFlag Python SDK to create a batch, emit structured data from two tests, and view the results. **Time:** about 10 minutes **Before you start:** - You have a SignalFlag account. Sign up at [app.signalflag.ai/signup](https://app.signalflag.ai/signup) if you don't have one. - You have Python 3.10+ installed. - You can authenticate in a browser (the script will prompt you with a URL the first time). ______________________________________________________________________ ## What you're building The [resim-sdk-example](https://github.com/resim-ai/resim-sdk-example) repo contains a minimal Python script that: 1. Creates a batch in your SignalFlag project 1. Runs two tests, each emitting a series of `(x, y)` position data points 1. Uploads the data to SignalFlag and triggers metrics processing You'll end up with a batch in the SignalFlag app showing a line chart of `y over time` per test, and a batch-level table of min/avg/max values across both tests. ______________________________________________________________________ ## Step 1: Create a project If you don't have a SignalFlag project yet, create one: Go to [app.signalflag.ai/projects/create](https://app.signalflag.ai/projects/create) and create a project. Shell ``` resim projects create --name "my-project" --description "My project" ``` Note the project name — you'll need it in Step 4. ______________________________________________________________________ ## Step 2: Clone the example repo Shell ``` git clone https://github.com/resim-ai/resim-sdk-example.git cd resim-sdk-example ``` ______________________________________________________________________ ## Step 3: Install dependencies Shell ``` pip install -r requirements.txt ``` Or, if you're using `uv`: Shell ``` uv sync ``` This installs `resim-sdk`, the SDK under its pre-rebrand name. It is the same package as `signalflag`, and the example's `resim.*` imports keep working either way. ______________________________________________________________________ ## Step 4: Set your project name Open `main.py` and update `PROJECT_NAME` to match the project you created in Step 1: main.py ``` PROJECT_NAME = "my-project" ``` ______________________________________________________________________ ## Step 5: Look at what you're running Before running, take a look at the two files that matter: **`main.py`** — creates the batch and emits test data: main.py ``` client = DeviceCodeClient() with Batch( client=client, project_name=project_name, branch="metrics-test-branch", metrics_set_name="my metrics", metrics_config_path="resim/config.resim.yml", ) as batch: with Test(client, batch, "test 1") as test: for i in range(10): test.emit("position", {"x": float(i), "y": float(i)}, i) with Test(client, batch, "test 2") as test: for i in range(10): test.emit("position", {"x": float(i), "y": float(i * 2)}, i) ``` **`resim/config.resim.yml`** — defines the data schema and metrics: resim/config.resim.yml ``` version: 1 topics: position: schema: x: float y: float metrics: y over time: type: test query_string: select 'y', timestamp, y from position order by timestamp template_type: system template: line min/avg/max: type: batch query_string: select min(y) as min, avg(y) as avg, max(y) as max from position template_type: system template: table metrics sets: my metrics: metrics: - y over time - min/avg/max ``` The config defines one topic (`position` with `x` and `y` floats), two metrics (a per-test line chart and a batch-level table), and a metrics set that groups them. ______________________________________________________________________ ## Step 6: Run it Shell ``` python main.py # or uv run main.py ``` The first time you run it, you'll be prompted to authenticate in a browser: Output ``` Authenticating by Device Code Please navigate to: https://resim.us.auth0.com/activate?user_code=XXXX-XXXX Created batch rejoicing-aquamarine-starfish with id dc71ebee-... test 1 done test 2 done Batch done. After a few minutes, you can view your metrics here: https://app.resim.ai/projects/.../batches/... ``` Visit the URL, authenticate with your SignalFlag account, and the script will continue automatically. Subsequent runs reuse the cached token. The script exits as soon as the data is uploaded; metrics processing runs in SignalFlag and takes a couple of minutes. Running in CI For non-interactive environments (CI pipelines), replace `DeviceCodeClient` with `UsernamePasswordClient` and pass SignalFlag API credentials as secrets. Contact [info@signalflag.ai](mailto:info@signalflag.ai) if you need API credentials. Python ``` from resim.sdk.auth.username_password_client import UsernamePasswordClient client = UsernamePasswordClient(username="...", password="...") ``` ______________________________________________________________________ ## Step 7: View results in the dashboard Open the URL printed by the script, or navigate to your project in [app.signalflag.ai](https://app.signalflag.ai) and find the batch. For each test you'll see: - **y over time** — a line chart of `y` values across the 10 data points At the batch level: - **min/avg/max** — a table comparing the minimum, average, and maximum `y` values across both tests `test 2` has `y = i * 2` (steeper slope) so its values will be higher than `test 1` (`y = i`), which you'll see clearly in the batch table. ______________________________________________________________________ ## What's next You've now pushed data into SignalFlag and seen it as metrics — without any Docker or cloud infrastructure setup. The natural next steps are: - **[Adapt this to your own data](../../guides/resim-sdk/)** — swap out the `position` topic for your own data schema and write SQL metrics that match your system # How-to Guides # Getting Started with Your Own System If you haven't used SignalFlag before, start with the [tutorials](../tutorials/). They fill a demo project with real results and need no Docker or AWS setup. This section covers the same steps with your own system: your Docker images, your experiences, and your metrics. - **Connect your experiences and builds** ______________________________________________________________________ Grant SignalFlag limited access to your data in S3 and your images in ECR. [Guide](resim-data-access/) - **Install the CLI** ______________________________________________________________________ Install the SignalFlag CLI to create resources and trigger batches from your terminal. [Guide](cli/) - **Create a project** ______________________________________________________________________ Create a [project](../core-concepts/#project) to house your work and collaborate with your team. [Guide](projects/) - **Create a system** ______________________________________________________________________ Define a [system](../core-concepts/#system) to run tests against. [Guide](systems/) - **Add experiences** ______________________________________________________________________ Register your own [experiences](../core-concepts/#experience) as inputs for your tests. [Guide](adding-experiences/) - **Build your system image** ______________________________________________________________________ Package your system into a Docker image with the right inputs/outputs contract. [Guide](build-images/) - **Create test suites** ______________________________________________________________________ Create [test suites](../core-concepts/#test-suite) that pair a system with a fixed set of experiences. [Guide](test-suites/) - **(Optional) Set up CI** ______________________________________________________________________ Automatically trigger batches on pull requests using GitHub Actions or GitLab CI. [Guide](ci/) For SignalFlag to run your tests, it needs to be able to access four types of data: [experiences](../../core-concepts/#experience), [assets](../../core-concepts/#asset), [system build images](../../core-concepts/#build), and [metrics build images](../metrics-builds/). - [**Experience Data Sources**](../experience-data-sources/) ______________________________________________________________________ Configure access to experience data stored in AWS S3, Google Cloud Storage, the Foxglove Data Platform, or locally in your build container. - [**Asset Data Sources**](../assets/#prerequisites) ______________________________________________________________________ Assets use the same cloud storage sources as experiences. If you have already configured experience data access, no additional setup is needed. - [**Container Registry Access**](../container-registry-access/) ______________________________________________________________________ Configure access to container images stored in AWS ECR, Docker Hub, GitHub Container Registry, or Google Artifact Registry. ## IAM roles SignalFlag will provide you with an AWS IAM role unique to your organization that is used for various AWS APIs, depending on the features you use. This role is referenced in the setup instructions for [AWS S3](../experience-data-sources/#aws-s3), [Foxglove](../../guides/foxglove-primary-sites/#self-hosted-primary-sites) and [AWS ECR](../container-registry-access/#aws-ecr). The role will look like this: Text ``` arn:aws:iam::083585393365:role/resim-customer-yourorg.ai-prod ``` SignalFlag needs to pull your [build images](../../core-concepts/#build) and [metrics build images](../metrics-builds/) from a container registry. This page describes how to configure access for each supported registry. ## AWS ECR To use images hosted in AWS Elastic Container Registry (ECR) with SignalFlag, you can set up access to an existing ECR repository, or create a new ECR repository. If you're creating a new repository, we recommend creating it in `us-east-1` as that's where our app runs. You can follow [the AWS documentation for creating ECR repositories](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-create.html). The repository needs a policy applied so that SignalFlag components can pull images from it. Use your customer-specific role as described in [IAM Roles](../resim-data-access/#iam-roles). ### New repository, or existing repository without policy JSON ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "AllowSignalFlagToPull", "Effect": "Allow", "Principal": { "AWS": ["arn:aws:iam::083585393365:role/resim-customer-yourorg.ai-prod"] }, "Action": ["ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer"] } ] } ``` ### Existing repository with policy If you have an existing ECR repository with images you would like to test, and that repository already has a policy set, add the below statement to the repository's policy. Check whether you are using a generic role or a customer-specific role as described in [IAM Roles](../resim-data-access/#iam-roles). JSON ``` "Statement": [{ "Sid": "AllowSignalFlagToPull", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::083585393365:role/resim-customer-yourorg.ai-prod" ] }, "Action": [ "ecr:BatchGetImage", "ecr:GetDownloadUrlForLayer" ] }] ``` ## Docker Hub You can use an existing repository or [create a new one](https://docs.docker.com/docker-hub/repos/create/). Once it's ready, you will need to [set up a new service account](https://docs.docker.com/docker-hub/service-accounts/#creating-a-new-service-account) and share the Docker ID and personal access token (PAT) securely with us. Docker Image Pull Rate We use a mechanism to pull and mirror your container images securely inside our platform. This means image retrieval is quicker, and also means that we only make requests to Docker Hub when you publish images we haven't yet mirrored. Thus the limit of 5000 pulls per day Docker imposes should be more than enough. ## GitHub Container Registry To use SignalFlag with GitHub Container Registry (`ghcr.io`), [create a Personal Access Token (classic)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-personal-access-token-classic) with access to the relevant repository and the `read:packages` scope. Share the token and your GitHub username with your contact at SignalFlag and we will configure secure storage and automated use of your credentials. To restrict the access this token grants, create a new GitHub account and add it as an [outside collaborator](https://docs.github.com/en/organizations/managing-user-access-to-your-organizations-repositories/managing-outside-collaborators/adding-outside-collaborators-to-repositories-in-your-organization) with access only to the relevant repository/repositories. ## Google Artifact Registry You can use an existing repository or [create a new one](https://cloud.google.com/artifact-registry/docs/docker). Once it's ready, you will need to [set up a new service account](https://cloud.google.com/iam/docs/service-accounts-create) and [generate a key](https://cloud.google.com/iam/docs/keys-create-delete) for it. SignalFlag needs the service account to have `Artifact Reader` access to the repository in question. To grant SignalFlag access, please share a base64 encoded version of the `key.json`, [which can be used to login to the artifact registry](https://cloud.google.com/artifact-registry/docs/docker/authentication#json-key). ## AWS GovCloud or private cloud solutions Please get in touch with SignalFlag if you would like to discuss other, more bespoke, access patterns or if you are a user of AWS GovCloud, which we also support. SignalFlag supports multiple sources for [experience](../../core-concepts/#experience) data. This page describes how to configure access for each supported source. ## AWS S3 You can use an existing bucket where you already have input data, or create a new one for SignalFlag. If you create a new bucket, we recommend creating it in the `us-east-1` region as that's where our app runs. You can follow the [AWS documentation for creating a bucket](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html). We don't require any specific settings except that the data isn't stored in [Glacier archive](https://aws.amazon.com/s3/storage-classes/glacier/). Whether you have an existing bucket, or have created one specifically for use with SignalFlag, you need to give a SignalFlag IAM role access to read from the bucket, as described in [IAM Roles](../resim-data-access/#iam-roles). The IAM role is used by components of the SignalFlag platform if they need to fetch Experience data and looks like this: Text ``` arn:aws:iam::083585393365:role/resim-customer-yourorg.ai-prod ``` ### New bucket, or bucket without existing policy The following policy needs to be applied to your bucket. Make sure you replace the `bucket-name` placeholder value with the name of your bucket, and the `Principal` with your customer-specific role as described above. JSON ``` { "Version": "2012-10-17", "Id": "SignalFlagAccess", "Statement": [ { "Sid": "SignalFlagAccess", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::083585393365:role/resim-customer-yourorg.ai-prod" }, "Action": ["s3:GetObject", "s3:ListBucket"], "Resource": ["arn:aws:s3:::bucket-name", "arn:aws:s3:::bucket-name/*"] } ] } ``` ### Existing bucket with policy If your bucket already has a bucket policy set, update the policy to add the following statement. Make sure you replace the `bucket-name` placeholder value with the name of your bucket, and the `Principal` with your customer-specific role as described above. JSON ``` "Statement": [ { "Sid": "SignalFlagAccess", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::083585393365:role/resim-customer-yourorg.ai-prod" }, "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::bucket-name", "arn:aws:s3:::bucket-name/*" ] } ] ``` ### Using S3 experiences When creating experiences, use the `s3://` prefix for your location: Bash ``` resim experiences create \ --project "my-project" \ --name "My S3 Experience" \ --locations "s3://my-bucket/path/to/experience/" ``` Important Any location that points to an S3 bucket must be prefixed with `s3://` rather than the `https://...` form. ## Google Cloud Storage You can use an existing GCS bucket or create a new one for SignalFlag. If creating a new bucket, we recommend using the `us-east1` region for optimal performance. ### Setting up access To grant SignalFlag access to your GCS bucket, you need to create a service account and share its credentials with us. 1. **Create a service account** following the [Google Cloud documentation](https://cloud.google.com/iam/docs/service-accounts-create). 1. **Grant the service account read access** to your bucket. The service account needs the `Storage Object Viewer` role on the bucket containing your experiences. 1. **Generate a key** for the service account following the [Google Cloud documentation](https://cloud.google.com/iam/docs/keys-create-delete). 1. **Share the key with SignalFlag**. Provide the contents of the `key.json` file to your SignalFlag contact. We will configure secure storage and automated use of your credentials. ### Using GCS experiences When creating experiences, use the `gs://` prefix for your location: Bash ``` resim experiences create \ --project "my-project" \ --name "My GCS Experience" \ --locations "gs://my-gcs-bucket/path/to/experience/" ``` ## Foxglove Data Platform SignalFlag integrates with the [Foxglove Data Platform](https://foxglove.dev/data-platform), allowing you to use recordings stored in Foxglove as experience data for your tests. ### Setting up access Foxglove access is configured per SignalFlag project using an API key. 1. **Generate an API key** in your Foxglove organization settings. Navigate to your [Foxglove settings](https://app.foxglove.dev/~/settings) and create a new API key with read access to the recordings you want to use. 1. **Share the API key with SignalFlag**. Provide the API key to your SignalFlag contact along with the SignalFlag project you want to associate it with. We will configure secure storage and automated use of your credentials. One API key per project Each SignalFlag project can have one Foxglove API key associated with it. If you need to access recordings from multiple Foxglove organizations, please contact us to discuss your requirements. ### Using Foxglove experiences When creating experiences, use the `foxglove://` prefix with the recording ID: Bash ``` resim experiences create \ --project "my-project" \ --name "My Foxglove Experience" \ --locations "foxglove://rec_abc123def456" ``` You can find the recording ID in the Foxglove web app when viewing a recording. ## Local experiences Experience data can also be stored locally within your [build container](../../core-concepts/#build). This is useful when experience data is tightly coupled to changes in your system and lives in the same repository. ### Providing access to local experiences in your container In your [experience build](../build-images/) Dockerfile, copy the files from your repository to your container: Dockerfile ``` COPY //// /experiences/ ``` ### Creating local experiences When creating a local experience, provide the path where the data will be located inside the container: Bash ``` resim experiences create \ --project "my-project" \ --name "My Local Experience" \ --locations "/experiences/my-scenario/" ``` ### Programmatically accessing local experiences For local experiences, SignalFlag does not perform any automatic file copying. Instead, the `/tmp/resim/test_config.json` file will contain an `experienceLocation` field that you can use to locate the experience data: Python ``` import json # Read the test_config.json file try: with open("/tmp/resim/test_config.json", "r") as f: test_config = json.load(f) except FileNotFoundError: raise FileNotFoundError("test_config.json not found in /tmp/resim") # Get the experience location experience_location = test_config["experienceLocation"] ``` Cloud vs Local experiences For cloud-based experiences (S3, GCS, Foxglove), SignalFlag automatically copies the experience files into `/tmp/resim/inputs/` before your build runs. For local experiences, files remain at their original location and you must access them directly using the path from `test_config.json`. # Install the CLI SignalFlag has released an [open source repository](https://github.com/resim-ai/api-client) for the SignalFlag Command Line Interface, which can be used to easily access our API endpoints. Follow the [installation](https://github.com/resim-ai/api-client#installation) and [authentication](https://github.com/resim-ai/api-client#installation) instructions in that repo before continuing to the subsequent articles. The first thing for us to do is to create a [project](../../core-concepts/#project) to work in. As discussed in the Core Concepts, we advise you use a project to reflect an entire stack, but otherwise, the project is arbitrary. ## Creating a project You can create a project using the SignalFlag CLI. Bash ``` resim projects create --name "autonomy-stack" --description "Our autonomy stack" ``` ## Using the CLI with projects All further commands will expect you to declare a project, for example: Bash ``` resim experiences create --project "autonomy-stack" --name "my experience" --description "My experience description" --locations "s3://..." ``` In order to make this a better user experience, you can set a default project: Bash ``` resim projects select "autonomy-stack" ``` Once selected, the previous command can be written: Bash ``` resim experiences create --name "my experience" --description "My experience description" --locations "s3://..." ``` To find out what project you have selected and to select another, you can use: Bash ``` resim projects list ``` ## Navigating between projects In the [SignalFlag app](https://app.signalflag.ai), the project name is displayed in the top right hand corner. You can select your project via a dropdown. Warning This dropdown is currently not persisted through user sessions: the app will, by default, auto-select the most recently created project. If you have feedback on this feature or requests for more functionality around projects, feel free to reach out to info@signalflag.ai Next, we will create a [system](../../core-concepts/#system) in order to define a particular software stack to test. ## Creating a system You can create a system using the SignalFlag CLI. Bash ``` resim systems create --project "autonomy-stack" --name "perception" \ --description "The perception subsystem of our autonomy stack" \ --architecture "amd64" --build-vcpus 4 --build-memory-mib 16384 --build-gpus 0 ``` The above example details some common resource requirements. Sensible defaults are provided for those unspecified. For a full list of available flags and their defaults, run `resim systems create --help`. ## Architectures SignalFlag supports both the `amd64` and `arm64` architecture. This can be specified at the system level via the `--architecture` flag. ## Listing systems via CLI To list all the systems in your project: Bash ``` resim systems list --project "autonomy-stack" ``` To test our [system](../../core-concepts/#system) we need some [experiences](../../core-concepts/#experience). ## Prerequisites Before creating experiences, you need to configure SignalFlag to access your experience data. See the [Experience Data Sources](../experience-data-sources/) guide for instructions on setting up access to: - [AWS S3](../experience-data-sources/#aws-s3) - [Google Cloud Storage](../experience-data-sources/#google-cloud-storage) - [Foxglove Data Platform](../experience-data-sources/#foxglove-data-platform) - [Local paths](../experience-data-sources/#local-experiences) in your build container ## Experience locations Each experience can have multiple locations. In many cases, a single location is sufficient, but it can be helpful to store MCAPs recorded from different sources in different locations. [Assets](../assets/) that are common across all (or many) experiences are also supported in SignalFlag via a separate path. SignalFlag supports the following location prefixes: | Source | Prefix | Example | | -------------------- | ------------- | ------------------------------------- | | AWS S3 | `s3://` | `s3://my-bucket/experiences/exp_001/` | | Google Cloud Storage | `gs://` | `gs://my-bucket/experiences/exp_001/` | | Foxglove | `foxglove://` | `foxglove://rec_abc123def456` | | Local | None | `/experiences/my-scenario/` | For cloud storage locations (S3, GCS), each location should have a unique prefix containing the input files. Anything contained in that prefix will be treated as the same experience and downloaded into your tasks. We recommend using a prefix structure that represents your SignalFlag projects: Text ``` 📂 . ├── 📂 project_a │ └── 📂 experiences │ └── 📂 experience_56ccce99 │ └── ⚙️ test_file │ └── 📂 project_b └── 📂 experiences └── 📂 experience_679bef80 └── ⚙️ test_file ``` ### How experience files are accessed For cloud-based experiences (S3, GCS, Foxglove), files are copied into `/tmp/resim/inputs/` when the experience build runs. If you use *multiple locations*, all files are copied directly into the `inputs` directory. Take care to ensure there are no file or directory name clashes at the root of the locations. Avoiding file clashes with multiple locations If your experience has two locations with the same directory structure: Text ``` s3://your-bucket/experience1 s3://your-bucket/experience2 └── mcaps └── mcaps └── log.mcap └── log.mcap ``` The `mcaps/log.mcap` file will clash - one will overwrite the other in `/tmp/resim/inputs/`. **Solution:** Change the path or filename to ensure uniqueness. For example, use distinct directory names: Text ``` s3://your-bucket/experience1 s3://your-bucket/config └── mcaps └── params.yaml └── log.mcap ``` This results in: Text ``` /tmp/resim/inputs/ ├── mcaps/ │ └── log.mcap └── params.yaml ``` [Experience caching](../../guides/experience-caching/) is enabled by default, so we only copy files that have been added or changed since the last run. For local experiences, files remain at their original location and you access them using the path from `/tmp/resim/test_config.json`. See [Local Experiences](../experience-data-sources/#local-experiences) for details. ## Experience timeouts Each experience has a timeout. This is the maximum amount of time that a build can run for a given experience. If the build does not finish running before the timeout is reached, it will be sent a SIGTERM and then a SIGKILL after a grace period. The default timeout is 1 hour, but you can specify a different timeout when creating an experience. ## Creating an experience You can create new experiences like so: Bash ``` resim experiences create \ --project "my-project" \ --name "Experience_56ccce99" \ --description "In this experience, we test that our robot can navigate to the goal without colliding with any other agents." \ --locations "s3://my-experiences-bucket/project_hal9000/experiences/experience_56ccce99/" \ --systems "perception" "localization" \ --tags "navigation" "collision-avoidance" ``` This command should return a message containing the UUID of the created experience, and it should now be browsable through the [SignalFlag app](https://app.signalflag.ai). The `--systems` flag enables you to specify any [systems](../../core-concepts/#system) that this experience is compatible with. You can specify multiple systems by providing multiple system names or IDs. The `--tags` flag allows you to apply tags to the experience for better organization and filtering. You can specify multiple tags by providing multiple tag names or IDs. Optional flags are: Bash ``` --timeout "1h10m" # the maximum expected duration of the container \ --environment-variable "SYSTEM_CONFIGURATION=Foo" # many allowed \ --environment-variable "AN=other" \ --profile "left-arm" # a docker compose profile to select the subset of compose services to run \ ``` ### Examples with different sources **AWS S3:** Bash ``` resim experiences create \ --project "my-project" \ --name "S3 Experience" \ --description "An experience stored in S3" \ --locations "s3://my-bucket/experiences/exp_001/" ``` **Google Cloud Storage:** Bash ``` resim experiences create \ --project "my-project" \ --name "GCS Experience" \ --description "An experience stored in GCS" \ --locations "gs://my-gcs-bucket/experiences/exp_001/" ``` **Foxglove:** Bash ``` resim experiences create \ --project "my-project" \ --name "Foxglove Experience" \ --description "An experience stored in Foxglove" \ --locations "foxglove://rec_abc123def456" ``` **Local:** Bash ``` resim experiences create \ --project "my-project" \ --name "Local Experience" \ --description "An experience stored locally in the build container" \ --locations "/experiences/my-scenario/" ``` ## Updating an experience You can update various aspects of an experience, including its systems and tags: Bash ``` resim experiences update \ --project "my-project" \ --experience "Experience_56ccce99" \ --name "Updated Experience Name" \ --description "Updated description" \ --locations "s3://my-experiences-bucket/project_hal9000/experiences/updated_experience/" \ --timeout "2h" \ --systems "perception" "planning" \ --tags "updated-tag" "new-functionality" ``` When updating systems or tags, the provided lists will replace the existing systems and tags associated with the experience. You can also add or remove individual systems from an experience: Bash ``` resim experiences add-system \ --project "my-project" \ --system "" \ --experience "Experience_56ccce99" resim experiences remove-system \ --project "my-project" \ --system "" \ --experience "Experience_56ccce99" ``` Similarly you can add and remove individual tags: Bash ``` resim experiences tag \ --project "my-project" \ --tag "" \ --experience "Experience_56ccce99" resim experiences untag \ --project "my-project" \ --tag "" \ --experience "Experience_56ccce99" ``` To share common data across [builds](../../core-concepts/#build) and [experiences](../../core-concepts/#experience), SignalFlag supports [assets](../../core-concepts/#asset). ## Why use assets? Consider a simulation stack that requires a large HD map, 3D environment meshes, and ML model weights. Every experience in your test suite needs this data, but none of it changes between experiences, it only changes when your simulation environment or models are updated. Without assets, you have two options, neither ideal: - **Bake the data into your Docker image** - This can bloat the image, slow down CI builds, and force a full image rebuild every time a map or model is updated even if the code hasn't changed - **Duplicate the data across experience locations** - This can waste storage, make updates error-prone, and increase data transfer costs *Assets* solve this by letting you define shared data once, revision it independently of your code, and mount it into any build at runtime. Your Docker image stays lean, your experiences stay focused on per-test inputs, and your shared data is versioned and cached. ``` flowchart LR Asset["Asset\n(e.g. HD Map)"] -->|"linked to"| Build[Build] Experience["Experience\n(e.g. Scenario)"] -->|"run with"| Test[Test] Build -->|"run in"| Test Test -->|"part of"| Batch[Test Batch] subgraph container ["Container at Runtime"] AssetMount["/tmp/resim/assets/world/"] InputMount["/tmp/resim/inputs/"] end Asset -->|"mounted at"| AssetMount Experience -->|"mounted at"| InputMount ``` ## Prerequisites Assets are stored in cloud blob storage, using the same data sources as experiences. If you have already configured access for your experience data, no additional setup is needed. If not, see the [Experience Data Sources](../experience-data-sources/) guide for instructions on setting up access to: - [AWS S3](../experience-data-sources/#aws-s3) - [Google Cloud Storage](../experience-data-sources/#google-cloud-storage) ## Asset locations Each asset can have multiple locations. SignalFlag supports the same location prefixes as experiences: | Source | Prefix | Example | | -------------------- | ------- | ---------------------------------- | | AWS S3 | `s3://` | `s3://my-bucket/assets/sim-world/` | | Google Cloud Storage | `gs://` | `gs://my-bucket/assets/sim-world/` | | Local | None | `/assets/sim-world/` | For cloud storage locations, each location should have a unique prefix containing the asset files. Anything within that prefix will be synced. We recommend a prefix structure that mirrors your SignalFlag projects: Text ``` 📂 . └── 📂 my-project └── 📂 assets ├── 📂 sim-world │ ├── 🗺️ hd_map.bin │ └── 📂 meshes │ └── 🏙️ urban_env.glb └── 📂 ml-models └── 🤖 perception_v3.onnx ``` ## How asset files are accessed For cloud-based assets, files are synced from cloud storage and mounted **read-only** at `/tmp/resim/assets//` when a build runs. The `mountFolder` is specified when creating the asset and determines where inside the container the files appear. For example, an asset with `mountFolder` set to `world` and location `s3://my-bucket/assets/sim-world/` results in: Text ``` /tmp/resim/assets/world/ ├── hd_map.bin └── meshes/ └── urban_env.glb ``` Your entrypoint script can then read from `/tmp/resim/assets/world/` just as it would from any local directory. If an asset has multiple locations, all files are synced into the same mount directory. Take care to avoid file or directory name clashes at the root of the locations, just as with [experience locations](../adding-experiences/#experience-locations). [Asset caching](../../guides/asset-caching/) is enabled by default, so only files that have been added or changed since the last run are downloaded. Note Because assets are shared across potentially concurrent jobs, the `/tmp/resim/assets//` directory is mounted **read-only**. If your workload needs to modify asset data, copy the files to a writable location first. ## Creating an asset You can create a new asset like so: Bash ``` resim assets create \ --project "my-project" \ --name "Simulation World" \ --description "HD map and 3D meshes for urban simulation environment" \ --locations "s3://my-assets-bucket/my-project/assets/sim-world/" \ --mount-folder "world" \ --version "1.0.0" ``` This command returns the UUID and revision number of the created asset. The asset starts at *revision* `0`. Revisions are automatically incremented. | Flag | Required | Description | | ---------------- | -------- | --------------------------------------------------- | | `--name` | Yes | A human-readable name for the asset | | `--description` | Yes | A description of what the asset contains | | `--locations` | Yes | One or more cloud storage locations | | `--mount-folder` | Yes | The subdirectory name under `/tmp/resim/assets/` | | `--version` | Yes | A version string (e.g. `1.0.0`, `v2`, a commit SHA) | | `--cache-exempt` | No | If set, the asset will not be cached between runs | ### Examples with different sources **AWS S3:** Bash ``` resim assets create \ --project "my-project" \ --name "ML Models" \ --description "Perception model weights" \ --locations "s3://my-bucket/assets/ml-models/" \ --mount-folder "models" \ --version "v3.1" ``` **Google Cloud Storage:** Bash ``` resim assets create \ --project "my-project" \ --name "Calibration Data" \ --description "Sensor calibration parameters" \ --locations "gs://my-gcs-bucket/assets/calibration/" \ --mount-folder "calibration" \ --version "2024-Q4" ``` ## Revising an asset When the underlying data changes (e.g. a new map version or updated model weights), we create a **new revision** of the asset rather than modifying it in place. This preserves an immutable history of every version of the asset data, enabling reproducibility. Bash ``` resim assets revise \ --project "my-project" \ --asset "Simulation World" \ --locations "s3://my-assets-bucket/my-project/assets/sim-world-v2/" \ --version "2.0.0" ``` Each revision increments the revision number (0, 1, 2, ...). The `--version` flag is required when revising -- it should reflect the new state of the data (e.g. a semantic version, date, or commit SHA). You can also optionally update the `mountFolder`: Bash ``` resim assets revise \ --project "my-project" \ --asset "Simulation World" \ --locations "s3://my-assets-bucket/my-project/assets/sim-world-v2/" \ --mount-folder "world-v2" \ --version "2.0.0" ``` You can list all revisions of an asset: Bash ``` resim assets list-revisions \ --project "my-project" \ --asset "Simulation World" ``` ## Updating asset metadata To update an asset's name or description without changing its data: Bash ``` resim assets update \ --project "my-project" \ --asset "Simulation World" \ --name "Urban Simulation World" \ --description "HD map and 3D meshes for the urban simulation environment (downtown)" ``` ## Associating assets with builds Assets must be explicitly **linked to a build** before they are available at runtime. This association specifies which asset revision the build should use. ### At build creation time (recommended) The simplest approach is to attach assets when you create the build. The `resim builds create` command accepts an `--assets` flag with a comma-separated list of asset references: Bash ``` resim builds create \ --image "123456789.dkr.ecr.us-east-1.amazonaws.com/my-sim:abc1234" \ --version "abc1234" \ --name "Autonomy Build" \ --description "Path planning v2.3" \ --branch "my-branch" \ --project "my-project" \ --system "full-stack" \ --auto-create-branch \ --assets "Simulation World:2,ML Models:5" ``` This creates the build and links the specified asset revisions in a single step. Each asset will be mounted at its respective `mountFolder` when the build runs. Each entry in the comma-separated list can take one of four forms where the `uuid` is the revision ID: | Format | Example | Behaviour | | --------------- | -------------------- | ----------------------------- | | `name` | `Simulation World` | Links the **latest** revision | | `name:revision` | `Simulation World:2` | Links a specific revision | | `uuid` | `a1b2c3d4-...` | Links the **latest** revision | | `uuid:revision` | `a1b2c3d4-...:2` | Links a specific revision | When the revision is omitted, the latest revision at the time of build creation is used as the static revision for that build. We recommend pinning explicit revision numbers in CI/CD for reproducibility. ### Adding assets to an existing build If you need to add assets to a build after it has been created, use `resim builds add-assets`: Bash ``` resim builds add-assets \ --project "my-project" \ --build "" \ --assets "Simulation World:2" ``` ### Listing assets for a build Bash ``` resim builds list-assets \ --project "my-project" \ --build "" ``` ### Removing assets from a build Bash ``` resim builds remove-assets \ --project "my-project" \ --build "" \ --asset "Simulation World" ``` ### Querying which builds use an asset Bash ``` resim assets list-builds \ --project "my-project" \ --asset "Simulation World" ``` ## Archiving assets Assets that are no longer needed can be archived. Archived assets are hidden from default list views but are not deleted. Bash ``` resim assets archive \ --project "my-project" \ --asset "Simulation World" ``` To restore an archived asset: Bash ``` resim assets restore \ --project "my-project" \ --asset "Simulation World" ``` To include archived assets in a list: Bash ``` resim assets list \ --project "my-project" \ --archived ``` ## Recommended CI/CD workflow We recommend associating assets with builds as part of your CI/CD pipeline. This ensures that every build is pinned to specific asset revisions, making test results fully reproducible. A typical workflow: 1. **Create and revise assets independently of your code** - When a map, model, or calibration dataset is updated, create a new asset revision. This can be done manually or in a separate pipeline triggered by changes to your data repository 1. **Pin asset revisions in CI** - When your CI pipeline creates a new build, associate it with the desired asset revisions. Store the asset revision numbers as CI variables or derive them from a configuration file in your repository 1. **Update asset revisions deliberately** - When you want to test against a new map or model, bump the revision number in your CI configuration and let the pipeline pick it up ### GitHub Actions example Extending the [GitHub Actions workflow](../ci/github/), you can pass asset revisions directly when creating the build using the `--assets` flag: YAML ``` - name: Create build with assets and launch batch env: RESIM_CLIENT_ID: ${{ secrets.RESIM_CLIENT_ID }} RESIM_CLIENT_SECRET: ${{ secrets.RESIM_CLIENT_SECRET }} run: | resim builds create \ --project "my-project" \ --system "full-stack" \ --image "${{ steps.docker_meta.outputs.tags }}" \ --version "${{ github.sha }}" \ --branch "${{ github.head_ref || github.ref_name }}" \ --auto-create-branch \ --assets "Simulation World:${{ vars.SIM_WORLD_REVISION }},ML Models:${{ vars.ML_MODELS_REVISION }}" ``` Store `SIM_WORLD_REVISION` and `ML_MODELS_REVISION` as [GitHub Actions variables](https://docs.github.com/en/actions/learn-github-actions/variables) so they can be updated without changing the workflow file. Tip Separating asset revisions from code changes means you can update your simulation environment without rebuilding your Docker image, and vice versa. This significantly speeds up iteration when only data has changed. ## End-to-end example Here is a complete walkthrough using a simulation world asset shared across many experiences. ### 1. Create the asset Bash ``` resim assets create \ --project "my-project" \ --name "Simulation World" \ --description "Downtown urban environment: HD map, 3D meshes, traffic light configs" \ --locations "s3://my-assets-bucket/my-project/assets/sim-world/" \ --mount-folder "world" \ --version "1.0.0" ``` ### 2. Create a build with the asset attached Bash ``` resim builds create \ --image "123456789.dkr.ecr.us-east-1.amazonaws.com/my-sim:abc1234" \ --version "abc1234" \ --name "Autonomy Build" \ --description "Path planning v2.3" \ --branch "my-branch" \ --project "my-project" \ --system "full-stack" \ --auto-create-branch \ --assets "Simulation World:0" ``` ### 3. Run a batch Bash ``` resim batches create \ --project "my-project" \ --build "Autonomy Build" \ --test-suite "Nightly Regression" ``` When each test in the batch runs, the simulation world data is automatically mounted at `/tmp/resim/assets/world/`. ### 4. Access asset data in your entrypoint Python ``` import pathlib map_path = pathlib.Path("/tmp/resim/assets/world/hd_map.bin") mesh_dir = pathlib.Path("/tmp/resim/assets/world/meshes/") map_data = map_path.read_bytes() meshes = list(mesh_dir.glob("*.glb")) ``` Every experience in the test suite gets the same simulation world without any duplication. In order to run our [experiences](../../core-concepts/#experience) against our [system](../../core-concepts/#system) we need a system [build](../../core-concepts/#build) available as a container image hosted somewhere we can access. ## Prerequisites For your build image to be usable in SignalFlag, it needs to be hosted in an image registry that we are granted permission to access. Uploading the image to a registry is documented below, but you should ensure you have followed [SignalFlag Data Access](../resim-data-access/) to set up an AWS ECR repository to work with SignalFlag. Furthermore, you must have permission to push to that registry. If you're using an IAM user, the easiest way to get the needed permissions is to attach the AWS-managed IAM policy called `AmazonEC2ContainerRegistryPowerUser` to the user. You can do this through the [AWS browser UI](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html#users_change_permissions-add-console) (see "Adding permissions by attaching policies directly to the user") or using the CLI: Bash ``` aws iam attach-user-policy --policy-arn "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser" --user-name ``` If you're using an IAM role instead, it's a [similar operation](https://docs.aws.amazon.com/IAM/latest/UserGuide/roles-managingrole-editing-console.html#roles-modify_permissions-policy). If you'd prefer to be stricter with your permissions, you really only need the permissions from the policy [here](https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-push.html#image-push-iam) rather than all of those in `AmazonEC2ContainerRegistryPowerUser`. If there is a [repository policy](https://docs.aws.amazon.com/AmazonECR/latest/userguide/repository-policies.html#repository-policy-vs-iam-policy) on the repository already, check it doesn't explicitly deny access by the IAM identity (user or role) you're using. ## Entrypoint The first step in creating a build image is to identify a shell command that you would like to run in order to test your software. If your software test involves multiple different processes, this is fine, but you may want to write a shell script or another piece of software to correctly initialize and run them. For instance, if you are using ros2 with gazebo, you might write a launch file for this and then your test command might be a call to a script containing something like: Bash ``` #!/bin/bash # run_sim_test.sh # Source our ros2 distribution source /opt/ros//setup.bash # Navigate to, build, and source the ros2 package we're building cd colcon build source install/setup.bash # Launch our test sim ros2 launch my_package launch_test_sim.py ``` But really, any script which your docker container can execute will suffice. Once you have identified such a command, your goal is to create a [Dockerfile](https://docs.docker.com/engine/reference/builder/) which builds an image that has everything it needs to run your test command. Then, that command should be denoted as the `ENTRYPOINT` of that Dockerfile. ## Inputs and outputs It is typically desirable to run the same *Build* with a variety of different *Experiences* (sets of input files). Therefore, the SignalFlag worker will mount a volume containing the inputs for each run at `/tmp/resim/inputs/`. Using the ros2 example above, the `launch_test_sim.py` launch file might reference ROS2 Parameter YAML files stored in `/tmp/resim/inputs/`. You would then put these YAML files in an Amazon S3 bucket and add them as experiences as described [here](../adding-experiences/). Again, there's nothing special about using ros2 in this case and any file stored correctly in the experiences bucket will be available there. In addition to experience inputs, any [assets](../../core-concepts/#asset) associated with the build are mounted read-only at `/tmp/resim/assets//`. See the [assets setup guide](../assets/) for details on creating assets and linking them to builds. The SignalFlag platform additionally places a file called `test_config.json` in the `/tmp/resim/` directory. This JSON contains the following fields: JSON ``` { "experienceID": "the unique id of the experience", "experienceName": "the name of the experience", "experienceLocation": "the location of the experience", "branchName": "the name of the branch", "buildVersion": "the version of the build running", "buildID": "the unique id of the build running", "systemName": "the name of the system the build running belongs to", "systemID": "the unique id of the system" } ``` This config file can be helpful for determining the experience location, if the experience is not stored in S3 but within the build image, by reading `/tmp/resim/test_config.json` If custom parameters are used in the test command, they should be stored in the `/tmp/resim/parameters.json` file. When the test command is run, it will produce some sort of valuable outputs that can be accessed through the SignalFlag app. Any such outputs (e.g. console logs or rosbags) should be stored in the `/tmp/resim/outputs/` directory. If this is done correctly, you will be able to download them using a link in the SignalFlag app. If you place files in nested subdirectories beneath `/tmp/resim/outputs/`, SignalFlag will archive those nested directories into `.zip` files before uploading them. If you want log files to remain unarchived, place them in `/tmp/resim/outputs/logs/`. Files in that directory are uploaded as logs without being zipped and are browsable on the web app. There are two exceptions to the zipping of nested subdirectories. First, if a subdirectory contains only a single file, that file is uploaded individually rather than being zipped. Second, any image or video referenced in your emissions file is uploaded individually — with its original path preserved — so that metrics can resolve it by filename; if such a subdirectory contains other files as well, it is still zipped in addition. This means you can reference images and videos in your metrics even when they live in nested subdirectories, without having to place them at the top level of `outputs/` or in `logs/`. To avoid overwhelming uploads, at most 100 emission-referenced media files are uploaded individually per run. It can, however, be the case that you generate logs that you do not wish to upload, but wish to use for computing metrics. If you place any logs in `/tmp/resim/outputs/ignore` these will be made available in the metrics phase, but are not uploaded to the app. If your image exits `0` in the context of a SignalFlag batch, we consider it a successful run and will automatically compute the next, metrics, stage. If your image exits non-zero, we will create an `Error` status on that test and will not run metrics. It is possible to provide a custom error message that will be reported, alongside the exit code, on the SignalFlag web app by leaving a file called `error.txt` in the `/tmp/resim/outputs` folder. ## Developing an image For an example of a very simple build image please refer to [this directory](https://github.com/resim-ai/open-core/tree/main/resim/examples/sim_runner) which contains directions and example scripts to create and test run a build image. In this case, we use SignalFlag's open source simulator to run a simple scripted simulation of two quadcopter drones. The example uses a build script to build our test executable and then uses Docker's `COPY` command in its `Dockerfile` to move the resulting executable into the finished image. The general rule is that the Docker image must contain everything your test needs to run *except* for the the inputs (i.e. the experience). In order to develop a working test image, a good workflow is generally to iterate on it locally. This involves the following steps: ### 1. Inputs and Outputs Create local `inputs/` and `outputs/` folders with the `inputs/` folder containing a set of representative input files. **If you are doing this from within a Docker container**, creating volumes and copying content to them is probably better. Here's how you would do that: Bash ``` # Create a helper container to copy content over docker container create \ --name tmp \ --volume test_inputs:/inputs \ busybox # Copy test data to the volume docker cp path/to/test/inputs tmp:/inputs # Remove the helper container docker rm tmp ``` ### 2. Build Assuming your Dockerfile is in a folder called `runner/`, build your Docker image using: Bash ``` docker build runner/ -t test-runner-image:latest ``` Note that the name `test-runner-image` here is arbitrary. You can use whatever name you would like as long as you're consistent when developing because the image will be re-tagged with `docker tag` before it's pushed. ### 3. Run Try out a run with your Docker image using: Bash ``` docker run \ --volume /absolute/path/to/inputs:/tmp/resim/inputs \ --volume /absolute/path/to/outputs:/tmp/resim/outputs \ test-runner-image:latest ``` The `--volume` flags here ensure that the inputs and outputs are available to the container at `/tmp/resim/inputs` and `/tmp/resim/outputs`. Note that `/absolute/path/to/inputs` and `/absolute/path/to/outputs` are paths on the *host* system. **If you are iterating within a Docker container**, you would use the Docker volumes approach mentioned in step 1. In this case, you would **instead** run: Bash ``` docker run \ --volume test_inputs:/tmp/resim/inputs \ --volume test_outputs:/tmp/resim/outputs \ test-runner-image:latest ``` And getting your outputs back would look like: Bash ``` # Create a helper container to copy content back: docker container create \ --name tmp \ --volume test_outputs:/outputs \ busybox # Copy outputs from the volume docker cp tmp:/outputs . # Remove the helper container docker rm tmp ``` ### 4. Iterate If any step is unsuccessful, make modifications and return to step 2. ## Pushing Once the build image is created and working, the next step is to push it to your AWS Elastic Container Registry (set up according to [SignalFlag Data Access](../resim-data-access/)) like so: Bash ``` docker tag test-runner-image .dkr.ecr..amazonaws.com/test-runner-image: aws ecr get-login-password --region | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com docker push .dkr.ecr..amazonaws.com/: ``` Note the use of a commit SHA here. This is what we recommend as a unique ID associated with your software version if you're using `git` for version control. You can get a short form of the current commit hash by running: Bash ``` git rev-parse --short HEAD ``` On a successful push, the command will report something like: Text ``` The push refers to repository [.dkr.ecr..amazonaws.com/] a9950558cded: Pushed 29563d2ab359: Pushed aa5d6a7236b9: Pushed ... bce45ce613d3: Pushed latest: digest: sha256:aa47e2d0c6dcc5d48d3c9b8c9c8a57433b2d241c27ef92aa53de885683f06487 size: 7480 ``` If you instead get something like the following, it is an indicator either that the ECR repository URL (account number or repository name) you are trying to push to is incorrect, or that you don't have the required permissions to push to it. Check the [Prerequisites](#prerequisites) for additional info. Text ``` The push refers to repository [.dkr.ecr..amazonaws.com/] a9950558cded: Retrying in 7 seconds 29563d2ab359: Retrying in 7 seconds aa5d6a7236b9: Retrying in 7 seconds ... bce45ce613d3: Waiting EOF ``` You might want to automate these build and push steps by adding them to a new or existing CI pipeline. See our articles on [Pushing Build Images from CI](../ci/) for how. ## External links - [AWS ECR authentication documentation](https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html) ## Creating a branch Each build is associated with a branch. The branch represents the particular change you're testing. A build can be thought of as one implementation of that change. A given branch may have many builds associated with it. You can manually create a branch with the CLI: Bash ``` resim branches create \ --name "my-branch" \ --type CHANGE_REQUEST \ --project "my-project" ``` It's often easier to use the `--auto-create-branch` flag when creating a build which is the approach taken below. The SignalFlag app supports three types of branch, for three common git workflows: - `CHANGE_REQUEST`: a pull or merge request, a short-lived branch - `MAIN`: a long-lived, protected branch that most change requests will land to - `RELEASE`: a release of a piece of software The `--auto-create-branch` flag will create a `CHANGE_REQUEST` branch, but if the branch is named `main` it will make it a `MAIN` branch. ## Creating a build Now that the build image has been pushed to your elastic container registry, the final step is to formally create the build by informing the SignalFlag app as to its location. This is done using the [SignalFlag CLI](../cli/). We assume that we already have a [project](../projects/) called `my-project`, a [system](../systems/) called `my-system` and a [branch](../../core-concepts/#branch) called `my-branch`. These are required for build creation. After [authenticating](https://github.com/resim-ai/api-client#authentication) we can create a new build like so: Bash ``` resim builds create \ --image ".dkr.ecr..amazonaws.com/:" \ --version "" \ --name "Autonomy Build" \ --description "Optimized path planning to avoid unexpected squirrels" \ --branch "my-branch" \ --project "my-project" \ --system "my-system" \ --auto-create-branch ``` The `--project`, `--branch`, and `--system` flags will accept either a name or an ID. When using `--auto-create-branch`, one must also pass a branch name using `--branch`. If you have selected a project via the CLI, you can omit the `--project` for convenience. This command should return a message containing the UUID of the created build, and it should now be browsable through the [SignalFlag app](https://app.signalflag.ai). Note that the image itself requires the full tagged URI, since we allow an arbitrary version to be stored that need not reflect the image tag. ## Multi-container builds You can also submit a [Docker Compose](https://docs.docker.com/reference/compose-file/) file when creating a build, instead of a single build image (`--image`). Bash ``` resim builds create \ --build-spec=my-docker-compose.yml [...] ``` See [the guide](../../guides/multi-container-builds/) for more details on how to set up a Compose file to execute your code during the **Experience** phase. When executing multiple containers like this, all containers have access to the `/tmp/resim` directory for inputs and outputs, although it is certainly common for only a subset of containers to actually read/write there as necessary. All images specified in a Compose file must be in an accessible Container Registry as per above. The Compose file used is passed via the API at build creation time and does not need to be hosted anywhere. You may not need this If your emitted data from your experience build does not require post-processing you can skip this step and go straight to [Metrics](../../guides/metrics/). A metrics build is only needed if your workflow requires post-processing of raw experience build outputs before metrics can be computed. This document describes how the SignalFlag platform supports executing metrics builds. ## Metrics in the SignalFlag platform Similar to [builds](../../core-concepts/#build), customers can register a **Metrics Build** with SignalFlag, which will be executed after each test. A metrics build is also a container image, with a similar contract to [builds](../build-images/#1-inputs-and-outputs). After each test, the SignalFlag platform runs the metrics build with the `/tmp/resim/inputs` directory populated with: - a subdirectory, `experience`, which contains any files as part of the experience - a subdirectory, `logs`, which contains any files generated as logs from the execution of the simulation. The metrics build post-processes these and writes its results as emissions with the [SignalFlag SDK](../../guides/resim-sdk/), to a file in `/tmp/resim/outputs` whose name ends in `.resim.jsonl`, such as `emissions.resim.jsonl`. SignalFlag ingests these emissions alongside those from the experience build, and the [metrics config](../../guides/metrics/) synced to the branch computes metrics from both. Any additional files placed in the outputs directory will also be uploaded to the SignalFlag platform as logs. ## Creating metrics builds It is possible to create metrics builds in CI/CD or with the CLI tool: Bash ``` resim metrics-builds create \ --project "my-project" \ --name "My Metrics Build" \ --version \ --image \ --systems "perception" "localization" ``` Where, again, if one has a project selected, that parameter can be omitted. Similarly to builds, SignalFlag recommend the set-up of CI/CD workflows to generate metrics builds when the metrics generation code is updated. ## Systems and metrics builds Since metrics are closely tied to the test application which runs, the SignalFlag app enables metrics builds to be aligned to a given system, to enable compatibility checking at test creation time. One can register an existing metrics build against a given system: Bash ``` resim metrics-builds add-system \ --project "my-project" \ --system "my-system" \ --metrics-build-id "" ``` and remove them with: Bash ``` resim metrics-builds remove-system \ --project "my-project" \ --system "my-system" \ --metrics-build-id "" ``` ## Debugging metrics To check how your metrics render without running a batch, generate an emissions file and pass it to `signalflag metrics debug`, which renders it against a metrics set in a temporary dashboard. See [Adding your first metric](../../guides/metrics/#adding-your-first-metric). If you've gone through all of the previous articles, you are now ready to begin testing your builds with some [experiences](../../core-concepts/#experience) and generating metrics! We do this by creating and running a [**Test Suite**](../../core-concepts/#test-suite). ## Creating a test suite To create a test suite, you need to have a [system](../../core-concepts/#system) in mind, a [metrics set](../../guides/metrics/) and a set of experiences you would like to use. You can create a test suite using the [SignalFlag CLI](../cli/) with this command, assuming you already have a [project selected](../projects/#using-the-cli-with-projects) Bash ``` resim suites create \ --name "Perception Nightly Regression" \ --description "A test suite for perception system nightly regression tests" \ --system "perception" \ --metrics-set "" \ --experiences ", , ..." ``` You can `get` or `list` the test suites for that project. For example, the following command displays the latest revision: Bash ``` resim suites get \ --test-suite "Perception Nightly Regression" ``` To get a specific revision append the `--revision 1` key. To get a list of all revisions, pass `--all-revisions`. In order to `revise` the test suite, you simply need to pass the elements that you would like to change. For example, to change the experience list, you would use the following command: Bash ``` resim suites revise \ --test-suite "Perception Nightly Regression" --experiences ", , ..." ``` Warning You need to specify the exact list of experiences. The CLI will be extended in the near future to support specifying a query to describe the list of experiences. In the short term, the [SignalFlag app](https://app.signalflag.ai) supports query-based test suite creation and building. ## Running a test suite If you have a test suite, you can then run it to create a **Test Batch**: Bash ``` resim suites run \ --test-suite "Perception Nightly Regression" --build-id "" ``` If you would like to specify a particular revision, you can pass `--revision 0`. You can list the test batches that have been created from this test suite via: Bash ``` resim suites batches \ --test-suite "Perception Nightly Regression" ``` This will list all the test batches from all revisions of the test suite. Simply pass the `--revision` flag to see test batches from a specific test suite revision. ## Tracking a test batch Once you have created a test batch by running a test suite, the CLI lets you track the status of that batch. You can list all of the batch details -- as JSON -- either by passing the friendly name of the batch (e.g. rejoicing-aquamarine-starfish) or its id: Bash ``` resim batches get \ --batch-id ``` or Bash ``` resim batches get \ --batch-name ``` If you additionally pass the exit status key: Bash ``` resim batches get \ --batch-id \ --exit-status ``` The application will return only the exit status, which are: - 0: `Succeeded`: the batch has completed. This does not indicate whether the metrics have passed or failed. - 1: `Internal Error`: the CLI command errored (which is nothing to do with the test batch). - 2: `Error`: the batch has finished with an error. - 3: `Submitted`: the batch is in the submitted state. - 4: `Running`: the batch is still running. - 5: `Cancelled`: the batch has been cancelled. This is commonly used in CI to trigger a failing test. You can also list the individual tests in that batch with: Bash ``` resim batches tests \ --batch-name ``` This will return the JSON associated with the test to track the individual status and return their IDs. The IDs are helpful to, for example, list the log files associated with that test: Bash ``` resim logs list \ --batch-id \ --test-id ``` The output of this includes pre-signed URLs that would enable you to download logs directly from the command line. In particular, every test contains the `experience-container.log` and `metrics-container.log` files which contain the console logs from your container (`experience-container.log`) and the metrics stage (`metrics-container.log`). ## Ad hoc test suites It is not always desirable to create a test suite: sometimes, you may simply want to try a few experiences on an experimental branch to make sure that you haven't broken everything. For this, the SignalFlag App supports the concept of an ad hoc test suite. That is, an unnamed test suite. You can create this in the CLI as follows: Bash ``` resim batches create \ --project "my-project" \ --build-id \ --metrics-build-id \ --experiences ", , ..." \ --experience-tags ", " ``` Use `--experiences` and/or `--experience-tags` to control which experiences are run in the batch. ## Configuration options for running test suites There are several configuration options available when one is running a test suite or creating an ad-hoc batch: - **A custom batch name:** Rather than use the auto-generated friendly name (e.g. `rejoicing-aquamarine-starfish`), you can provide your own. In the CLI, this comes from the `--batch-name ` flag. - **An allowable error rate:** By default, if any tests in the test suite `ERROR` -- that is, have a failure at the execution phase -- the SignalFlag orchestration system will mark the whole test batch as `ERROR` and not compute any batch metrics. In many cases, this is the correct default behaviour, but it can be overridden with `--allowable-failure-percent `. By default, tests which error will not have test metrics associated with them, but if you set `--allowable-failure-percent 100`, SignalFlag will attempt to compute metrics even when a test has errored. - **Parameter overrides:** One can pass key-value parameters to your build to change how it runs via a parameter override flag e.g. `--parameters "key=value"`. ## Rerunning a subset of tests in a test suite It can be frustrating if one or two tests within a test suite has a flaky error or unexpected behavior. The SignalFlag platform makes it easy to re-run just a subset of tests so that the whole suite doesn't need to be run again. In the CLI, this can be achieved with Bash ``` resim batches rerun --batch-name --test-ids "," ``` # Metrics SignalFlag's Metrics framework recently underwent a major refactor to provide a more flexible and powerful way to collect, plot, and track data in SignalFlag. At its core, Metrics works by ingesting data from various sources in your system; ROS topics, log files, or custom instrumentation, into a centralized data lake. When you emit data during a test run, it's automatically captured and stored in this data lake. After each run completes, the system automatically queries this data lake using SQL queries you've defined to generate metrics, visualizations, and pass/fail status checks. The data lake persists over time, which means you can continue to explore, query, and visualize your test data long after the run has finished, enabling powerful post-hoc analysis and trend tracking across releases. ### Why use Metrics? The data lake architecture provides several key benefits. Because your data persists, you're not limited to viewing only the metrics you defined before the test ran. You can return to the SignalFlag UI days, weeks, or months later to explore the raw data, write new queries, create new visualizations, and answer questions you didn't anticipate when you first ran the test. This makes Metrics ideal for debugging unexpected failures, conducting retrospective analysis, and iterating on your metrics definitions without needing to re-run expensive tests. ### Branch isolation Metrics configurations are tied to specific branches in your project. This means your `main` branch can maintain a stable, well-tested set of metrics and data schemas, while development branches can experiment with new metrics, modify schemas, or test breaking changes without affecting production workflows. Each branch maintains its own independent metrics configuration, allowing engineers to iterate safely and merge changes only when they're ready. ### What you'll create To use Metrics, you'll create a single configuration file (`.resim/metrics/config.resim.yml`) that defines three key components: 1. **Topics** — The schema for data you'll emit from your system (e.g., robot velocity, localization error, goal status). Topics define the structure and types of the data flowing into the data lake. 1. **Metrics** — SQL queries that transform your emitted data into visualizations and scalar values. Metrics can be test-level (computed per individual test run) or batch-level (aggregated across multiple test runs). Metrics can have status checks - optional thresholds attached to metrics that determine pass/fail/warning states. These allow you to automatically flag tests that miss performance requirements. 1. **Metrics sets** — Collections of metrics that are run against a set of tests, helping you to validate performance and learn about robot behavior. The rest of this guide will walk you through each of these components step by step, starting with identifying the data you want to collect. ## Identifying Key Data The first step to adopting SignalFlag's Metrics framework is to identify the key data that you want to collect, plot, and track over time in the SignalFlag Platform. To start with, we'd recommend simple data that you're interested in plotting simply; i.e. your robot's velocity over time. Identify where this data comes from; in our example, we'll be reading it from our robot's `/odom` topic. ## Writing the schema Firstly, we strongly recommend installing the [SignalFlag VSCode Extension](https://marketplace.visualstudio.com/items?itemName=resim-ai.resim) (or from [Open VSX Registry](https://open-vsx.org/extension/resim-ai/resim) for VSCode forks); it provides language support for SignalFlag config files and will help you avoid mistakes when emitting your data! Look at the structure of the data you want to emit to SignalFlag; in our example case, we're looking at the `Odometry.twist.twist.linear` data, which is a `Vector3` representing the velocity of the robot in the x, y and z axis. We can write a SignalFlag config for this like so: YAML ``` # required version header version: 1 topics: odom_linear_velocity: schema: x: float y: float z: float ``` To ease your experience with writing queries, we recommend flattening nested data structures into their own topics. Additionally, we don't support slash characters `/` in topic names (sorry!). Use the content assist from the VSCode extension to explore the available data types. In-depth descriptions are provided by the extension on hover. Some column types don't hold raw data at all: `image`, `video`, and `artifact` columns hold the file names of files written to the job outputs, resolved when the metric renders. See [Image](#image), [Video](#video), and [Artifact](#artifact) under [Metric Templates](#metric-templates) below. Save the config file to `.resim/metrics/config.resim.yml` in your repository. The SignalFlag CLI and the Python SDK both read this path by default. You can save it to another path: pass `--metrics-config-path=` to the CLI, or `metrics_config_path=` to the SDK's `Batch`. Both accept more than one file; the files are merged, and each topic may appear in only one of them. The VSCode extension finds config files through its `resim.configFilePaths` setting, which accepts `.yml` and `.yaml` files. Its default entry is `.resim/metrics/config.resim.yaml`, so if you use the `.yml` default above, add your path to that setting. ## Emitting your data There are two approaches to take here; you can either emit the data live from your stack or post-hoc when your test is complete. We'll discuss both here. The `emit` API is primarily used through the `Emitter` class. You can pass it a `config_path` and an `output_path`. `output_path` is where we will write your emissions file - by default it is `/tmp/resim/outputs/emissions.resim.jsonl`. When `config_path` is both specified and is visible to the application, we will validate the types & shape of the data being emitted to ensure it will be ingested by the SignalFlag platform correctly. See below for an example: Python ``` from resim.metrics.python.emissions import Emitter emitter = Emitter(config_path=".resim/metrics/config.resim.yml") emitter.emit("odom_linear_velocity", {"x": 1.0, "y": 2.0, "z": 3.0}, timestamp=0) emitter.emit("odom_linear_velocity", {"x": 1.1, "y": 1.9, "z": 0.0}, timestamp=500000000) # rejected emitter.emit("odom_linear_velocity", {"x": "left", "y": "right", "z": "up"}, timestamp=1000000000) ``` When validation is on, the `Emitter` rejects a row locally, before anything is uploaded, if: - a field is not in the topic's schema, or a schema field is missing from the row. Every emit must carry every column. If only some rows have a field (for example, a column that a newer logger added), put that field in its own topic, or emit long-format rows such as `(name, value)`. - a value is `None`. There are no optional columns. - a value has the wrong type. `True`/`False` is rejected for `int` and `float` fields (use `1`/`0`). A float such as `1.0` is rejected for an `int` field, and so are numpy integers such as `numpy.int64`; convert them with `int()` or `.tolist()`. An `int` is accepted for a `float` field. - the config uses a type that the `Emitter` does not know. The known types are `boolean`, `int`, `float`, `string`, `status`, `image`, `video`, `string[]` and `metric[]`. A rejected row means the config and the data do not agree. Fix the config or the data; do not turn validation off. Timestamps are integers in nanoseconds. When your data has simulation time, use it: for ROS messages, use `header.stamp`, not the time the message was logged. In a simulation that runs slower than real time, log time stretches the run (at 8% of real time, a 17 s run looks like 205 s). Under the hood, we append these messages to your `output_path`. In this example, the lines would look like this: /tmp/resim/outputs/emissions.resim.jsonl ``` {"$metadata": {"topic": "odom_linear_velocity", "timestamp": 0}, "$data": {"x": 1.0, "y": 2.0, "z": 3.0}} {"$metadata": {"topic": "odom_linear_velocity", "timestamp": 500000000}, "$data": {"x": 1.1, "y": 1.9, "z": 0.0}} ... ``` ### Stack Emissions Here's a more fully fleshed-out code example of a stack-based emission pattern. Continuing with our ROS example, we will create a new node for our stack, the `MetricsEmitter` which will receive messages from the `/odom` ROS topic and emit them to the `odom_linear_velocity` SignalFlag topic. Python ``` from rclpy.node import Node from resim.metrics.python.emissions import Emitter from nav_msgs.msg import Odometry from builtin_interfaces.msg import Time as MsgTime from rclpy.time import Time from typing import Optional class MetricsEmitter(Node): def __init__(self): super().__init__('metrics_emitter') self.emitter = Emitter(config_path=".resim/metrics/config.resim.yml") self.first_timestamp = None self.odom_subscriber = self.create_subscription(Odometry, '/odom', self.odom_callback, 10) def get_relative_timestamp(self, msg_time: Optional[MsgTime] = None) -> Optional[int]: if self.first_goal_received_time is None: return None current_time: Time if msg_time is not None: current_time = Time.from_msg(msg_time) else: current_time = self.get_clock().now() return current_time.nanoseconds - self.first_timestamp.nanoseconds def odom_callback(self, msg: Odometry): if self.first_timestamp == None: self.first_timestamp = Time.from_msg(msg.header.stamp).nanoseconds self.emitter.emit("odom_linear_velocity", { "x": msg.twist.twist.linear.x, "y": msg.twist.twist.linear.y, "z": msg.twist.twist.linear.z }, timestamp=self.get_relative_timestamp(msg.header.stamp)) ``` ### Post-hoc Emissions If you're already recording log files from your simulations, you might want to take advantage of SignalFlag's metrics builds. We can follow much the same pattern, but instead read the data from log files like MCAPs or rosbags. See a developed example below using ROS: Python ``` from resim.metrics.python.emissions import Emitter from rosidl_runtime_py.utilities import get_message import rosbag2_py from pathlib import Path import rclpy.serialization def emit_velocity_data(emitter: Emitter, input_bag: Path): reader = rosbag2_py.SequentialReader() reader.open( rosbag2_py.StorageOptions(uri=str(input_bag), storage_id="mcap"), rosbag2_py.ConverterOptions( input_serialization_format="cdr", output_serialization_format="cdr" ), ) # Create a dictionary mapping topic names to their types topic_type_map = {topic.name: topic.type for topic in reader.get_all_topics_and_types()} odom_topic = "/odom" if odom_topic not in topic_type_map: raise ValueError(f"topic {odom_topic} not in bag") msg_type = get_message(topic_type_map[odom_topic]) while reader.has_next(): topic, data, timestamp = reader.read_next() if topic == odom_topic: # Only emit if enough time has passed since last emission msg = rclpy.serialization.deserialize_message(data, msg_type) emitter.emit('odom_linear_velocity', { 'x': msg.twist.twist.linear.x, 'y': msg.twist.twist.linear.y, 'z': msg.twist.twist.linear.z, }, timestamp=timestamp) if __name__ == "__main__": input_bag = Path("/tmp/resim/inputs/logs/record.mcap") with Emitter(config_path=".resim/metrics/config.resim.yml") as emitter: emit_velocity_data(emitter, Path(input_bag)) ``` ### Series Emissions If you're emitting a series of data from a single call, there is an additional helper function - `Emitter.emit_series`. This will spread the data from this single emit call to multiple emission lines, to aid with query simplicity. For the initial example above, this can be written as: Python ``` emitter = Emitter(config_path=".resim/metrics/config.resim.yml") emitter.emit_series("odom_linear_velocity", { "x": [1.0, 1.1, 1.1, 1.2], "y": [2.0, 1.9, 1.8, 1.7], "z": [3.0, 3.0, 3.0, 3.0] }, timestamps=[0, 500000000, 1000000000, 1500000000]) # Equivalent to emitter.emit("odom_linear_velocity", {"x": 1.0, "y": 2.0, "z": 3.0}, timestamp=0) emitter.emit("odom_linear_velocity", {"x": 1.1, "y": 1.9, "z": 3.0}, timestamp=500000000) emitter.emit("odom_linear_velocity", {"x": 1.1, "y": 1.8, "z": 3.0}, timestamp=1000000000) emitter.emit("odom_linear_velocity", {"x": 1.2, "y": 1.7, "z": 3.0}, timestamp=1500000000) ``` The config file remains the same as the original example, as the resulting emissions match the existing definition of the topic. The function will validate the types of the data as it is emitted & ensure all series are of the same length. ### Event Emissions SignalFlag supports special types of emissions called "events". These cause an event to be registered for your job at the given timestamp, which will appear in the Events tab in the SignalFlag app. Each event refers to a specific point in time - such as reaching a goal, or when a system stop occurs. Metrics can be attached to each event by including them in the emission to help you understand the system state at that time. Events are emitted using the `emitter.emit_event(topic_name, data, timestamp)` method. The topic must have `event: true` in the config, and each event takes exactly one timestamp. See the example below: YAML ``` topics: # ... goal_reached: event: true schema: name: string description: string status: status tags: string[] metrics: metric[] ``` Python ``` emitter = Emitter(config_path=".resim/metrics/config.resim.yml") emitter.emit_event("goal_reached", { "name": "Goal 1 Reached", "description": "Robot successfully reached goal number 1", "status": "PASSED", "tags": ["navigation"], "metrics": [{ "name": "Stereo View", "type": "image", "value": ["goal_1.png"], "status": "PASSED", }], }, timestamp=1750000000) ``` The structure of the `metric[]` object is defined as JSON ``` "metrics": [ { "name": "string", "description": "optional", "status": "optional. PASSED | FAIL_BLOCK | etc.", "type": "scalar | image | plotly | text | video | artifact", "value": "the value of the metric. Changes based on the type.", // examples: // Scalar and text metrics are treated the same way. // "type": "scalar" | "text", // "value": "2.37", // "type": "image", // "value": ["1.jpg", "2.jpg"] // "type": "plotly" // "value": "{ a valid plotly blob}", ie. the output of fig.to_json() // "type": "video", // "value": ["clip.mp4"] // "type": "artifact", // "value": ["trace.mcap", "notes.txt"], // "timestamp": 1773351267250000000 (optional, see "Seeking to a point in time" under Artifact) } ] ``` Each entry in `metrics` must be an object Each entry needs at least `name`, `type` and `value`. Do not send a list of metric names, such as `["Error Over Time"]`. The SDK does not check this, so the upload succeeds but the batch's metrics fail to compute. If an event has no evidence to attach, send `[]` and put the details in `description`. ### Custom emissions API If Python isn't your jam, you're more than welcome to write your own emitter in the language of your choice. The minimal implementation of the `emit` API (without validation) in Python is [here](https://github.com/resim-ai/open-core/blob/538b339550dc8c63c57dfc8a9cee6659bbb5da36/resim/sdk/metrics/emissions.py#L86). If you'd like official support for your language of choice, please reach out to our ProServe team and we'll look into it! ## Metrics development ### Adding your first metric We recommend developing your first couple of metrics using debug dashboards. The first step is to create an empty metrics set at the end of your config file: YAML ``` ... metrics sets: My Metrics: metrics: [] ``` Next, create the debug dashboard using the [SignalFlag CLI](../../setup/cli/) - see the linked docs for installation instructions. Once you've generated an emissions file, it can be ingested using the `resim metrics debug` command, invoked like so: Bash ``` resim metrics debug --project "metrics-demo" --emissions-file "emissions.resim.jsonl" --metrics-config-path ".resim/metrics/config.resim.yml" --metrics-set "My Metrics" ``` After a short time, this will print a URL to the debug dashboard which is linked to the config and emissions file provided. From here, you can start to develop your metrics. See the video below for a demonstration. \[ Your browser does not support the video tag. \](https://resim-public-assets.s3.us-east-1.amazonaws.com/Adding_your_first_metric.webm) Once you're happy with your metrics, you can export the current state of the dashboard by clicking the "Export Config" button. This will download a zip file containing the config file and any custom templates configured. You can then replace your local config file with this - you may need to do some formatting & reordering to get it how you like it. You can also develop your metrics in the config file directly, and then reusing `resim metrics debug` to test them. See the [metric templates](#metric-templates) below for more details. Once you've added a metric in the config file, make sure that it's included in your metrics set: YAML ``` ... metrics: Robot Speed: type: test description: Robot speed over time query_string: | SELECT 'Speed', timestamp / 1E9 AS "Time (s)", SQRT(POWER(x, 2) + POWER(y, 2)) AS "Speed (m/s)" FROM odom_linear_velocity; template_type: system template: line metrics sets: My Metrics: metrics: - Robot Speed ``` And then you can view it in the debug dashboard by running: Bash ``` resim metrics debug --project "metrics-demo" --emissions-file "emissions.resim.jsonl" --metrics-config-path ".resim/metrics/config.resim.yml" --metrics-set "My Metrics" ``` #### Image, Video and Artifact Metrics Debug dashboards can resolve image, video and artifact metrics too. Upload each file referenced in your emissions file with a repeatable `--media-file` flag alongside `resim metrics debug`: Bash ``` resim metrics debug --project "metrics-demo" --emissions-file "emissions.resim.jsonl" --metrics-config-path ".resim/metrics/config.resim.yml" --metrics-set "My Metrics" --media-file "screenshot.png" --media-file "clip.mp4" ``` `--media-file` works the same for artifact files. Each `--media-file` should point to a local file whose name matches a filename referenced by an image, video, or artifact metric's value in your emissions file. #### Status checks You can make a metric affect pass/fail by adding a status block. Status checks run a separate query that is evaluated against threshold values: if the query returns any rows, the metric is marked as blocking (or warning). Under `status` you configure: - **`query_string`** — A SQL query with exactly one `?` parameter. The `?` is replaced with the `block` or `warn` threshold value when the check runs (e.g. `HAVING COUNT(*) < ?` or `WHERE value > ?`). - **`block`** — Threshold value for a blocking failure. If the status query returns any rows when run with this value, the job is marked as failed (blocker). - **`warn`** — Optional. Threshold value for a warning. If the query returns any rows when run with this value, the metric is marked as a warning instead of a pass. Example: require at least 3 goals to be reached; otherwise the metric blocks. The status query returns a row when the count is *below* the threshold, so we use `HAVING COUNT(*) < ?` with `block: 3`: YAML ``` Time to reach final goal: type: test description: Time between receiving first goal and reaching final goal. Blocks if less than 3 goals are reached. query_string: | SELECT CASE WHEN (SELECT COUNT(*) FROM time_to_goal) != 3 THEN 75.0 ELSE sum(time_s) END FROM time_to_goal; template_type: system template: scalar status: query_string: SELECT '1' FROM time_to_goal HAVING COUNT(*) < ? block: 3 ``` How the check runs: - The `block` threshold is tried first. If the query returns no rows, the `warn` threshold is tried next. - The query takes exactly one `?`. To use two different conditions for block and warn, compute a severity code in SQL and compare against it, for example `SELECT 1 FROM (SELECT CASE WHEN err > 2 THEN 2 WHEN err > 1 THEN 1 ELSE 0 END AS severity FROM summary) s WHERE s.severity >= ?` with `block: 2` and `warn: 1`. - The status check runs only when the metric's own query returns data. If the topic emitted nothing in a test, the metric shows `NO_DATA` and the check does not run, so the job can still pass. See [Required and optional data](#required-and-optional-data). #### Required and optional data Each metric can set `skip_if_no_data`. When it is `true`, a metric whose query returns no rows is hidden for that test. Use it for **optional** data, which some tests do not have: a topic that only some test types emit, events, or a camera that only some rigs carry. Without it, these metrics show as `NO_DATA` on every test that does not emit them. For **required** data (the verdict, the values that have status checks, the headline numbers), set `skip_if_no_data: false`. This alone does not fail a test whose data is missing. With a topic that emitted nothing: | Setup | Metric | Job | | ---------------------------------------------------------------------------- | ------------------------------------------------------------ | ------- | | `skip_if_no_data: true` | Hidden | Passed | | `skip_if_no_data: false`, no status check | `NO_DATA` | Passed | | `skip_if_no_data: false`, status check that blocks when data is absent | `NO_DATA` (the status check does not run on an empty result) | Passed | | `skip_if_no_data: false`, query such as `MIN(x)` that returns one `NULL` row | `NO_DATA` | Passed | | `skip_if_no_data: false`, `COUNT(*)` query with a status check on the count | `FAIL_BLOCK` | Blocker | To make missing data fail the test, give each topic that a required metric reads a presence check, built from a `COUNT(*)` query: YAML ``` Run Summary Present: type: test description: Rows in run_summary. Blocks when this test emitted none. query_string: SELECT COUNT(*) AS value FROM run_summary template_type: system template: scalar units: rows skip_if_no_data: false status: query_string: SELECT 1 FROM (SELECT COUNT(*) AS n FROM run_summary) s WHERE s.n <= ? block: 0 ``` If whole groups of metrics apply only to some tests (for example, detection metrics only for perception tests), consider one metrics set per test type instead of one shared set full of optional metrics. Each batch picks its set by name, so each test type can keep its required metrics strict. ### Running your first Metrics batch Once you're happy with your metrics set, you can give your stack its first run in SignalFlag. First step is to register your config with the SignalFlag platform: Bash ``` $ resim metrics sync --project "metrics-demo" --branch "metrics-setup" ``` If you have a test suite you want to run your metrics set against, you can update the test suite as so: Bash ``` $ resim suite revise \ --project "metrics-demo" \ --test-suite "Nightly Suite" \ --metrics-set "My Metrics" # Now run the test suite: $ resim suite run \ --project "metrics-demo" \ --test-suite "Nightly Suite" \ --build-id "dc71ebee-1601-4d98-aab4-d4bdecfafca1" ``` If running a batch directly, the parameters are similar: Bash ``` $ resim batches create --project "metrics-demo" --metrics-set "My Metrics" ... ``` **Note**: Each branch in your project has its own copy of the config file, which might result in lots of CLI calls to re-register config files. To help with this, at the time of triggering a batch your config can automatically be synced to your branch with the `--sync-metrics-config` flag: Bash ``` $ resim suite run --project "metrics-demo" \ --test-suite "Nightly Suite" \ --build-id "dc71ebee-1601-4d98-aab4-d4bdecfafca1" \ --sync-metrics-config ``` With that, your batch should be running. Wait for it to complete, and the metrics you have configured will be shown automatically in the SignalFlag app. If you want to make more changes, you can simply repeat the process of exporting the config and replacing your local config file. #### Changing your metrics If you make changes to your metrics on a completed batch, the changes will be saved for the instance of metrics you are currently viewing. For example, if you make a change to the metrics for a single job, it will not be reflected on other jobs in the batch or going forward until you synchronise the config file manually. ### Removing topics (archiving or deprecating a topic) If you remove a topic from your config, `resim metrics sync` treats this as **removing** that topic - you'll also see this referred to as archiving or deprecating a topic. Removal hides that topic's historical emissions from queries and charts going forward, without deleting them. This is useful for retiring a topic that's no longer relevant without losing the ability to reintroduce a topic of the same name later - reintroducing a topic with a name that was previously removed starts a fresh, independent incarnation of it. Only data emitted after the reintroduction is visible; data from before the removal stays hidden. Because removal hides data other people may be relying on (batch/job metric charts, dashboards), a sync that would remove a topic first previews the impact and asks you to confirm: Bash ``` $ resim metrics sync --project "metrics-demo" --branch "metrics-setup" This sync would remove the following topics: - odom_linear_velocity: 128 emission(s) would be removed, 3 batch/job metric chart(s) reference it Affected dashboards: Nightly Overview (3393a727-0de9-4414-83e9-420b24dcceb5) sync would remove topic(s) odom_linear_velocity; re-run with --allow-topic-removal to confirm ``` This is a destructive operation you cannot undo yourself Once you re-run the sync with `--allow-topic-removal` to confirm, there's no flag or command that reverses it. Reintroducing a topic of the same name only makes *new* data visible again - the emissions hidden by the removal stay hidden, and the charts/dashboards called out in the preview cannot be pointed back at that data on your end. If you remove a topic by mistake, reach out to SignalFlag support - don't rely on being able to fix it yourself. Once you've reviewed the impact, re-run the sync with `--allow-topic-removal` to confirm it: Bash ``` $ resim metrics sync --project "metrics-demo" --branch "metrics-setup" --allow-topic-removal This sync will remove the following topics: - odom_linear_velocity: 128 emission(s) would be removed, 3 batch/job metric chart(s) reference it Affected dashboards: Nightly Overview (3393a727-0de9-4414-83e9-420b24dcceb5) ``` The batch/job metric charts and dashboards called out in the preview keep showing what was already rendered for them - removal doesn't retroactively clear anything from view. If one of those charts is re-rendered afterwards, though, it will fail, since it queries a topic that's no longer part of the config. Removal is scoped to the branch, [same as the rest of your metrics config](#branch-isolation) - removing a topic on a development branch has no effect on `main` or any other branch. `--allow-topic-removal` is accepted on `main` too. ### Authoring batch metrics When defining a metric in the config file, you may have spotted the `type: test` definition. This helps our metrics system determine where that metric should be displayed - in this case, it will be shown as a test result. The other option is `type: batch`, which will be shown as a batch metric - which are used for creating aggregate metrics across an entire batch. This changes the data which is available to the metric when authoring the query. This allows really easy authoring of high level metrics - such as "What was the average speed of the robot across all jobs in the batch?" which can be defined as: YAML ``` Average Speed: type: batch description: Average speed of the robot across all jobs in the batch. query_string: SELECT AVG(speed) FROM robot_speed template_type: system template: scalar ``` Along with the data you have emitted, you may also want to join against the metadata table to get more human-readable information. See the [metadata schema](#metadata-schema) below for more details. ### Dashboards Dashboards aggregate metrics across many batches over a trailing time window, so you can track how your system, tests, or test suite are performing as a whole. See the [Dashboards guide](dashboards/) for details. ## Metric Templates Each metric uses a **template** that defines how its query results are rendered. The following system templates are available; in all cases the query runs against your emitted topic data, and column order (and for charts, column aliases) determine layout and axis titles. Chart templates also accept a `template_settings` block for axis labels, fixed ranges, series colors and per-chart display options — see [Customizing charts](#customizing-charts). Queries use Trino SQL (for example `ARBITRARY`, `CAST(x AS VARCHAR)` and window functions). Give every column an alias: chart templates use the aliases as axis titles. The columns each template expects, in order: | Template | Columns | | --------------------------------- | ----------------------------------------- | | [Line](#line-chart) | series, x, y | | [Bar](#bar-chart) | series, x, y | | [Table](#table) | any | | [Scalar](#scalar) | one value (first row, first column) | | [Image](#image) | file name, from a column typed `image` | | [Video](#video) | file name, from a column typed `video` | | [Artifact](#artifact) | file name, from a column typed `artifact` | | [State Timeline](#state-timeline) | system, timestamp, state | | [Histogram](#histogram) | value, or series and value | | [Pie Chart](#pie-chart) | label, or label and value | A chart that no system template draws, such as an XY path where both axes are positions, needs a [custom template](custom-templates/). ### Line Chart The line template expects exactly three columns in order: a series identifier, then the x value, then the y value. Each row is `[series_name, x_value, y_value]`. Different `series_name` values produce multiple series on the same chart. The axis titles are taken from the column names at positions 2 and 3, so use descriptive aliases with units (e.g. `"Time (s)"`, `"Speed (m/s)"`). Emitted timestamps are in nanoseconds; divide by `1E9` to show seconds. Example config: YAML ``` Distance to Goal: type: test description: Distance to each goal over time, measured from odometry. Each line represents a different goal. query_string: | SELECT goal_name as group_name, timestamp / 1E9 AS "Time (s)", distance_m as "Distance to Goal (m)" FROM goal_distance; template_type: system template: line ``` ### Bar Chart The bar template uses the same shape as the line template: three columns `[series, x, y]` and rows `[series_name, x_value, y_value]`. It is well suited to aggregated or categorical comparisons (e.g. per experience, per goal). Example config: YAML ``` Maximum Localization Error: type: batch description: Maximum position difference between odometry and AMCL localization across all experiences. query_string: | SELECT 'Maximum', m.experience_name as "Experience Name", MAX(position_diff_m) as "Maximum Localization Error (m)" FROM pose_difference pd JOIN metadata m on pd.job_id = m.job_id GROUP BY m.experience_name ORDER BY m.experience_name; template_type: system template: bar ``` #### Click-through links Bar charts support an optional click-through feature: each bar can link to a specific page within SignalFlag (e.g. a batch, test, or experience). Click-through is most useful in [Dashboards](dashboards/) where you'll be aggregating data from tests over time. To enable click-through, include a `link_path` column in your query. Hovering over a bar will expose the link, and clicking it will navigate to the target page: The `link_path` value must be a path under SignalFlag (third-party URLs are not supported). You are responsible for constructing the path yourself by interpolating the relevant IDs into the query. All the data you need to construct the path is available when running queries: project_id, batch_id, test_id, experience_id, and more. View the `metadata` topic in the app to see what is available. Example config, for the metric seen above: YAML ``` altitude over time: type: batch description: Maximum drone altitude per batch, with click-through to the source batch. query_string: | SELECT 'altitude', timestamp, altitudes, '/projects/' || project_id || '/batches/' || batch_id AS link_path FROM drone_altitude template_type: system template: bar ``` ### Table The table template accepts any number of columns and any number of rows. Column names become the table headers and each row is rendered as a table row. Use it for summary statistics, key-value pairs, or any tabular output. Example config: YAML ``` AMCL Covariance Calibration: type: test description: Evaluates whether AMCL's uncertainty estimates are realistic. query_string: | SELECT "Metric", "Value" FROM ( SELECT 1 as sort_order, 'Total Samples' as "Metric", CAST(COUNT(*) AS VARCHAR) as "Value" FROM covariance_accuracy UNION ALL SELECT 2 as sort_order, 'Within 1σ (target: ~68%)' as "Metric", CONCAT(CAST(ROUND(100.0 * AVG(CASE WHEN within_1_sigma = 1 THEN 1.0 ELSE 0.0 END), 1) AS VARCHAR), '%') as "Value" FROM covariance_accuracy ORDER BY sort_order ); template_type: system template: table ``` ### Scalar The scalar template expects the query to return a single value. The renderer uses the first row and first column. Optional `status` thresholds can block or warn based on a separate query (e.g. checking that a count meets a minimum). You can also provide `units`, which will be displayed after the value. Example config: YAML ``` Time to reach final goal: type: test description: Time between receiving first goal and reaching final goal. query_string: | SELECT CASE WHEN (SELECT COUNT(*) FROM time_to_goal) != 3 THEN 75.0 ELSE sum(time_s) END FROM time_to_goal; template_type: system template: scalar units: "s" status: query_string: SELECT '1' FROM time_to_goal HAVING COUNT(*) < ? block: 3 ``` ### Image The image template expects one column whose values are image identifiers (e.g. filenames or paths) available in the job logs. Each row produces one image in the metric output. Use the `image` type and template for `.png`, `.jpg` and `.gif` files, animated GIFs included. Do not type a GIF as `video`: it syncs, uploads and renders with no error, but the app shows a video player that cannot play it. Note Any image referenced in your emissions file is uploaded individually so that the metric can resolve it by filename, even if you write it to a nested subdirectory beneath `/tmp/resim/outputs/` (which is otherwise zipped on upload). At most 100 emission-referenced media files are uploaded individually per run. See [Build Images](../../setup/build-images/) for details on output handling. Example config: YAML ``` Stereo Camera Feed: type: test description: Camera feed from navigation, sped up 2x, starting after first goal received. query_string: SELECT filename FROM camera_gif template_type: system template: image ``` ### Video The video template expects one column whose values are video identifiers (e.g. filenames or paths) available in the job logs. Each row produces one video in the metric output. Use the `video` type and template only for `.mp4` and `.webm` files. GIFs are images; see [Image](#image). Note As with image metrics, any video referenced in your emissions file is uploaded individually so that the metric can resolve it by filename, even if you write it to a nested subdirectory beneath `/tmp/resim/outputs/`. At most 100 emission-referenced media files are uploaded individually per run. Example config: YAML ``` Stereo Camera Feed: type: test description: Camera feed from navigation, starting after first goal received. query_string: SELECT filename FROM camera_video template_type: system template: video ``` ### Artifact The artifact template expects one column whose values are file names of files in the job outputs. Each row produces one artifact. Each artifact shows its name and size alongside a download button. `.mcap` and `.bag` files also get an "Open in Foxglove" button. Text files — `.txt`, `.log`, `.md`, `.json`, `.yaml`, `.yml`, `.csv`, `.xml`, `.toml` — can be expanded to read their contents inline, up to 1 MB; larger files are download only. A topic backing an artifact metric just needs a column of file names: YAML ``` topics: recordings: schema: filename: artifact ``` JSON ``` {"$metadata": {"topic": "recordings", "timestamp": 0}, "$data": {"filename": "trace.mcap"}} ``` Note As with image and video metrics, any file referenced by an artifact metric in your emissions file is uploaded individually so that the metric can resolve it by filename, even if you write it to a nested subdirectory beneath `/tmp/resim/outputs/`. This counts toward the same 100 emission-referenced media file cap shared with image and video. Example config: YAML ``` Recordings: type: test description: Recording and notes captured during the run. query_string: SELECT filename FROM recordings template_type: system template: artifact ``` #### Seeking to a point in time An artifact metric on an [event](#event-emissions) can set an optional `timestamp`. The artifacts then show it, and the "Open in Foxglove" button opens the recording at that moment. JSON ``` {"name": "recording", "type": "artifact", "value": ["trace.mcap"], "timestamp": 1773351267250000000} ``` `timestamp` accepts: - Epoch nanoseconds, the same clock as MCAP log times, as an integer or a string of digits: `1773351267250000000` or `"1773351267250000000"`. Floats such as `1773351267.25` or `1.77335126725e18` are rejected because they cannot hold epoch nanoseconds exactly. In Python, `time.time_ns()` returns an integer. - An RFC 3339 datetime with a `Z` or `±hh:mm` offset and up to nine fractional digits: `"2026-03-12T21:34:27.25Z"` or `"2026-03-12T18:34:27.25-03:00"`. Every format is stored as a UTC time with nanosecond precision, for example `2026-03-12T21:34:27.250000000Z`. The artifact list shows it to the millisecond, such as "Mar 12, 2026 21:34:27.250 UTC", and the full value on hover. Use the recording's own clock: the timestamp is not derived from the event's timestamp, and it is not checked against the recording's time range. A value in any other format, such as a float or a datetime without an offset, makes that metric an error chart. ### State Timeline The state timeline template shows discrete states over time for one or more systems. The query must return exactly three columns in order: system identifier, timestamp, and state name. Each row is `[system_name, timestamp, state_name]`. Use different `system_name` values to show multiple systems. Consecutive rows with the same state are merged into segments. The y-axis title comes from the alias you give the system column, and each unique state is assigned a color from the SignalFlag palette. The timestamp column may be either nanoseconds counted from the Unix epoch, or a datetime string such as `2026-03-15T00:00:00Z` (`from_unixtime(...)` produces one). The two formats may not be mixed within a single result. Example config: YAML ``` Goal Status: type: test description: Status of the goals over time. query_string: | SELECT 'Navigation' as "System", timestamp, state FROM goal_status; template_type: system template: state_timeline ``` #### Elapsed time or timestamps The x-axis can show either elapsed time or wall-clock time, set with the `time_axis` display setting (see [Customizing charts](#customizing-charts)): | `time_axis` | X-axis | | ----------- | ----------------------------------------------------------------------------------- | | `elapsed` | Seconds since the earliest row, so the axis starts at 0. Titled "Elapsed time (s)". | | `absolute` | Wall-clock timestamps. Titled "Time". | Omit it and the timestamp column decides: a numeric column is drawn as elapsed time, a datetime column as timestamps. Set it explicitly to get the other one — for example, to draw a column of epoch nanoseconds as wall-clock time: YAML ``` Goal Status: type: test description: Status of the goals over wall-clock time. query_string: | SELECT 'Navigation' as "System", timestamp, state FROM goal_status; template_type: system template: state_timeline template_settings: display: time_axis: absolute ``` ### Histogram The histogram template shows the distribution of a single numeric column. The query returns either one column (the values) or two columns `[series, value]` to compare multiple distributions on the same chart; each row is `[value]` or `[series_name, value]`. Different `series_name` values are overlaid with reduced opacity so overlapping distributions stay legible. The x-axis title comes from the alias you give the value column, so use a descriptive one with units (e.g. `"Speed (m/s)"`). Example config: YAML ``` Speed histogram: type: test description: Distribution of drone speeds. Single value column, so the series group is auto-injected. query_string: SELECT speeds AS "Speed (m/s)" FROM drone_speed template_type: system template: histogram ``` Bins are chosen automatically, and the y-axis shows the raw count of values per bin. Both are adjustable from **Customize Chart** on the metric — the chart above is this metric with percent normalization applied: - **Bins** — the maximum number of bins (1–1000). Leave it empty to keep automatic binning. - **Normalization** — what the y-axis measures: `Count` (raw values per bin, the default), `Percent` (each bin's share of the total), or `Density` (count per x-axis unit, useful when comparing distributions of differing sample sizes). The y-axis title follows this setting. ### Pie Chart The pie template shows how a whole divides into parts. The query returns either one column (the labels, each row counted) or two columns `[label, value]`, where the value column is summed per label. Slice names come from the values in the first column. Example config: YAML ``` Speed pie: type: test description: Share of samples in each speed band query_string: > SELECT CASE WHEN speeds > 13 THEN 'fast' WHEN speeds > 8 THEN 'cruising' ELSE 'slow' END AS "Band", count(*) AS "Samples" FROM drone_speed GROUP BY 1 template_type: system template: pie ``` Slices are ordered largest first, labelled with their percentage, and capped at 25 — anything past the cap combines into a single `Other` slice. All three are adjustable from **Customize Chart** on the metric: - **Slice labels** — `Percentage` (the default), `Value`, `Label and percentage`, or `None`. - **Max slices** — how many slices to draw (1–100) before the remainder folds into `Other`. - **Slice order** — `By size` (the default) or `Query order`, which preserves the order your query returned. A pie has no axes, so it takes no `xaxis` or `yaxis` settings and has no gridlines. ### Customizing charts Chart templates read an optional `template_settings` block on the metric. It has four sections, all optional: YAML ``` Distance to Goal: type: test query_string: ... template_type: system template: line template_settings: xaxis: label: Time (s) yaxis: label: Distance (m) min: 0 max: 50 series: - group_name: goal_a color: "#D3FAA3" display: legend: bottom grid: false ``` | Section | Keys | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `xaxis`, `yaxis` | `label` overrides the column-derived axis title. `min` and `max` fix the axis range; they must be given together, both numeric, with `min` less than `max`. | | `series` | A list of per-series overrides, each with a `group_name` matching a value from the query's series column, and a `color` as a quoted hex string (e.g. `"#FF0000"`). Series with no override keep cycling the default palette. | | `display` | Chart-level options, listed below. | #### Display options | Option | Values | Meaning | | ------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `legend` | `top` (default), `bottom`, `left`, `right`, `hidden` | Where the legend sits, or `hidden` to omit it. | | `grid` | `true` (default), `false` | Show or hide axis gridlines. | | `bar_mode` | `grouped` (default), `stacked` | Bar charts only: group bars side by side or stack them. | | `bins` | 1–1000 | Histograms only: maximum number of bins. Omit to let the chart choose. | | `histnorm` | `count` (default), `percent`, `density` | Histograms only: raw bin counts, each bin's share of the total, or count per x-axis unit. | | `text_info` | `percent` (default), `value`, `label_percent`, `none` | Pie charts only: what each slice is labelled with. | | `max_slices` | 1–100 (default 25) | Pie charts only: how many slices to draw before the remainder folds into a single "Other" slice. | | `sort_slices` | `true` (default), `false` | Pie charts only: sort slices descending by value, or keep the order the query returned. | | `time_axis` | `elapsed`, `absolute` | State timelines only: elapsed seconds since the earliest row, or wall-clock timestamps. See [Elapsed time or timestamps](#elapsed-time-or-timestamps). | #### What each template accepts Settings that a template does not support are rejected when the config is validated, with the offending key named in the error. | Template | `xaxis` | `yaxis` | `series` | `display` | | ------------------------------------- | --------------------- | --------------------- | -------- | -------------------------------------------------- | | Line | `label`, `min`, `max` | `label`, `min`, `max` | `color` | `legend`, `grid` | | Bar | `label`, `min`, `max` | `label`, `min`, `max` | `color` | `legend`, `grid`, `bar_mode` | | Histogram | `label`, `min`, `max` | — | `color` | `legend`, `grid`, `bins`, `histnorm` | | Pie | — | — | `color` | `legend`, `text_info`, `max_slices`, `sort_slices` | | State Timeline | `label` | — | — | `legend`, `grid`, `time_axis` | | Table, Scalar, Image, Video, Artifact | — | — | — | — | Histograms have no `yaxis` section: the chart bins the x values itself, and the y-axis title follows `histnorm`. Pie charts have no axes at all, so neither axis section nor `grid` applies, and `series` overrides key off the slice label rather than a series column. ### Custom templates If the system templates above don't fit, you can author a Plotly chart yourself with a Liquid template. See [Custom Metric Templates](custom-templates/) for the full guide. ## Metadata Schema When you `SELECT *` on your topics, you will notice some additional metadata such as `batch_id` and `job_id` are returned. These can be used to join against a special `metadata` table which contains information about your build, experience, and more. The `metadata` table has one row per job in the batch. Batch metrics (`type: batch`) query across all rows. This allows you to, in something like a batch metric, compute - what was the min, average, and max speed, for each experience in my test set? - how did the performance of this metric vary across sunny, cloudy, and rainy test scenarios? An example batch metric: "How does the min, avg, and max speed vary across each experience?" SQL ``` SELECT m.experience_name, MIN(speed), AVG(speed), MAX(speed) FROM drone_speed d JOIN metadata m on d.job_id = m.job_id GROUP BY m.experience_name; ``` If you want to see all metadata available you can run this query: SQL ``` SELECT * FROM metadata ``` ### Column reference | Column | Type | Description / Observed values | | -------------------------- | ------------- | ---------------------------------------------------------------------------------------------------- | | `org_id` | string | Organisation identifier, e.g. `"resim.ai"` | | `batch_id` | string (UUID) | ID of the batch this job belongs to | | `job_id` | string (UUID) | Unique identifier for the individual job | | `project_id` | string (UUID) | SignalFlag project ID | | `branch_id` | string (UUID) | Branch ID within the project | | `branch_name` | string | Human-readable branch name, e.g. `"main"` | | `build_id` | string (UUID) | ID of the build under test | | `build_name` | string | Human-readable build name | | `build_creation_timestamp` | timestamp | When the build was created | | `experience_id` | string (UUID) | ID of the experience (scenario) for this job | | `experience_name` | string | Human-readable experience name | | `experience_tag_ids` | string[] | UUIDs of the experience tags attached to this job | | `experience_tag_names` | string[] | Names of the experience tags attached to this job | | `test_suite_id` | string (UUID) | ID of the test suite | | `test_suite_name` | string | Human-readable test suite name | | `test_suite_revision` | int | Revision number of the test suite | | `job_status` | string | Raw execution status — e.g.: `'SUCCEEDED'`, `'FAILED'` | | `job_metrics_status` | string | Metrics evaluation status — e.g. `'PASSED'`, `'FAILED'` | | `job_conflated_status` | string | Combined status used for pass/fail decisions — e.g.: `'PASSED'`, `'ERROR'`, `'BLOCKER'`, `'WARNING'` | | `custom_field_keys` | string[] | Custom field names set on the job (may be `[]`) | | `custom_field_values` | string[] | Custom field values, parallel to `custom_field_keys` (may be `[]`) | | `time` | timestamp | When the relevant job started | ### Array column usage `experience_tag_names` and `experience_tag_ids` are Trino arrays. Common patterns: SQL ``` -- Check membership WHERE contains(experience_tag_names, 'my_tag') -- Filter to tags matching a prefix, take the first result (1-based indexing) filter(experience_tag_names, x -> starts_with(x, 'my_prefix_'))[1] -- Conditional count across jobs count_if(job_conflated_status = 'PASSED') -- Group jobs by a tag prefix WITH classified AS ( SELECT filter(experience_tag_names, x -> starts_with(x, 'my_prefix_'))[1] AS group_name, job_conflated_status FROM metadata WHERE contains(experience_tag_names, 'my_tag') ) SELECT group_name, count_if(job_conflated_status = 'PASSED') AS passed FROM classified WHERE group_name IS NOT NULL GROUP BY group_name ``` # Sessions A **session** is a recording of your robot operating in the real world — one or more MCAP files in cloud storage — registered in SignalFlag so the platform can evaluate it. An evaluated session's field data lands in the same data lake as your simulation results, so the same [metrics](../metrics/), events and [A/B comparison](../ab-comparison/) tools apply to it. Getting your recordings into SignalFlag SignalFlag registers your recordings as sessions and runs their evaluations; these pages show the results. To get your recordings connected, talk to us in your shared Slack channel or at [info@signalflag.ai](mailto:info@signalflag.ai). Sessions and their evaluations will then appear in the library automatically. ## The session library Sessions live under **Organize > Sessions**. Each card shows the session's name, description, recording location, creation time and tags, plus the state of its latest evaluation. A session that has never been evaluated reads **Unprocessed**; otherwise the card shows the evaluation's status, such as **Passed** or **Warning**, and its event count. ## The session evaluation page Opening an evaluated session goes straight to its **Session Evaluation** page. A session that has not been evaluated opens a simpler page marked **Not evaluated**. **Recording bounds** — **Start (UTC)**, **End (UTC)** and **Duration** — come from the recording inventory of this evaluation, not from when the job ran. The **event timeline** places the session's events on an elapsed axis from the start of the recording to its end. Hovering a marker shows the event's name, assessment and time; clicking it opens the event. Events too close together to draw apart share a marker, and clicking it lists each of them. Events without a usable recording timestamp are not plotted but remain on the **Events** tab. A session always shows its latest evaluation. ## Comparing two sessions Select two evaluated sessions in the library and choose **Compare**, or choose **Compare** on an evaluation page and pick the other session. **Session Compare** opens the latest evaluation of each, with the **Metrics**, **Merged Metrics**, **Events** and **Details** tabs of the [A/B comparison](../ab-comparison/) view. Unlike ordinary tests, two sessions can be compared even though they are different recordings. Each keeps its own timeline, measured from its own recording start. ## Where to go next - [Play a session recording back](playback/) - [Create session metrics with an agent](agent-metrics/) # Session recording playback SignalFlag opens a [session's](../) MCAP recordings in [Foxglove](https://app.foxglove.dev), so you can see what a metric or event looked like. ## Opening a recording Choose **View in Foxglove** at the top of a session's page. SignalFlag creates a short-lived signed link to the recording and opens it in Foxglove in a new tab. The link expires, so click again whenever you need a fresh one. If the session's folder holds several recordings, a **Recording file** prompt asks for the MCAP file name within that folder. Enter it and choose **Open recording**. ## If playback does not open - **"Allow pop-ups to open this recording, then try again."** Allow pop-ups for the SignalFlag app and click again. - **"Unable to open this recording. Try again to request a fresh link."** The request failed or named a file that is not in the session's folder. Check the file name and click again. ## Streaming large recordings Playback loads the file from cloud storage on demand. To stream large recordings instead, use [Foxglove primary sites](../../foxglove-primary-sites/). # Create session metrics with an agent The interesting question about a [session](../) is often one nobody wrote a metric for. An agent connected to the SignalFlag [MCP server](../../mcp/server/) can take your previously evaluated field session data, propose a metric over it, show you the real chart, and save it once you approve. ## Connecting In Claude Code, install the Signal Flag plugin and add the MCP endpoint under the server name `signal-flag`: Bash ``` claude plugin marketplace add resim-ai/signal-flag-plugin claude plugin install signal-flag@signal-flag-plugins --scope user claude mcp add --scope local --transport http signal-flag https://bff.resim.ai/mcp claude --chrome ``` Authenticate with `/mcp`, connect Chrome with `/chrome`, and start the workflow: Text ``` /signal-flag:explore-field-sessions ``` Local MCP configuration belongs to the directory you launch from, so run these where you intend to work. ## How it works - **It resolves what you mean.** The agent confirms your organization and resolves the project, branch and session from a link, a list, or the page you have open in the web app. - **It inspects real data.** It reads the branch's topics and schemas, the existing metric definitions, and actual rows from the evaluation. When data is missing, it names what is absent and stops that proposal rather than guessing. - **It asks about meaning.** Units, clocks, frames, state values and how to treat gaps are agreed with you, not assumed. - **It proposes new names.** Each proposal uses a metric name not already used anywhere in the branch's metrics configuration, so existing definitions are never rewritten. - **It shows the real chart.** Where your MCP host renders charts, the agent draws one there. Otherwise it uses the **Edit metric** screen in your signed-in browser, which previews without saving. If neither is possible, it says so. ## Approving and saving Nothing is saved until you approve a specific definition for a specific evaluation. Changing either needs a fresh approval. Once approved, the agent adds the metric to the evaluation you are looking at, where it appears on the **Metrics** tab. Other evaluations and existing metrics are unchanged. After saving, the agent reads the result back and reports what was stored. ## Scope and limitations - **Existing evaluations only.** The agent adds metrics to evaluations that already exist. It does not register sessions, run evaluations, inspect local MCAP files or repositories, or author emissions or metrics builds. - **New metrics only.** Editing an existing metric is ordinary [metric authoring](../../metrics/). To draft metrics inside the web app, use [Reeves](../../reeves/). ## Introduction This document is intended to help you build and push **build images** to an image registry from a CI service so that SignalFlag can use the images to run tests. Currently SignalFlag supports authenticating against, and pulling images from, an AWS ECR, Dockerhub or Google Artifact Registry repository. If you use a different image registry provider and would like SignalFlag to integrate with it, please get in touch. The example workflows provided are for GitLab and GitHub but almost any CI service should be able to push to ECR. If you have a suggested service to add to this list, or that you would like help configuring to push build images, please reach out to SignalFlag. Note If you already have well-established processes for pushing images build in a CI/CD environment, this guide is not required. Instead, follow the [Next Steps](./#next-steps) described below. ## Prerequisites - We recommend (although it's not mandatory) that you have built, pushed and tested an image from a local machine before setting up a CI pipeline to do so. See [Build Images](../build-images/) for guidance. - You need to have set up an image repository that SignalFlag can access, as described in [SignalFlag Data Access](../resim-data-access/). You will need the **repository URI** or: your AWS account number, the AWS region in which the account was created, and the repository name. - Your SignalFlag contact should have provided you with a **client ID** and **client secret**. These are the credentials your CI service will use to authenticate with SignalFlag. - You will need a project in the SignalFlag app with which to associate your builds. See [Projects](../projects/). ## Creating an IAM user with AWS ECR permissions Your CI pipeline will need to authenticate with AWS to push images. A thorough exploration of AWS configuration is outside the scope of this article, but as a starting point if you do not have existing AWS identities that your CI service uses, you can create an IAM user with the required permissions. You will need to create a user, create security credentials for the user, and assign the `AmazonEC2ContainerRegistryPowerUser` managed IAM policy to the user. If you have the AWS CLI installed and configured, we recommend using it rather than the browser UI for this task. See the [Command line interface](#command-line-interface) section below. If you don't have the AWS CLI set up, you can create the user in the AWS browser interface. ### Browser interface (AWS Console) Following along with the [AWS documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users.html) as required, you need to: 1. [Create a user](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_create.html#id_users_create_console), named `ci-user` or any other name you choose. You **do not** need to enable console access for this user. 1. [Create credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#Using_CreateAccessKey) in the form of an **access key** and **secret access key**. Store these locally in a password manager or other secure location, they are equivalent to a username and password for this CI user. 1. [Assign permissions](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_users_change-permissions.html#users_change_permissions-add-console) to the user so it can push to ECR. Following the instructions to "add permissions by attaching policies directly to the user", assign the `AmazonEC2ContainerRegistryPowerUser` IAM policy. This policy is managed by AWS. ### Command line interface Create a new user: Bash ``` $ aws iam create-user --user-name ci-user { "User": { "Path": "/", "UserName": "ci-user", "UserId": "AIDAWRXCWKBZSLQOSKX4V", "Arn": "arn:aws:iam:::user/ci-user", "CreateDate": "2023-08-10T18:47:20+00:00" } } ``` Create an access key pair: Note Store the access key and secret access key from the following command somewhere secure - they are static credentials equivalent to a username and password. We will add them to your CI service's secrets configuration later. Bash ``` $ aws iam create-access-key --user-name ci-user { "AccessKey": { "UserName": "ci-user", "AccessKeyId": "AKIAWRXCWKBZVEXAMPLE", "Status": "Active", "SecretAccessKey": "kKEZbvjb6UV/jEFPEED5sdcSiJUGeezDuEXAMPLE", "CreateDate": "2023-08-10T18:47:53+00:00" } } ``` Attach a policy to the new user which will allow it to push to ECR repositories. Bash ``` $ aws iam attach-user-policy --policy-arn "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser" --user-name ci-user ``` ### Testing permissions Once you have created a user, generated credentials for it, and assigned permissions to it following one of the above methods, you may wish to test the configuration locally before configuring your CI service. To log in to your ECR repository as the CI user: Bash ``` AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= aws ecr get-login-password --region | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com ``` Then pull and tag an example image and push it to your repository: Bash ``` docker pull public.ecr.aws/docker/library/hello-world docker tag public.ecr.aws/docker/library/hello-world .dkr.ecr..amazonaws.com/:hello-world docker push .dkr.ecr..amazonaws.com/:hello-world ``` Try pulling the image: Note This will not pull any data because the image already exists locally, but it will confirm that the image exists in the remote repository. Bash ``` docker pull .dkr.ecr..amazonaws.com/:hello-world ``` Log out from ECR in your local environment: Bash ``` docker logout .dkr.ecr..amazonaws.com ``` You have now confirmed that the CI user you've created can manage images in your ECR repository. (For more information about working with ECR images, see https://docs.aws.amazon.com/AmazonECR/latest/userguide/images.html) ## Next steps The next step is to configure your CI system to build and push images and run tests in SignalFlag: - [GitHub](github/) - [GitLab](gitlab/) # GitHub In this example, you have a GitHub repository which contains the application you would like to test. Note that you will need a SignalFlag project. You can [create a project](../../projects/) with the SignalFlag CLI. This example uses [our GitHub action](https://github.com/resim-ai/action). ## Prerequisites Referring to [GitHub's documentation about setting repository secrets](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) as required, navigate to your repository's Settings and go to the "Secrets and variables" page in the Security section. Add the following secrets: - `AWS_ACCESS_KEY_ID` - set the value of this to the access key generated when you [set up your ECR repository for use with SignalFlag](../#creating-an-iam-user-with-aws-ecr-permissions) - `AWS_SECRET_ACCESS_KEY` - set the value of this to the secret access key generated when you [set up your ECR repository for use with SignalFlag](../#creating-an-iam-user-with-aws-ecr-permissions) - `SLACK_WEBHOOK` - set this to a Slack webhook URL (see the [Slack documentation](https://api.slack.com/messaging/webhooks)), if you want nightly summaries Also add one of these sets of secrets. Your SignalFlag contact will provide you with either a client ID and secret, or a username and password: - `RESIM_CLIENT_ID` and `RESIM_CLIENT_SECRET`, or - `RESIM_USERNAME` and `RESIM_PASSWORD` Add a variable containing your ECR registry URL: - `ECR_REGISTRY_URL` - for example `123456789.dkr.ecr.us-east-1.amazonaws.com` ## Workflow Now you need to create a GitHub workflow by adding a file to `.github/workflows/` in your repository. Below is an example of a workflow that builds a Docker image, pushes it to an ECR repository and launches a batch in SignalFlag to test it. If you use this example, make sure to replace the following placeholder values: - `` - any commands that need to run before the Docker build, e.g. `bazel build` - `` - replace with the name of the ECR repository, e.g. the name of the image/application - `` - replace with the name of your SignalFlag project - `` - replace with the name of your SignalFlag system - `` - the test suite you want to test against (see [Test Suites](../../test-suites/)) YAML ``` name: build-and-test on: pull_request: push: branches: - "main" schedule: - cron: '0 0 * * *' jobs: build_and_test: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v3 - name: Build run: - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - name: Log in to ECR uses: docker/login-action@v2 with: registry: ${{ vars.ECR_REGISTRY_URL }} username: ${{ secrets.AWS_ACCESS_KEY_ID }} password: ${{ secrets.AWS_SECRET_ACCESS_KEY }} - name: Prepare Docker metadata id: docker_meta uses: docker/metadata-action@v5 with: images: | ${{ vars.ECR_REGISTRY_URL }}/ tags: | type=sha - name: Build and push image id: docker_build uses: docker/build-push-action@v5 with: context: . file: ./Dockerfile push: true tags: ${{ steps.docker_meta.outputs.tags }} - name: Launch Batch in SignalFlag uses: resim-ai/action@v1 id: launch_batch with: client_id: ${{ secrets.RESIM_CLIENT_ID }} client_secret: ${{ secrets.RESIM_CLIENT_SECRET }} project: system: image: ${{ steps.docker_meta.outputs.tags }} test_suite: # You may use experience names or tags instead # experiences: # experience_tags: metrics-build-id: ## If you would like the action to post a link to the results on PRs: comment_on_pr: true github_token: ${{ secrets.GITHUB_TOKEN }} - name: Fetch SignalFlag CLI if: github.event_name == 'schedule' run: | curl -L https://github.com/resim-ai/api-client/releases/latest/download/resim-linux-amd64 -o resim-cli chmod +x resim-cli - name: Wait for completion and send Slack summary if: github.event_name == 'schedule' env: PROJECT_ID: ${{ steps.launch_batch.outputs.project_id }} BATCH_ID: ${{ steps.launch_batch.outputs.batch_id }} RESIM_CLIENT_ID: ${{ secrets.RESIM_CLIENT_ID }} RESIM_CLIENT_SECRET: ${{ secrets.RESIM_CLIENT_SECRET }} SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }} run: | # the default wait time is one hour. If you think your batch will reliably take longer, you may need to edit this ./resim batch wait --project "$PROJECT_ID" --batch-id "$BATCH_ID" || exit_status=$? ./resim batch get --project "$PROJECT_ID" --batch-id "$BATCH_ID" --slack | \ curl -X POST -H "content-type: application/json" --data @- $SLACK_WEBHOOK exit "${exit_status:-0}" ``` # GitLab In this example, you will configure a GitLab project which contains the test application that you would like to run. Note that you will need a SignalFlag project. You can [create a project](../../projects/) with the SignalFlag CLI. Referring to [GitLab's documentation about CI variables](https://docs.gitlab.com/ee/ci/variables/) as required, navigate to your project's CI/CD Settings and scroll to and expand the Variables section. Add the following variables: - `AWS_ACCESS_KEY_ID` - set the value of this to the access key generated [above](../#creating-an-iam-user-with-aws-ecr-permissions) - `AWS_SECRET_ACCESS_KEY` - set the value of this to the secret access key generated [above](../#creating-an-iam-user-with-aws-ecr-permissions) - `SLACK_WEBHOOK` - if you want to run nightly tests with summaries, add a Slack webhook URL (see the [Slack documentation](https://api.slack.com/messaging/webhooks)) Also add one of these sets of variables. Your SignalFlag contact will provide you with either a client ID and secret, or a username and password: - `RESIM_CLIENT_ID` and `RESIM_CLIENT_SECRET`, or - `RESIM_USERNAME` and `RESIM_PASSWORD` If you plan to use a merge request workflow, these variables should **not** be `protected` (so that branches can use them to push images and run tests in SignalFlag), or the branches you raise MRs for should be covered by a branch protection rule. If you would like CI pipelines to be able to comment on merge requests, you also need to set a variable containing a [GitLab authentication token](https://docs.gitlab.com/ee/security/token_overview.html), either a personal access token or a project token, with at least the `api` scope. Add a [`.gitlab-ci.yml`](https://docs.gitlab.com/ee/ci/quick_start/#create-a-gitlab-ciyml-file) file to the root of the repository. Below is an example of building and pushing a Docker image to an ECR repository. If you use this example, make sure to replace the following placeholder values: - `` - replace with your AWS account number - `` - replace with the AWS region in which you have deployed your ECR repository, e.g. `us-east-1` - `` - replace with the name of the ECR repository, e.g. the name of the application - `` - replace with the name of your SignalFlag project - `` - replace with the id of your SignalFlag project - `` - the commands you run to build a Docker image containing your application - `` - the name of the Docker image created by your build - `` - the experience(s) you want to test against (see [Adding Experiences](../../adding-experiences/)), comma-separated YAML ``` stages: - build - comment # Optional - configures this workflow to post a results link if it's triggered by an MR - notify # Optional - configures this workflow to send the results of a completed nightly batch to a slack webhook variables: PROJECT_ID: push-image: stage: build # Start a Docker daemon we can use to build images services: - name: docker:dind alias: dockerdaemon # Run this job if this is an MR, a tag, or the default branch rules: - if: $CI_PIPELINE_SOURCE == 'merge_request_event' - if: $CI_COMMIT_TAG - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH variables: DOCKER_HOST: tcp://dockerdaemon:2375/ DOCKER_DRIVER: overlay2 DOCKER_TLS_CERTDIR: "" SYSTEM_NAME: script: # First, you need to build a Docker image containing your application. Put your build commands below. # For example # - docker build . -t :latest # Store our new image's URI as a variable to use later. Make sure you update these placeholder values - IMAGE_URI=.dkr.ecr..amazonaws.com/:$CI_COMMIT_SHORT_SHA # We are using $CI_COMMIT_SHORT_SHA as image tag - this variable is set automatically by GitLab to the first 8 characters of the commit that triggered the run # Using this tag lets us map built images to code (and the inverse) easily - docker tag :latest $IMAGE_URI # Log in to ECR so we can push the image. Make sure you update these placeholder values - aws ecr get-login-password --region | docker login --username AWS --password-stdin .dkr.ecr..amazonaws.com - docker push $IMAGE_URI # Install the latest version of the SignalFlag CLI - curl -L https://github.com/resim-ai/api-client/releases/latest/download/resim-linux-amd64 -o resim - chmod +x resim # Register our new image with SignalFlag as a build - BUILD_ID_OUTPUT=$(./resim builds create --system $SYSTEM_NAME --description "A quadcopter simulation" --version "$CI_COMMIT_SHORT_SHA" --project $PROJECT_ID --branch "$CI_COMMIT_REF_NAME" --image "$IMAGE_URI" --auto-create-branch --github) # Run a batch with the new build - BATCH_ID_OUTPUT=$(./resim batches create --project $PROJECT_ID --build-id ${BUILD_ID_OUTPUT#build_id=} --experiences "" --github) - echo $BATCH_ID_OUTPUT >> build.env # This passes data to the next job artifacts: reports: dotenv: build.env # To comment a results link for the batch above on an MR, this command uses a GitLab token from the variable $MY_GITLAB_TOKEN. You may need to change the name of this variable here or in your project's settings. comment-on-mr: stage: comment # Configure this job to run only on MRs rules: - if: $CI_PIPELINE_SOURCE == 'merge_request_event' script: # this syntax (a folded block scalar) stops YAML complaining about the colons in this string - > curl --location --request POST "https://gitlab.com/api/v4/projects/$CI_MERGE_REQUEST_PROJECT_ID/merge_requests/$CI_MERGE_REQUEST_IID/notes" \ --header "PRIVATE-TOKEN: $MY_GITLAB_TOKEN" --header "Content-Type: application/json" \ --data-raw "{ \"body\": \"View results on [app.signalflag.ai](https://app.signalflag.ai/projects/$PROJECT_ID/batches/$batch_id)\" }" slack-notify: stage: notify # Run only on a schedule - for nightly tests rules: - if: $CI_PIPELINE_SOURCE == 'schedule' script: # Install the latest version of the SignalFlag CLI - curl -L https://github.com/resim-ai/api-client/releases/latest/download/resim-linux-amd64 -o resim - chmod +x resim # Wait for the job to complete and capture the exit code # the default wait time is one hour. If you think your batch will reliably take longer, you may need to edit this - ./resim batch wait --project "$PROJECT_ID" --batch-id "$BATCH_ID" || exit_status=$? # Build the slack payload and post it using curl - > ./resim batch get --project "$PROJECT_ID" --batch-id "$BATCH_ID" --slack | \ curl -X POST -H "content-type: application/json" --data @- $SLACK_WEBHOOK # Exit with error code - exit "${exit_status:-0}" # optional - you may want to permit job failure for batch errors allow_failure: exit_codes: - 2 # batch failure - 5 # batch cancelled ``` # Managing your artifacts This tutorial will cover the starting point for managing your tests and other data artifacts within SignalFlag. This is a quick illustration of what those concepts are within the SignalFlag app, and how you might use them for your robot. ## Problem When you begin to scale your offline testing, you begin to generate many artifacts in the form of [experiences](../../core-concepts/#experience), versions of code to test, and metrics you want to collect. Often, you end up with many artifacts that are not relevant or compatible, so managing those artifacts becomes a burden. During development, you want to only see the tests that matter to you and your [system](../../core-concepts/#system), and ensure your metrics are managed correctly. Before you run any tests and waste time and resources doing so, you want to ensure that your [tests](../../core-concepts/#test), [metrics](../../core-concepts/#metrics), and [system build](../../core-concepts/#build) are all compatible. ## Examples This section runs through some examples of the above concepts, applied to an example basic autonomy architecture. ### Systems In this diagram, you see the sensors feed into a perception module, which feeds down into motion planning and ultimately controls. This is a basic representation of the pieces of an autonomy system, but sometimes these modules are combined or there are additional modules. The above example can be defined as its own System, which represents the end to end modules that make up the on-robot system. In the SignalFlag Platform, you could define a set of Systems as follows: **Perception**: A perception system is likely concerned with detecting everything of interest and tracking those objects over time. A perception system is likely to require high fidelity images (either real or simulated), but the metrics you care about are likely to be independent of the motion of the robot. By defining a perception System, you can isolate the tests you run and metrics you generate to only those that apply to a perception system. **Multi-Module Systems**: You can define a System in any way that is relevant to your robot. For example, the motion planner and controller in a robot system are closely tied together, but separating them for testing can still be useful. So you can define Systems for both individual modules, or the combination together to represent more of the realistic interaction of those modules on the robot. ### Metrics In this example autonomy stack, we have defined two main systems: the Perception System and the Multi Module (Motion Planning) System. These two systems would have very different metrics. For example, if you are not processing image data, it doesn't make sense to attempt to compute image processing time. To support efficiently running metrics, you could define Metrics Builds for each of these Systems that capture the types of metrics that are appropriate such as: Perception: - Classification accuracy - Precision & recall - Time to detect Multi-Module Motion Planning: - Robot State and Derivatives such as pose, velocity, acceleration of the robot. - Collision Checking - Planning metrics associated with the plans produced and safety constraints # Re-running partial batches You might have a need to re-run some subset of tests inside a batch. Maybe one experience encountered a transient error. Maybe it has non-deterministic behaviour. But you don't want to re-run all of the tests in a batch, just one or a few problematic ones. ## Limitations The tests will be re-run as if they were being freshly run now. They do not take a snapshot of what the configuration was when they were first run and re-use that data. This means that if you have updated any of the dependencies, including **system**, **image**, **experience data**, etc - then the test will run with the latest definitions of that data. When re-running tests, both the experience and metrics stages for that test will be re-run. The metrics stage will run against only the results from the new run; results from previous runs will be ignored. It is not currently possible to only re-run the metrics stage for a test re-using previously generated experience output. The Batch Metrics stage will always be re-run. It will reuse the output from any test that did not need to be re-run, combining it with the fresh output from any re-run tests, ignoring old output for anything that was superseded. Batch Metrics may also be re-run in isolation, without needing to re-run any tests within the batch. ## CLI instructions There are two ways to re-run batches using the CLI. ### Automated reruns using `batches supervise` The `supervise` command under resim batches is responsible for monitoring a batch run until completion, and then re-runs the subset of failed tests automatically for a set number of times. #### Configuration The `supervise` command takes several required and optional parameters. To see full documentation of each, please refer to `resim batches supervise --help`. ### Required parameters - `project`: the name or ID of the project to supervise - `max-rerun-attempts`: maximum number of rerun attempts for failed tests (default: 1) - `rerun-max-failure-percent`: maximum percentage of failed jobs before stopping (1-100, default: 50). This is intended to guard against re-running and wasting compute on a faulty version and only to rerun for flakiness of tests. For example, if more than 50% of tests fail, it is more likely that the code under test is failing vs failure due to non-determinism. - `rerun-on-states`: Status of test results to trigger automatic reruns on (e.g. Warning, Error, Blocker) #### Optional parameters - `batch-id`: the ID of the batch to supervise - `batch-name`: the name of the batch to supervise. One of `batch-name` or `batch-id` is necessary. - `poll-every`: interval between checking batch status, expressed in Golang duration string (default: "30s") - `wait-timeout`: amount of time to wait for a batch to finish, expressed in Golang duration string (default: "1h") #### Exit codes The supervise command returns an exit code corresponding to the final batch status: - `0`: Success - `1`: Internal error - `2`: Error - `5`: Cancelled - `6`: Timed out The exit codes can be used on CI pipelines to pass or fail a stage. #### Example Bash ``` resim batches supervise \ --project "my-project" \ --batch-name "my-batch-name" \ --max-rerun-attempts 3 \ --rerun-max-failure-percent 30.0 \ --rerun-on-states "Error,Warning" \ --poll-every "1m" \ --wait-timeout "2h" ``` ### Manual re-run using `batches rerun` If there is a need to partially re-run certain tests to test for flaky failures manually, we can use the `rerun` command. To re-run one or more tests: Bash ``` resim batches rerun --batch-id={batch id} --test-ids={test id 1, test id 2, ...} ``` Specify the test IDs (aka Job IDs) from the batch in question. Do not use (eg) experience IDs. To re-run just the Batch Metrics stage, simply specify no test IDs: Bash ``` resim batches rerun --batch-id={batch id} ``` # A/B Comparison An A/B comparison puts two runs next to each other and shows what changed between them: a new build against the last one that passed, a branch against main, or a single test that behaves differently in two batches. A/B comparisons work at two levels. Comparing two [test batches](../../core-concepts/#test-batch) tells you how a build performed overall. Comparing two [tests](../../core-concepts/#test) — the same [experience](../../core-concepts/#experience) run in two different batches — tells you what happened inside a single scenario. ## What the comparison shows Both runs are summarized in a pair of cards at the top of the page, grouped into **System Under Test** (**System**, **Branch**, **Build Version**) and **Test Suite** (**Name**, **Revision**, **Metrics Build**, **Version**). Any field whose value differs between the two sides is highlighted. Long values such as build versions are truncated, so it is the highlight rather than the value that tells you the two runs differ. ## Metrics: shared and unique Metrics are matched between the two runs **by name**. Metrics present on both sides are paired under **Shared Metrics**, so the same chart from each run sits at the same place on the page. Everything else appears under **Unique Metrics**, in the column of the run that produced it. Because the match is by name, renaming a metric splits the pair into two different entries, and the side-by-side reading of that metric is lost. Stable metric names keep comparisons useful over time. ## Merged Metrics **Merged Metrics** goes a step further and overlays a shared pair onto a single plot, so both runs are drawn on the same axes, and the difference between them is a distance on the chart rather than something you reconstruct across two images. Not every shared pair can be overlaid. A pair qualifies when both sides are of the same chart type and that type has a meaningful overlay. A line chart cannot be drawn onto a bar chart, and scalars, images, videos, tables and state timelines have no sensible overlaid form, so those pairs are left out. ## Comparing test outcomes When you compare two batches, the **Tests** tab holds **Test Performance Comparison**: every experience that ran in both batches, with its status on each side, under the branch and build version of the run it belongs to. The tests are split into four groups: - **Failing on just one batch** — the tests whose outcome actually changed. In a comparison against the last passing build, this is the regression and fix list, and it is normally the fastest read on a new build. - **Failing on both batches** — pre-existing failures, which is what you check to confirm that a change did not fix what it was meant to fix. - **Passing on both batches** — the stable set. - **All tests** — every experience present in both batches. Each row has its own **Compare** action, which opens the test-level comparison for that experience. That takes you from "this test regressed" to the two runs' metrics side by side without going back through the batch list. # SignalFlag HiL Agent ## Introduction An important part of testing embodied AI is hardware-in-the-loop testing or HiL. This is testing that involves communicating with real hardware. HiL testing complements simulation-based testing and is often run when changes are merged to a default branch, or before a release. The SignalFlag Agent is designed to facilitate HiL testing by integrating with your existing suite of tests, allowing HiL tests to be triggered and the results to be analyzed using the same tools as simulation-based testing. The SignalFlag Agent works by polling SignalFlag for tasks so there is no need to open ports or adjust firewall rules to allow inbound traffic. ## Prerequisites The SignalFlag Agent is currently tested against Linux on amd64 and requires Docker (version 27 or later). Note The user running the Agent needs permission to use the Docker API. In some environments this means adding the user to the `docker` group. If you can do `docker run hello-world` you're ready to go. ## Installation Pre-built binaries are available for linux-amd64, linux-arm64, darwin-arm64 (Mac OS): For Linux on AMD64: Text ``` curl -L https://github.com/resim-ai/agent/releases/latest/download/agent-linux-amd64 -o agent chmod +x agent ``` For Linux on ARM64: Text ``` curl -L https://github.com/resim-ai/agent/releases/latest/download/agent-linux-arm64 -o agent chmod +x agent ``` For Mac OS on Apple Silicon/ARM: Text ``` curl -L https://github.com/resim-ai/agent/releases/latest/download/agent-darwin-arm64 -o agent chmod +x agent ``` The Agent is open-source so you can inspect the code before you run it. You can also install the Agent directly using Go: Bash ``` go install github.com/resim-ai/agent ``` ## Configuration The Agent loads configuration from `~/resim/config.yaml`. The minimal configuration you need is as follows: Text ``` name: robot-arm-1 pool-labels: - arm username: password: ``` The name of the agent will be used to identify which agent instance ran a particular job (although uniqueness is not currently enforced.) Setting pool-labels as above will configure the agent to request jobs from SignalFlag that have been put into the pool called `arm`. This is a way to group jobs, for example based on the hardware they require. This value is a list, and the agent will receive jobs with *any* of the configured labels. Info Pool labels that begin with the prefix `resim` are reserved, so please ensure your pool labels do not have this prefix. A username and password for use by your agents will be provided by SignalFlag. Run the binary and it will start polling SignalFlag for work. By default, the agent logs to stdout and to `~/resim/agent.log` ### Network mode By default, your containers will be run as part of the default bridge network, which matches the Docker default. From this network, containers can reach services on the same network (or the internet), but not contact anything else running on the host. If you need to communicate with services running on the same host, set the following option in the agent configuration file: Text ``` docker-network-mode: host ``` This is equivalent to `docker run --net=host`. ### Privileged mode By default, your containers will not be run with elevated privileges, which matches the Docker default. If you would like your containers to be run with elevated privileges, set the following option in the agent configuration file: Text ``` privileged: true ``` This is equivalent to `docker run --privileged`. ### Passing mounts and environment variables It is possible to provide volume or socket mounts and environment variables to your test container by providing additional entries in the config, for example: Text ``` mounts: - /path/on/host:/mounted/path ... environment-variables: - HIL=true ... ``` ### Experience caching As of v1.1.0, the Agent will keep downloaded experience data on disk, avoiding repeat downloads at the start of each test. The two options of importance and their defaults are: Text ``` remove-experience-cache: false experience-cache-dir: /tmp/resim/cache ``` You can set `experience-cache-dir` to a custom path, overriding the default path to ensure the cache folder is kept between reboots of the host system. If `remove-experience-cache` is set to `true`, the Agent will clean up any cached experiences when the agent exits, keeping disk usage down. ### Other options See the [repo README file](https://github.com/resim-ai/agent) for a full list of configuration options. ## Build images The SignalFlag Agent uses the Docker daemon on the host to pull and launch images. If the image repository requires authentication, make sure the host is authenticated before running the agent by running [`docker login`](https://docs.docker.com/reference/cli/docker/login). For automatic authentication, the agent supports the [AWS ECR credential helper](https://github.com/awslabs/amazon-ecr-credential-helper/). To enable its use, ensure credentials/configuration are available in `~/.aws` and that `~/.docker/config.json` is [configured as required](https://github.com/awslabs/amazon-ecr-credential-helper/tree/main?tab=readme-ov-file#docker). ## Launching batches You can now launch batches to be run by the agent, by specifying a matching pool label: Text ``` resim batch create --project robot-arm --build-id 4b50f4f9-3b86-49a6-84b2-7813fd4bbcbe --experience-tags hardware-validation --pool-labels arm ``` Note that when creating a batch, the pool labels are used as an `AND`, that is, an agent will only run the batch if it is running with *all* of the labels on the batch. This is to enable selecting agents with certain capabilities, without having to create unique labels for each permutation. The status and results of the batch will be presented in SignalFlag as normal, and you can see which agent ran each batch from [HiL Agent Status](../agent-status/). ### Setting batch priority When launching a batch, you can control the order in which it will be picked up by an agent by passing the `--priority` flag: Text ``` resim batch create --project robot-arm --build-id 4b50f4f9-3b86-49a6-84b2-7813fd4bbcbe --experience-tags hardware-validation --pool-labels arm --priority 100 ``` The priority is an integer from `0` to `32767`. **Lower values indicate higher priority**, so a batch with `--priority 0` will be picked up before a batch with `--priority 100`. If the flag is omitted, batches use a default priority of `1000`. This is useful when you have several batches queued for the same pool of agents and you want to ensure that critical or time-sensitive work is run first, while lower-priority work (e.g. nightly regression runs) waits in the queue. ## Monitoring your agents Once your agents are checking in and running batches, you can monitor the whole fleet: which agents are online, what test each agent is running, what's queued for every pool label, and how busy your agents have been. Your HiL agents can be monitored from either the [`resim` CLI](../../setup/cli/) (`resim agents` and `resim pool-labels`) or the SignalFlag web app. See [HiL Agent Status](../agent-status/). ## Designing experiences The workflow we anticipate is that the agent is running on the hardware to be tested, or that the hardware is accessible over the network from the container that will run your experience. The experience container will send commands to the hardware and receive data back which it will process as required and store in preparation for the metrics step. Experience input data is currently expected to be built into the image, or accessible via S3. ## Running in the background Here is an example systemd unit file for running the agent as a systemd service: INI ``` [Unit] Description=SignalFlag Agent After=network-online.target [Service] ExecStart=/usr/local/bin/agent # Or other path as required Restart=always User=ubuntu # Replace with the user that will be running the agent [Install] WantedBy=multi-user.target ``` # SignalFlag SDK ## Introduction SignalFlag's standard workflow runs your sim (i.e. [build](../../core-concepts/#build)) inside our platform so we can manage execution, collect outputs, and generate metrics. However, if you can't or don't want to dockerize your sim, you can still generate metrics for analysis. SignalFlag's SDK allows you to run tests outside of SignalFlag, on your own infrastructure, export the results, and generate metrics for these tests. There are a few different reasons you may want to run tests on your own infrastructure, outside of SignalFlag: - You are just getting started, and want to explore our metrics framework - You want to run tests on actual hardware, and only use SignalFlag as a visualization/analysis tool - Your system is difficult to dockerize, perhaps due to special hardware requirements This guide walks you through how to use SignalFlag's Python SDK. You can run your tests wherever you like, emit structured data from each test, then upload the results to SignalFlag. The tests will appear in the SignalFlag app exactly like any other batch. Metrics required This feature utilizes the metrics framework. It is recommended to familiarize yourself with [Metrics](../metrics/). You need at least a valid metrics config file (by default `.resim/metrics/config.resim.yml`) with topics defined before running batches this way. A small, functional example is provided in the Github repo below to get you started. ## Installation Running batches outside of SignalFlag is supported via our Python SDK: Bash ``` pip install signalflag # or, if you're using uv uv add signalflag ``` Note: The signalflag package requires Python >= 3.10. Coming from `resim-sdk`? `signalflag` is the same SDK under its new name, and installing it over an existing `resim-sdk` is all the upgrade there is. The `resim.*` import paths still resolve to the same modules, so code written against `resim-sdk` runs unchanged. `pip install resim-sdk` also keeps working — see [the rename page](../../rename/) for what else the name change does and does not touch. ## Quickstart The following repository contains a working example that creates a batch with a few tests and generates some metrics: The example in this repo will create a small batch, with a couple of tests, and generate some simple metrics. ## API Documentation Basic usage involves create a `Batch`, with `Test`s inside, and `emit`-ing relevant data under each test. Python ``` from signalflag.sdk.batch import Batch from signalflag.sdk.test import Test # `DeviceCodeClient` will trigger an interactive prompt to authenticate. from signalflag.sdk.auth.device_code_client import DeviceCodeClient client = DeviceCodeClient() # OR, if you want non-interactive authentication, like for a CI system, use `UsernamePasswordClient` # instead, and request credentials from SignalFlag. The username/password below are NOT your personal # credentials. # from signalflag.sdk.auth.username_password_client import UsernamePasswordClient # client = UsernamePasswordClient(username="*****", password="*****") # Create a batch, and run the "my metrics" metrics set from your metrics config file with Batch( client=client, project_name="my-project", branch="metrics-test-branch", metrics_set_name="my metrics", # The SDK reads .resim/metrics/config.resim.yml by default. Pass metrics_config_path to use another file. ) as batch: print(f"Created batch {batch.friendly_name} with id {batch.id}") # Create a test, and emit some random data for the "position" topic with Test(client, batch, "test 1") as test: for i in range(10): test.emit("position", {"x": float(i), "y": float(i)}, i) # When finished the emitted data will be uploaded to SignalFlag so metrics processing can begin ``` Branches and versions The Batch constructor accepts a `branch` (required) and an optional `version` string. Common values for `version` include a commit SHA, a semver tag, or a build number. Using these fields can enable longitudinal analysis, such as: - How has metric X changed in my branch versus main? - How has metric Y changed between release versions 2.0 and 3.0? `branch` is required because SignalFlag performs a backwards-compatibility check when syncing a new config, ensuring you don't make destructive changes to your topics - for example, changing an string field to a integer could potentially break queries against historical data. For this reason, we recommend experimenting on a branch other than `main` when getting started. ### `Batch` parameters | Parameter | Type | Description | | --------------------- | --------------------- | ------------------------------------------------------------------------------------------------------- | | `client` | `AuthenticatedClient` | **Required.** Authenticated API client. | | `project_name` | `str` | **Required.** Name of the project to associate this batch with. Provide this or `project_id`, not both. | | `project_id` | `str` | **Required.** UUID of the project to associate this batch with. | | `branch` | `str` | **Required.** Branch name to associate this batch with. | | `metrics_set_name` | `str` | `None` | | `name` | `str` | `None` | | `version` | `str` | `None` | | `metrics_config_path` | `str` | `list[str]` | | `templates_path` | `str` | `None` | | `system` | `str` | `None` | | `test_suite` | `str` | `None` | | `test_suite_revision` | `int` | `None` | ### `Test` parameters | Parameter | Type | Description | | --------- | --------------------- | ------------------------------------------ | | `client` | `AuthenticatedClient` | **Required.** Authenticated API client. | | `batch` | `Batch` | **Required.** The parent `Batch` instance. | | `name` | `str` | **Required.** Name of the test. | **Note:** Each test is tied 1-to-1 with an Experience. The name of a test comes from the experience. This means when you do `Test(name="hello world")`, the SDK will get-or-create an experience with that name. This is mainly an implementation detail, but useful to know if you want to associate any tests with pre-existing experiences. Because of this, the test name is what links a test across batches. Two batches can be compared test by test only where the test names match, so keep test names stable from one run to the next, and put anything that changes between runs (a build, a checkpoint, a parameter) in the batch `version` or in the emitted data instead. Each `Test` writes its emissions to `emissions_.resim.jsonl` in the current working directory before uploading it. To keep these files out of your repository, run from a temporary directory and give `Batch` absolute paths for `metrics_config_path` and `templates_path`. ## Emitting data `test.emit()` writes a data point to the emissions log for that test. Topic names and schemas must match your metrics config file. You do not need to manage the emissions file yourself. The SDK handles creating and uploading it to SignalFlag automatically. The test object supports all emit methods. The metrics guide will cover these in greater depth. Python ``` test.emit(topic_name, data, timestamp) test.emit_series("odom_linear_velocity", { "x": [1.0, 1.1, 1.1, 1.2], "y": [2.0, 1.9, 1.8, 1.7], "z": [3.0, 3.0, 3.0, 3.0] }, timestamps=[0, 500000000, 1000000000, 1500000000]) test.emit_event("goal_reached", {"name": "Goal 1 Reached", ...}, timestamp=1750000000) ``` For more details on emitting data, see [emitting your data](../metrics/#emitting-your-data) ## Attaching files Use `test.attach_log()` to upload a file (such as an image, GIF, or other artifact) and associate it with a test result: Python ``` test.attach_log("run_001.mcap") ``` The file is uploaded to SignalFlag's storage and registered under the test. You can attach any file type — images, logs, MCAPs, videos, etc. `attach_log` also accepts: | Parameter | Type | Description | | ----------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `log_type` | `LogType` | `None` | | `file_name` | `str` | `None` | | `wait` | `bool` | Keyword only. Upload now and return when done, instead of in the background. Use it when you are about to delete or change the file. | Uploads run in the background by default, so keep each file in place and unchanged until the test closes. Closing the test waits for every upload, and raises `LogUploadError` if any of them failed. ### Image and Video metrics For image and video metrics, attaching the file is **required**. Your metric references the image by filename, so the file must be uploaded for the image metric to render: Python ``` test.attach_log("screen_capture_123.png") test.emit("my-image-topic", {"img": "screen_capture_123.png"}, time.time_ns()) ``` The image's filename in the emission must match the filename passed to `attach_log`: use the file's base name, not its path. The metrics engine uses this reference to retrieve and render the image. Here is how the metrics config would look for the above image metric: YAML ``` topics: images: schema: img: image metrics: images: type: test query_string: select img from images template_type: system template: image ``` ## Authentication `DeviceCodeClient` signs you in through your browser. The first time it needs a token, it prints `Please navigate to: ` and then waits until you approve the sign-in in your browser. Run it where you can see that output: in a background job or an upload script, the link reaches no one and the process seems to hang. The token is cached in `~/.signalflag/token.json` (or in `~/.resim/token.json`, if only that directory exists from an earlier install). Later runs reuse it. A token within one hour of its expiry counts as expired, and the next client starts a new sign-in. Do not share or print the cache file; it is a credential. ## Syncing a config without a batch `Batch` syncs your metrics config when it starts. To sync a config without creating a batch, for example to check that a new config is accepted, use `metrics.sync_config`: Python ``` from signalflag.sdk.auth.device_code_client import DeviceCodeClient from signalflag.sdk.bff_client import metrics metrics.sync_config( DeviceCodeClient(), "", "my-scratch-branch", config_path=".resim/metrics/config.resim.yml", templates_path=".resim/metrics/templates", ) ``` Each branch keeps its own config, and the sync checks a new config against the branch's current one. Changing a column's type is refused, and removing a topic hides its history on that branch (see [Removing topics](../metrics/#removing-topics-archiving-or-deprecating-a-topic)). Try new configs on a scratch branch first. ## Systems and test suites `Batch` can attach a batch to a [system](../../core-concepts/#system) and a [test suite](../../core-concepts/#test-suite), by name. Both must exist before the batch is created. A test suite is built from experiences that already exist, and the SDK creates an experience the first time it sees a test name. So the order is: 1. Create the system. 1. Push a first batch without `test_suite`. This creates an experience for each test. 1. Register those experiences on the system and create the test suite from them, with the metrics set it should run. See [Create Test Suites](../../setup/test-suites/). 1. Pass `test_suite=` on later batches, and `test_suite_revision=` to use a specific revision. When the list of tests changes, revise the suite to make a new revision. ## Viewing results Once the `Batch` context exits, and all metrics processing is complete, navigate to your project in the [SignalFlag app](https://app.signalflag.ai) to view the batch, its tests, and generated metrics — the same as any batch run inside SignalFlag. # Set Up SignalFlag with an AI Coding Agent An AI coding agent can help you set up tests in SignalFlag quickly with minimal tokens to onboard and use our platform to its ideal capacity with useful graphs and metrics. You point it at a project that already produces results (simulation runs, test reports, recorded robot logs, data files, model evaluations), and it does the setup work for you. The SignalFlag skills give your agent these functions: - Reviews what your runs produce. - Asks a few questions in plain language. - Writes the code that sends your results to SignalFlag, and picks charts that fit your data. - Sends a small trial first, checks that the numbers match your files, then sends the real results and gives you the links. You don't need to know how SignalFlag is organized to start. This page covers installing the SignalFlag skills, connecting your agent to your SignalFlag account, and what to ask for. **Reading time:** about 6 minutes. **Setup time:** 15 to 45 minutes, depending on how many questions you choose to answer. ## Before you start - **An AI coding agent** that supports skills and remote MCP servers, such as [Claude Code](https://code.claude.com/docs/en/overview), installed and signed in. - **A SignalFlag account**, and ideally the name of the SignalFlag project your results should go into. If you don't remember the name, say so. The agent notes it in the plan and looks it up after you approve, when it signs in. - **Python 3** on your machine. The agent creates a separate Python environment for the upload code, so your project's own packages aren't touched. - **A project with results on disk**, such as a folder of runs, a test suite you run locally, recorded logs, or CSV or Parquet files. ## Step 1: Install the SignalFlag skills Skills teach your agent how to set up and use SignalFlag. They live in the [signalskills](https://github.com/resim-ai/signalskills) repository and cover getting results you already have into SignalFlag. The MCP server also ships its own skills for running tests, diagnosing failures and other day-to-day work, listed in [Agent Capabilities](../capabilities/#skills). In Claude Code, the skills come as a plugin, which also adds the connection to SignalFlag. 1. Open Claude Code in a terminal. 1. Type these two commands, one at a time: Claude Code ``` /plugin marketplace add resim-ai/signalskills /plugin install signalskills@signalskills ``` 1. Close Claude Code and open it again, so the plugin loads. To check it worked, type `/plugin`. You should see `signalskills` in the list, enabled. 1. Clone the repository: Shell ``` git clone https://github.com/resim-ai/signalskills.git ~/signalskills ``` 1. Link the skills into `~/.agents/skills`, the shared skills folder that Codex, Cursor, Devin CLI and Devin Desktop all read: Shell ``` mkdir -p ~/.agents/skills for s in ~/signalskills/skills/signalflag-*; do ln -s "$s" ~/.agents/skills/; done ``` Linking rather than copying lets a later `git pull` update the skills in place. If your agent reads a different folder (see the table below these steps), put that folder in place of `~/.agents/skills` in both lines. 1. Restart your agent so it picks up the new skills. | Agent | Skills folder in your home directory | | ------------------ | ---------------------------------------------- | | Codex | `~/.agents/skills` | | Cursor | `~/.agents/skills` or `~/.cursor/skills` | | Devin CLI | `~/.agents/skills` or `~/.config/devin/skills` | | Devin Desktop | `~/.agents/skills` | | Gemini CLI | `~/.gemini/skills` | | GitHub Copilot CLI | `~/.copilot/skills` | | OpenCode | `~/.config/opencode/skills` | | Windsurf | `~/.codeium/windsurf/skills` | Devin in the cloud reads skills from the repositories it works in, not from your home directory, so this setup doesn't apply to it. ## Step 2: Connect your agent to your SignalFlag account The agent uses the [SignalFlag MCP server](../server/) to read your results back (statuses, numbers, charts) and confirm everything arrived correctly. The server can also set things up, such as projects and metrics configs, but launching, rerunning, canceling, archiving and deleting always come back to you as a draft to approve in the web app. See [Approving consequential actions](../capabilities/#approving-consequential-actions). The plugin added this connection in Step 1. 1. In Claude Code, type `/mcp`. 1. Choose `signalflag` from the list and select **Authenticate**. 1. Your browser opens. Sign in with your SignalFlag account and approve. 1. Back in Claude Code, `signalflag` should now show as connected. 1. Add the SignalFlag MCP server to your agent's MCP configuration, using this URL: Text ``` https://bff.resim.ai/mcp ``` The [MCP server guide](../server/#connect) has the configuration for several agents, including one-click setup for Cursor and VS Code. Any server name works, since the skills find the server by its tools. 1. Start the sign-in from your agent. Your browser opens. Sign in with your SignalFlag account and approve. You only need to do this once. Tip Separately, when the agent first sends results, it may need you to sign in once more for uploads. It will post a link in the chat. Open it, approve, and the agent carries on by itself. If your agent can't run commands in the background, it asks you to run the sign-in command yourself instead. You never need to share a password with your agent. ## Step 3: Start with your project 1. In a terminal, go to the folder of the project that has your results. 1. Start your agent there. 1. Ask for what you want in your own words. See the examples in the next section. You don't need to name the skills or use any special command, because the agent recognizes the request and starts the setup. ## Example prompts Pick the one closest to your situation, and change the details to match yours. | Your situation | Try asking | | ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | You run simulations that save a folder per run | "Get our simulation runs into SignalFlag. They're in runs/." | | You have a test suite (for example pytest) you run locally or in CI | "Make my pytest run report its results to SignalFlag." | | You have recorded robot logs (for example ROS bags or MCAP files) | "Put these recordings into SignalFlag and show me how the robot did on each one." | | You have data files (CSV, Parquet, HDF5) | "Upload our drive telemetry to SignalFlag so we can explore it later." | | You train models and evaluate checkpoints | "Hook my checkpoint evaluations up to SignalFlag and rank the checkpoints." | | You want results to upload every time your test script runs | "Wire SignalFlag into `run_suite.py` so every run reports automatically." | | You replay recorded camera or sensor data through your perception stack | "Wire SignalFlag into our perception replay harness. I want to see what got worse per class and range, and scrub to the missed frames." | | You compare model versions on a labeled photo set | "Evaluate each new detector version on our photo set in SignalFlag and show me where it got better or worse." | | Your results already upload, but something looks wrong | "Our tests already upload to SignalFlag but some metrics show errors. Am I organizing this right?" | | It's already set up and you want to improve something | "Tune the controller gain and show me the progress in SignalFlag." | Once your results are in SignalFlag, the [MCP Server](../server/#your-first-run) guide has prompts for running, diagnosing and comparing tests. It helps to mention: - **Your checks after a run**: "I check whether it reached the goal and how far off the path it went." - **Who will use the results**: "My lead wants to see whether we're getting better over time." The agent sizes the charts to the reader, with a few headline numbers for a lead and more depth for an engineer debugging. - **Different kinds of tests**: "These are regression tests, and these are known failures we're tracking", or "same stack, but on sim drives and real drives". The agent keeps each kind in its own test suite so comparisons stay like for like. - **Anything off-limits**: "Don't change `run_tests.sh`." ## FAQ How does setup work after I ask? 1. The agent reads your project and your results before asking anything. 1. It asks how involved you want to be. - **"You decide"**: the agent makes the choices from what it finds, and only asks which SignalFlag project to use. Before anything is sent, it stops once and shows you everything it chose. This suits people new to SignalFlag. - **"Walk me through it"**: the agent asks one question at a time, each with a few options and its recommendation. This suits people who want to understand and shape each decision. 1. It writes a short plan in your project, named after your type of test, for example `docs/signalflag/route-test-brief.md`. It describes your tests, how they map into SignalFlag, and which charts and pass/fail checks it will set up. You can read and edit it. 1. It reads your data and proposes metrics and charts. It adds checks that flag a bad run and point to the likely cause, puts video or a replay first where your runs have them, and keeps everything else available for digging in later. It shows you the plan and waits for your go. Nothing is installed and nothing is signed in to before that. 1. It builds the upload code in your project, in its own Python environment. 1. It sends a small trial first to a separate test area in SignalFlag, and checks that the numbers there match your files. 1. It sends your real results and gives you links to the results page and the dashboard. If your project already sends results to SignalFlag, the agent reads your existing setup first, keeps what you built, answers your question, and fixes only what's wrong. Which files does the agent add to my project? - A plan file for each type of test, `docs/signalflag/-test-brief.md`. - Upload code, usually a small script or a few files, plus a settings file in `.resim/metrics/`. - A separate Python environment (for example `.venv-signalflag/`), which it adds to `.gitignore`. It only changes your existing code if you agree to it, and says exactly what it changed. Nothing is committed for you, so review the changes and commit them when you're happy. How do I keep the skills up to date? We improve the skills regularly. Type these in Claude Code, then close and reopen it: Claude Code ``` /plugin marketplace update signalskills /plugin update signalskills@signalskills ``` Type `/plugin` to see which version you have. Pull the latest version into your clone, then restart your agent: Shell ``` git -C ~/signalskills pull ``` If you copied the skill folders instead of linking them, copy them again after pulling. Which SignalFlag terms will I see? | Word | Meaning | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | Project | Your team's space in SignalFlag. Your admin usually sets it up. | | Branch | A named line of results that get charted together over time, for example all the rounds of one test suite. The agent suggests a name. | | Batch | One upload, for example one round of your test suite. | | Test | One item inside a batch, for example one scenario, one route or one seed. Keeping the same name across rounds lets SignalFlag compare them. | | Metric | A number, chart or video shown for a test or batch, sometimes with a pass/fail check attached. | | Dashboard | Charts across many batches, for example "how is this getting better over time". | The SignalFlag server doesn't show up in my agent Restart your agent after installing the skills or adding the server. In Claude Code, run `/plugin` and check that `signalskills` is enabled, then look for `signalflag` in `/mcp`. In other agents, check that the server URL in your MCP configuration is `https://bff.resim.ai/mcp`. The [MCP Server troubleshooting](../server/#troubleshooting) section explains the error codes the server returns. The sign-in link expired Links last about 15 minutes. Tell the agent "the login link expired" and it will post a new one. The dashboard looks empty or out of date Dashboards refresh on their own and can take a few minutes after an upload. Open it again a little later. With only one round of results, trend charts show a single point until the next round. The agent asks about a run it can't identify If a run is missing information (for example which model checkpoint it used), the agent leaves it out rather than guessing. Tell it what the run was, or let it skip that run. I see extra test areas in my SignalFlag project During setup the agent sends trial uploads to separate areas whose names end in -lowres or -scratch, so your real results stay clean. You can archive them once you're happy. I'm not sure about something the agent chose Ask it, for example "Why did you pick that branch name?" or "Change the warning limit to 5 cm." Every choice is written in the plan file (`docs/signalflag/-test-brief.md`), so you can also edit it there and ask the agent to update the setup. ## Getting help Contact your SignalFlag representative or our support team, and include what you asked the agent and what happened. If your agent can export the conversation, attach that too. In Claude Code, `/export` saves it to a file. # MCP server SignalFlag's MCP server lets an agent operate the platform on your behalf. That might be a coding agent in your repository, a chat client asking about results, or a step in your CI pipeline. There is nothing to install and no API key to create. Your client prompts you to sign in through your browser on first use. LLM-friendly docs Pair the MCP server with our LLM-friendly documentation, published as plain markdown following the [llms.txt convention](https://llmstxt.org): - [`/llms.txt`](/llms.txt), a curated index of the docs - [`/llms-full.txt`](/llms-full.txt), the full corpus as a single file - Any page is also available as raw markdown by appending `.md` to its URL (for example [`/setup/cli.md`](../../../setup/cli/)) Point your agent at these alongside the MCP server for full context on the platform. ## Reads, writes, and approvals Your agent can read anything you can read, and it can set things up on its own. That covers creating projects and branches, registering builds and experiences, assembling test suites, pushing metrics configs, and building dashboards. Your agent cannot launch a batch, rerun a job, cancel a run, archive anything, delete anything, or change a project's agent instructions. Those six actions produce a draft that you review and submit in the web app. The agent never executes them, no matter how it is prompted. See [Approving consequential actions](../capabilities/#approving-consequential-actions). Every write and every draft is recorded against your account and can be listed at any time. For the full picture of what an agent can do with the server, see [Agent Capabilities](../capabilities/). ## Connect Every client needs the same one thing, the server URL: Text ``` https://bff.resim.ai/mcp ``` If you use Cursor or VS Code, one click does it: [Add to Cursor](cursor://anysphere.cursor-deeplink/mcp/install?name=signalflag&config=eyJ1cmwiOiJodHRwczovL2JmZi5yZXNpbS5haS9tY3AifQ%3D%3D) [Add to VS Code](https://insiders.vscode.dev/redirect/mcp/install?name=signalflag&config=%7B%22type%22%3A%22http%22%2C%22url%22%3A%22https%3A%2F%2Fbff.resim.ai%2Fmcp%22%7D) Your editor asks you to confirm, then prompts for sign-in on first use. The manual steps for every client are below. ### Claude Code Bash ``` claude mcp add --transport http signalflag https://bff.resim.ai/mcp ``` Your browser opens a consent page, then SignalFlag sign-in. Tokens refresh on their own. If you set up SignalFlag with the signalskills plugin, it registers this server for you. See [Set Up SignalFlag with an AI Coding Agent](../coding-agent/). ### claude.ai Go to Settings, then Connectors, then Add custom connector, and enter the server URL. Use this client for asking questions about results rather than working in a repository. It can render charts inline. ### Claude Desktop Open Settings, then Developer, then Edit Config, and add the `signalflag` entry to your `mcpServers`: JSON ``` { "mcpServers": { "signalflag": { "url": "https://bff.resim.ai/mcp" } } } ``` Restart Claude Desktop. You will be prompted to authenticate through your browser on first use. ### Cursor Open Settings, then MCP, and click Add new global MCP server: JSON ``` { "mcpServers": { "signalflag": { "url": "https://bff.resim.ai/mcp" } } } ``` Cursor's callbacks are already registered. ### CI and other machine credentials A pipeline cannot open a browser, so it uses a machine credential. Ask us for one bound to your organization, then send the token as a bearer header: Text ``` Authorization: Bearer ``` Machine credentials get reads, writes and drafts, with two differences: - A draft created by a machine names a reviewer, and that person submits it in the web app. Your pipeline can prepare a run, but a human still starts it. - Machine credentials cannot see or steer your browser tabs. ## Scopes On first connection you consent to three scopes: | Scope | Allows | | ---------- | -------------------------------------------------------------------------------- | | `read` | Listing and fetching anything your account can already see | | `write` | Setting up entities: projects, experiences, suites, metrics configs, dashboards | | `ui:steer` | Moving a view in a SignalFlag browser tab you have open, and showing you a toast | `ui:steer` lets an agent navigate a tab you already have open. It cannot submit anything, each tab has a steering toggle you control, and every command is recorded. It is still rolling out, so it may not be available to your organization yet. See [Agent Capabilities](../capabilities/#working-alongside-you-in-the-app). ## Your first run Ask your agent something about your tests and it should just work. There is nothing to configure and no first command for you to run. If you want somewhere to start, pick the row that matches what you have today. | Where you are | Try asking | | ------------------------------------ | ---------------------------------------------------------------------------------------- | | Nothing set up yet | "What projects do I have?" then "Create a project called nav-regression." | | A project, no experiences | "Register the scenarios in `s3://my-bucket/nav/` as experiences and tag them `nav`." | | Experiences and a build, nothing run | "Build a test suite from the `nav` experiences and draft a run against my latest build." | | Tests already running | "How did my last batch do?" or "Why did the failing tests in it fail?" | | Metrics you want to change | "Add a metric tracking median goal distance per test, and validate it before pushing." | Ask the first question whatever stage you are at. It confirms the connection works and tells you what your agent can see. Creating a project, registering experiences and pushing a metrics config all happen immediately. Running the suite does not. That comes back as a draft with a link, and nothing runs until you submit it. If your results already live on disk, such as sim runs, test reports or recorded logs, the [Set Up SignalFlag with an AI Coding Agent](../coding-agent/) guide walks your agent through getting them in. ## Rate limits and budgets | Limit | Value | | --------------------------------------- | -------------- | | Calls per user | 120 per minute | | Calls per organization | 600 per minute | | Data queries per organization | 200 per day | | Log and batch analyses per organization | 100 per day | Exceeding a per-minute limit returns HTTP 429 and a JSON-RPC error. Back off and retry after a short delay. The per-day budgets cover the tools that run real queries or call a model. Exhausting one returns a `budget_exhausted` error rather than failing quietly. ## Troubleshooting Errors come back with a code your agent can act on. | Code | Meaning | | ------------------------- | ----------------------------------------------------- | | `not_found` | The id does not exist, or is not in your organization | | `invalid_argument` | The arguments do not satisfy the tool's schema | | `insufficient_permission` | Your account cannot perform this operation | | `rate_limited` | Slow down and retry | | `budget_exhausted` | A daily query or analysis budget is spent | | `membership_required` | Your account is not a member of an organization yet | A missing authorization scope comes back as HTTP 403 with a `WWW-Authenticate` header, which compliant clients use to prompt you for consent again. Two things to try when a session goes badly rather than errors outright. If your agent is guessing at project names or calling tools at random, it skipped orienting itself. Tell it to run `whoami` first, then ask again. If it reports that it cannot write anything, check `whoami` for your permissions. An account with read-only permissions sees every read tool and no others. # Agent Capabilities The SignalFlag MCP server gives your agent 126 tools. You do not need to learn them. This page covers what you can ask an agent to do, which tools it will reach for, and where it has to stop and wait for you. Setting up the connection This page assumes your agent is already connected. For Claude Code, Cursor, the claude.ai connector and CI credentials, see [MCP Server](../server/). ## Three kinds of tool Every tool falls into one of three classes, and the class decides whether your agent acts or asks. | Class | What happens | Covers | | ----- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | Read | Runs immediately | 77 tools. Listing and fetching anything, log contents, metrics, analyses, quota | | Write | Runs immediately, and is recorded | 42 tools. Setting things up: projects, branches, builds, systems, experiences, tags, test suites, metrics configs, dashboards, notes | | Draft | Never runs | 6 tools. Launching, rerunning, canceling, archiving, deleting, and changing agent instructions | One more sits outside the three: `steer_ui_session` moves a view in a browser tab you have open, and is covered under [Working alongside you in the app](#working-alongside-you-in-the-app). Your agent can create an experience or push a metrics config on its own. It cannot launch a batch, rerun a job, cancel a run, archive an experience or delete anything. Those produce a draft that you review and submit in the web app. Every write and every draft is recorded against the account whose token the agent is using, and you can list that history at any time. ## Skills Skills are task instructions the server ships to your agent, so you do not have to explain how SignalFlag works every time. Your agent fetches one with `get_skill` after orienting itself, without being asked. See [MCP Server](../server/#your-first-run) for what that looks like from your side. | Skill | For | | ----------------------- | ---------------------------------------------------------- | | `onboard` | Getting a repository from nothing to a first drafted batch | | `run-tests` | Choosing a build and suite and drafting a run | | `diagnose-batch` | Working a failing batch back to a root cause | | `regression-check` | Comparing a branch against main | | `author-metrics-config` | Writing or fixing a metrics config | | `dashboards` | Building and refreshing dashboards | | `stage-experiences` | Bulk-registering and tagging experiences | | `containerize-sim` | Packaging a simulator into a build image | | `fleet` | Checking hardware agents and queues | | `drafts` | Working with actions that are waiting for a human | | `ui-sessions` | Working alongside you in an open browser tab | These are separate from the [signalskills](https://github.com/resim-ai/signalskills) skills, which you install in your agent to get results you already have into SignalFlag. See [Set Up SignalFlag with an AI Coding Agent](../coding-agent/). ## Capabilities by task ### Getting oriented Ask what projects exist, what changed recently, or where something lives. `list_projects`, `get_project`, `list_branches`, `list_builds`, `get_build`, `list_systems`, `get_system`. `get_project` also returns your project's agent instructions, which is how you give every agent working on a project the same standing context. ### Setting up a project to test Ask it to register experiences from a bucket, tag them, and assemble a suite. `upsert_experience`, `validate_experience_location`, `upsert_experience_tag`, `add_experience_tags`, `create_test_suite`, `revise_test_suite`, `register_build`, `upsert_system`, `create_branch`. These run without asking you. They are upsert-by-name, so re-running the same request updates rather than duplicating, which makes them safe to repeat when a setup script is half-finished. ### Running tests Ask it to run a suite against a build. `draft_launch_batch`, then you submit. Your agent assembles the run, validates the ids, and hands you a link. You open it, see a prefilled form with a banner naming the agent and what it intends to do, change anything you like, and submit. The agent can then read back what you actually ran. ### Seeing how a run went Ask how the last batch did, or whether a suite is passing. `list_batches`, `get_batch`, `list_jobs`, `get_job`, `get_metrics_summary`, `get_metric_detail`, `get_metric_chart`, `list_batch_runs`, `get_batch_usage`. `get_batch` accepts a wait, so an agent can watch a running batch to completion and summarize it when it lands rather than polling in a loop. In claude.ai and Claude Desktop, `get_metric_chart` returns a chart your agent can render inline. In a terminal-based coding agent it will describe the figure instead. ### Diagnosing a failure Ask why a test failed. `list_batch_errors`, `get_job`, `read_log`, `list_logs`, `get_log`, `list_events`, `get_event`, `request_log_analysis`, `get_log_analysis`, `get_batch_analysis`. Error codes distinguish problems in your code and data from problems on our side, so an agent can tell whether to fix something or simply rerun. `read_log` reads a window of a log with search and tailing, so a large log does not have to be pulled down whole. ### Catching regressions Ask whether a branch broke anything relative to main. `get_batch_suggestions` finds the right baseline to compare against, then `compare_batches` returns the tests whose status changed. ### Authoring metrics and dashboards Ask it to add a metric, fix a failing one, or build a dashboard. `get_metrics_config`, `get_metrics_config_schema`, `list_topics`, `get_topic_schema`, `preview_topic_data`, `preview_metric`, `validate_metrics_config`, `push_metrics_config`, `list_chart_templates`, `list_metrics_sets`, `validate_status_query`, `list_dashboards`, `get_dashboard`, `upsert_dashboard`, `refresh_dashboard`. Your agent can validate and preview a metric against real data before pushing anything, so ask it to preview before it pushes. ### Asking questions of your data Ask a question that no existing metric answers. `query_emissions` runs read-only SQL over your emitted data. `preview_topic_data` samples a topic when you need to see its shape first. These cost money to run, so they are budgeted. See Limits below. ### Capturing what you learned Ask it to write down what it found so the next investigation starts further along. `save_org_note`, `list_org_notes`, `get_org_note`. Notes an agent writes are marked unreviewed until a person confirms them, and agents treat unreviewed notes with suspicion. Confirm the ones you want relied on. ### Watching your fleet Ask about hardware agents and queue depth. `list_agents`, `get_agent`, `get_pool_labels`. ### Working alongside you in the app An agent acting for you can see the SignalFlag tabs you have open, and nothing else. Once it can see a tab, it can move that tab around so the two of you are looking at the same thing. Ask for it in the terms you would use with a person. | Try asking | What happens | | ---------------------------------- | ---------------------------------------------------------------- | | "What am I looking at?" | It reads the open tab and answers from the same data you can see | | "Take me to the failing job" | The tab navigates to that page | | "Show me the metrics tab" | The tab switches tabs where the page has them | | "Tell me when this batch finishes" | It waits for the page to change, then shows you a toast | `list_ui_sessions`, `get_ui_session` and `steer_ui_session` are the tools behind those. Look for the pill at the bottom left of the app, labeled "Agent steering on" or "Agent steering off". It is on by default, so an agent you are already talking to can move your view. Switch it off and the server refuses commands for that tab. The setting is per tab and survives a reload, so turning it off in one tab does not affect another and does not quietly come back. Four things bound it. Steering only moves a view; forms, launches and deletions stay behind your own click. Every command shows a toast naming the agent that sent it. An agent sees only the tabs belonging to the account whose token it holds. And machine credentials cannot see your tabs at all, so a CI pipeline can never look at your browser. ### Auditing what agents have done Ask what the agents working in your organization have actually changed. `list_agent_actions` returns the full trail with who requested each item, recorded against the account whose token the agent was using. It covers the writes that happened without asking you as well as the proposals that waited. `get_draft` returns the state of any single proposal. ## Approving consequential actions Six actions never happen on their own. Your agent proposes them, and you approve or decline them in the web app. | Your agent proposes | Tool | | ----------------------------------------------------------- | --------------------------- | | Launching a build against experiences, tags or a test suite | `draft_launch_batch` | | Rerunning a whole batch, or named jobs within it | `draft_rerun` | | Canceling a running batch | `draft_cancel` | | Archiving entities of one type | `draft_archive` | | Permanently deleting one entity | `draft_delete` | | Replacing a project's agent instructions | `draft_update_instructions` | Everything else an agent can write, it simply writes. There is no permission that turns this off and no override for trusted agents. ### Approving one You get a notification in the web app and a link. Opening it takes you to the form for that action, already filled in, with a banner naming who proposed it and what it will do. Change anything you want to change, then submit. The action runs as if you had filled in the form yourself, and your agent can see what you changed and carries on against what you actually ran. If you do not want it, close it. A proposal you never submit expires after seven days. An agent can also withdraw its own, so one you were about to look at may occasionally disappear. ### Proposals from CI A pipeline running on a machine credential can prepare work but cannot approve it. When a machine proposes something it names a reviewer, and that person submits it in the web app. Your pipeline can register the build, assemble the suite and propose the run, and a human decides whether to spend the compute. ### Why one is waiting, or is not | State | Meaning | | ------------ | ---------------------------------------------------- | | `draft` | Waiting for a person | | `submitting` | Someone has submitted it and it is being carried out | | `submitted` | Done, the action ran | | `discarded` | Withdrawn by the agent, or declined | | `expired` | Nobody acted within seven days | | `stale` | Submitting it would no longer do anything | A proposal goes stale when the world moves on, such as a cancel for a batch that has since finished. ## Budgets Two of the groups above cost money to run, so they are budgeted per organization per day: metrics compute through `query_emissions`, `preview_metric` and `preview_topic_data`, and token usage through `request_log_analysis` and `request_batch_analysis`. An agent that exhausts one gets a clear error rather than a silent failure. See [MCP Server](../server/#rate-limits-and-budgets) for the current limits. ## When no tool fits Two escape hatches cover cases the tool surface does not reach yet. `call_customer_api` makes a read-only call to the public REST API, and `query_graphql` runs a read-only GraphQL query. Both are available only over MCP. If you find yourself relying on them, tell us which tool is missing. # Reeves Reeves is the AI assistant built into the SignalFlag web app. It's powered by Claude and is available throughout the UI to help you triage failures, explore your data, draft metrics, and assemble dashboards — without leaving the page you're on. ## Where to find it Reeves lives in a chat panel inside the SignalFlag web app. Open it from any page — batches, jobs, dashboards, test suites, project overview — and it picks up the context of what you're looking at automatically. You can ask "why did this batch fail?" or "what does this metric mean?" without restating which batch or metric you mean. Your conversation is persistent and per-user. History, page context, and any in-flight work (for example, a dashboard being drafted) survive page navigation and reloads, so you can keep working as you click around the app. ## What Reeves can do ### Triage failures - Summarize errors across a batch and surface common failure modes. - Pull up the [agentic log summary](../log-summary/) for a failed job and reason over it. - Compare jobs and batches to narrow down regressions. ### Explore your data - List and inspect projects, branches, builds, systems, experiences, and test suites. - Query batches and jobs with status filtering (passing, failing, in-progress, etc.). - Navigate you directly to the relevant page when that's the fastest answer. ### Draft metrics - List available metric topics and inspect their schemas. - Preview sample rows against your real data. - Co-write SQL with you and preview what the draft query would return before saving. - Push a pre-filled draft into the metric editor for you to review and save. Reeves never saves metrics on your behalf. - Edit existing metrics the same way — Reeves opens them in the editor with the proposed changes staged. ### Build dashboards The dashboard-creation workflow is a multi-phase agent: it researches your project's data, proposes a layout, and assembles the dashboard with appropriate metrics. It pauses at checkpoints for your approval before moving on, so you stay in control of the result. ### Capture team knowledge Reeves can save org-level notes — project conventions, naming, gotchas it has learned from you — so they're available to everyone on your team in future sessions. You can ask Reeves to update or add a note at any time. ## Scope and limitations - **No destructive actions.** Reeves proposes metrics, dashboards, and notes as drafts. It does not delete or destructively edit anything on its own. - **Web app only.** Reeves runs inside the SignalFlag web app. For read-only programmatic access from your IDE or other tools, use the [MCP server](../mcp/server/) instead. - **Time and iteration budgets.** Long-running subagents (dashboard creation, metric drafting) enforce per-phase iteration and wall-clock budgets to prevent runaway loops. If a phase hits its budget, Reeves stops and reports back rather than continuing silently. - **Best with page context.** Reeves is most useful when you ask it questions from a relevant page — the batch you're triaging, the dashboard you're building, the test suite you're reviewing. Open-ended questions with no page anchor work too, but may need more prompting. ## Feature availability Reeves is enabled on a per-organization basis. If you don't see the chat panel in the web app, contact [info@signalflag.ai](mailto:info@signalflag.ai) to have it enabled for your organization. # Foxglove primary sites If you produce large simulation results, you may benefit from streaming them from a Foxglove [Primary Site](https://docs.foxglove.dev/docs/primary-sites/introduction) when visualizing. Once set up, any `.mcap` files saved to the `/tmp/resim/outputs/foxglove` directory at the end of a job will be automatically uploaded to your inbox and processed by your primary site for streaming. Clicking the `Open in Foxglove` button in your job results within the SignalFlag app will still take you to Foxglove, but will now stream your results. This will mean your visualizations will load faster and play back more smoothly. Your results will still be uploaded for storage with SignalFlag - they’re safe with us and still available to open from our cloud if you’d like. Foxglove-hosted Primary Sites We don't currently support [Foxglove-hosted Primary Sites](https://docs.foxglove.dev/docs/primary-sites/introduction/#foxglove-hosted), but they're on our roadmap. Please get in touch if you're interested in using them. ### Self-hosted primary sites If you have a [self-hosted primary site](https://docs.foxglove.dev/docs/primary-sites/introduction#self-hosted) in S3, SignalFlag can upload your MCAPs to your inbox bucket so they’re ready to stream when your simulations complete. To give us access, extend your bucket’s access policy to add the following statement. Make sure you replace the `inbox-bucket-name` placeholder value with the name of your bucket and the `Principal` with your customer-specific role as described [here](../../setup/resim-data-access/#iam-roles). JSON ``` "Statement": [ { "Sid": "AllowSignalFlagUploads", "Effect": "Allow", "Principal": { "AWS": ["arn:aws:iam::083585393365:role/resim-customer-yourorg.ai-prod"] }, "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::inbox-bucket-name", "arn:aws:s3:::inbox-bucket-name/*" ] } ] ``` Contact us to let us know what S3 URI your MCAPs should be uploaded to, i.e. `s3://inbox-bucket-name` and your Foxglove organization slug (see the top of your [organization settings page](https://app.foxglove.dev/~/settings/general)). We'll get you set up, and you'll be streaming your results in no time. # SignalFlag log ingestion ## Introduction An important part of testing embodied AI is field testing and gathering examples from real-world deployments. Historically this type of testing requires manual inspection of the generated logs and is decoupled from the simulation testing workflows. The SignalFlag platform, however, makes it possible to ingest these logs and compute metrics automatically as part of a SignalFlag metrics flow. One can then further process these logs to generate replay tests for interesting subsets. This guide explains how log ingestion works with the SignalFlag CLI. ## Prerequisites The SignalFlag CLI requires that any logs to be ingested have already been uploaded to a cloud bucket and that SignalFlag has been granted access. This is described in the [setup guide](../../setup/resim-data-access/). ## Core concepts The log ingestion process in SignalFlag will: 1. Store the logs as experiences, each with a user-supplied name, tagged as `ingested-via-resim` 1. Create a test batch that will ingest the logs and run a given metrics build on them. The batch will contain a test for every log ingested and will, by default, attempt to also calculate batch metrics on the logs 1. Once completed, the test results will be available to visualize in the usual way, in the results page ## Configuration The SignalFlag CLI functionality takes several required and optional parameters. To see full documentation of each, please refer to `resim ingest help`. ### Required parameters - `project`: the name of the project to ingest the log to (can be omitted if you have used `resim project select` to set your default project) - `system`: the name of the system that this log belongs to e.g. `full-stack` - `log`: the name and location of the logs you would like to ingest, specified as `log-name=log-location`. This accepts multiple log name and location pairs to ingest multiple logs. For ingesting large numbers of logs, it is recommended to use the [config file](#yaml-log-specification-file) approach - `metrics-build-id`: the unique ID for the metrics build to run, which can be found via `resim metrics-builds list` or in the SignalFlag app. ### Optional (and recommended) parameters - `branch`: the name of the branch that you would like this log ingestion to be related to (e.g. what branch of your system created it). If no branch name is supplied, it will automatically be ingested under `log-ingest-branch`. - `version`: what version of your code you would like this log ingestion to be related to (e.g. what release or commit sha). If no version is supplied, then `latest` is used. - `tags`: a comma-separated list of tags to be added to the ingested log; by default SignalFlag applies `ingested-via-resim`, but additional metadata tags can be provided here. ### Alternative approaches #### YAML log specification file The SignalFlag CLI supports providing a location to a YAML file with the log names and locations via `--config-file `. The syntax of the YAML input is: YAML ``` # logs.yaml logs: - name: "log1" location: "s3://bucket/path1" - name: "log2" location: "s3://bucket/path2" ``` #### Deprecated single-log ingestion It is possible to use two flags to ingest a single log, rather than the `--log "name=location` syntax: - `log-name`: the unique name that you would like to give to this log (often a field test id or timestamped) - `log-location`: a cloud storage path to find the log (currently only `s3` is supported) ### Example Bash ``` resim ingest \ --project "my-project" \ --system "My Drone Stack" \ --log "Drone-Test-42=s3://my-bucket/prefix" \ --log "Drone-Test-43=s3://my-bucket/prefix2" \ --metrics-build-id "" \ --branch "main" \ --version "1.1.0" \ --tags "field-test,hardware-v2" ``` ## Using a custom ingestion build If you would like to pre-process a log before ingestion and cannot do that within a metrics build (due to wishing to reuse it in non-ingestion settings), the SignalFlag CLI allows you to run the `resim ingest` command with your own **build**. In this case, you would simply use: Bash ``` resim ingest \ --project "my-project" \ --build-id "" \ --log "Drone-Test-42=s3://my-bucket/prefix" \ --metrics-build-id "" \ --tags "field-test,hardware-v2" ``` In this case, one cannot supply the `--system`, `--branch`, or `--version` flags as that information is derived from the build. # Agentic Log Summary When a test fails, SignalFlag automatically generates a log summary to help you understand why. Rather than manually digging through your test container's logs, you get a concise explanation of the test failure mode alongside specific points of interest — each annotated with the file name, line number, and a brief note on its relevance. ## Feature availability This feature is available under the **Logs** tab for any test with the `error` state. Log summaries are enabled on a per-organization basis. If you don't see the Log Summary panel for failed jobs, please contact [info@signalflag.ai](mailto:info@signalflag.ai) to have it enabled for your organization. # Open in IDE When a test fails, you can send the failure context directly to your IDE for AI-assisted debugging. SignalFlag packages the log summary, relevant log lines, and test identifiers into a prompt and stages it in your IDE's AI chat panel — ready for you to review and submit. ## Supported IDEs | IDE | Requirements | | ------- | ---------------------------------------------------------------------------------------------------------- | | VS Code | [SignalFlag VS Code extension](https://marketplace.visualstudio.com/items?itemName=resim-ai.resim) v0.1.3+ | | Cursor | [SignalFlag Cursor extension](https://open-vsx.org/extension/resim-ai/resim) v0.1.3+ | ## Using the feature 1. Navigate to a test with an **Error** status. 1. Click the **Open in IDE** button near the top of the page. 1. Select your IDE from the dropdown menu. 1. Your IDE will open with a pre-filled prompt in the AI chat panel. 1. Review the prompt, then submit it to start investigating the failure. Note This prompt is staged as a draft — it won't be sent to the AI assistant until you review and submit it. ### Copy Prompt If you use an AI tool that isn't listed in the dropdown (Claude Code, Codex, etc) - select **Copy Prompt** from the dropdown instead. This copies the failure context to your clipboard as a ready-to-use prompt that you can paste into any AI assistant. ## Troubleshooting **I don't see the "Open in IDE" button** The button only appears on tests in the Error state. This feature is also enabled on a per-organization basis — contact [info@signalflag.ai](mailto:info@signalflag.ai) if you don't see it on failed tests in the Error state. **My IDE didn't open** For VS Code and Cursor, make sure the SignalFlag extension is installed and active. If the link expires (it's valid for 30 seconds), click **Open in IDE** again. # Debug mode ## Introduction Debugging why a batch has failed by making changes to the configuration and rerunning can be fiddly. To help with debugging in certain situations, SignalFlag has developed a debug mode which can be used to interact with your image in a live environment. ## Prerequisites - The [SignalFlag CLI](https://github.com/resim-ai/api-client) must be installed on your local device - Your image must include a shell (e.g. `sh` or `bash`) and the `sleep` command. - You need an experience ID or name - You also need a batch name or a build ID ## Running a debug session Run the debug command: Bash ``` resim debug --project --build --experience my-test-data ``` In the background, SignalFlag will launch a new ephemeral environment and run your build image in it. The experience data will be available (read-only) in `/tmp/resim/inputs` as in a standard batch. If your image is very large, it can take a few minutes to set up the debug environment. By default, the command that runs in your container will be `sh`, but you can override this by setting the `--command` flag on the `debug` command. Once the batch is running, you will be dropped into a shell. From this shell, you can run commands as normal to inspect the environment and configuration. For example, you could: - check dependencies are installed - check the input data in `/tmp/resim/inputs` is present and structured correctly - run entrypoint scripts to see immediate output and debug errors The debug batch will run for 1 hour or your experience's timeout setting, whichever is longer. After the timeout or when you exit the shell by running `exit` or pressing Ctrl-D, SignalFlag will terminate the debug batch. Any files you write to `/tmp/resim/outputs` will be uploaded as in a standard batch for later inspection. Nested output directories are archived by default after the run, except for `/tmp/resim/outputs/logs/`, whose contents remain unarchived. ### Debug mode for multi-container builds If you have a multi-container build setup, which is described by a docker compose file, you can specify the container you want to bash in using the `--container` flag, which should be the `container_name` parameter that you specify in the compose file for a service. # Multi-container builds ## Introduction A truly single-node system is hard to find in the real world. Many robots are made up of multiple compute nodes working together, and even a robot that had just a single node onboard would typically interact with an offboard system to some degree. The SignalFlag platform supports running multiple containers together in a single experience, to better simulate your real world systems. This may be useful as a representation of your multi-node robot, or multiple robots working together, or your robot communicating with an offboard cloud service, or similar. ## Docker compose We support defining your experience's multiple containers via a [docker-compose](https://docs.docker.com/reference/compose-file/) file. You can see an example that we use for testing [here](https://github.com/resim-ai/api-client/blob/main/testing/data/test_build_spec.yaml). Our execution environment supports a subset of the configuration options of docker-compose. The expectation is that you can pass SignalFlag the same compose file that you would use for running your system locally on Docker, without any changes. However, not all options in the file will take effect in the SignalFlag platform. Most unsupported options will be ignored, but a small number that are strictly incompatible will surface as errors that will prevent your batch from running. ### Volumes We support the definition of named volumes in your docker-compose file. Each volume specified in this way will be created ephemerally for the duration of your running experience. It is available to mount as read/write into any or all services in your experience as per normal docker-compose behavior. In addition to any named volumes, the standard [/tmp/resim/](../../setup/build-images/#inputs-and-outputs) input/output directories will be automatically mounted into every container. ### Networks Custom networks are not supported. All services will be run in a single default network. Your experience will not expose any ports to the outside world. Each service that listens on a port must listen on a unique port, as containers may share the same host and port space, and thus conflict. ### Services Each service is run as a separate container. Services may use the same image, or different images. See [build image setup](../../setup/build-images/) for more information on how to create a build image for your service and make it available to SignalFlag. #### Startup dependencies Unlike Docker, `depends_on` does not control start up ordering. If your containers have startup dependencies they must fail healthchecks (or exit) and expect to be restarted continuously until their dependencies are running and they can start up correctly. See [Container Lifecycle](#container-lifecycle) below for how `depends_on` works with container restarts. #### Restart See [Container Lifecycle](#container-lifecycle) below for `restart`. #### Resources Service resource requests are supported. The resource requests of all containers must sum up to less than the total amount of resources requested in your [system](../../setup/systems/). If per-container resource requests are not specified, your containers will share the available resources without any resource guarantees - they may, for example, be OOM-killed. You can request GPUs for a given service. YAML ``` services: my-sim: deploy: resources: reservations: devices: - driver: nvidia capabilities: [gpu] count: 1 ``` Services will not share GPUs; each service reserving a GPU will receive the requested number of dedicated GPUs. #### Other Other compose service blocks that should function as expected: - `entrypoint` / `command` - `workingdir` - `healthcheck` ### Environment variables Environment variables can be passed in from your compose file. They may also be set at the experience level. Environment variables set at the experience level will take precedence over those set in your compose file. Environment variables set at the experience level will be set across all services. ### Profiles [Profiles](https://docs.docker.com/reference/compose-file/profiles/) can be used for services. The specific profile to use can be set at the experience level. Bash ``` resim experiences create [...] --profile {name} ``` ## Container lifecycle When testing a multi-node system, it's typical to have a number of nodes representing the various parts of the system, plus a single node that actually controls the running of the test, often referred to as the "test orchestrator". Nodes representing parts of a robot, for example, may not have things like "graceful shutdown" handling. So if we wanted to wait for all containers to exit to finish a test, that might not be possible. Our system focuses on the "test orchestrator" for lifecycle management, such that you do not need to add complex startup or shutdown logic to other containers just for test purposes. At least one container **must** have a label of `resim.ai/isTestOrchestrator` = `true`. When monitoring a test, a non-zero exit code in any orchestrator container will fail the test. When all orchestrator containers have exited with a zero exit code, the test is considered successful. For non-orchestrator containers, their lifecycle is handled in two phases. Initially, if a container `depends_on` another container, any failure of that container will be ignored until after their dependency is satisfied - when the other container is running. After a container has satisfied dependencies (if any), any failure of that container will fail the test, **unless** it has `restart` defined, in which case it will happily restart without affecting the test. An orchestrator will not restart. Non-orchestrator containers may exit successfully at any time. **Summary of capabilities** | | Orchestrator Containers | Non-Orchestrator Containers | | ------------------------------------ | --------------------------------------------------- | --------------------------------------------------- | | `depends_on` | Exits are ignored before dependencies are satisfied | Exits are ignored before dependencies are satisfied | | exit code non-zero with `restart` | Not possible | Container restarts | | exit code non-zero without `restart` | Fails the test | Fails the test | | exit code zero | Succeeds the test when all exit | Ignored, not restarted | ### Termination on failure When the test enters a failed state due to an inappropriate container exit or timeout, any other running containers will be terminated. ## Runtime considerations ### Container logs Logs from each container will be available in the SignalFlag web app, as `experience-CONTAINERNAME-container.log` ### Container state Resource metrics (CPU, memory) are automatically collected for each container. They can be viewed in the SignalFlag web app. ### Cross-container communication Your containers will be able to communicate with each other over the default network, by using the service name as the hostname. For example, if you have a service defined as `my-service`, you can communicate with port `8080` on that container with by connecting to `my-service:8080`. ## Creating a multi-container build Once you have a compose file created, it is as simple as passing it to `resim builds create` in place of where you would use `---image` to pass in a single image URI: Bash ``` resim builds create --branch=my-branch --system=my-system --version=1.0.0 --build-spec=my-docker-compose.yml ``` Once created, you can define and run test suites and batches as normal using this build ID. The SignalFlag web app will show the full compose file for each experience defined this way. ### Direct API usage If you use the API directly, the compose file is passed in as the `buildSpecification` field as part of the `createBuildForSystem` or `createBuildForBranch` inputs. It can be passed in as either YAML or JSON string content. ## Helpful tips ### Image tags Every image in the compose file should be fully specified with the tag for the desired version of that image to be pulled and run. Make sure the compose file specifies these, they are not parameterized later. In some testing scenarios, you may not need all images in the environment to be freshly built for each test, and you might find it more efficient to use different tags across images, and only change some of the image tags on each test. YAML ``` services: onboard-node-a: image: my-image-a:test-build-1-sha onboard-node-b: image: my-image-b:test-build-1-sha cloud-service-node: image: my-image-c:stable-v1 ``` ## Archive The **Archive** feature in SignalFlag enables soft deletion of artifacts such as [Experiences](../../core-concepts/#experience) and [Test Suites](../../core-concepts/#test-suite), allowing users to manage their workspace without permanently losing data. Archiving helps reduce clutter and enforce data hygiene, especially as autonomy development scales, while preserving traceability and compatibility metadata. Archiving is *non-destructive* — the data remains available for inspection and can be restored at any time. However, archived items are excluded from default workflows and cannot be used in most UI-based actions. Test results (Batch/Test details, Reports, etc) will continue to reflect their association with any archived entity. However these entities will no longer appear in experience lists in either the CLI or web UI and cannot be selected in the UI for running tests. They can still be invoked directly for experimentation or validation through the CLI. Archived test suites and experiences retain their associated [System](../../core-concepts/#system) compatibility. ### Archiving experiences Archiving an **Experience** removes it from active use and visibility in the SignalFlag platform’s UI and default CLI queries. Upon archiving, the experience is automatically removed from all [Test Suites](../../core-concepts/#test-suite) it belongs to and results in a new version of the test suite. Restoration does *not* re-add it to these test suites — this must be done manually. Since active experiences must have unique names, restoring an archived experience with a duplicate name will result in a numeric suffix being added to the restored experience (e.g. `Experience Name (1)`). ### Archiving test suites Archiving a **Test Suite** disables its use in future workflows but retains all historical versions and associations. If a Test Suite is linked to an **Executive Dashboard** on the Overview tab, the UI prevents archiving. # Experience caching Fetching [experience data](../../setup/adding-experiences/) from cloud storage can potentially involve a large amount of data transfer depending on the size of the experience. In a lot of cases the experience data either doesn't change or at least doesn't change very often. Thus it is not particularly efficient or cost-effective to pull identical experience data for every test that requires it. For this reason we have introduced the experience caching feature. ## How does it work? When we run your workloads, they are supervised by a worker. That worker does jobs like pulling the experience data, collecting container metrics and collating and storing your logs and metrics. For the experience data, we mount a shared storage location that is unique and isolated to your organization and create a directory with the ID of the target experience. We then sync the contents of your experience location with that directory. That directory is then mounted in your workload container(s) at `/tmp/resim/inputs` in the case of your experience build and `/tmp/resim/inputs/experience` for metrics builds. When we do this for the first time, we record a timestamp for when it was last updated. For subsequent uses of that experience, when we come to the sync job, we check each file in the cloud storage location's `lastModified` timestamp against our timestamp and only update or download files that are newer than that. Any files that do not exist in the cloud storage location are then deleted from our cache as well. This means we're pulling the minimum subset of data every time, but ensuring we check the status of every file individually for changes. ## What do I need to do? Absolutely nothing. We'll let you know if this feature is available to you, but please reach out if you need more info. ## What do I need to know? ### The experience data is read-only Because we share the experience data with potentially concurrent running jobs, we have to ensure its integrity. Thus the directory is mounted read-only. To clarify, in an experience job, the `/tmp/resim/inputs` directory is read-only, and in a metrics job, the `/tmp/resim/inputs/experience` directory is read-only. ### How do I invalidate the cache? The easiest way to ensure the cached copy of a particular experience is pulled fresh is to "touch" (i.e. modify in some way) every file in the remote location. As we check the `lastModified` timestamp for every file individually, this will ensure it is definitely newer than the cached copy. It should be noted that this shouldn't be necessary under normal circumstances, only if you experience any unexpected behaviour in your tests. If you've done this and you're still experiencing unexpected behaviour, reach out to us and we'll help. # Asset caching Fetching [asset data](../../core-concepts/#asset) from cloud storage can involve a large amount of data transfer, especially for assets like HD maps, 3D meshes, or ML model weights. In most cases asset data changes infrequently, often much less frequently than your code. To avoid redundant downloads, assets are cached by default. This works in the same way as [experience caching](../experience-caching/), so if you are already familiar with that feature, the same principles apply. ## How does it work? When we run your workloads, the SignalFlag worker mounts a shared storage location that is unique and isolated to your organization. For each asset, we create a directory identified by the asset ID and revision. We then sync the contents of your asset's cloud storage location into that directory, which is mounted read-only in your workload container(s) at `/tmp/resim/assets/`. On the first sync, all files are downloaded and we record a timestamp. On subsequent uses of the same asset revision: 1. We check each file in the cloud storage location's `lastModified` timestamp against our recorded timestamp and the file size. 1. Only files that do not match. 1. Files that no longer exist in the cloud storage location are removed from the cache. This means we always pull the minimum subset of data while ensuring every file is individually checked for changes. ## Cache-exempt assets Some assets change frequently enough that caching is counterproductive. When creating an asset, you can mark it as **cache-exempt**: Bash ``` resim assets create \ --project "my-project" \ --name "Rapidly Changing Data" \ --description "Data that updates every run" \ --locations "s3://my-bucket/volatile-data/" \ --mount-folder "volatile" \ --version "1.0.0" \ --cache-exempt ``` Cache-exempt assets are always fetched fresh from cloud storage, bypassing the caching mechanism entirely. ## Asset data is read-only Because cached asset data is shared across potentially concurrent running jobs, we must ensure its integrity. The `/tmp/resim/assets/` directory is mounted **read-only**. If your workload needs to modify asset data at runtime, copy the relevant files to a writable location (such as `/tmp/resim/outputs/` or any other writable directory) before modifying them. ## How do I invalidate the cache? The easiest way to ensure a cached copy of a particular asset is pulled fresh is to "touch" (i.e. modify in some way) every file in the remote location. Since we check the `lastModified` timestamp for every file individually, this ensures it is definitely newer than the cached copy. Under normal circumstances this should not be necessary as the sync mechanism handles changes automatically. If you have touched the files and are still experiencing unexpected behavior, reach out to us and we'll help. # Experience syncing For larger projects, it's often convenient to store a registry of [experiences](/core-concepts/#experience) and their properties locally for a number of reasons. For example: - Other tools besides SignalFlag may leverage this configuration to run local (or HiL) sims or manage test sets. - It may be useful for additional organization-specific metadata to be attached to experiences, even if said data isn't as relevant within the SignalFlag app. - Experience history can be stored in version control alongside code which can be useful when inspecting historical sim behavior. In order support these sorts of use cases, and many more, SignalFlag supports *Experience Syncing*. We define a schema which contains a list of experiences, their tags, systems, and test suites. Users can then produce config files matching that schema in a number of ways (more on this below). The key to making this config useful, however, is the [SignalFlag CLI](https://github.com/resim-ai/api-client), which can be used to mirror the config state to the SignalFlag project of your choice. This makes it very easy for users to make bulk programmatic updates to experiences (e.g. renaming, switching s3 paths, retagging) without having to manually navigate all the api endpoints needed for the specific category of update they're trying to accomplish. ## A simple syncing example The config schema is relatively simple and looks like this in practice: YAML ``` experiences: - name: scenario-survey-alpha description: Aerial survey over test zone locations: - s3://drone-missions/surveys/alpha-test-zone tags: - regression - my_custom_tag - name: new-scenario-system-check description: Regression validation run locations: - s3://drone-missions/system-checks/regression-1 tags: - progression systems: - demo_system customFields: - name: weather type: text values: - cloudy - rainy - name: version id type: number values: - 1.23 managedTestSuites: - name: Basic Suite experiences: - scenario-survey-alpha - new-scenario-system-check managedExperienceTags: - regression - progression ``` The full schema for this file is provided below in the [Appendix](#appendix-config-schema). This config can be mirrored onto an *existing* SignalFlag project like so: Warning This command should be run with care. It *will* affect other users of whatever project you run it on, so it's advisable to set up a test project if you're just testing out the tool. You can do this using the `resim projects create` command in the CLI. Bash ``` resim experiences sync \ --project \ --experiences-config ``` Let's walk through what this command will do. We hope that this is relatively intuitive, but there are a lot of edge cases so we break it down in detail: 1. The experiences in the project will be updated so that every experience in the config will exist with the specified parameters. Existing experiences will be updated and new experiences will be created. Experiences that *aren't* in the config will be archived, and any archived experiences that *are* in the config will be restored as they are updated. Experiences are typically identified in the config by unique name, but their `experience_id` can also be provided as a field to enable renaming of existing experiences so long as doing so doesn't violate the uniqueness constraint on names. Note that swapping or permuting names is not currently allowed even if it preserves uniqueness because computing a valid ordering for the updates would be complex. 1. The experiences tags are updated in the following manner: All experience tags listed for each experience are added to that experience if they don't already exist on that experience. All managed experience tags that aren't listed on each experience are removed from that experience if it does currently have them. The managed experience tags are the only tags that are ever removed from any experience. None of these changes apply to experiences we're archiving. **All of the tags listed must already exist in the project**. 1. The systems are updated similarly to the experience tags, but they're always additive. We never remove systems from experiences. **All of the systems listed must already exist in the project**. 1. All test suites in the `managed_test_suites` are revised to contain exactly the experiences listed in the config by their (possibly new) names. Other test suites may be revised, but this will only happen if experiences they currently contain get archived according to the earlier rules, forcing a revision. **All of the test suites listed must already exist in the project**. ## Where should I get a config? If you'd like to use this feature, there are a number of ways to obtain a useful config. First, you can just write one manually. That's probably not feasible for large projects however. In practice, you will likely want to generate your config using python or another language of your choice. The source data for this could be: - A directory containing all of your experiences. In this case, your config-generating script can walk through the directory parsing each experience to compute the right tags and systems for it, and then write out the config to a yaml file for you. - Another config. If you have another config describing your tests, the translation into the SignalFlag config should be a pretty straightforward script to convert one schema to another. - An s3 bucket containing your experiences. This is similar to the first option, although you'll likely be using `boto3` to get the information you need. As outlined in the [experience docs](/setup/adding-experiences/), experiences in s3 buckets work very well in SignalFlag. - An existing project with your experiences already in them. For this see the `--clone` option below. ## Other options Some other useful options bear mentioning: ### The `--update-config` flag When syncing a config to the app, you can update the provided config to include the `experience_id`s and other information for the existing and new experiences by passing the Boolean `--update-config` flag when calling `resim experiences sync`. This is useful if you want to check the config into version control and rename experiences over time. ### The `--clone` flag Similarly, you can also clone the existing state of the experiences in the app to the passed-in config using the Boolean `--clone` flag. E.g. Bash ``` resim experiences sync \ --project \ --experiences-config \ --clone ``` If the config path exists already, the experiences field in it will be replaced while the managed test suites and tags are unaffected. This is useful if you want to clone the experiences from an existing project to add them to another, or simply as a convenient way to get a list of all experiences with their tags and systems. Note This currently only populates the experiences. It does not try to determine which experience tags you'd like to be `managed_experience_tags` or which test suites you'd like to be in `managed_test_suites`. Furthermore, if the config exists and has the `managed_test_suites` field, the test suites therein will not be updated to reflect their current membership. This is because we don't currently have to fetch the current test suite membership during the normal sync operation, so we'd have to add it specifically for this. This is a feature we hope to add in the near future. ### The `--no-archive` flag You may not necessarily want to archive experiences that aren't listed in the config. This can be useful when batch adding a large number of experiences or when you don't generate an exhaustive list of them when generating the config. In this case, the `--no-archive` flag can be used to prevent unlisted experiences from being archived as they otherwise would. Note that such experiences will still be removed from managed test suites if they aren't included in the `experiences` field of such test suites. ## Appendix: Config schema ### ExperienceSyncConfig | Field | Type | Notes | | --------------------- | --------------------------------- | -------- | | experiences | list\[[Experience](#experience)\] | optional | | managedTestSuites | list\[[TestSuite](#testsuite)\] | optional | | managedExperienceTags | list[string] | optional | ### Experience | Field | Type | Notes | | ----------------------- | ----------------------------------- | ------------------------------------ | | name | string | **required, must be unique** | | description | string | **required** | | locations | list[string] | optional | | tags | list[string] | optional | | systems | list[string] | optional | | profile | string | optional | | experienceID | string | optional, must be unique if provided | | environmentVariables | list[object] | optional (fields: `name`, `value`) | | customFields | list\[[CustomField](#customfield)\] | optional | | cacheExempt | bool | optional | | containerTimeoutSeconds | int | optional | ______________________________________________________________________ ### CustomField | Field | Type | Notes | | ------ | ------------ | ------------------------------------------------------- | | name | string | **required** | | type | string | **required** (`text`, `number`, `timestamp`, or `json`) | | values | list[string] | optional | ______________________________________________________________________ ### TestSuite | Field | Type | Notes | | ----------- | ------------ | -------- | | name | string | optional | | experiences | list[string] | optional | ______________________________________________________________________ # Custom fields Experiences support adding **custom fields**. Custom fields are similar to tags, but allow specifying both keys and values. This allows you to add flexible metadata to your experiences. These can be used for organizational purposes when launching batches or managing your test suites. For example, you could add a custom field for "version", along with a version number, to experiences. You could then launch a batch containing all experiences where version is greater than 10. In general, we recommend working with custom fields via [experience syncing](../experience-syncing/), as it prevents the need to maintain the fields via manually-run CLI commands. Custom fields support various **types** of data: - text - numbers, e.g. 2, 4, or 3.283 - timestamps e.g. 2025-10-01T12:24:00Z - raw JSON e.g. {"hello": "world"} Types are supported in order to allow for more advanced filtering. For example, numbers and timestamps can be filtered using greater than/less than operators, in addition to the usual equals/not equals. When adding a custom field you will specify the type as well. **Note:** JSON fields do not support filtering. These are mainly useful in the context of tests or metrics builds, where you could store arbitrary metadata on the experience and use it at runtime. ## Adding custom fields to experiences You can add custom fields via our CLI or when using [experience syncing](../experience-syncing/). We do not support adding custom fields via our webapp yet. You can add a Custom Field to an experience when creating or editing an experience: Bash ``` resim experiences update \ --experience 2fa0453f-749f-4803-bfcc-91690a831c12 \ --custom-field weather=sunny \ --custom-field weather=hot \ # a field can have multiple values --custom-field age=23 \ --custom-field "run time=2025-06-01T12:23:45Z" \ # must be a RFC3339 formatted timestamp --custom-field 'raw json={"foo": "bar"}' ``` You will see the Custom Fields listed under the relevant experience in the webapp. Custom fields can have a type of **text, number, timestamp**, or **JSON**. The specific type changes the filtering options available to you in the UI, see the filtering section for more details. In certain cases, you may want to explicitly mark the type of a custom field, instead of letting the CLI infer it for you. You can do this by using a colon : in the custom field: Bash ``` resim experiences update \ --experience 2fa0453f-749f-4803-bfcc-91690a831c12 \ --custom-field weather:text=cloudy \ --custom-field version:number=239.1 \ --custom-field import_time:timestamp=2025-10-05T12:15:00Z ``` ## Removing custom fields from experiences When updating custom fields on an experience, it replaces all existing custom fields. To remove a field, simple leave it out when performing an update: Bash ``` resim experiences update \ --experience 2fa0453f-749f-4803-bfcc-91690a831c12 \ --custom-field weather=sunny \ --custom-field weather=hot \ --custom-field age=23 # "run time" and "raw json" fields will be removed ``` ## Filtering You can filter experiences by their custom values on the Experiences page, as well as when launching a batch or creating/editing a Test Suite. **number** and **timestamp** fields support additional filtering options: | Type | Supported filters | | --------- | ------------------------------------ | | text | =, != | | number | =, !=, \<, > | | timestamp | =, !=, \<, > | | json | none, json values are not filterable | # Workflows ## Introduction SignalFlag workflows provide a powerful way to organize and manage collections of test suites for CI/CD pipelines. Instead of managing individual test suites separately, workflows allow you to group related test suites together and run them as a single unit. Where a single batch answers "how does this build perform on these experiences?", a workflow answers "how does this version of our software perform across all the test suites we care about?" — producing one batch per enabled test suite in a single operation. ### Why use workflows? SignalFlag is frequently used with CI/CD platforms like GitHub Actions and GitLab CI to trigger test suites on pull requests, releases, and scheduled runs (e.g., nightly tests). While you can trigger individual test suites directly, this approach has limitations: - **Configuration Management**: Adding new test suites or temporarily disabling tests requires updating CI workflow files and pushing changes - **Complexity**: Managing multiple test suite triggers across different CI events becomes unwieldy - **Flexibility**: Making runtime changes to test configurations requires code changes ### How workflows help Workflows solve these problems by providing: - **Named Collections**: Group related test suites (e.g., "nightly", "regression", "smoke tests") under meaningful names - **Runtime Configuration**: Enable/disable test suites or add new ones through the SignalFlag UI without code changes - **Simplified CI**: Trigger entire test collections with a single command - **Multi-System Testing**: Test several [systems](../../core-concepts/#system) (e.g., a planner and a perception stack) in one run by supplying one build per system - **Flexible Management**: Update test suite configurations independently of your CI pipeline ### Key concepts | Concept | Description | | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Workflow** | A project-scoped collection of test suites, each marked enabled or disabled. Has a name, description, and an optional CI link. | | **Workflow test suite** | A membership record tying a test suite to a workflow, with an `enabled` flag. Disabling a suite keeps it associated with the workflow but skips it at run time. | | **Workflow run** | One execution of a workflow: you supply one [build](../../core-concepts/#build) per [system](../../core-concepts/#system) covered by the workflow's enabled suites, and SignalFlag launches one batch per enabled suite, pairing each suite with the build for its system. | Workflows build on a couple of core SignalFlag concepts: every test suite targets exactly one [system](../../core-concepts/#system), and every [build](../../core-concepts/#build) belongs to exactly one system. (See [Core concepts](../../core-concepts/) for the full definitions.) Because each test suite targets a system, a workflow that contains suites for several different systems implicitly *requires* a build for each of those systems at run time. The API exposes this as the workflow's `requiredSystems`. Nothing prevents — and nothing special is needed to allow — a workflow from spanning multiple systems; the requirement only materializes when you run it. ### Example use case Consider a nightly CI job that runs progression, regression, and smoke tests. Instead of managing three separate test suite triggers, you can: 1. Create a "Nightly Tests" workflow containing all three test suites 1. Configure your CI to run: `resim workflows runs create --workflow "Nightly Tests"` 1. Later, add new test suites or disable progression tests entirely through the SignalFlag UI 1. Your CI pipeline remains unchanged while test configurations evolve ## Workflow management SignalFlag provides comprehensive CLI commands for managing workflows. You can create, update, list, and retrieve workflows, as well as manage workflow runs. ### Creating workflows Create a new workflow with the `create` command: Bash ``` resim workflows create \ --project "my-project" \ --name "nightly-regression" \ --description "Full regression: planner + perception" \ --ci-link "https://github.com/myorg/myrepo/actions/workflows/nightly.yml" \ --suites '[ {"testSuite": "planner-regression", "enabled": true}, {"testSuite": "perception-smoke", "enabled": true}, {"testSuite": "slow-soak-tests", "enabled": false} ]' ``` **Required Parameters:** - `--project`: The name or ID of the project to create the workflow in - `--name`: The name of the workflow - `--description`: A description of the workflow - `--suites` OR `--suites-file`: JSON array of test suites (**exactly one required**) **Optional Parameters:** - `--ci-link`: A link to the CI pipeline that triggers this workflow (e.g., a GitHub Actions URL) **Test Suite Configuration:** You can specify test suites in two ways: 1. **Inline JSON** (using `--suites`): Bash ``` --suites '[{"testSuite": "planner-regression", "enabled": true}, {"testSuite": "perception-smoke", "enabled": false}]' ``` 1. **JSON File** (using `--suites-file`): Bash ``` --suites-file ./workflow-suites.json ``` The `testSuite` field accepts either a test suite UUID or its name. Test suite names are unique identifiers, so you can use whichever is more convenient — names are often more readable and maintainable. ### Listing workflows List all workflows in a project: Bash ``` resim workflows list --project "my-project" ``` This command returns a summary of each workflow including: - Workflow ID and name - Description - CI workflow link (if set) - Associated test suites with their enabled/disabled status ### Getting workflow details Retrieve detailed information about a specific workflow: Bash ``` resim workflows get --project "my-project" --workflow "nightly-regression" ``` You can specify the workflow by either name or UUID. The output includes the workflow's suites and their enabled states. ### Updating workflows Update an existing workflow: Bash ``` resim workflows update \ --project "my-project" \ --workflow "nightly-regression" \ --name "nightly-regression-v2" \ --description "Updated description" \ --ci-link "https://new-ci-link.com" ``` **Optional Parameters:** - `--name`: New name for the workflow - `--description`: New description - `--ci-link`: New CI workflow link - `--suites` OR `--suites-file`: Replace all test suites (see create command for format) You must provide at least one update parameter, and the `--suites` and `--suites-file` flags are mutually exclusive. **Suite updates are declarative.** When you pass `--suites` or `--suites-file`, the CLI diffs the supplied list against the workflow's current state and adds, removes, and toggles suites so that the workflow matches the full list you provided. Omitting a suite from the list removes it from the workflow. ## Workflow runs Once you have a workflow defined, you can execute it to run all enabled test suites. A workflow run launches one batch per enabled suite, pairing each suite with the build for its system. ### Creating workflow runs Bash ``` resim workflows runs create --project --workflow [build flags...] ``` A run needs **one build per system** covered by the workflow's enabled suites. There are three ways to supply builds, exactly one of which must be used: #### 1. Repeatable `--build` (simple multi-build) Bash ``` resim workflows runs create \ --project my-project \ --workflow nightly-regression \ --build 11111111-1111-1111-1111-111111111111 \ --build 22222222-2222-2222-2222-222222222222 ``` Each `--build` is a build ID; the system each build covers is derived server-side from the build itself, so no system needs to be named on the command line. With this form, the run-level flags apply to *every* build in the run: - `--parameter =` (repeatable): parameter overrides passed to each batch - `--pool-labels `: where to run; labels are a logical AND - `--allowable-failure-percent <0-100>`: maximum percentage of tests that may hit an execution error while still computing aggregate metrics and counting the run as complete Parameters can be specified multiple times or comma-separated: Bash ``` --parameter "env=production,debug=false" # or --parameter "env=production" --parameter "debug=false" ``` #### 2. `--builds` / `--builds-file` JSON (per-build configuration) When different systems need different parameters, pool labels, or failure tolerances, describe each build as a JSON object: Bash ``` resim workflows runs create \ --project my-project \ --workflow nightly-regression \ --builds '[ { "buildID": "11111111-1111-1111-1111-111111111111", "parameters": {"speed": "fast"}, "poolLabels": ["gpu", "large"], "allowableFailurePercent": 10 }, {"buildID": "22222222-2222-2222-2222-222222222222"} ]' ``` (`--builds-file path/to/builds.json` is equivalent.) All entry fields except `buildID` are optional and apply only to that build's batches. Because configuration is per-entry here, the run-level `--parameter`, `--pool-labels`, and `--allowable-failure-percent` flags cannot be combined with `--builds`/`--builds-file`. #### 3. `--build-id` (deprecated) The legacy single-build flag still works — it is converted internally to a one-entry builds list — but prints a deprecation warning and is hidden from `--help`. Use `--build` or `--builds` instead, even for single-system workflows. #### Validation rules The CLI rejects, before calling the API: - More than one build mechanism at once (e.g. `--build` together with `--builds`) - Run-level override flags combined with `--builds`/`--builds-file` - Duplicate build IDs, malformed UUIDs, the reserved pool label `resim`, and `allowableFailurePercent` outside 0–100 The API additionally rejects: - Two builds resolving to the same system (HTTP 400) - An enabled suite whose system has no matching build (HTTP 422) — the run is not partially created If you supply a build whose system is not covered by any enabled suite, the run is still created but that build is simply not exercised; the response reports it under `unusedBuilds` and the CLI prints a warning. This lets CI pipelines send their full set of builds without tracking exactly which suites are currently enabled. #### Other run flags - `--account `: associate a CI/CD platform account with the run (otherwise inferred from CI environment variables) - `--github`: machine-readable output (`workflow_run_id=`) for GitHub Actions ### Listing workflow runs List all runs for a specific workflow (paginated): Bash ``` resim workflows runs list --project "my-project" --workflow "nightly-regression" ``` ### Getting workflow run details Bash ``` resim workflows runs get \ --project "my-project" \ --workflow "nightly-regression" \ --run-id "run-uuid-here" ``` This prints one entry per test suite in the run: JSON ``` [ { "testSuiteID": "aaaaaaaa-...", "systemID": "bbbbbbbb-...", "buildID": "11111111-...", "batchID": "cccccccc-...", "batchURL": "https://app.resim.ai/projects/.../batches/cccccccc-..." } ] ``` - `systemID` is always present (copied from the test suite at run creation) - `buildID` shows which build was paired with this suite — important for multi-system runs - `batchID`/`batchURL` are omitted for suites that produced no batch: suites disabled at run time, or suites with no active experiences (recorded for traceability, but nothing is launched) `resim workflows runs get --slack` instead emits a Slack webhook payload summarizing every batch in the run. ### Supervising runs from CI `resim workflows runs supervise` waits for every batch in a run to finish, rerunning failed tests, and exits with a code reflecting the worst final batch state — suitable as the gating step of a CI job: Bash ``` resim workflows runs supervise \ --project my-project \ --workflow nightly-regression \ --run-id \ --max-rerun-attempts 2 \ --rerun-max-failure-percent 50 \ --rerun-on-states Error,Blocker \ --wait-timeout 2h ``` All batches are supervised in parallel. Tests whose conflated status matches `--rerun-on-states` are rerun up to `--max-rerun-attempts` times (skipped if more than `--rerun-max-failure-percent` of jobs failed). The exit code is derived from each batch's conflated status, filtered to `--fail-on-states` if set, otherwise to `--rerun-on-states`: | Exit code | Meaning | | --------- | ----------------------------------------------------------- | | 0 | All batches complete (no remaining jobs in the fail filter) | | 1 | Internal CLI error | | 2 | ERROR (orchestration error or unresolved ERROR jobs) | | 5 | Cancelled | | 6 | Timed out | | 7 | Unresolved BLOCKER jobs | | 8 | Unresolved WARNING jobs | Suites that produced no batch are skipped. `--quiet` suppresses informational logging; `--poll-every` controls the polling interval. For supervising a single batch rather than a whole workflow run, see [re-running batches](../re-running-batches/). ## Typical CI flow ``` flowchart TD ci[CI pipeline triggers on commit] --> buildA["resim builds create (system A image)"] ci --> buildB["resim builds create (system B image)"] buildA --> runCreate["resim workflows runs create --build idA --build idB"] buildB --> runCreate runCreate --> batchA[Batch per enabled suite of system A] runCreate --> batchB[Batch per enabled suite of system B] batchA --> supervise["resim workflows runs supervise (reruns + exit code)"] batchB --> supervise supervise --> gate[CI job passes or fails on exit code] ``` 1. CI builds and pushes one image per system, then registers each with `resim builds create --system ... --image ...` (or `--build-spec` for compose-based builds), capturing the build IDs via `--github`. 1. `resim workflows runs create` launches the workflow with one `--build` per system (or `--builds` JSON for per-system configuration), capturing `workflow_run_id`. 1. `resim workflows runs supervise` blocks until all batches settle and gates the pipeline on the exit code. ## Examples ### Example 1: Creating a multi-system nightly workflow Bash ``` # Create a workflow covering both the planner and the perception stack resim workflows create \ --project "my-robot-project" \ --name "Nightly Tests" \ --description "Nightly regression across planner and perception" \ --ci-link "https://github.com/myorg/robot-repo/actions/workflows/nightly.yml" \ --suites-file ./nightly-suites.json ``` Where `nightly-suites.json` contains: JSON ``` [ {"testSuite": "planner-regression", "enabled": true}, {"testSuite": "perception-smoke", "enabled": true}, {"testSuite": "slow-soak-tests", "enabled": false} ] ``` Because `planner-regression` and `perception-smoke` target different systems, every run of this workflow will require one build for each of those systems. ### Example 2: Running and supervising a workflow in CI Bash ``` # Register a build for each system, capturing the IDs PLANNER_BUILD=$(resim builds create --project "my-robot-project" \ --system "planner" --image "$PLANNER_IMAGE" --branch "$BRANCH" \ --version "$GIT_SHA" --github | cut -d= -f2) PERCEPTION_BUILD=$(resim builds create --project "my-robot-project" \ --system "perception" --image "$PERCEPTION_IMAGE" --branch "$BRANCH" \ --version "$GIT_SHA" --github | cut -d= -f2) # Launch the workflow run with one build per system RUN_ID=$(resim workflows runs create \ --project "my-robot-project" \ --workflow "Nightly Tests" \ --build "$PLANNER_BUILD" \ --build "$PERCEPTION_BUILD" \ --parameter "test_environment=staging" \ --allowable-failure-percent 10 \ --github | cut -d= -f2) # Block until all batches finish, rerunning flaky failures, and gate CI on the result resim workflows runs supervise \ --project "my-robot-project" \ --workflow "Nightly Tests" \ --run-id "$RUN_ID" \ --max-rerun-attempts 2 \ --rerun-max-failure-percent 50 \ --rerun-on-states "Error,Blocker" \ --wait-timeout 2h ``` ### Example 3: Per-build configuration with `--builds` Bash ``` # Perception batches need GPU pools and tolerate some flakiness; # the planner runs with defaults resim workflows runs create \ --project "my-robot-project" \ --workflow "Nightly Tests" \ --builds '[ { "buildID": "'$PERCEPTION_BUILD'", "poolLabels": ["gpu", "large"], "allowableFailurePercent": 10 }, {"buildID": "'$PLANNER_BUILD'"} ]' ``` ### Example 4: Updating workflow configuration Bash ``` # Disable performance tests and update description. The suite list is # declarative: any suite omitted here is removed from the workflow. resim workflows update \ --project "my-robot-project" \ --workflow "Nightly Tests" \ --description "Nightly regression and smoke tests (performance tests moved to weekly)" \ --suites '[{"testSuite": "planner-regression", "enabled": true}, {"testSuite": "perception-smoke", "enabled": true}]' ``` ## API notes For integrations that call the API directly rather than through the CLI: - `createWorkflowRunInput` accepts either the deprecated top-level `buildID` (with run-level `parameters`/`poolLabels`/`allowableFailurePercent`) **or** the preferred `builds` array of `{buildID, parameters, poolLabels, allowableFailurePercent}` — never both (HTTP 400). New integrations should always use `builds`, even for single-system workflows. - With `builds`, the deprecated run-level `parameters`/`poolLabels`/`allowableFailurePercent` must be omitted; configure each entry instead. - The `workflowRun` response carries `workflowRunTestSuites[]` with per-suite `systemID`, `buildID`, and `batchID`, plus `unusedBuilds[]` for supplied-but-unneeded builds. The top-level `workflowRun.buildID` is a deprecated convenience echo populated only when every enabled suite used the same build. - The `workflow` resource exposes `requiredSystems[]` — the systems covered by its enabled suites — which callers can use to determine which builds a run will need. ## Best practices ### Workflow organization - **Use Descriptive Names**: Choose workflow names that clearly indicate their purpose (e.g., "Nightly Regression", "PR Smoke Tests", "Release Validation") - **Group Related Tests**: Organize test suites logically by functionality, environment, or execution frequency — a single workflow can span multiple systems - **Document with Descriptions**: Provide clear descriptions explaining what each workflow tests and when it should be used ### CI/CD integration - **Link CI Workflows**: Use the `--ci-link` parameter to connect SignalFlag workflows with your CI pipeline for easy navigation - **Send All Your Builds**: It is safe for CI to pass every build it produces — builds whose system is not covered by any enabled suite are reported as unused rather than causing an error - **Gate on `supervise`**: Use `resim workflows runs supervise` as the final step of a CI job so the pipeline passes or fails based on test outcomes - **Use Environment Variables**: Leverage CI environment variables for dynamic parameters like build IDs and test environments - **Set Appropriate Failure Thresholds**: Use `--allowable-failure-percent` to handle expected test failures in non-critical workflows ### Test suite management - **Start with Core Tests**: Begin with essential test suites enabled, then add optional tests as needed - **Use Test Suite Names**: Prefer test suite names over UUIDs for better readability and maintainability - **Use JSON Files**: For complex configurations, use `--suites-file` instead of inline JSON for better maintainability - **Remember Updates Are Declarative**: When updating suites, supply the complete desired list — omitted suites are removed - **Regular Review**: Periodically review and update workflow configurations to ensure they remain relevant ### Parameter management - **Consistent Naming**: Use consistent parameter naming conventions across your workflows - **Per-Build Configuration**: When systems need different parameters or pool labels, use `--builds` JSON instead of run-level flags - **Environment-Specific Values**: Use parameters to customize test behavior for different environments - **Document Parameters**: Keep track of what parameters each workflow expects and their valid values ### Monitoring and maintenance - **Track Workflow Runs**: Regularly review workflow run results to identify patterns and issues - **Update Descriptions**: Keep workflow descriptions current as test suites evolve - **Archive Unused Workflows**: Remove or archive workflows that are no longer needed to keep your project organized # Optimizing build images ## Introduction Running your tests at scale in SignalFlag involves running the image as quickly as possible on multiple cloud instances. This document is intended to explain the way images are handled in SignalFlag and to assist customers in optimising their images. Spending some time to make your images smaller will save time and money, not only when running tests in SignalFlag but also when storing and distributing images in your own environments. N.B.: This document refers to single images, however note that if you are using our [multi-container builds](../multi-container-builds/) feature, images are processed in the same way. ### SignalFlag's image handling architecture When you register a build image with SignalFlag, our Mirror service will pull the image from your registry and store it in SignalFlag's platform. Then once a batch is launched with that build image, the Mirror service checks it has an up-to-date copy of the image, and the SignalFlag platform launches cloud compute **instances** to run the image with the experiences configured in the batch. This means that your image is transferred from one place to another at the following times: 1. When a batch is launched, the Mirror service checks that it has an image in its storage that matches the tag and digest. If not, it authenticates and pulls the image. - If the Mirror service determines that it needs to update its copy of the image, it transfers only the layers of the image that have changed. This indicates a significant potential optimisation, discussed below (see [Order layers](#order-layers) below). - If your image is hosted in a registry other than AWS ECR in `us-east-1`, for example in AWS ECR in `us-west-2` or Google Artifact Registry, this will incur egress costs for you according to the size of the image, meaning that reducing the size of your image will save time and money here. This cost is also potentially incurred in other cases, for example when you transfer images into hardware in the field. 1. When tests are running, the instances running the test each pull the image from our mirror service. - The number of instances we launch is carefully optimized - once you are running more than a small number of tests in a batch, we launch fewer instances and re-use them for tests in the same batch. This means that the image is pulled a number of times equal to the number of instances launched, and possibly not equal to the number of tests. Because pulling and extracting images - even from our Mirror service which is closely co-located with test instances - takes time, smaller images will reduce the time it takes for your batches to run. ``` architecture-beta group customer_cloud(cloud)[Customer Cloud] service customer_registry(disk)[Image Registry] in customer_cloud group resim(cloud)[SignalFlag] service mirror(disk)[Image Mirror] in resim service instance1(server)[Instance] in resim service instance2(server)[Instance] in resim service instance3(server)[Instance] in resim mirror:R -- L:instance1 mirror:T -- L:instance2 mirror:B -- L:instance3 customer_registry:R -- L:mirror ``` ## Potential optimizations There is a lot of discussion online about optimizing container images (we've provided some links to further reading below). Here we have outlined some of the more impactful changes you can make, in our experience. Some of these are trade-offs, where you may be trading complexity or maintenance overhead for smaller images that are cheaper and quicker to handle, and some are outright improvements. ### Use a lean base image Generic base images can be very large, as they include packages to suit the potential needs of many users (text editors, SSH servers, compilers etc.). Consider starting with a minimal image and adding only the packages and dependencies you need. Many images publish a `slim` variant which is useful for this purpose. ### Order layers In your Dockerfile, commands that modify the filesystem (such as `ADD`, `COPY` and `RUN`) cause a new layer to be created. Those layers are cached "in order", so if a command at the top of the file means that next time the build is run, the layer it produces has changed, all layers "below" that one in the Dockerfile will be invalidated and rebuilt. When using the image in SignalFlag, this means that our Mirror service will need to pull all of the newly-invalidated layers, and not just the most recently changed layer. As a concrete example, suppose you had a Dockerfile like this: Dockerfile ``` FROM ubuntu # let's record which commit this is (argument value provided by build command) ARG GIT_SHA RUN echo $GIT_SHA > /.build-version # install dependencies and build our application RUN apt-get update && apt-get install -y build-essential COPY main.c Makefile /src/ WORKDIR /src/ RUN make build ``` This is a contrived example, but in this case every time the image is built with a different `GIT_SHA` value, the `RUN echo $GIT_SHA > /.build-version` layer will change, and all layers that run after it (the apt-get install, ADD, and make build commands) will be invalidated and rebuilt, even though those dependencies and source files may not have changed. This means that all of: running the build, uploading the image to long-term storage (because `docker push` and similar commands are layer-aware), and transferring the image to SignalFlag will take longer than they need to. If you really wanted that `.build-version` file, putting it at the end of the Dockerfile (or at least after the expensive operations) would mean that the `RUN apt-get...` and later commands would not be invalidated by it. ### Don't install unnecessary packages You should only install the packages you need to run your application. Development tools and other utilities make images less secure, slower to build, and larger. See [Consider using multi-stage builds](#consider-using-multi-stage-builds) below for an example of how to build a smaller image that only contains your application and its runtime dependencies. ### Install packages in one command If you are installing multiple packages, install them in one command. Splitting the installations into separate commands leads to inefficient layer caching, affecting both build time and overall image size. For example: Dockerfile ``` RUN apt-get update && apt-get install -y build-essential \ python3 \ python3-pip \ libcudnn8 \ <...other required packages> \ && rm -rf /var/lib/apt/lists/* # See below ``` ### Clear or disable package manager caches Package managers cache information about the packages available in configured repositories. This can take up a surprising amount of space. For `apt`: Dockerfile ``` RUN apt-get update && apt-get install -y --no-install-recommends \ build-essential \ <...other required packages> \ && rm -rf /var/lib/apt/lists/* ``` Note that if you split the installation and cache removal into separate `RUN` commands the cache will appear to be removed in the final image, but it will still be cached in the prior layers, taking up space. For `apk` (Alpine's package manager): Dockerfile ``` RUN apk add --no-cache ``` This both updates package information and prevents `apk` storing any. ### Consider using multi-stage builds This is an advanced topic, but for image size optimization and security it can be best to produce an image that only contains your compiled binary, if it suits your application. For example: Dockerfile ``` FROM golang:1.25 AS builder WORKDIR /src COPY main.go . RUN go build -o /bin/hello ./main.go FROM scratch COPY --from=builder /bin/hello /bin/hello CMD ["/bin/hello"] ``` ## Conclusion As you can see, it's possible to spend a small amount of time working on your Dockerfile and save significant time and money in return - not just when running in SignalFlag, but also when building, storing and distributing images for your own use. In addition, images with only the necessary packages installed present a smaller attack surface and therefore can be more secure. ## Further reading - [Dockerfile best practices](https://docs.docker.com/build/building/best-practices/) - [Layer caching](https://docs.docker.com/build/cache/) - [Multi-stage Builds](https://docs.docker.com/build/building/best-practices/#use-multi-stage-builds) # Reference # Reference Reference material is **information-oriented**: precise, complete, and authoritative. Use this section when you need to look something up — not to learn a concept, but to verify a detail. This section contains: - **Core Concepts** — the SignalFlag data model: projects, systems, experiences, builds, batches, and metrics - **Metrics Library** — the full API reference for the SignalFlag open-core metrics library (Python and C++) - **Open Core Libraries** — API references for transforms, curves, math, time, visualization, and autonomy-log utilities In order to easily manage the execution of your code on the cloud, SignalFlag introduces a few basic concepts that are useful to understand. ## Project A **Project** is a flexible container for all related concepts within the SignalFlag Ecosystem. All data within the SignalFlag platform is isolated across projects, enabling different teams or embodied AI applications to be developed within a dedicated space. Projects can be used to represent a full embodied AI system, for an individual subsystem or as a personal sandbox. Projects typically map to repositories, in which case it is helpful to use the same name for the project as the repository so that CI services associated with repositories (e.g. GitHub, GitLab) have easy ways to reference repository names inside CI workflows and commands can be parameterised. It is not mandatory to maintain this distinction. We recommend that a **Project** be used to reflect an autonomy stack as a whole, and that [systems](#system) be used to partition individual components of that autonomy stack e.g. perception, localization, etc. In this way, the reporting functionalities of the SignalFlag app can be used to offer an overview of the state of your stack. Projects have a name and an ID. - [Create a project](../setup/projects/) ## Experience An **experience** is a set of interchangeable input files for a test you want to execute in the SignalFlag app. For a simulation test, an experience will likely include a scenario description (e.g. where the ego is in the world and what its goals are), autonomy stack parameters, and any other assets the simulation test needs to run. Experiences can be stored in cloud blob storage (such as AWS S3 or Google Cloud Storage), in the Foxglove Data Platform, or at a local path in the [build container](#build). You will need to [configure SignalFlag to access your experience data](../setup/experience-data-sources/) in order for us to consume it. Experiences can be stored in multiple locations, all of which is fetched by SignalFlag. Experiences can be tagged to help filter and search them. - [Add experiences](../setup/adding-experiences/) Experiences also support important metadata for when they are executed as a test. In particular, you can set: - A set of environment variables that can be injected into your [build](#build) - A timeout (in seconds) that can be used as an upper bound on your expected experience runtime, to guard against infinite loops - A [docker compose profile](https://docs.docker.com/compose/how-tos/profiles/) that should be used to run this experience if you are using a multiple container build. Note: we do not support multiple profiles right now, please contact us if this is important. ## Asset An **asset** is a set of shared data files that a [build](#build) needs at runtime but that are not specific to any single [experience](#experience). Common examples include HD maps, 3D environment meshes, ML model weights, physics parameters, and calibration data -- anything that remains constant across many experiences. While experiences represent per-test inputs (mounted at `/tmp/resim/inputs/`), assets represent shared data that is associated with a **build** and mounted read-only at `/tmp/resim/assets//`. This separation means you don't need to duplicate large shared files across experience locations or bake them into your Docker image. Assets support **revisions**: each time the source data changes, a new revision is created with a version string, preserving an immutable history. Assets are [cached](../guides/asset-caching/) by default, so only changed files are synchronized between runs. Assets are explicitly linked to builds, and we recommend managing these associations in your CI/CD pipeline to ensure reproducibility. - [Set up assets](../setup/assets/) ## System A **System** is an abstraction of a software component that can be run on a specific set of data with its own resource needs, such as an embodied AI solution. This solution may be composed of several subsystems that can be tested individually or collectively at scale. Examples might include perception system, motion planning, full stack. A **system** definition includes the hardware resources requirements for a particular component, such as CPUs, memory, and GPUs. It can then be run against a suite of [experiences](#experience) using a [build](#build) definition, with [metrics](#metrics) evaluating the results. You may wish to use experiences and metrics builds with multiple systems. SignalFlag allows you to tag experiences and metrics builds with one or more systems to show that a particular system is compatible with them. This flexible representation enables users to filter individual test results based on which system is being tested, or get a unified view of the status of the systems that make up a given [project](#project). Note We have a feature on the roadmap to introduce interactive prompts and compatibility validation to the UI and CLI, but for now, you'll need to visually check - [Define a system](../setup/systems/) ## Build A **Build** is an instantiation of a system in time. It is a snapshot of the code, represented by a container image, that can actually run the [system](#system) under test with inputs provided by an [experience](#experience). Builds are typically versioned, as the executable test is modified over time. For instance, in continuous integration workflows, you might generate a new build for every new commit to your autonomy code repository so that you can run tests on the newest software. The name of a build usually references the relevant system (e.g. "Perception Build") whereas its description contains a descriptive commit message or SHA. The executable and dependencies for the build is packaged up in a [container image](https://opencontainers.org/) which is pushed to a container registry and [registered with the SignalFlag app](../setup/build-images/). SignalFlag supports images from AWS ECR, Docker Hub, GitHub Container Registry, and Google Artifact Registry. You will need to [grant us access](../setup/container-registry-access/) to any non-public container registry you use. - [Build a system image](../setup/build-images/) - [Push a system image from CI](../setup/ci/) ### Multi-container builds In place of a single container image, a **Build** may also be a collection of container images tied together by a [Docker Compose](https://docs.docker.com/reference/compose-file/) file. Defining a multi-container build like this allows you to have multiple containers executed together during the **Experience** phase. These containers can communicate with each other, share data, etc, and may allow you to better simulate a more complex system. ## Metrics **Metrics** quantify how your system performed. During a test, your build emits data with the [SignalFlag SDK](../guides/resim-sdk/). A metrics config synced to your branch defines the metrics computed from that data, each a SQL query with the chart that draws its result, and status checks on those metrics decide whether a test passes. SignalFlag computes metrics at two levels of granularity. Test metrics evaluate a single [*test*](#test), and batch metrics aggregate across every test in a [*test batch*](#test-batch). For example, a test metric might compute *mean localization error magnitude* for one experience, while a batch metric computes the *batch mean localization error magnitude* across hundreds of them. Metrics are grouped into *metrics sets*, which test suites and batches run against. If your raw outputs need post-processing before metrics can be computed, a [metrics build](../setup/metrics-builds/) can run after each test and emit the processed data. - [Add metrics](../guides/metrics/) ## Branch A **branch** is a collection of builds. Branches in the SignalFlag app are similar to the idea of a branch in [git](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell), although there is no automatic association between branches in the two systems. Branches are useful for associating builds that represent different iterations of the same work in the same way that git branches associate a graph of commits. In a continuous integration workflow, each pull request could therefore have its own branch in the SignalFlag app to contain associated builds. - [Create a branch](../setup/build-images/#creating-a-branch) ## Test A **test** is the fundamental unit of cloud-based work at SignalFlag. It is a single execution of a given build with a given experience. It is canonical terminology in software engineering to refer to a **Test** as a single function that checks some functionality of your system and a **Test Suite** as a... well, a suite of tests. A test returns either a pass or a fail, and a test suite passes if and only if all of its tests pass. At SignalFlag, we adopt this terminology with a small modification: - A **Test** is the execution of a single [*experience*](#experience) against a [*build*](#build), where performance is evaluated by [*metrics*](#metrics) - A **Test Suite** is a *specification* of a number of tests to run and evaluate with the same metrics set. The aggregate performance of the test suite is also determined by its *metrics set*. A test suite can then be run with a build to generate tests. Running a test suite will generate a test for each experience in the test suite and then generate aggregate metrics. To distinguish between the test suite *qua* specification and group of tests it creates, we use the term [**Test Batch**](#test-batch). A test batch is a group of tests that have been run in the SignalFlag platform. As has been discussed in the [Metrics Build](../setup/metrics-builds/) page, the test suite's metrics build can also be instrumented to compute so-called batch metrics, which aggregate the metrics from each individual test to display, for example, the average performances of the build or a list of the worst performing experiences. ## Test suite A **test suite** is a named template for a set of [tests](#test). It is an *organizational* component to allow you to specify a commonly used set of [experiences](#experience) to run. A common example is to define the set of regression tests for a given [system](#system). A test suite has five main pieces of data: - The **name** of the test suite e.g. Nightly Regression Test. - An optional **description** to provide some more detail about the intent and purpose of this test suite. - The **system** that this test suite is intended to be run with. The SignalFlag app will warn if you attempt to run the test suite with a build that is not from that system, since it is likely that there will be compatibility issues and it will fail. - The **metrics build** that will be used to compute the metrics for this test suite. - A list of **experiences** that will be run against the build you choose. In order to provide traceability of previous results, test suites are *versioned*. Any time a change is made to a test suite (updating the metrics build, adding or removing experiences), a new *revision* is created. Test suites start at `revision 0` and increment. One of the main benefits for using test suites -- other than the time saving from having to choose what experiences and metrics you want repeatedly -- is that it makes it very easy to: - Filter and Compare your tests. If there are many test batches created in your project, it can be helpful to filter a list of results by test suite to find what you are looking for and to then [compare](../guides/ab-comparison/) like for like e.g. compare the last two times a test suite has been run against main - Control your tests. If a systems and test engineering team exists separately to engineering and have responsibility for defining acceptance criteria, test suites can be used to define what tests the engineering team should be passing for a given feature. - Track performance over time. If a test suite is run regularly against a particular branch, a [dashboard](../guides/metrics/dashboards/) can chart how its metrics evolve across batches, showing whether your system is getting better. - [Create test suites](../setup/test-suites/) ## Test batch A **test batch**, often referred to as simply a **batch**, is a collection of tests that have been triggered simultaneously using the same build. In a continuous integration workflow, a batch of simulation tests could be kicked off on each change to the code under development. These would all use the same build (based on the latest iteration of the software), but each could use a different experience. - [Run a batch](../setup/test-suites/#running-a-test-suite) ## Session A **session** is a recording of a robot operating in the real world, registered in SignalFlag as an [experience](#experience) whose input files are the recorded MCAPs. SignalFlag evaluates sessions for you, which puts their field data into the same data lake, and therefore in reach of the same metrics, events and comparisons, as a simulation [test](#test). Sessions are distinct from ordinary experiences in what the app offers on top of them: recording bounds and an event timeline in real recording time, playback of the underlying MCAPs, and comparison of two sessions that are different recordings rather than two runs of the same scenario. - [Sessions](../guides/sessions/) ## View A **view** is a set of visualizable 3D content generated by SignalFlag View. This content can be visualized using Foxglove Studio by clicking the **Open in Foxglove** link provided on the view's page in the SignalFlag app. SignalFlag View is a C++ library that users can link to their code to make it easy to visualize 3D transforms in their code. - [Visualization](../open-core/visualization/) # ReSim Open Libraries ReSim's `open-core` repository contains the open source subset of ReSim's C++ code intended to accelerate robotics development. ## Getting Started A number of our libraries have Python bindings or implementations, such as a subset of the `transforms` and `metrics` libraries as well as a Python client for the ReSim API. These are distributed as part of our [Python package](https://pypi.org/project/resim-open-core/) which can be easily installed using `pip`. Bash ``` pip install resim-open-core ``` Note Currently, this package is only built for Python 3.10, supporting Linux for x86_64 and aarch64 architectures. If you're using C++ and Bazel, you can follow the development instructions below to build some examples and explore our libraries. ## Development ### Install and Setup Docker These instructions vary by host platform: - [Mac](https://docs.docker.com/desktop/mac/install/) - [Windows](https://docs.docker.com/desktop/windows/install/) - [Linux](https://docs.docker.com/engine/install/ubuntu/) - [post-install](https://docs.docker.com/engine/install/linux-postinstall/) ### Get the Code Bash ``` git clone git@github.com:resim-ai/open-core.git ``` ### Setting up the Environment In order to easily build and interact with ReSim's code, you need our development docker image. To fetch this image, navigate to the open source repo and call the following script. Bash ``` cd open-core ./.devcontainer/pull.sh ``` If you have any trouble pulling here, please see [this section](#building-the-docker-image-locally) below which shows how to build your own image locally. Next, we can start up a development docker container locally: Bash ``` ./.devcontainer/run.sh ``` ### Building the Code We use [Bazel](https://bazel.build/) as our build system. Therefore, to build all of the example binaries you simply have to execute: Bash ``` bazel build //resim/examples/... ``` Now, all of the binaries will be present in `bazel-bin/resim/examples/`. Individual binaries can be built and run in a single step using `bazel run`. For example: Bash ``` bazel run //resim/examples:liegroups ``` ### Autocomplete via `clangd`, `clang-tidy`, and `compile_commands.json` Many developer tools (such as `clangd` and `clang-tidy`) require a `compile_commands.json` file in the root of the tree. This file is large and prone to change, so we don't check it in and each developer is responsible for generating it themselves. You will need to regenerate it whenever you change your BUILD files. We use [Hedron's Compile Commands Extractor](https://github.com/hedronvision/bazel-compile-commands-extractor) to generate `compile_commands.json`. It can be invoked with: Bash ``` bazel run @hedron_compile_commands//:refresh_all ``` ### Using Dotfiles If you'd like to use your own dotfiles for customization inside the container, our recommendation is to create a dotfiles repository and write an `install.sh` script to automatically move those dotfiles to the appropriate location (and perform whatever other workspace customization you'd like). Note: do not put credentials of any kind in this repository. If you'd like to streamline this system a bit, you can check out your dotfiles repo to `$HOME/dotfiles` and use `.devcontainer/run_dotfiles.sh` (instead of `.devcontainer/run.sh`) to have it automatically mounted and installed in your container. ### Building the Docker Image Locally We also provide scripts to build the development Docker image locally. To do this, simply do: Bash ``` ./.devcontainer/build.sh ``` And then once this is complete simply use `.devcontainer/run_local.sh` or `.devcontainer/run_dotfiles_local.sh` instead of `.devcontainer/run.sh` or `.devcontainer/run_dotfiles.sh`. # Lie Groups ## Introduction Disclaimer *This is not written from the perspective of a mathematician or physicist, but rather from the perspective of an engineer that has found Lie groups to be an indespensible tool for robotics. The knowledge here is primarily practical knowledge built up over experience, so it is very likely that some of the mathematical details are not precisely correct as written. Please don't hesitate to reach out and let us know if you find any errors!* [**Lie groups**](https://en.wikipedia.org/wiki/Lie_group) are mathematical objects with a rich history of study and application. They are named for the Norwegian mathematician [Sophus Lie](https://en.wikipedia.org/wiki/Sophus_Lie) who pioneered the mathematics of continuous transformation groups. Why should we use Lie groups in robotics? The answer is that robotics is replete with examples of continuous transform groups. One example is the fundamental concept of pose. A pose describes a rigid-body transformation from one set of coordinates to another. Most commonly, a robot's location and orientation relative to the world is represented by the pose (or rigid transformation) relating the "world" and "robot" frames. There are, of course, many ways of representing a robot's pose. Orientation alone can be represented by Euler angles, quaternions, rotation matrices, etc. Using Lie groups affords us two primary advantages: - First, the set of all orientations and the set of all poses are both Lie groups. Representing them as such allows us to write generic algorithms (like Hermite spline construction or Newtonian dynamics) that operate equally well on both since both are equipped with the same basic Lie group properties. This would be impossible to do, for example, if we represented orientations as quaternions (which otherwise have very nice properties) and poses as tuples of (quaternion, translation vector). - Second, every Lie group has a corresponding Lie algebra (a vector space), which can be used to describe its structure locally (i.e. near any given element in the group). This means that we can do a lot of things on Lie groups that we normally know how to do on vector spaces. For instance, we can: - Construct Gaussian distributions on the Lie group of rigid body transformations (SE(3)) and sample from them. - Interpolate between recorded actor poses to produce a trajectory without worrying about wrap-around issues that might result if we interpolated poses with Euler angles describing their orientations. - Write loss functions on orientations and use them for optimal control, once again without worrying about wrap-around or singularities. - Take the derivatives of our poses and represent them using elements of the corresponding Lie algebra. We've glossed over everything exceptionally briefly here, so I would encourage the curious reader to peruse the [external links](#external-links) below. ## LieGroup Interface In an effort to make Lie-groups-related code more reusable, we require each implementation of a Lie group object (that is an object representing an element of a Lie group) to satify a few constraints. First, the implementation must inherit from the LieGroup template. This base class is templated on the dimensionality of the input to the transform (e.g. 3 for 3D rotations), and the number of degrees of freedom for this group (e.g. 6 = {3 rotational} + {3 translational} for rigid body transformations). This base template also defines some helpful aliases for tangent vectors (i.e. members of the Lie algebra). Second, the implementation should satisfy the LieGroupType concept which enforces a number of the group axioms (e.g. group action, invertability, identity, etc.). We implement these checks as a concept since different groups need different signatures so inheritance doesn't allow us to enforce them. ## Lie Groups Implemented **SO(3)** - The Special Orthogonal Group in 3D. The set of all rotations. **SE(3)** - The Special Euclidian Group in 3D. The set of all rigid transformations. ## Framed Groups As discussed above, Lie group elements are frequently used to represent transformations between coordinate frames. In practice, this can often lead to mistakes when users of Lie group libraries compose group elements in the wrong order (e.g. multiplying `robot_from_sensor` times `scene_from_robot`). These small mistakes are simple enough that they should be possible to catch, at least at runtime. We therefore assign a unique id to each coordinate frame. Group elements thus have two frame ids that they track (`into` and `from`). When two elements are multiplied, consistency is enforced between them. More precisely, if a pose that transforms points in frame B's coordinates to frame A's coordinates (call the transform `A_from_B`) and a similar transform `C_from_D` then we should fail if the user ever tries to multiply `A_from_B * C_from_D`. However `A_from_B * B_from_D` is fine. Unframed groups are supported, by setting the frame to a null (0) id, in which case they are not checked. Be aware that multiplying an unframed group by a framed group will always result in an unframed group, so unframed groups can propagate quickly! ## More Information For more information on the features our Lie group classes provide for working with the derivatives of functions involving Lie groups, please refer to [this guide](../liegroup_derivatives/). ## External Links For more information, please take a look at the following links: - [A micro Lie theory for state estimation in robotics](https://arxiv.org/pdf/1812.01537) - [ethaneade.com](https://ethaneade.com/) - Jakob Schwichtenberg: - [Lie Group Theory - A Completely Naive Introduction](https://jakobschwichtenberg.com/naive-introduction-lie-theory/) - [How is a Lie Algebra able to describe a group](https://jakobschwichtenberg.com/lie-algebra-able-describe-group/) - [What's so special about the adjoint representation of a Lie group?](https://jakobschwichtenberg.com/adjoint-representation/) # Using SO(3) and SE(3) ## Introduction This document is intended to breifly outline the basics of using ReSim's main Lie group libraries. For an introduction to Lie groups and the reasons why we think they're worth using, the reader is highly encouraged to read [Lie Groups](../liegroups/). As a motivating example, we imagine that we have the pose of a robot in 3D space given by Tait-Bryan [Euler Angles](https://en.wikipedia.org/wiki/Euler_angles) (yaw, pitch, and roll) (\\psi), (\\theta), and (\\varphi) and the actor's location coordinates (x), (y), and (z) expressed in some stationary scene coordinate frame. We will show how to create an SO3 describing the orientation of this robot and an SE3 describing its full pose. ## Orientation / SO3 Before we get to the full pose, it's simpler to take on the orientation part separately. Our Euler angles are defined as the composition of three rotations. If we have a vector (v\_{\\text{robot}}) in our robot's coordinates and we want to express it in our scene coordinates, we have: [ v\_\\text{scene} = R_x(\\varphi) R_y(\\theta) R_z(\\psi) v\_\\text{robot} ] Where (R_x(\\varphi)) is a rotation matrix for a rotation around the (x) axis by (\\varphi) radians. Hence the overall rotation matrix for our orientation is given by: [ R = R_x(\\varphi) R_y(\\theta) R_z(\\psi) ] One can identify the colums of this matrix as the axes of the robot coordinate frame expressed in scene coordinates. Since rotation matrices are a *representation* of (\\text{SO(3)}), we can use this equation to compute an `SO3` for our robot pose like so: C++ ``` #include #include "resim/transforms/so3.hh" #include "resim/visualization/view.hh" using namespace resim::transforms; // ... const double psi = M_PI_4; const double theta = 0.5; const double phi = 0.1; const SO3 scene_from_robot_rotation = SO3(phi, {1., 0., 0.}) * SO3(theta, {0., 1., 0.}) * SO3(psi, {0., 0., 1.}); // Visualize with ReSim View VIEW(scene_from_robot_rotation) << "My rotation"; ``` Here, we're constructing `SO3`s representing the rotations around each axis. Note that we have to treat each rotation in order if we're using Euler angles. If we use the exponential map on `SO3` (which converts angle/axis representations of an orientation into an `SO3` object) with a vector of the angles, we will get a different result: C++ ``` const SO3 scene_from_robot_rotation_wrong = SO3::exp({phi, theta, psi}); // The following assertion will not fail except in the degenerate case of two of // the angles being zero. REASSERT( not scene_from_robot_rotation.is_approx(scene_from_robot_rotation_wrong)); ``` The naming here (i.e. `scene_from_robot`) is conventional in ReSim's libraries as it reflects the fact that the SO3 is a transformation from robot coordinates into scene coordinates. In fact, one can use the `SO3` to directly transform vectors. C++ ``` using Vec3 = Eigen::Vector3d; const Vec3 robot_forward_in_robot_coordinates{1.0, 0.0, 0.0}; Vec3 robot_forward_in_scene_coordinates = scene_from_robot_rotation * robot_forward_in_robot_coordinates; // Can also be more explicit. This is equivalent to the above. robot_forward_in_scene_coordinates = scene_from_robot_rotation.rotate(robot_forward_in_robot_coordinates); ``` Quaternions are also often used to represent orientations, and have many benefits over Euler angles. Fortunately, `SO3` objects can also be converted to and from quaternions: C++ ``` const Eigen::Quaterniond scene_from_robot_quat{ scene_from_robot_rotation.quaternion()}; REASSERT(SO3(scene_from_robot_quat).is_approx(scene_from_robot_rotation)); ``` Furthermore, like quaternions, SO3 affords smooth interpolation without wrap-around issues. To do this, we use the `SO3::interp()` member function: C++ ``` constexpr double EPSILON = 1e-2; const Vec3 axis = Vec3(0.1, 0.2, 0.3).normalized(); const SO3 scene_from_a(M_PI - EPSILON, axis); const SO3 scene_from_b(-M_PI + EPSILON, axis); constexpr double FRACTION = 0.5; const SO3 scene_from_interped_rotation{ scene_from_a * (scene_from_a.inverse() * scene_from_b).interp(FRACTION)}; const SO3 expected(M_PI, axis); REASSERT(scene_from_interped_rotation.is_approx(expected)); ``` ## Pose / SE3 Now, if we want to express the robot's whole pose, most of the work is already done: C++ ``` #include "resim/transforms/se3.hh" // ... const Vec3 scene_from_robot_translation{x, y, z}; const SE3 scene_from_robot{ scene_from_robot_rotation, scene_from_robot_translation}; ``` This can still be used to transform points as above, but as one might expect, the translation is now included: C++ ``` const Vec3 point_in_robot_coordinates{Vec3::Random()}; const Vec3 point_in_scene_coordinates{ scene_from_robot * point_in_robot_coordinates}; REASSERT( point_in_scene_coordinates == scene_from_robot_rotation * point_in_robot_coordinates + scene_from_robot_translation); ``` Sometimes, we don't want to transform a point, but instead we want to transform a vector (e.g. robot velocity). In this case, we instead use: C++ ``` const Vec3 vector_in_robot_coordinates{Vec3::Random()}; const Vec3 vector_in_scene_coordinates{ scene_from_robot.rotate(vector_in_robot_coordinates)}; REASSERT( vector_in_scene_coordinates == scene_from_robot_rotation * vector_in_robot_coordinates); ``` Note that the `rotate()` member function is different than `operator*()` for `SE3` even though it is equivalent for `SO3`. Like `SO3`, `SE3` also has an `interp()` member function which can be used to interpolate elements of the group: C++ ``` const SE3 scene_from_interped{scene_from_robot.interp(0.5)}; REASSERT( scene_from_robot.is_approx(scene_from_interped * scene_from_interped)); ``` Note that interpolation of the Lie group (\\text{SE(3)}) follows a smooth geodesic curve between the two frames. This property can confer some advantages - especially in robotics applications - but, it's important the caller is aware of this. ## Derivatives So far, we have dealt with the static pose of an example robot relative to the scene. A natural next step is to discuss how to work with time derivatives of poses since robots typically have components that change their poses over time. We very frequently use objects called *tangent vectors* to express this velocity information. We'll define these vectors here without diving deeply into the math (for more see [Lie Group Derivatives](../liegroup_derivatives/)). For (\\text{SE(3)}), we can represent any element/object (g) with a 4x4 matrix of the form: [ g = \\begin{bmatrix} R & t \\ 0 & 1 \\ \\end{bmatrix} ] Where (R) is a 3x3 [rotation matrix](https://en.wikipedia.org/wiki/Rotation_matrix), and (t) is the 3x1 translation vector. To operate on a point (p), we simply append 1 to it and we can see: [ g * p= \\begin{bmatrix} R & t \\ 0 & 1 \\ \\end{bmatrix} \\begin{bmatrix} p \\ 1 \\ \\end{bmatrix} = \\begin{bmatrix} Rp + t \\ 1 \\end{bmatrix} ] Because we can represent our poses using a matrix, we now have a way to represent their derivatives. We can just store the ordinary time derivatives of the matrix. This will involve us storing 12 floating point numbers: 9 for the rotation matrix and 3 for the translation. We can do better however. After all, we know that the rotation time derivatives (a.k.a. the angular velocity) should be expressible with only three numbers. There's a trick that we can use here. If we multiply our ordinary time derivative (\\dot g) by (g^{-1}), we get a simpler form: [ g^{-1} \\dot g = \\begin{bmatrix} 0& -\\omega_3 & \\omega_2 & v_1\\ \\omega_3 & 0& -\\omega_1 & v_2\\ -\\omega_2 & \\omega_1 & 0& v_3\\ 0 & 0 & 0 & 0& \\ \\end{bmatrix} ] Now we only have to store six number in what we call a **right** tangent vector: [ \\begin{bmatrix} \\omega_1 & \\omega_2 & \\omega_3 & v_1 & v_2 & v_3 \\end{bmatrix} ^T ] This is a "right" tangent vector because the ordinary time derivative (\\dot g) appears on the right side of the expression (g^{-1} \\dot g). As you might imagine, there are also "left" tangent vectors because the expression (\\dot g g^{-1}) happens to also produce matrices of exactly the above form. However, the left and right tangent vector representations of the time derivative are in general different for a given robot trajectory, so care must be taken when dealing with them. For simplicity, we only deal with right tangent vectors here. One can verify that if (g) as written above is `scene_from_robot`, then (\\omega_1), (\\omega_2), and (\\omega_3) in the right tangent vector represent the components of the robot's angular velocity expressed in its own coordinates. If, for example, they are ([1, 0, 0]), then the robot is rolling to its right (assuming the x axis is forward). One can further verify that (v_1), (v_2), and (v_3) are the robot's linear velocity expressed in its own coordinates. If they are ([1, 0, 0]) the actor is moving along its own x axis. When working with these tangent vectors in code, we often want to easily access the components from them. For example, let's say we want to know what the velocity (v) and angular velocity (\\omega) are in **scene** coordinates. The paragraph above tells us how to get these velocities in robot coordinates, and then we need to rotate them to get them into scene coordinates. The variable `d_scene_from_robot` holds a right tangent vector representing our time derivative (e.g. (g^{-1}\\dot g) if (g) represents `scene_from_robot`). `SE3` is equipped with helpers to get these components from a tangent vector: C++ ``` const SE3::TangentVector d_scene_from_robot = SE3::TangentVector::Random(); const Vec3 robot_angular_velocity_in_robot_coordinates{ SE3::tangent_vector_rotation_part(d_scene_from_robot)}; const Vec3 robot_velocity_in_robot_coordinates{ SE3::tangent_vector_translation_part(d_scene_from_robot)}; const Vec3 robot_angular_velocity_in_scene_coordinates{ scene_from_robot.rotation() * robot_angular_velocity_in_robot_coordinates}; const Vec3 robot_velocity_in_scene_coordinates{ scene_from_robot.rotation() * robot_velocity_in_robot_coordinates}; ``` If we want to go in the opposite direcion, we can also do that: C++ ``` REASSERT( d_scene_from_robot == SE3::tangent_vector_from_parts( robot_angular_velocity_in_robot_coordinates, robot_velocity_in_robot_coordinates)); ``` Note For (\\text{SO(3)}), the tangent vector matrices have a simpler form: [ g^{-1} \\dot g = \\begin{bmatrix} & -\\omega_3 & \\omega_2 \\ \\omega_3 & & -\\omega_1 \\ -\\omega_2 & \\omega_1 & \\ \\end{bmatrix} ] Where (\\omega) is once again the angular velocity as described above. ## Exp and Log We touched breifly above on the fact that the exponential on (\\text{SO(3)}) converts angle axis representations to elements of (\\text{SO(3)}), but we did not fully explain this exponential operation that the `SE3` and `SO3` classes both implement. This operation takes a tangent vector (i.e. a time derivative) and tells you where you end up in the group following that constant time derivative for one unit of time. More explicitly, if we think about the idea of the the matrix representation of the right tangent vector (g^{-1} \\dot g) (as described above) being constant along a trajectory, we realize that this statement is a matrix ordinary differential equation (ODE): [ g^{-1} \\dot g = \\text{Constant} = X ] [ \\dot g = g X ] This matrix ODE has a known solution given by the [Matrix exponential](https://en.wikipedia.org/wiki/Matrix_exponential): [ g = g_0\\text{Exp}(Xt) = g_0\\left( \\sum\_{k = 0}^{\\infty} \\frac{1}{k!} X^k t^k\\right) ] Where (g_0) is some initial condition for (g). The exponential map (\\exp()) defined for `SE3` and `SO3` is the same as this matrix exponential up to the input formatting. For instance, for (\\text{SO(3)}): [ \\exp\\left(\\begin{bmatrix} \\omega_1 \\ \\omega_2 \\ \\omega_3 \\end{bmatrix}\\right) = \\text{Exp}\\left(\\begin{bmatrix} & -\\omega_3 & \\omega_2 \\ \\omega_3 & & -\\omega_1 \\ -\\omega_2 & \\omega_1 & \\ \\end{bmatrix}\\right) ] In practice, we don't have to compute an infinite series for (\\exp()) because there are closed forms for (\\text{SO(3)}) (the [Rodrigues formula](https://en.wikipedia.org/wiki/Rodrigues%27_rotation_formula)) and (\\text{SE(3)}). We also provide the logarithm function, which is the inverse of (\\exp()). In other words, it gives the constant velocity needed to arrive at the given element from the identity in one time unit. For their complex definitions, using these in code is quite simple: C++ ``` REASSERT(my_tangent_vector.isApprox(SE3::exp(my_tangent_vector).log())); ``` ## Frame Checking So far we've ignored the fact that our `SO3` and `SE3` classes come equipped with frame checking capabilities (as alluded to in [Lie Groups](../liegroups/)). In brief, each `SO3` and `SE3` object can be assigned two coordinate frames, represented by `Frame` objects. This is done by passing the coordinate frames into the constructor or into the exponential or identity member functions. When assigned, these objects ensure that frame consistency is maintained when composing Lie groups. Here's an example: C++ ``` #include "resim/assert/assert.hh" #include "resim/transforms/frame.hh" #include "resim/transforms/se3.hh" #include "resim/transforms/so3.hh" #include "resim/visualization/view.hh" using resim::transforms::SE3; using resim::transforms::SO3; using Frame = resim::transforms::Frame; /* SE3::DIMS == 3 */ // ... const Frame world{Frame::new_frame()}; const Frame robot{Frame::new_frame()}; const Frame sensor{Frame::new_frame()}; // The pose of the robot in the world const SE3 world_from_robot{SO3::identity(), {5., 5., 0.}, world, robot}; // Visualize with ReSim View VIEW(world) << "World frame"; VIEW(robot) << "Robot frame"; VIEW(world_from_robot) << "World from robot"; // The pose of a sensor mounted on the robot const SE3 robot_from_sensor{ SO3{M_PI_2, {0., 0., 1.}}, {0., 0., 1.}, robot, sensor}; // Visualize with ReSim View VIEW(sensor) << "Sensor frame"; VIEW(robot_from_sensor) << "Robot from sensor"; const SE3 world_from_sensor{world_from_robot * robot_from_sensor}; REASSERT(world_from_sensor.is_framed()); REASSERT(world_from_sensor.into() == world); REASSERT(world_from_sensor.from() == sensor); // Whoops! This fails at run time because we forgot to invert // robot_from_sensor! // const SE3 robot_from_world{world_from_sensor * robot_from_sensor}; // We should have done: const SE3 robot_from_world{world_from_sensor * robot_from_sensor.inverse()}; ``` Using frame checking is generally good practice as it makes it less likely for silly bugs to occur. In the future, we plan on making it possible to deactivate frame checking as an optional performance optimization. Note that composition with an unframed `SO3` or `SE3` *always* results in an unframed `SO3`/`SE3`. Consequently, unframed objects can propogate rapidly if one is not deliberate about using framed objects. Note Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/liegroups.cc) for the examples above. # Lie Group Derivatives ## Introduction This document provides some mathematical background on Lie groups and Lie algebras. In particular, this document covers how differentiation works on Lie groups, the Lie group exponential, left and right tangent spaces, the adjoint representations of a Lie group and its algebra, and the chain rule for Lie groups. ## Differentiating Curves on Lie Groups By definition, a Lie group is any differentiable manifold which is also a group. A differentiable manifold is not generally [diffeomorphic](https://en.wikipedia.org/wiki/Diffeomorphism) to Euclidean space (e.g. (\\mathbb{R}^N)), but it is locally diffeomorphic to Euclidean space in the sense that we can define a diffeomorphism (\\phi_p) with (\\mathbb{R}^N) in the neighborhood of any point (p) in our group (G). We don't know how to take derivatives directly on the group (G), but we *do* know how to do this on Euclidean space. So we define the derivative with respect to a **particular** choice of (\\phi_p) in the neighborhood of the point (p) where we're taking the derivative. For a trajectory (g:\\mathbb{R}\\rightarrow G) with (g(t) = p), this looks like: [ \\frac{d^{\\phi_p}g(t)}{dt} \\equiv \\frac{d}{dt}(\\phi_p \\circ g)(t) ] Since (\\phi_p \\circ g) is just a function from (\\mathbb{R}) to (\\mathbb{R}^N), we can take its derivative in the standard way, and the typical properties of derivatives (e.g. the chain rule) all apply. We just need to come up with a consistent way of picking (\\phi_p) for whatever point (p) we're at. For a Lie group, one way we can do this is to first define (\\phi_e) for the identity element (e\\in G) and then we can define (\\phi_p) in the neighborhood of any point (p), since Lie group elements are guaranteed to have an inverse. We can do this like so: [ \\phi_p(g) = \\phi_e(p^{-1} g) ] So our derivative at a point (p = g(t)) along our trajectory is given by: [ \\frac{d^{\\phi_p}g(t)}{dt} = \\frac{d}{d\\epsilon} \\phi_e (g(t)^{-1} g (t + \\epsilon)) ] This happens to be a nice choice for (\\phi_p) for each point (p) because it has a property called left invariance. This means that the derivative is the same for a curve that is left multiplied by a **constant** element of (G). Say that we're differentiating (f(t) = hg(t)) for constant (h\\in G) and (g:\\mathbb{R}\\rightarrow G). In this case, (p = f(t) = h g(t)) and we have: [ \\begin{aligned} \\frac{d^{\\phi_p} f}{dt} &= \\frac{d}{dt} \\phi_p(h g(t)) \\ &= \\frac{d}{dt} \\phi_e(p^{-1} h g(t)) \\ &= \\frac{d}{d\\epsilon} \\phi_e((g(t)^{-1} h^{-1}) h g(t + \\epsilon)) \\ &= \\frac{d}{d\\epsilon} \\phi_e(g(t)^{-1} g(t + \\epsilon)) \\ &= \\frac{d^{\\phi\_{g(t)}} g}{dt} \\end{aligned} ] Which demonstrates the left invariance. ## Lie Group Exponential Now, let's define a curve (\\gamma: \\mathbb{R} \\rightarrow G) with the properties that ((d^{\\phi_p}\\gamma / dt)) as defined above is **constant** at all points along the curve and that (\\gamma(0) = e). We claim without proof that it's possible to uniquely define such a curve. If we define (\\gamma_2(t) = g^{-1} \\gamma(t)) for some (g) in the range of (\\gamma), then we can conclude that (\\gamma_2(t) = \\gamma(t - \\gamma^{-1}(p))) because (\\gamma_2) also goes through the origin (at (t = \\gamma^{-1}(p))) and has the same constant value for ((d^{\\phi_p} \\gamma_2/dt)). Therefore, the curves have the same range, and (\\gamma_2(0) = p^{-1}) is also a member of the range of (\\gamma). We can do this for any (p) in the range of (\\gamma). We can conclude that the range of (\\gamma) is a subgroup of (G) because it contains the identity, every element in the subgroup has an inverse, and it is closed under composition with other members of the subgroup. This is called a **one parameter subgroup** of (G). Now, we can go further by defining the *Lie Group Exponential* as the function (\\exp: \\mathbb{R}^N \\rightarrow G) satisfying (\\exp(X) = \\gamma(1)) where (\\gamma) is defined as above with (X = (d^{\\phi_p}\\gamma /dt)). In other terms, the exponential tells you where you end up if you follow the same constant derivative for one unit of time through the Lie group. The standard exponential on real numbers falls out of this definition if you consider curves with constant derivatives that intersect the multiplicative identity: \[ \\begin{aligned} \\frac{d^{\\phi_p}y}{dt} &= \\frac{d}{d\\epsilon} \\left[y(t)^{-1} y(t + \\epsilon)\\right] = k \\ y(0) &= 1 \\ \\end{aligned} \] With some rearrangement and variable substitution: [ \\begin{aligned} y'(t) &= ky \\ y(0) &= 1 \\ \\end{aligned} ] which has a known solution (y = \\exp(t)). Therefore, the Lie group exponential can be interpreted as a natural extension of the familiar concept of the exponential. In fact, for matrix Lie groups (Lie groups which have a matrix representation), the Lie group exponential is identical to the matrix exponential. Furthermore, just as the ordinary exponential has an inverse, the logarithm, there is a Lie group logarithm, which is the inverse of the Lie group exponential. Conceptually, the logarithm tells you "what constant velocity would I have to move at to arrive at this point in one unit of time from the origin." It's worth noting that (\\mathbb{R}^N) is not formally the input to the exponential. The domain of the exponential is actually called the **Lie Algebra** (\\mathfrak{g}) corresponding to the Lie group (G). It is a vector space and hence it is homeomorphic to (\\mathbb{R}^N), so we casually treat them as interchangeable. The exponential and logarithm define a correspondence between the Lie group and a vector space (\\mathbb{R}^N) called the Lie algebra. This connection allows us to leverage much of the mathematics derived in vector spaces on Lie groups which can be incredibly powerful. For instance, one can define a Gaussian probability distribution on the Lie algebra (\\mathfrak{so}(3)) and take the exponential of samples from it to sample orientations in the Lie group (\\text{SO(3)}). Because they are commonly useful, we provide implementations of the exponential and logarithm for each Lie group we implement. ## Left and Right Tangents So far, we have been working with the left invariant definition for the derivative: [ \\frac{d^{\\phi_p}g(t)}{dt} = \\frac{d}{d\\epsilon} \\phi_e (g(t)^{-1} g (t + \\epsilon)) ] However, we've been very vague about what we may pick for (\\phi_e). We know it needs to be a diffeomorphism defined in an open set around (e). It turns out that the Lie group logarithm is a reasonable choice for this, so this is what we use. Hence: [ \\phi_p(g) = \\log(p^{-1} g) ] As noted before, this definition is left invariant since we can multiply on the left by any member of the group without changing the value of the derivative. This is practically quite useful because if (g) represents a transform of `world_from_robot` as a function of time, it doesn't actually matter where we decide the "world" frame is when we're computing derivatives as long as our choice does not move relative to some agreed-upon world frame (remember that (h) must be constant for the above to work. Confusingly, although this choice for (\\phi_p) yields *left* invariance, ((d^{\\phi_p}g/dt)) is often referred to as the "right tangent" or "right tangent space derivative" of (g). This is because it can also be defined in terms of the *right* perturbation to (g) that the trajectory is "following" at that moment. \[ g(t + \\epsilon) = g(t) \\exp\\left[\\frac{d^{\\phi_p}g}{dt} \\epsilon\\right] \] which when rearranged gives: \[ \\frac{d^{\\phi_p}g}{dt} = \\frac{\\log\\left[g(t)^{-1} g(t + \\epsilon )\\right]}{\\epsilon} \] In the limit as (\\epsilon \\rightarrow 0), we get: [ \\frac{d^{\\phi_p}g}{dt} = \\frac{d}{d\\epsilon} \\log(g(t)^{-1} g(t + \\epsilon)) ] which is equivalent to what we had before. For this reason, we henceforth refer to this definition of (\\phi_p) as (\\phi_p^R). The reason for the special notation, is that we could equally well have chosen: [ \\phi_p(g) = \\phi_p^L(g) = \\log(g p^{-1}) ] As before, (g p^{-1}) is in the neighborhood of the identity (e) when (g) is in the neighborhood of (p), so this works. With this, we get: [ \\begin{aligned} \\frac{d^{\\phi_p^L} g}{dt} &= \\frac{d}{dt} \\log(gp^{-1}) \\ &= \\frac{d}{d\\epsilon} \\log(g(t + \\epsilon) g(t)^{-1}) \\end{aligned} ] ((d^{\\phi_p}g/dt)) is often referred to as the "left tangent" or "left tangent space derivative" of (g), and one can verify that it can be defined in terms of a left perturbation to (g) that the trajectory is following and that it has a **right** invariance property such that multiplying the trajectory on the **right** by a constant element of the group does not affect its value. In summary, we have left and right tangent space derivatives that have right and left invariance respectively. ## The Adjoint In practice both the left and right tangent space derivatives happen to be useful in particular cases, so one might ask if it's possible to easily convert between the right tangent of a trajectory and the left tangent of the trajectory. If we want the left tangent space derivative, we can see: \[ \\begin{aligned} \\frac{d^{\\phi_p^L}g}{dt} &= \\frac{d}{dt}\\log(g p^{-1}) \\ &= \\frac{d}{dt}\\log((p p^{-1}) g p^{-1}) \\ &= \\frac{d}{dt}\\log(p (p^{-1} g) p^{-1}) \\ &= \\frac{d}{dt}\\log(p \\exp[\\log(p^{-1} g)] p^{-1}) \\ \\end{aligned} \] Applying the chain rule to (h(k(t))) where (k(t) = \\log(p^{-1} g)) and (h(k) = \\log(p \\exp(k) p^{-1})) are both functions to and from Euclidean space, we have: \[ \\begin{aligned} \\frac{d^{\\phi_p^L}g}{dt} &= \\left[\\frac{d}{dk} \\log(p \\exp(k) p^{-1})\\right] \\frac{d^{\\phi_p^R}g}{dt} \\end{aligned} \] so we can convert from right to left tangent space by multiplying by this Jacobian matrix: \[ \\text{Ad}\_g \\equiv \\left[\\frac{d}{dk} \\log(g \\exp(k) g^{-1})\\right] \] where we define this Jacobian to be the adjoint representation of the element (g\\in G). Technically, the adjoint representation is the map that produces such matrices given inputs from (G). This map can also be defined as the derivative of a curve (f(g) = p g p^{-1}) at the identity. Since this is a map from (G) to (G), one can show this by assuming that (g) is a curve passing through the identity and using the chain rule as above to find what ((df/dg) = \\text{Ad}\_p) is. There are a number of properties that the adjoint has that are worth noting: [ \\text{Ad}\_{g^{-1}} = \\text{Ad}\_g^{-1} ] \[ \\text{Ad}_{gh} = \\text{Ad}_{g} \\text{Ad}\_{h} \] which is basically just a reminder that the adjoint is a **representation** of G. In the case where the Lie group has a matrix representation (which is true for all the Lie groups we use), one can simplify our definition to be: [ \\text{Ad}\_g X = g X g^{-1};\\quad\\quad g\\in G,,,X\\in \\mathfrak{g} ] for any (X) in the Lie Algebra (\\mathfrak{g}) of (G). Note that this is a matrix representation of the Lie algebra element, not just a vector in (\\mathbb{R}^N). Expressions for (\\text{Ad}\_g) are derived for all the Lie groups we implement since it is so commonly needed. It's often the case that we need to take the derivative of (\\text{Ad}\_g) with respect to time. We define the adjoint representation of the algebra (\\text{ad}\_X) based on a curve (g(t)) going through the identity. \[ \\left.\\frac{d \\text{Ad}_g}{dt}\\right|_{g = e} = \\text{ad}_X;\\quad\\quad X = \\left.\\frac{dg}{dt}\\right|_{g = e} \] It doesn't matter whether we use the right or left tangent space derivative here for (g) since they are equivalent at the identity, as one can verify by inspecting their definitions. The algebra adjoint is related to the Lie bracket: \[ \\text{ad}\_X Y = [X, Y] \] For matrix Lie groups, the bracket is the commutator on the matrix representation of algebra elements. \[ \\text{ad}\_X Y = [X, Y] = XY - YX \] The algebra adjoint is somewhat commonly used, so we provide it as a static member function in our Lie group objects. ## The Chain Rule Let's look at how one might differentiate the composition of two Lie group elements in right tangent space, as an example. In other words, take the time derivative of (f(t) = g(t)h(t)): \[ \\begin{aligned} \\frac{d^Rf}{dt} &= \\frac{d^R}{dt} [g(t)h(t)] \\ &= \\frac{d}{d \\epsilon} \\log [ h(t)^{-1} g(t)^{-1} g(t + \\epsilon) h(t + \\epsilon)] \\end{aligned} \] To help ourselves, let's define a function (c(\\delta_1, \\delta_2)) like so: \[ c(\\delta_1, \\delta_2) = \\log [ h(t)^{-1} g(t)^{-1} g(t + \\delta_1) h(t + \\delta_2)] \] where (\\delta_1) and (\\delta_2) are functions of (\\epsilon). Taking the derivative with respect to (\\epsilon) with the multivariable chain rule gives: [ \\frac{d}{d\\epsilon}c(\\delta_1, \\delta_2) = \\text{Ad}\_{h^{-1}} \\frac{d^R g}{d t} \\frac{d\\delta_1}{d\\epsilon} + \\frac{d^R h}{dt} \\frac{d\\delta_2}{d\\epsilon} ] Of course, the derivative of (c) is useful to us if (\\delta_1 = \\delta_2 = \\epsilon) so: [ \\frac{d^R f}{dt} = \\text{Ad}\_{h^{-1}} \\frac{d^R g}{dt} + \\frac{d^R h}{dt} ] which is the chain rule in the right tangent space. There is also a chain rule for the left tangent space: [ \\frac{d^L f}{dt} = \\frac{d^L g}{dt} + \\text{Ad}\_g \\frac{d^L h}{dt} ] One can verify the left and right invariance properties using these expressions and assuming one of the group elements is constant. # Curves Curves are functions from a single real scalar variable (such as time) to another manifold (such as 3D space or a [Lie group](../transforms/liegroups/)). Consequently, they end up being very useful for representing the trajectory of a rigid body over time (among other things), so we provide two primary classes to represent curves through Lie groups: - **DCurve**: This class represents an arc-length-parameterized curve through `SE3`. It gives an `SE3` object as a function of the distance travelled along the curve. - **TCurve**: This class gives a twice continuously differentiable time-parameterized curve built up from a sequence of quintic Hermite spline segments. Each of these has its own peculiarities and uses which we will outline in detail. ## DCurve A `DCurve` is a sequence of geodesic segments that interpolate between a vector of `SE3` objects passed to its constructor. More precisely we can make a piecewise geodesic curve connecting the frames described by each `SE3` object we passed in order. We can query the curve as a function of the arc length along it using the `DCurve::point_at()` function. As the arc length increases from zero, the curve will move through each of the `SE3`s we passed into the constructor creating a continuous curve. `DCurve`s are useful whenever one has a set of points with headings (or full orientations) and they want to produce a reasonable continuous geometry connecting them. This could be the case if a user wants to represent static geometry, such as the center-line of a road, or a boundary around an area. However, one should note that the derivatives of the motion along a `DCurve` are not continuous when switching from one segment to the next. Furthermore, they do not work well if they have zero-length segments. Here's an example of how to construct and use a `DCurve`: C++ ``` #include "resim/assert/assert.hh" #include "resim/curves/d_curve.hh" #include "resim/transforms/se3.hh" #include "resim/transforms/so3.hh" #include "resim/visualization/view.hh" // ... using resim::transforms::SE3; using resim::transforms::SO3; using Frame = resim::transforms::Frame<3>; const Frame world_frame{Frame::new_frame()}; VIEW(world_frame) << "World frame"; const Frame d_curve_frame{Frame::new_frame()}; VIEW(d_curve_frame) << "DCurve"; // Make a DCurve that goes straight for one unit, turns left along a unit // circle for pi/2 radians, continues straight in the y direction for one // unit, and then turns to the right back to its original heading along a unit // circle. const resim::curves::DCurve d_curve{ SE3::identity(world_frame, d_curve_frame), SE3{{1., 0., 0.}, world_frame, d_curve_frame}, SE3{SO3{M_PI_2, {0., 0., 1.}}, {2., 1., 0.}, world_frame, d_curve_frame}, SE3{SO3{M_PI_2, {0., 0., 1.}}, {2., 2., 0.}, world_frame, d_curve_frame}, SE3{{3., 3., 0.}, world_frame, d_curve_frame}, }; // Query a point halfway along the third edge: const double QUERY_ARC_LENTH = 1.0 + M_PI_2 + 0.5; const SE3 reference_from_queried{d_curve.point_at(QUERY_ARC_LENTH)}; REASSERT( reference_from_queried.rotation().is_approx(SO3{M_PI_2, {0., 0., 1.}})); REASSERT(reference_from_queried.translation().isApprox( Eigen::Vector3d{2., 1.5, 0.})); VIEW(d_curve) << "My DCurve"; ``` This produces a curve that looks like this: ## Two Jet Before we address `TCurve` objects, we need to discuss the `TwoJet` object. Briefly, a `TwoJet` is a struct containing a Lie group element along with its first and second derivatives. There are two varieties of `TwoJet`, namely `TwoJetR` and `TwoJetL`. The distinction between them is that `TwoJetR` represents its first and second derivatives as *right tangent vectors* (described fully in [Using SO(3) and SE(3)](../transforms/using_liegroups/) and [Lie Group Derivatives](../transforms/liegroup_derivatives/)), whereas `TwoJetL` uses *left tangent vectors*. For simplicity, we'll use the `actor::state::RigidBodyState` abstraction in our example to make defining these easy. That way, we can write clearer code that doesn't require us reason about tangent vectors. ## TCurve A `TCurve` represents a time-based trajectory using a sequence of quintic [Hermite splines](https://en.wikipedia.org/wiki/Hermite_interpolation) on the group (\\text{SE(3)}) (or (\\text{SO(3)})). The details of this operation are outlined [here](https://ethaneade.com/lie_spline.pdf), but the main take-away is that this creates a curve through (\\text{SE(3)}) which is twice continuously differentiable ((\\text{C}^2)) as it proceeds through a set of user-provided control points. All we need to create a `TCurve` is a sequence of control points which each have a time along with a `TwoJetL` that describes the desired pose and derivatives at that time. One major use of `TCurve`s is to replay observed actor trajectories in a simulation. One simply has to take the observed states of the actors from an autonomy log and convert them into control points from which they can make a `TCurve`. Then, the simulation just has to query this `TCurve` over time to replay the trajectory. Usually, it is recommended to use the `actor::state::Trajectory` class, which is a higher-level wrapper for `TCurve` in such cases. Note that the control points must be monotonically increasing in time for the curve to be valid. Here's an example of how we can create one: C++ ``` #include "resim/actor/state/rigid_body_state.hh" #include "resim/assert/assert.hh" #include "resim/curves/t_curve.hh" #include "resim/transforms/frame.hh" #include "resim/transforms/se3.hh" #include "resim/transforms/so3.hh" #include "resim/visualization/view.hh" // ... using resim::transforms::SE3; using resim::transforms::SO3; using RigidBodyState = resim::actor::state::RigidBodyState; using TCurve = resim::curves::TCurve; using Frame = resim::transforms::Frame<3>; const Frame t_curve_frame{Frame::new_frame()}; VIEW(t_curve_frame) << "TCurve"; // Define some states that we want to pass through // A state at the origin with a small forward velocity RigidBodyState state_a{SE3::identity(world_frame, t_curve_frame)}; state_a.set_body_linear_velocity_mps({0.1, 0., 0.}); // A state at (1, 1, 0) oriented along the x axis with a small forward // velocity and a small angular velocity about the x axis. RigidBodyState state_b{SE3{{1., 1., 0.}, world_frame, t_curve_frame}}; state_b.set_body_linear_velocity_mps({0.1, 0., 0.}); state_b.set_body_angular_velocity_radps({0.1, 0., 0.}); // A state at (3, 0, 0.5) with a rotation of pi/4 about the z axis RigidBodyState state_c{ SE3{SO3{M_PI_4, {0., 0., 1.}}, {3., 0., 0.5}, world_frame, t_curve_frame}}; state_c.set_body_linear_velocity_mps({0.1, 0., 0.}); // Create a t_curve by getting left two jets from the states: const TCurve t_curve{{ TCurve::Control{ .time = 0., .point = state_a.ref_from_body_two_jet().left_two_jet(), }, TCurve::Control{ .time = 10., .point = state_b.ref_from_body_two_jet().left_two_jet(), }, TCurve::Control{ .time = 20., .point = state_c.ref_from_body_two_jet().left_two_jet(), }, }}; // Visualize VIEW(t_curve) << "My TCurve"; ``` The resulting curve looks like this: Note Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/curves.cc) for the examples above. # Rigid Body State & Trajectory Two very common tasks in robotics are: - Representing the position and orientation (i.e. pose) of a body along with its derivatives (velocity & acceleration) at a particular point in time. We call this a "rigid body state". - Representing rigid body states over a bounded period of time. In other words: a trajectory. Rigid body states and trajectories can be adequately represented by the `resim::curves::TwoJetR` and `resim::curves::TCurve` classes respectively. However, these are low-level mathematical representations that do not provide all the functionality we would like when reasoning about rigid body states and trajectories. Therefore, we provide two higher-level utility classes called `RigidBodyState` and `Trajectory`. These wrap `TwoJetR` and `TCurve` respectively and provide that functionality. In particular, using `RigidBodyState` objects, one can work directly with linear and angular velocities and accelerations expressed in the body's frame without having to reason directly about `SE3` tangent vectors. If we have a time series of `RigidBodyState` objects representing a trajectory, we can bundle these into a `Trajectory` object. The `Trajectory` object merely contains a `TCurve` under the hood, so the interpolation is performed in exactly the same way (i.e. using quintic Hermite splines). As an example of how to use these objects, lets make a unit circle trajectory: C++ ``` #include #include #include "resim/actor/state/rigid_body_state.hh" #include "resim/actor/state/trajectory.hh" #include "resim/time/timestamp.hh" #include "resim/transforms/se3.hh" #include "resim/transforms/so3.hh" #include "resim/visualization/view.hh" // ... using resim::transforms::SE3; using resim::transforms::SO3; using RigidBodyState = resim::actor::state::RigidBodyState; using Trajectory = resim::actor::state::Trajectory; using Vec3 = Eigen::Vector3d; // ... // We will be orbiting the unit circle at a rate of one radian per second. // Derivatives are constant along the unit circle: // Moving forward at one unit per second const Vec3 linear_velocity_mps = {1., 0., 0.}; // Yawing to the left at one unit per second const Vec3 angular_velocity_radps = {0., 0., 1.}; // Importantly, the acceleration here is the derivative of the velocity // expressed in the body's coordinate frame. Because the velocity expressed // in the body's frame is constant in this case, the acceleration is zero. // Because this coordinate frame is itself accelerating, the centripetal // acceleration (i.e. mv^2 / r) *does not appear*. const Vec3 linear_acceleration_mpss = {0., 0., 0.}; const Vec3 angular_acceleration_radpss = {0., 0., 0.}; const RigidBodyState::StateDerivatives derivatives{ .velocity = { .linear_mps = linear_velocity_mps, .angular_radps = angular_velocity_radps, }, .acceleration = { .linear_mpss = linear_acceleration_mpss, .angular_radpss = angular_acceleration_radpss, }, }; // We want to divide the circle into 4 segments, so we need to compute the // amount of time it should take per second. Since we're orbiting at 1 radian // per second, this time is just a quarter of the unit circle's circumference: constexpr double TIME_PER_SEGMENT = M_PI_2; constexpr resim::time::Timestamp START_TIME; const Vec3 z_axis = {0., 0., 1.}; const Trajectory::Control control_a{ .at_time = START_TIME, .state = RigidBodyState{SE3{SO3{M_PI_2, z_axis}, {1., 0., 0.}}, derivatives}, }; const Trajectory::Control control_b{ .at_time = START_TIME + resim::time::as_duration(TIME_PER_SEGMENT), .state = RigidBodyState{SE3{SO3{M_PI, z_axis}, {0., 1., 0.}}, derivatives}, }; const Trajectory::Control control_c{ .at_time = START_TIME + resim::time::as_duration(2. * TIME_PER_SEGMENT), .state = RigidBodyState{SE3{SO3{-M_PI_2, z_axis}, {-1., 0., 0.}}, derivatives}, }; const Trajectory::Control control_d{ .at_time = START_TIME + resim::time::as_duration(3. * TIME_PER_SEGMENT), .state = RigidBodyState{SE3{SO3::identity(), {0., -1., 0.}}, derivatives}, }; const Trajectory::Control control_e{ .at_time = START_TIME + resim::time::as_duration(4. * TIME_PER_SEGMENT), .state = RigidBodyState{SE3{SO3{M_PI_2, z_axis}, {1., 0., 0.}}, derivatives}, }; // Construct the trajectory Trajectory unit_circle_trajectory{{ control_a, control_b, control_c, control_d, control_e, }}; // Query our unit circle and make sure we always are one unit from the origin: const auto is_one = [](double x) { constexpr double TOLERANCE = 1e-12; return std::fabs(x - 1.) < TOLERANCE; }; for (resim::time::Timestamp t = unit_circle_trajectory.start_time(); t < unit_circle_trajectory.end_time(); t += std::chrono::milliseconds(100)) { REASSERT(is_one(unit_circle_trajectory.point_at(t) .ref_from_body() .translation() .norm())); } // Visualize the trajectory VIEW(unit_circle_trajectory) << "My trajectory"; ``` Running this code shows us our unit circle trajectory which looks like this: Note Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/trajectory.cc) for the example above. # Multivariate Gaussian [Gaussian (i.e. normal) distributions](https://en.wikipedia.org/wiki/Normal_distribution) are very useful in robotics (and in engineering more broadly) because physical quantities are quite often distributed in this way. This is because the sample mean of a collection of independent and identically distributed (iid) random variables is normally distributed assuming the underlying distribution has finite variance. Gaussian assumptions underlie the frequently-used [Kalman filter](https://en.wikipedia.org/wiki/Kalman_filter) algorithm, [optimal control theory](https://en.wikipedia.org/wiki/Linear%E2%80%93quadratic%E2%80%93Gaussian_control), and many other useful mathematical tools for robotics. Accordingly, it can be quite useful to use a Gaussian distribution when generating plausible physical values for use in a simulation. Of course, it is often the case that the values we might like to generate for a simulation are not independently distributed. For example, let's say we're trying to generate a realistic and representative road-using vehicle agent for our robot to interact with, and we want to generate values for the agent's height as well as a parameter representing their average speed. Intuitively we may expect that these parameters are negatively correlated since short sports cars tend to drive faster than tall class 8 trucks. Therefore, we shouldn't independently sample each parameter from a scalar Gaussian distribution. If we did, we would end up with as many fast-moving trucks as fast-moving sports cars! We instead want to sample from a realistic joint distribution for the two quantities which we approximate as a Gaussian with mean and covariance: [ \\mu = \\begin{bmatrix} 20 \\ 2 \\end{bmatrix}\\quad\\quad \\Sigma = \\begin{bmatrix} 4.8 & -0.96 \\ -0.96 & 0.25 \\ \\end{bmatrix} ] To sample from this distribution, we could do the following: C++ ``` #include #include #include #include "resim/math/multivariate_gaussian.hh" // ... using resim::math::Gaussian; using Vec = Gaussian::Vec; using Mat = Gaussian::Mat; // Set up our Gaussian sampler Vec mean = Vec::Zero(2); mean << 20., 2.; Mat covariance = Mat::Zero(2, 2); covariance << 4.8, -0.96, -0.96, 0.25; Gaussian gaussian{mean, covariance}; // Sample the distribution: Gaussian::SamplesMat samples = gaussian.samples(1000); // Write our data out std::ofstream output; output.open("gaussian_samples.csv"); for (const auto &sample : samples.rowwise()) { output << fmt::format("{0}, {1}", sample.x(), sample.y()) << std::endl; } output.close(); ``` Which gives the following samples: As you can see, we've produced a qualitatively reasonable set of values where agents with smaller heights tend to have a higher average speed. We could now use these to generate a set of simulation scenarios with agents like this. In reality we would likely want to use sample means and covariances based on real world data rather than making up a qualitatively nice mean and covariance like we did for this example, but hopefully this gives a taste of how this library can be used in production code. Note Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/gaussian.cc) for the example above. # Time ## Basic Types Within the ReSim libraries, we generally represent durations using `resim::time::Duration`, which is an alias for `std::chrono::nanoseconds` and timestamps as `resim::time::Timestamp` which is an alias for `std::chrono::sys_time`. ## Converters ### Floating Point Representation While we generally use the aforementioned integer-based timestamps and durations where possible, it is very often convenient to store times as double-precision floating point values, especially when using them in physical computations. If the values are sufficiently small (e.g. elapsed time in a sim), this can be done with no loss of accuracy. To facilitate, this, we have convenience converters: C++ ``` #include "resim/time/timestamp.hh" #include "resim/assert/assert.hh" // ... using namespace resim::time; Duration my_duration{std::chrono::nanoseconds(10000U)}; double my_duration_s = as_seconds(my_duration); // Should pass since my_duration is small REASSERT(as_duration(my_duration_s) == my_duration); my_duration += std::chrono::system_clock::now().time_since_epoch(); my_duration_s = as_seconds(my_duration); // Will likely not pass because my_duration is big and precision is lost // converting it to a double. REASSERT(as_duration(my_duration_s) == my_duration); ``` ### Seconds & Nanoseconds Struct Representation It's common in time serialization formats (e.g. [google::protobuf::Timestamp](https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/timestamp.proto) and ROS2's [builtin_interfaces/msg/Time.msg](https://github.com/ros2/rcl_interfaces/blob/master/builtin_interfaces/msg/Time.msg)) for the seconds and nanoseconds to be stored as separate integer counts. For `google::protobuf::Timestamp`, we provide standard packers and unpackers: C++ ``` const time::Timestamp time{}; google::protobuf::Timestamp time_msg{}; pack(time, &time_msg); const time::Timestamp unpacked_time = unpack(time_msg); ``` To facilitate other conversions to and from such serialization types, we have a time representation called `SecsAndNanos`: C++ ``` struct SecsAndNanos { int64_t secs = 0; int32_t nanos = 0; }; ``` And converters to and from it: C++ ``` const SecsAndNanos my_secs_and_nanos = to_seconds_and_nanos(my_duration); REASSERT(from_seconds_and_nanos(my_secs_and_nanos) == my_duration); ``` ## Event Scheduling Running simulations or processing logs often requires iterating through an ordered set of times in increasing order. These sets of interesting timestamps are not always uniformly spaced, and sometimes we want to add future times to this set as we are iterating through it. For instance, when modeling message transmission latency in a simulated system (i.e. one where we are explicitly controlling a simulated time), we may want to schedule a future time when we expect the message to arrive so we can properly transmit it to its receiver and keep the order of messages consistent with the real world system. To support this functionality, we define the `EventSchedule` class template which can store a time-ordered queue of events with arbitrary payloads. Here's a simple example of how to use it: C++ ``` #include #include "resim/time/event_schedule.hh" // ... EventSchedule my_messages; my_messages.schedule_event(Timestamp(std::chrono::nanoseconds(200)), "ReSim"); my_messages.schedule_event(Timestamp(std::chrono::nanoseconds(100)), "Hello"); my_messages.schedule_event(Timestamp(std::chrono::nanoseconds(200)), "user!"); std::cout.precision(9); std::cout << std::fixed; while (my_messages.size() > 0U) { const auto event = my_messages.top_event(); std::cout << "[" << event.time.time_since_epoch().count() / 1e9 << "] " << event.payload << std::endl; my_messages.pop_event(); } ``` Note that the event schedule is "stable" in the sense that it's first-in-first-out for events with the same timestamp. Hence the code above will print out: Text ``` [0.000000100] Hello [0.000000200] ReSim [0.000000200] user! ``` ## Interval Sampling Another common operation in running simulations and collecing log metrics is sampling an interval uniformly. For example, one may want to numerically compute a robot's average deviation from its desired pose over the course of a simulation. If a user wants to compute this average with Reimann rectangles, they could pick a maximum rectangle width (dt) that they are willing to accept (depending on what accuracy they desire) and then compute the integral using `resim::time::sample_interval()` and the associated `resim::time::num_samples()` function like so: C++ ``` #include #include "resim/time/sample_interval.hh" // ... const Timestamp start_time; const Timestamp end_time{start_time + std::chrono::seconds(30)}; const Duration max_dt = std::chrono::microseconds(100); // A stand in function for illustration purposes. const auto deviation = [](const Timestamp &t) { return std::cos(std::sqrt(as_seconds(t.time_since_epoch()))); }; double integral = 0.; sample_interval(start_time, end_time, max_dt, [&](const Timestamp &t) { // Left rectangles so we leave off the value at the end time if (t != end_time) { integral += deviation(t); } }); // The actual dt per rectangle is the total interval divided by (N - 1) const int N = num_samples(start_time, end_time, max_dt); const double dt_s = as_seconds(end_time - start_time) / (N - 1); integral *= dt_s; // Make sure we match the analytical result const double analytical_result = 2. * (-1. + std::cos(std::sqrt(30.)) + std::sqrt(30.) * std::sin(std::sqrt(30.))); REASSERT(std::fabs(integral - analytical_result) < 1e-4); ``` Note Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/time.cc) for the examples above. # Error Handling ## Introduction It is practically inevitable that errors will be encountered in any sufficiently complex software. In order to handle errors when they occur, we divide errors into the following categories: - **Recoverable Errors:** This is the class of errors where this is some possibility of the error being handled gracefully by the calling code. An example might be the case where a client is trying to connect to a server and it should re-try if the connection fails. - **Unrecoverable Errors:** This is the class of errors where there is (and should be) no way for the program to continue if encountered. Such an error means that the program is fundamentally invalid in some way. An example might be the case where I insert something into a `std::set` and then `std::map::contains()` returns false for that element. In this case, the contract of a specific interface has failed and there's no way for the caller to gracefully handle that. Within ReSim's libraries, we have specific ways of handling each of these error types outlined below. ## Unrecoverable Errors: REASSERT() For unrecoverable errors we use the `REASSERT()` macro. Using this macro is very simple: C++ ``` #include "resim/assert/assert.hh" // ... REASSERT(some_condition); REASSERT(some_other_condition, "Some failure message."); ``` If `some_condition` and `some_other_condition` are both true, then this code runs with no problem. If `some_condition` is false, the program will exit immediately with: Bash ``` terminate called after throwing an instance of 'resim::AssertException' what(): - ReAssertion failed: (some_condition). Message: Aborted ``` If `some_other_condition` is false, then we'll get the Message field populated: Bash ``` terminate called after throwing an instance of 'resim::AssertException' what(): - ReAssertion failed: (some_other_condition). Message: Some failure message. Aborted ``` Note that the text of the condition is always copied verbatim into the output. E.g.: Text ``` REASSERT(2 + 2 == 5); ``` Outputs: Bash ``` terminate called after throwing an instance of 'resim::AssertException' what(): - ReAssertion failed: (2 + 2 == 5). Message: Aborted ``` Under the hood, `REASSERT()` throws a `resim::AssertException` if the condition is false. As noted below, we never catch this exception in production code, so it always terminates the program. The reason we use an exception at all is so that we can verify in unit tests that the assertion would terminate the program when it should do so. We use Google Test's `EXPECT_THROW()` macro for this: C++ ``` // my_library.cc // ... void my_failing_subroutine() { REASSERT(false); } // my_library_test.cc // ... EXPECT_THROW(my_failing_subroutine(), AssertException); ``` ## Recoverable Errors: Status & StatusValue ### Why not Exception Handling? C++ has a built in language feature to handle this sort of error. Namely [exception handling](https://en.cppreference.com/w/cpp/language/exceptions). Developers can indicate that such an error has occurred by `throw`ing exceptions which can then be caught in a `try` block and gracefully handled. The [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html#Exceptions) has a detailed description of why using exceptions in this way is less than ideal. In brief, they can lead to very non-linear control flow which makes the code harder to reason about. Consequently, to the extent possible, we **never catch exceptions in our production libraries**. ### The Alternative Instead, we use function return values to pass error information up to the caller. This is a well-trodden path with languages like [Rust](https://doc.rust-lang.org/reference/expressions/operator-expr.html#the-question-mark-operator) even adding language features to make this sort of error handling syntactically nice. At ReSim we use `resim::Status` for this. For example: C++ ``` #include "resim/assert/assert.hh" #include "resim/utils/status.hh" using namespace resim; enum class Arg { GOOD_ARGUMENT = 0, BAD_ARGUMENT, }; Status my_subroutine(const Arg arg) { if (arg == Arg::BAD_ARGUMENT) { // Use this to make a new Status object with line number information. return MAKE_STATUS("Oh no! We failed!"); } return OKAY_STATUS; } // ... Status good_status = my_subroutine(Arg::GOOD_ARGUMENT); REASSERT(good_status.ok()); REASSERT(good_status.what() == "OKAY"); Status bad_status = my_subroutine(Arg::BAD_ARGUMENT); REASSERT(not bad_status.ok()); // Call this macro if we decide we want to exit on bad status. I.e. if we // don't want to gracefully handle the error, although that might be // possible. CHECK_STATUS_OK(bad_status); ``` When run, this code outputs the following: Bash ``` terminate called after throwing an instance of 'resim::AssertException' what(): - ReAssertion failed: ((bad_status).ok()). Message: {bad_status.what() == Oh no! We failed!} Aborted ``` Where `NN` is the line number where `CHECK_STATUS_OK()` is, and `MM` is the line number where `MAKE_STATUS()` is. Note that these might not generally be in the same source file, and we include both so that a user debugging the failure can more easily see where things began to go wrong and also where the issue became unrecoverable. It's commonly the case that a function may want to handle bad statuses by "passing the buck" up to its caller. Assuming that the enclosing function also returns a status, we can utilize the `RETURN_IF_NOT_OK()` macro to make this easy: C++ ``` Status my_wrapping_subroutine(const Arg arg) { // This is equivalent to: // Status s = my_subroutine(arg); // if (not s.ok()) { // return s; // } RETURN_IF_NOT_OK(my_subroutine(arg)); return OKAY_STATUS; } // ... Status good_status = my_wrapping_subroutine(Arg::GOOD_ARGUMENT); REASSERT(good_status.ok()); REASSERT(good_status.what() == "OKAY"); Status bad_status = my_wrapping_subroutine(Arg::BAD_ARGUMENT); REASSERT(not bad_status.ok()); ``` ### Working with Values Sometimes, we also want to return a value from a function that can fail. To handle this we use the `StatusValue` template where `T` is the type that we want to return. `StatusValue` is designed and tested to preserve `const`ness and `ref`ness of the contained value (i.e. you can do `StatusValue` and return a wrapped reference). It also implements the "buck-passing" behavior that `Status` does via the `RETURN_OR_ASSIGN()` macro which works in functions returning `Status` or `StatusValue` for any `T`. Here's an example of `StatusValue` in action: C++ ``` #include "resim/utils/status.hh" #include "resim/utils/status_value.hh" // ... StatusValue my_returning_subroutine(const Arg arg) { if (arg == Arg::BAD_ARGUMENT) { return MAKE_STATUS("Oh no! We failed!"); } return 3; } StatusValue my_returning_wrapper(const Arg arg) { int val = RETURN_OR_ASSIGN(my_returning_subroutine(arg)); return 2.0 * val; } Status my_outer_wrapper(const Arg arg) { double val = RETURN_OR_ASSIGN(my_returning_wrapper(arg)); std::cout << val << std::endl; return OKAY_STATUS; } // ... Status good_status = my_outer_wrapper(Arg::GOOD_ARGUMENT); REASSERT(good_status.ok()); REASSERT(good_status.what() == "OKAY"); Status bad_status = my_outer_wrapper(Arg::BAD_ARGUMENT); CHECK_STATUS_OK(bad_status); ``` This code first prints `6` (for the `GOOD_ARGUMENT` case) and then outputs: Bash ``` terminate called after throwing an instance of 'resim::AssertException' what(): - ReAssertion failed: ((bad_status).ok()). Message: {bad_status.what() == Oh no! We failed!} Aborted ``` As the error propogates out of the wrappers to the `CHECK_STATUS_OK()` macro which terminates the program. Note Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/assert_and_status.cc) for the examples above. # Variant Matching ## Overview When working with variants, one very frequently encounters cases of branching logic depending on what's present in the variant. Typically this looks like this: C++ ``` std::variant my_variant = 'm'; // ... if (std::holds_alternative(my_variant)) { const char my_char = std::get(my_variant); // // // } else if (std::holds_alternative(my_variant)) { const int my_int = std::get(my_variant); // // // } else { // // // } ``` It can be a bit annoying to set up this branching logic every time, and such an approach can sometimes be clunkier when multiple variant cases can be supported with the same logic (e.g. if we just want to convert all cases in `std::variant` to `double`). To address such cases, we include an implementation of pattern matching that allows users to provide a list of functors (referred to as branches) that can be applied to the current variant to the `match()` function. The branch which best matches the current case of the variant according to C++'s overload resolution rules will be selected and executed on the current case of the variant. The `match()` function then returns any value returned by the selected functor. Note that all functors must have the same return type and all variant cases must match at least one of the functors. ## Example Here's how you can perform pattern matching with the `match()` function: C++ ``` std::variant my_variant = 'm'; // ... std::cout << match( my_variant, // case: char [](const char c) { return "This variant contains a char!"; }, // case: int [](const int i) { return "This variant contains an int!"; }, // default: [](const auto x) { return "This variant contains a double or bool!"; }) << std::endl; ``` Gotchas / Pitfalls Due to the current implementation of this functionality, users should be aware of a few limitations. TL;DR just wrap every branch in a non-`mutable` lambda to be safe if you aren't sure. - The functors passed must be *functors*. Function pointers don't work. As a simple work-around, users can wrap these in a non-`mutable` lambda. - The functors should not define non-`const` `operator()`. This admonition includes any lambdas marked `mutable`. Because non-`const` member functions are preferred over `const` in overload resolution, this can cause ambiguities when other branches can potentially bind the current type (e.g. a `int` branch binding a `char`). We considered disabling all non-const `operator()` overloads, but decided that this would be more likely to cause silent errors in most cases (i.e. a default case might simply be used when a user didn't expect it). If a user must use a functor with a non-`const` `operator()`, they can work around it again by wrapping in a non-`mutable` lambda. - Functors that are passed as lvalues will end up being copied. If a user wishes to avoid this as an optimization, they can do so by reference capturing it in a lambda that they then pass in instead. Development Opportunities - It's not all that hard to add some functionality to convert any function pointers passed into `match()` into lambdas, but we haven't done it yet since that use case is very rare. - It's possible to avoid copying functors passed as lvalues, but we believe this would require a much more complex implementation than is warranted for this edge case. Note Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/match.cc) for the example above. # Output Parameters ## InOut Wrappers Although the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html#Inputs_and_Outputs) and consequently our style guide have a preference for return values over output parameters, there are still cases where having a function output data using a parameter is reasonable (e.g. when there are many outputs from a function and making a custom struct to return them all is inconvenient). Typically one could do this using pass-by-reference. Consider the following simple example: C++ ``` #include void set_to_three(int &x) { x = 3; } int main(int argc, char **argv) { int val; set_to_three(val); std::cout << val << std::endl; return 0; } ``` This code as written works perfectly well. However, in practice such code can be bugprone because developers cannot tell whether the call to `set_to_three()` will modify its arguments. To improve code clarity, we therefore prefer the use of our `InOut` wrapper when passing in arguments that may be modified by a function. Using it works like so: C++ ``` void set_to_three(resim::InOut x) { *x = 3; } int main(int argc, char **argv) { int val; set_to_three(resim::InOut{val}); std::cout << val << std::endl; return 0; } ``` In this case, the reader immediately knows that `val` may be changed by `set_to_three()`. Note that you can also use the arrow operator with `InOut`: C++ ``` #include #include "resim/utils/inout.hh" struct Foo { int x = 0; }; void set_to_three(resim::InOut f) { f->x = 3; } int main(int argc, char **argv) { Foo f; set_to_three(resim::InOut{f}); std::cout << f.x << std::endl; return 0; } ``` Under the hood, `InOut` is simply a pointer to the object passed in, so it should be treated with care to avoid dangling references. It's only use is to annotate the code so that users know which arguments they pass may be modified. Typically, there is no danger of memory issues if, as in this example, we're simply wrapping and passing a variable from the stack into our function. ## NullableReference Sometimes we only want a function to use an output parameter conditionally. Normally, one could use a raw pointer to do this. If the pointer is not `nullptr`, the function populates it, but otherwise leaves it alone: C++ ``` #include void maybe_set_to_three(int *x) { std::cout << "Ran maybe_set_to_three()!" << std::endl; if (x) { *x = 3; } } int main(int argc, char **argv) { int val; maybe_set_to_three(&val); std::cout << val << std::endl; maybe_set_to_three(nullptr); return 0; } ``` As above, however, we would like to make things more explicit. In this case, we want to make it clear that the pointer is being used as a nullable reference and that no memory ownership is being passed from the caller to the function. To do this, we use the `NullableReference` template: C++ ``` #include #include "resim/utils/nullable_reference.hh" void maybe_set_to_three(resim::NullableReference x) { std::cout << "Ran maybe_set_to_three()!" << std::endl; if (x.has_value()) { *x = 3; } } int main(int argc, char **argv) { int val; maybe_set_to_three(resim::NullableReference{val}); std::cout << val << std::endl; maybe_set_to_three(resim::null_reference); return 0; } ``` In some ways, this wrapper is almost the same as `InOut`, and one could use `NullableReference` anywhere that an `InOut` could be used. However, it is important to have these as separate wrappers because a function can express more about how it treats its parameters by its choice of `InOut` (where its clear that the function **will** output something there) or `NullableReference` (where it's clear the function **won't** if you don't ask it to). Note Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/output_parameters.cc) for the examples above. # ReSim View Visualization is a critical accelerator for robotics development. When developing and debugging robotics code there is simply no substitute for a good picture of what is going on in 3D space. Objects like vectors, transforms, geometries, and curves are very difficult to reason about up until the moment they are visualized. Then, the human brain can bring to bear its innate abilities to reason about 3D space and gain insights much more quickly and correctly. It is for this reason that visualization is of prime importance to our mission. The main visualization tool at the code level is ReSim View. In a single phrase, this tool is "print statements for 3D objects". In the same way that a print or log statement can help one reason about the behavior of non-robotics code, a View statement allows users a quick and lightweight way to see their 3D objects. Here's how it works: First, a user adds a `VIEW()` statement to their code, enclosing one of a set of supported types. The types currently include: 1. Named coordinate frames (`Frame`). 1. 3D rotations (`SO3`). 1. 3D poses (`SE3`). 1. Arc length parameterized curves (`DCurve`). 1. Time parameterized curves (`TCurve` and `Trajectory`). 1. Vectors in a coordinate frame (`FramedVector`). It's also possible to assign a name to each object being viewed which can make it much easier when viewing many different objects at once. This is done with the `<<` operator to pipe a name into the view statment. Here's an example of how one might visualize a transform with named coordinate frames: C++ ``` #include "resim/transforms/frame.hh" #include "resim/transforms/se3.hh" #include "resim/transforms/so3.hh" #include "resim/visualization/view.hh" // ... using resim::transforms::SE3; using resim::transforms::SO3; using Frame = resim::transforms::Frame; // Generate new frames with unique ids: const Frame scene = Frame::new_frame(); const Frame robot = Frame::new_frame(); const SE3 scene_from_robot{ SO3::identity(), // Rotation {1., 0., 1.}, // Translation scene, // Destination/Into Frame robot // Source/From Frame }; // We're using the << operator to add a name to our frame. If we don't use // it, the frame id is used instead. VIEW(scene) << "Scene Frame"; VIEW(robot); VIEW(scene_from_robot) << "My Transform"; ``` When this statement is executed at run time, it will prompt the user to authenticate (if they have not previously): Text ``` Authenticate at the following URL: https://resim.us.auth0.com/activate?user_code=XXXX-XXXX ``` And once authenticated the user will be given a link which they can follow to observe their visualization in Foxglove Studio. Text ``` View your data visualization in the ReSim App: https://app.signalflag.ai/view?bucket=resim-mcap&path=sessions/059787c0-3800-47e7-b95b-9d2e593d4908/view.mcap ``` Following this link opens Foxglove Studio in your browser with the visualization: If you don't initially see anything, you may have to enter the 3D panel settings and set its display frame to one of the frames in your scene (e.g. "Scene Frame" for this example): Foxglove Studio will automatically build a scene graph for you from the frames you provide. Consider the following example: C++ ``` #include "resim/transforms/frame.hh" #include "resim/transforms/se3.hh" #include "resim/transforms/so3.hh" #include "resim/visualization/view.hh" // ... using resim::transforms::SE3; using resim::transforms::SO3; using Frame = resim::transforms::Frame; // Generate new frames with unique ids: const Frame scene = Frame::new_frame(); const Frame robot = Frame::new_frame(); const Frame sensor = Frame::new_frame(); const SE3 scene_from_robot{ SO3::identity(), // Rotation {1., 0., 1.}, // Translation scene, // Destination/Into Frame robot // Source/From Frame }; const SE3 robot_from_sensor{ SO3(M_PI, {0., 0., 1}), // Rotation {-0.25, 0.25, 0.25}, // Translation robot, // Destination/Into Frame sensor // Source/From Frame }; // We're using the << operator to add a name to our frame. If we don't use // it, the id is used instead. VIEW(scene) << "Scene Frame"; VIEW(robot) << "Robot"; VIEW(scene_from_robot) << "Scene from Robot"; VIEW(sensor) << "Sensor"; VIEW(robot_from_sensor) << "Robot from Sensor"; ``` As mentioned above, ReSim View is capable of visualizing many more data types. Rather than go through these in detail here, we refer the reader to the examples in `resim/visualization/examples/` to explore and play around with the full capabilities of the tool. Don't forget to play around with the "play" button in Foxglove Studio when working with the time parameterized `TCurve` or `Trajectory` objects! Note Feel free to play around with the [source code](https://github.com/resim-ai/open-core/blob/main/resim/examples/view.cc) for the examples above. # ReSim Log Format The ReSim app is designed to work with any log format. This way, you can ensure that your simulation jobs produce the same types of logs that you might have from real-world data collection. This is often essential for tool or library re-use. In ReSim's open source library, however, we are working to provide a rich set of tools for computing and visualizing salient [metrics](https://github.com/resim-ai/open-core/tree/main/resim/metrics). We further aim to continue producing tools to analyze and leverage data from both simulation and real-world logs in the open source library. As we cannot write such tools to work with any log format, we maintain our own **ReSim Log Format**. We want this format to be as simple and portable as possible so that converters from other common robotics log formats are as trivial as possible. To this end we define the format like so: The ReSim Log Format is based on Foxglove's [MCAP](https://foxglove.dev/blog/introducing-the-mcap-file-format) file format. Using this file format furnishes us with a pre-existing set of open source tools and libraries for reading, writing, and visualizing our logs. In particular, [Foxglove Studio](https://foxglove.dev/) natively visualizes this format efficiently. Those familiar with the MCAP format will be aware that this format can hold arbitrary serialized robotics data with ROS1, ROS2, JSON, Flatbuffers, and Protobuf recieving first-class support from Foxglove's open-source libraries. Therefore, we define a set of protobuf message schemas in [open-core/resim/msg](https://github.com/resim-ai/open-core/tree/main/resim/msg) which our tools are designed to recognize and work with efficiently. This allows all of ReSim's metrics libraries to be leveraged for the small cost of writing log converters. For the ubiquitous [ROS2](https://ros.org/) middleware log format (CDR), such converters are [provided](../ros2/) for the defined ReSim message types. With that said, we have deliberately chosen to promulgate our own log format rather than using ROS's so that our users are not forced to depend on this large software ecosystem if they would prefer not to. Simultaneously, our provided converters eliminate integration friction for ROS users. A ReSim log therefore consists of an MCAP file containing ReSim messages for topics of interest. Other topics may be present and may have any schema/serialization. These topics will generally be ignored by ReSim tools, or copied unmodified by tools that input and output mcap files. There may be exceptions to this in the future. # ROS2 The [Robot Operating System](https://www.ros.org/) or ROS and its associated libraries represent one of the most ubiquitous collections of robotics software in use today. We therefore provide first class support for using ROS2 with ReSim's tools. To enable this without making ReSim's tools explicitly depend on ROS2 libraries, we provide a converter binary that can be used to convert ROS2 bags to the [ReSim Log Format](../msg) which our other tools and libraries natively consume. ## Basic Conversion If you're using common ROS2 message types (e.g. those in `std_msgs` or `geometry_msgs`) to log your topics of interest the conversion is as simple as cloning [resim/open-core](https://github.com/resim-ai/open-core/) and running the following commands. Bash ``` bazel run //resim/ros2:convert_log -- \ --log \ --output ``` If there are common public types that you need which we don't yet support, feel free to reach out and we can add them. A pybound version of this converter is also provided in the `resim_ros2` package distributed through our open-core [Releases](https://github.com/resim-ai/open-core/releases) and can be used like so: Python ``` import resim.ros2.resim_log_from_ros2_python as rlr2 from resim.ros2 import RESIM_DIR # ... rlr2.resim_log_from_ros2( str(RESIM_DIR / "ros2" / "default_converter_plugin.so"), # See plugin info below input_log_path, converted_log_path) ``` ## Custom Message Types & Converter Plugins Occasionally, it is necessary or just convenient to convert a custom message type to a serialized ReSim protobuf type so that its contents can be used with ReSim's open source metrics and analysis tools. To facilitate this, the aforementioned converter tools are designed with a plugin architecture to allow custom converters to be written. The core of the plugin interface is the following function: C ``` extern "C" ReSimConverterPluginStatus resim_convert_ros2_to_resim( const char *const ros2_message_type, const rcutils_uint8_array_t *const ros2_message, rcutils_uint8_array_t *const resim_message); ``` This essentially converts serialized ros2 message bytes to serialized resim message bytes based on the message type given as a string. There are also other functions that the binary needs to interrogate the plugin for: - Which types are supported for conversion. - Descriptions of the serialization format for the converted type (as required by Foxglove/MCAP tools). The best practice here is to simply copy [resim/ros2/default_converter_plugin.cc](https://github.com/resim-ai/open-core/blob/main/resim/ros2/default_converter_plugin.cc) and add the messages you care about to it, following its example. One can also create a minimal plugin which dynamically loads the default converter plugin and uses it to handle all the common message types. Once you've compiled such a plugin, it can be provided using the `--plugin` flag for `convert_log`: Bash ``` bazel run //resim/ros2:convert_log -- \ --plugin \ --log \ --output ``` or in Python: Python ``` import resim.ros2.resim_log_from_ros2_python as rlr2 from resim.ros2 import RESIM_DIR # ... rlr2.resim_log_from_ros2( my_custom_converter_plugin_path, input_log_path, converted_log_path, ) ``` # Explanation # Explanation Explanation is **understanding-oriented**: it illuminates the *why* behind SignalFlag's design. Read these articles when you want to build a mental model, not when you need step-by-step instructions. ## Articles - [ReSim Is Now SignalFlag](../rename/) — what the September 2026 rename changes for you, and what stays the same - [Understanding Batches & Tests](../guides/batches-and-tests/) — how SignalFlag structures test runs, what a batch is, and how individual tests relate to it ## SignalFlag U SignalFlag U is a collection of deeper technical articles on the mathematics underlying autonomous systems: - [Introduction to SignalFlag U](../resimu/introduction/) - [Lie Group Dynamics](../resimu/lie_group_dynamics/) - [Dual Quaternions for SE(3)](../resimu/dual_quaternions_for_se3/) More explanation articles are coming, covering topics like the SignalFlag testing model, the metrics pipeline, and how to design good test experiences. # ReSim is now SignalFlag On **Monday, September 21, 2026**, ReSim takes a new name: **SignalFlag**. It is the same product, the same team, and the same account you already have. This page lists exactly what changes for you and, more importantly, what does not. Short version Every URL, command, package, and credential you use today keeps working, and you do not need to change any of them. Two things are worth checking. If you filter inbound mail by sender domain, add `signalflag.ai` alongside `resim.ai`. If your network allowlists outbound domains and your team will use the new address, add `app.signalflag.ai`. ## What changes - **Web app address.** The app is also served at [app.signalflag.ai](https://app.signalflag.ai). Your existing bookmark to [app.resim.ai](https://app.resim.ai) keeps working and shows the same app. Neither address redirects to the other, so links you have already shared, in tickets, chat, or CI output, stay valid. - **Name and logo.** Starting the week of September 21, the app takes on the SignalFlag name, logo, and colors. Until that update lands you will still see ReSim branding, including at `app.signalflag.ai`. - **Website.** Our website is [signalflag.ai](https://signalflag.ai). - **Email from people on the team.** Mail written by SignalFlag staff will start coming from `@signalflag.ai` addresses. Automated mail from the platform, such as organization invitations, keeps coming from `no-reply@resim.ai` for now (see the note below). If you filter inbound mail by sender domain, allow `signalflag.ai` alongside `resim.ai`. Where you will still see the ReSim name for a while - **The sign-in page.** Signing in still goes through our identity provider at `resim.us.auth0.com`. That page still shows the ReSim logo and says **Log in to ReSim to continue to app.resim.ai**, even when you started from `app.signalflag.ai`. That is expected, and after you sign in you return to the address you started from. - **Organization invitations.** These still arrive from `ReSim `, mention ReSim in the subject and body, and link to `app.resim.ai`. They are legitimate. We will update them later in the transition. - **Command and package names.** The `resim` CLI and the `resim-*` Python packages keep working under their current names. The Python SDK is now also published as `signalflag`; see [What comes next](#what-comes-next). - **These docs.** The text now says SignalFlag, but the screenshots and diagrams still show the ReSim name, logo, and the previous chart colors. We are re-capturing them once the app itself is rebranded. Page URLs do not change. If your network restricts outbound domains `app.resim.ai` continues to work, so nothing needs to change. Allow `app.signalflag.ai` only if you want to use the new address. From either address, the app still calls `api.resim.ai`, `bff.resim.ai`, and `resim.us.auth0.com`, which you already allow. ## What does not change Everything below stays exactly as it is today. No configuration change is required on your side. ### Your account and data - Your organization, users, projects, systems, experiences, builds, batches, metrics, reports, and permissions are untouched. - Log in the same way you do today, whether with your email and password or with Google or GitHub sign-in. Both addresses use the same sign-in. - Pricing, contracts, and support contacts are unchanged. ### Integrations and developer tools | Component | Unchanged | | ------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | REST API | `https://api.resim.ai`, including the OpenAPI spec served there | | [MCP server](../guides/mcp/server/) | `https://bff.resim.ai/mcp` | | [CLI](../setup/cli/) | The `resim` command from [github.com/resim-ai/api-client](https://github.com/resim-ai/api-client), the `RESIM_*` environment variables, and the `~/.resim/resim.yaml` config file | | [Python packages](../guides/resim-sdk/) | `resim-sdk` and `resim-open-core` on PyPI, the `resim.*` imports, and your metrics config files. The SDK is now also published as `signalflag`, but nothing forces you to switch | | [HiL Agent](../guides/agent/) | The agent binary from [github.com/resim-ai/agent](https://github.com/resim-ai/agent) | | [GitHub Action](../setup/ci/github/) | `resim-ai/action` | | [IDE extension](../guides/ide-handoff/) | The `resim-ai.resim` extension for [VS Code](https://marketplace.visualstudio.com/items?itemName=resim-ai.resim) and [Cursor](https://open-vsx.org/extension/resim-ai/resim) | | Source code | [github.com/resim-ai](https://github.com/resim-ai) | | Documentation | [docs.resim.ai](https://docs.resim.ai), including `llms.txt` and the raw markdown URLs | | Status page | [status.resim.ai](https://status.resim.ai) | | [Cloud access](../setup/resim-data-access/) | Your organization's `resim-customer-...` IAM role and any CI credentials we have issued to you | ## What comes next We are changing the customer-facing name first. Every existing hostname, package name, and command name keeps working. Where we add SignalFlag-named equivalents, the `resim` ones stay alongside them. Over the coming weeks: - **Documentation.** These docs are now also served at `docs.signalflag.ai`. They stay at `docs.resim.ai`, so page URLs and existing links keep working. - **CLI.** The `resim` command keeps working, so existing scripts and CI pipelines need no changes. We plan to also publish the CLI under the `signalflag` name. When we do, we will post the install instructions here, and `resim` will keep working alongside it. - **Python packages.** The SDK is now on PyPI under the SignalFlag name: `pip install signalflag`. It is the same package, so installing it over `resim-sdk` is the whole upgrade. The `resim.*` import paths still resolve to the same modules and the `resim-demo` command is still installed alongside `signalflag-demo`, which means code and scripts written against `resim-sdk` run unchanged. It reads `SIGNALFLAG_*` environment variables and caches its login under `~/.signalflag`, falling back to the `RESIM_*` names and `~/.resim`, so existing CI secrets and cached logins keep working. `resim-sdk` also keeps working and remains maintained, so there is no deadline to switch. `resim-open-core` keeps its name for now; a SignalFlag-named metrics package will be listed here when it exists. - **GitHub.** Our repositories, release downloads, and the GitHub Action stay under `github.com/resim-ai`. If we move them, we will announce it in advance. - **API and backend hostnames.** `api.resim.ai` and `bff.resim.ai` stay as they are. There is no `api.signalflag.ai` or `bff.signalflag.ai`, so keep the `resim.ai` hostnames in API URLs, MCP configuration, and CI. If these hostnames change in the future, we will announce it in advance. ## Why the new name? When we launched, the platform orchestrated simulation, and ReSim made sense. Very quickly our customers asked for more: the same evaluations on hardware tests, on experiments run locally, and on production data. The platform has grown into a general set of tools that teams use to gain *signal* on the performance of their robots and to *flag* issues at the earliest opportunity. We are not pivoting. Every current capability remains under active development, and the name change coincides with new features that build on the same infrastructure. ## Questions? Reach out to your usual SignalFlag contact, or email [info@signalflag.ai](mailto:info@signalflag.ai). # Batches and tests In this document, we will cover two fundamental concepts in the SignalFlag infrastructure that enable customers to run a large number of tests and aggregate the results: [tests](../../core-concepts/#test) and [test batches](../../core-concepts/#test-batch). This guide will help you understand what is happening when you are running thousands of tests. ## Understanding batches You can trigger a batch in many ways in the SignalFlag platform, such as: - the SignalFlag CLI tool - SignalFlag App - GitHub Action If you need to run a thousand tests, you will initiate one batch that will encompass all those tasks. The Batch is the unit of work that processes an entire set of experiences. ## Understanding tests Within each batch, individual work items are broken down into tests. When running experiences, Each test corresponds to a specific experience that runs. ## Detailed workflow The following sections will illustrate each of the phases in the overall batch execution process defined here: ### Batch submission When you submit a batch, you define: - **Experiences**: the set of experiences you wish to run against your version of code. - **Build**: Contains your simulator and autonomy code. - **Metrics Build**: Used for analyzing test outputs. For each experience a separate, parallel, test will be created that runs that experience through the build and then computes metrics on the outputs from the experience using a metrics build. ### Test execution and metrics The execution of a test is split into two separate stages: an experience execution stage and a metrics stage: 1. **Experience Execution**: Each test runs a specific experience against that build, generating some outputs. The output of this stage is processed in the following metrics stage. 1. **Metrics Stage**: After execution, the test's output is analyzed using the metrics build. This is where your pass fail signal is configured and computed. ### Batch metrics and aggregation Once all tests within a batch are completed, SignalFlag moves to the batch metrics stage. During this stage, you can define aggregation methods such as weighted averages in whatever way is best suited to your system. ## Conclusion Running thousands of tests efficiently in the cloud is a challenge. Our use of batches and tests helps us organize the work and make it clear what state your test is in while it is running. # Introduction Welcome to our collection of SignalFlagU articles! As the name may suggest, this section of our documentation is intended to cover topics of a more academic interest relative to the rest of our documentation. More plainly, it's a place for us to share simulation-related derivations, demonstrations, and learnings which we feel to be intrinsically interesting and potentially useful for the typical robotics engineer to be aware of. Please enjoy and don't hesitate to reach out with any questions, observations, or errata. We're excited by the opportunity to improve these pages over time. So far, we have the following articles: - [Rigid Body Dynamics on Lie Groups](../lie_group_dynamics/) - [Dual Quaternions for SE(3)](../dual_quaternions_for_se3/) # Rigid body dynamics on Lie groups *October 11, 2024* *Michael Bauer* ## Introduction While one can often get a lot of value from a simple kinematic model of a system (e.g. the bicycle model for wheeled vehicles), there are many cases in robotics where a simulation must adopt an accurate Newtonian dynamics model in order to usefully test the system in question. The bicycle model, for instance, may lose fidelity in extreme maneuvers with high lateral acceleration. This is why many robotics simulation tools (e.g. [Dartsim](https://dartsim.github.io/), [Drake](https://drake.mit.edu/), [Gazebo](https://gazebosim.org/docs)) provide rich rigid body dynamics solvers. As we discuss and motivate in our [open core documentation](https://docs.signalflag.ai/open-core/transforms/liegroups/), SignalFlag uses Lie groups and their derivatives to describe the trajectories of simulated actors through the world. Given our extensive use of Lie groups for kinematic applications ([example](https://github.com/resim-ai/open-core/blob/85bb030069425bc5f446825be29afef2d5ddfcd5/resim/curves/t_curve_segment.hh#L9)), one may wonder if Lie groups can be used for dynamics problems as well. We can describe the pose and velocity of a rigid body using elements of the Lie group (\\text{SE}(3)) and the Lie algebra (\\mathfrak{se}(3)). However, having chosen these coordinates, is it feasible to predict how the system's state will evolve over time under the influence of external forces and torques? The answer is, of course, yes. ## Definitions Lets start by defining some parameters. We begin by describing a rigid body's position and orientation. We do this by attaching a Cartesian coordinate frame to our body. For convenience, we place the origin of this coordinate frame at its [center of mass](https://en.wikipedia.org/wiki/Center_of_mass) and align its axes with the [principal axes of rotation](https://en.wikipedia.org/wiki/Moment_of_inertia#Principal_axes) of our body. This coordinate frame is attached to our rigid body such that every part of the body is always stationary in this coordinate frame. We then associate a group element (g\\in \\text{SE}(3)) with the pose of the rigid body. Specifically, the group action of (g) on a point's coordinates in our body coordinate frame gives that same point's coordinates in an inertial reference frame. For example, applying (g) to ((0, 0, 0)\\in \\mathbb{R}^3) (the origin in body coordinates) gives the position of the rigid body our inertial reference frame. Now, lets define our generalized velocity (\\xi\\in\\mathfrak{se}(3)) as the time derivative of (g) in [right tangent space](https://docs.signalflag.ai/open-core/transforms/liegroup_derivatives/#left-and-right-tangents). Specifically, we define it as the real derivative: \[ \\xi(t) = \\frac{d}{d\\epsilon}\\log\\left[g(t)^{-1}g(t + \\epsilon)\\right] \] Although we often write this in shorthand: [ \\xi(t) = \\frac{dg(t)}{dt} ] See [Lie Group Derivatives](https://docs.signalflag.ai/open-core/transforms/liegroup_derivatives/) from our open core documentation for information on where the full expression comes from. The Chain Rule section of that document will be relevant as well. We choose a basis for our Lie algebra such that the first three entries of (\\xi) correspond to the angular velocity of the body in its own coordinate frame, and the last three entries of (\\xi) correspond to the linear velocity of the body in its own coordinate frame. In other words (\\xi^T=\\begin{bmatrix}\\omega^T & v^T\\end{bmatrix}) where (\\omega\\in\\mathbb{R}^3) is the angular velocity and (v\\in\\mathbb{R}^3) is the linear velocity. Finally, we define our generalized inertia tensor (I\\in\\mathbb{R}^{6\\times 6}) like so: [ \\mathcal{I} = \\begin{bmatrix}I\_{xx} \\ & I\_{yy}\\ & & I\_{zz} \\ & & & m \\ & & & & m \\ & & & & & m\\end{bmatrix} ] Where (I\_{xx}), (I\_{yy}), and (I\_{zz}) are the moments of inertia about the principle axes and (m) is the mass of the body. The inertia tensor has this nice diagonal form due to our choice of body coordinate frame (origin at the center of mass and axes aligned with the principal axes of rotation), but it can in general be a dense symmetric matrix if we choose another coordinate frame attached to our body. The important thing about these definitions is that we can now write the kinetic energy of our rigid body system like so: [ \\begin{aligned}T = \\frac{1}{2} \\xi^T \\mathcal{I} \\xi &= \\frac{1}{2}\\omega^T I_R\\omega + \\frac{1}{2}m||v||^2 \\&= KE_R + KE_T\\end{aligned} ] Where we can identify the terms as the rotational kinetic energy (KE_R) and the translational kinetic energy (KE_T). ## Forward dynamics Using this expression for kinetic energy, we can derive an ordinary differential equation for the forward dynamics of the system. This ordinary differential equation will relate the system's generalized acceleration (\\dot \\xi) to its generalized velocity (\\xi). To do this, we will utilize a variational approach. In particular, we define the Lagrangian without any conservative forces: [ L(g, \\xi, t) = T - V = \\frac{1}{2} \\xi^T \\mathcal{I} \\xi ] and then apply the [Lagrange-D'Alembert Principle](https://en.wikipedia.org/wiki/D%27Alembert%27s_principle#Formulation_using_the_Lagrangian) which states: \[ 0 = \\delta\\int\_{t_1}^{t_2} L(g,\\xi, t)dt - \\int\_{t_1}^{t_2} (F \\cdot \\delta g) dt \] Where (F\\in \\mathfrak{dse}(3)) is the generalized force on our rigid body (\\delta g\\in\\mathfrak{se}(3)) is an arbitrary infinitesimal variation of our system's six degree of freedom state. (\\mathfrak{dse}(3)) is the [*dual space*](https://en.wikipedia.org/wiki/Dual_space) of (\\mathfrak{se}(3)) and is also isomorphic to (\\mathbb{R}^6). One might guess that the first three components of (F) represent the torque on the system in body coordinates and the last three represent the linear force on the system in body coordinates and this is in fact correct. (F\\cdot \\delta g) is the virtual work done by the forces on our system applied along the virtual displacements (\\delta g). Now, let's start by plugging in the definition of (\\xi): \[ 0 = \\delta\\int\_{t_1}^{t_2} \\left[ \\frac{1}{2}\\left( \\frac{dg}{dt}\\right)^T \\mathcal{I} \\left(\\frac{dg}{dt} \\right) \\right]dt - \\int\_{t_1}^{t_2} (F \\cdot \\delta g) dt \] Now, let's take the functional derivative (\\delta) on the first term. Translating this into more familiar terms, we define the derivative like so: \[ 0 = \\frac{d}{d\\epsilon}\\left\[ \\int\_{t_1}^{t_2} \\left[ \\frac{1}{2}\\left( \\frac{dg\_\\epsilon}{dt} \\right)^T \\mathcal{I} \\left( \\frac{dg\_\\epsilon}{dt} \\right) \\right]dt \\right\]\_{\\epsilon = 0} - \\int\_{t_1}^{t_2} (F \\cdot \\delta g) dt \] Where we have substituted in the varied version of (g): [ g\_\\epsilon := g\\exp(\\epsilon \\delta g) ] Using the chain rule in right tangent space, we have: \[ \\frac{dg\_{\\epsilon}}{dt} = \\text{Ad}\_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt} - \\text{Dexp}(\\epsilon \\delta g) \\epsilon \\frac{d\\delta g}{dt} \] Where (\\text{Dexp}) is the differential of the exponential map in right tangent space and (\\text{Ad}\_{\\exp(-\\epsilon \\delta g)}) is the [adjoint representation](https://docs.signalflag.ai/open-core/transforms/liegroup_derivatives/#the-adjoint) of (\\exp(-\\epsilon\\delta g)). Plugging this in and expanding the first term gives: \[ \\begin{aligned} \\frac{1}{2}\\left( \\frac{dg\_\\epsilon}{dt} \\right)^T \\mathcal{I} \\left( \\frac{dg\_\\epsilon}{dt} \\right) =& \\frac{1}{2}\\left( \\text{Ad}_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\text{Ad}_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt} \\right) \\ &+ \\left( \\text{Ad}\_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\text{Dexp}(\\epsilon \\delta g) \\epsilon \\frac{d\\delta g}{dt} \\right) + O(\\epsilon^2) \\end{aligned} \] Differentiating this with respect to (\\epsilon) and setting (\\epsilon\\rightarrow0) we get: \[ \\frac{d}{d\\epsilon}\\left[ \\frac{1}{2}\\left( \\frac{dg\_\\epsilon}{dt} \\right)^T \\mathcal{I} \\left( \\frac{dg\_\\epsilon}{dt} \\right) \\right]_{\\epsilon=0} = -\\left( \\text{ad}_{ \\delta g} \\frac{dg}{dt} \\right)^T \\mathcal{I} \\frac{dg}{dt} + \\left( \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\frac{d \\delta g}{dt} \\right) \] This step merits some explanation. Looking at the first term, we recognize it as a quadratic form and compute the derivative of (\\left(\\text{Ad}\_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt}\\right)) with respect to (\\epsilon) first: # \[ \\frac{d}{d\\epsilon}\\left( \\text{Ad}\_{\\exp(-\\epsilon \\delta g)} \\frac{dg}{dt} \\right) -\\text{Ad}_{\\exp(-\\epsilon \\delta g)} \\text{ad}_{\\delta g}\\frac{dg}{dt} \] Then (\\text{Ad}\_{\\exp(-\\epsilon \\delta g)}=\\text{Ad}\_e = I_6) when (\\epsilon\\rightarrow 0) so this goes away. (\\text{ad}) is the adjoint representation of the Lie *algebra* which we get because we differentiated the group adjoint representation (\\text{Ad}). In the second term, we see that the whole term is multiplied by (\\epsilon) to begin with, so only one term resulting from the product rule will survive after (\\epsilon\\rightarrow 0). We then use the fact that the adjoints go to the identity and (\\text{Dexp}(0)=\\text{Identity}) also to arrive at the final expression above. Finally, one more manipulation is useful. Using the fact that (\\text{ad}\_uv = -\\text{ad}\_vu) we have: \[ \\frac{d}{d\\epsilon}\\left[ \\frac{1}{2}\\left( \\frac{dg\_\\epsilon}{dt} \\right)^T \\mathcal{I} \\left( \\frac{dg\_\\epsilon}{dt} \\right) \\right]_{\\epsilon=0} = \\left( \\text{ad}_{\\frac{dg}{dt}}\\delta g \\right)^T \\mathcal{I} \\frac{dg}{dt} + \\left( \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\frac{d \\delta g}{dt} \\right) \] Plugging this into the integral and grouping terms yields: \[ \\begin{aligned} 0 &= \\int\_{t_1}^{t_2} \\left\[ \\left( \\text{ad}_{\\frac{dg}{dt}}\\delta g \\right)^T \\mathcal{I} \\frac{dg}{dt} + \\left( \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\frac{d \\delta g}{dt} \\right) \\right\]dt + \\int_{t_1}^{t_2} (F \\cdot \\delta g) dt \\ 0&= \\int\_{t_1}^{t_2} \\left[ \\left( \\frac{dg}{dt} \\right)^T \\mathcal{I} \\left( \\frac{d \\delta g}{dt} \\right) \\right]dt + \\int\_{t_1}^{t_2} \\left[ \\left( F + \\text{ad}\_\\frac{dg}{dt}^T\\mathcal{I}\\frac{dg}{dt} \\right)\\cdot \\delta g \\right] dt \\end{aligned} \] Integrating the first term by parts, we get: \[ \\begin{aligned} 0 &= -\\int\_{t_1}^{t_2} \\left\[ \\left( \\frac{d}{dt}\\left[ \\mathcal{I}\\frac{dg}{dt} \\right] \\right)\\cdot \\delta g \\right\]dt + \\int\_{t_1}^{t_2} \\left\[ \\left( F + \\text{ad}_\\frac{dg}{dt}^T\\mathcal{I}\\frac{dg}{dt} \\right)\\cdot \\delta g \\right\] dt \\ 0 &= \\int_{t_1}^{t_2} \\left\[ \\left( F + \\text{ad}\_\\frac{dg}{dt}^T\\mathcal{I}\\frac{dg}{dt} - \\frac{d}{dt}\\left[ \\mathcal{I}\\frac{dg}{dt} \\right] \\right)\\cdot \\delta g \\right\] dt \\end{aligned} \] Note that the boundary terms are zero when integrating by parts since (\\delta g) is defined to vanish at (t_1) and (t_2). Using the fundamental theorem of variational calculus, we note that, because this must be true for any variations (\\delta g), we have: \[ \\begin{aligned} 0 &= F + \\text{ad}_\\frac{dg}{dt}^T\\mathcal{I}\\frac{dg}{dt} - \\frac{d}{dt}\\left[ \\mathcal{I}\\frac{dg}{dt} \\right] \\ 0&= F + \\text{ad}_\\xi^T\\mathcal{I}\\xi - \\mathcal{I} \\dot \\xi \\end{aligned} \] So we have our forward dynamics: [ \\boxed{\\mathcal{I}\\dot \\xi = F + \\text{ad}\_\\xi^T\\mathcal{I}\\xi} ] As a sanity check let's look at what this expression looks like on (\\text{SE}(3)). For this group we can express elements of the algebra in a basis such that: # \[ \\xi \\begin{bmatrix} \\omega \\ v \\end{bmatrix} \] Where (\\omega) and (v) are the angular and linear velocity expressed in body coordinates. We know: # \[ \\text{ad}\_\\xi \\begin{bmatrix} \\omega\_\\times & \\ v\_\\times & \\omega\_\\times \\end{bmatrix} \] Assuming that our inertia tensor (I) has the diagonal form shown above, plugging these in yields: # \[ \\begin{bmatrix} I\_\\text{rot} \\ & m \\end{bmatrix} \\begin{bmatrix} \\dot \\omega \\ \\dot v \\end{bmatrix} ## \\begin{bmatrix} \\tau \\ f \\end{bmatrix} \\begin{bmatrix} \\omega\_\\times & v\_\\times\\ & \\omega\_\\times \\end{bmatrix} \\begin{bmatrix} I\_\\text{rot} \\ & m \\end{bmatrix} \\begin{bmatrix} \\omega \\ v \\end{bmatrix} \] [ \\begin{aligned} I\_\\text{rot} \\dot \\omega &= \\tau - \\omega \\times I\_\\text{rot}\\omega \\ m\\dot v &= f - m\\omega\\times v \\end{aligned} ] The first equation is identifiable as the vector form of [Euler's equations]() for rotation, which is exactly what we expect! Now, if we look at the second equation we have for the linear velocity in body-attached coordinates: [ m\\dot v = f - m\\omega\\times v ] Now, we make a substitution into inertial coordinates. The velocity in inertial coordinates is related to the body-attached velocity by the rotation: (u=Rv). From this, we can also derive: [ \\begin{aligned} \\dot u &= \\frac{d}{dt}(Rv) = R(\\omega \\times v) + R\\dot v \\ \\dot v &= R^T\\dot u - \\omega\\times v \\end{aligned} ] Substituting this into our translational dynamics (along with transforming the force into inertial coordinates (\\varphi=Rf)) we get: [ \\begin{aligned} mR^T\\dot u - m\\omega\\times v &= R^T\\varphi - m\\omega \\times v \\ mR^T\\dot u &= R^T\\varphi \\ m\\dot u &= \\varphi \\end{aligned} ] In other terms, (F = ma), which is Newton's Second Law. Therefore, we see that this single equation is in fact equivalent to the combination of Newton's Second Law and Euler's Equations of rigid body rotation. The expression (I\\dot \\xi = F + \\text{ad}_\\xi^TI\\xi) is actually a generalization of both of these equations and is a more general statement of Newton's Second Law. By choosing (\\mathbb{R}^3) under addition as a Lie group, one can derive (F=ma) immediately, and by choosing (\\text{SO}(3)) as the Lie group one can derive (I_\\text{rot} \\dot \\omega = \\tau - \\omega \\times I\_\\text{rot}\\omega) immediately. ## Simulating We can then express our system's equations of motion like so: [ \\begin{aligned} \\frac{dg}{dt}&=\\xi \\ \\frac{d\\xi}{dt} &= \\mathcal{I}^{-1}(F + \\text{ad}\_\\xi^T \\mathcal{I}\\xi) \\end{aligned} ] This can be integrated using Forward Euler integration by using the Lie group exponential the (\\xi\\Delta t) step to (g): [ \\begin{aligned} g(t\_{i+1})&=g(t_i)\\exp(\\xi_i \\Delta t) \\ \\xi(t\_{i+1})&=\\xi(t_i) + \\mathcal{I}^{-1}(F + \\text{ad}\_{\\xi_i}^T \\mathcal{I})\\Delta t \\end{aligned} ] This will give first order accuracy as one would expect from Forward Euler. In the case of (\\text{SE}(3)), it happens to be the case that integrating (\\xi) in the above manner is not stable, although it's usually not a big deal if small enough timestamps are used and appropriate dissipative forces are present (e.g. friction or air resistance). Higher-order integration schemes (e.g. Ralston's Method, RK4) can also be employed to mitigate this, although we will leave the exploration of such integrators for another time as some care should be taken in applying them to Lie groups in general. ## Handling gravity Gravity is a very common force that's relevant when modeling rigid body dynamics, so it's worth taking a quick look on how to handle it. Due to our choice of body coordinates (i.e. coordinates whose origin is the center of mass), this is relatively doable. We know that the force of gravity is a linear force applied to the center of mass. In *reference* coordinates: [ f\_{g,\\text{ref}} = \\begin{bmatrix} 0 \\ 0 \\ -9.8\\text{m/s}^2 \\cdot m \\ \\end{bmatrix} ] However, to use this force in the equation above, we need the rotation (R) associated with (g). In particular (R) can be defined as the rotation that satisfies (Rx=gx-g\\begin{bmatrix} 0 & 0 & 0\\end{bmatrix}^T). Then we can write the generalized force due to gravity as: [ F= \\begin{bmatrix} 0 \\ 0 \\ 0 \\ R^{-1}\\begin{bmatrix} 0 \\ 0 \\ -9.8\\text{m/s}^2\\cdot m \\ \\end{bmatrix} \\end{bmatrix} ] Where (R^T = R^{-1}) can also be used if a matrix representation is employed for (R). Note that the first three entries are zero since gravity does not apply a torque to the center of mass. The final three entries are simply the gravitational force rotated into the coordinates of the body. ## Bringing it all together As a quick illustration of this formulation of rigid body dynamics, here's a quick sim that you can use to explore how such dynamics behave. We're using forward Euler to simulate the rotational degrees of freedom of the above equations of motion, and we've tweaked the translational dynamics integration to enhance stability. #### Controls - W/S: Pitch - A/D: Yaw - Q/E: Roll - G: Thrust - Space: Reset - T: Hold to point the engine downwards - Arrow Keys: Camera Acknowledgement This work is heavily inspired by [Lie Group Formulation of Articulated Rigid Body Dynamics](https://www.cs.cmu.edu/~junggon/tools/liegroupdynamics.pdf) by Junggon Kim. I highly recommend this work as a particularly lucid exposition of the topic. # Dual quaternions for SE(3) *February 25, 2025* *Michael Bauer* ## Introduction At SignalFlag we use Lie groups for a [variety of reasons](https://docs.signalflag.ai/open-core/transforms/liegroups/). One good reason for choosing to express our robotics algorithms and libraries in terms of Lie groups is that they are an abstraction for rigid body transformations, rotations, and other relevant transformations, meaning we can write common algorithms to solve a variety of problems. This, of course, is very appealing to software engineers, who always love a good abstraction. As an example, the [Sophus](https://github.com/strasdat/Sophus) library includes a single header which provides a template implementation of `ceres::Manifold` for any Lie group defined in a similar manner to the ones implemented in that library. For those unfamiliar with CERES, the upshot of this is that Lie-group-valued parameters (e.g. camera extrinsics) can be easily optimized based on logged robotics data if those parameters are expressed using Sophus's library. This abstraction goes even deeper: even for a particular Lie group such as (\\text{SO}(3)) or (\\text{SE}(3)), we have multiple choices of how to implement it. You may be familiar with how unit quaternions can be used to represent 3D orientation (if not I highly recomment [this video](https://www.youtube.com/watch?v=d4EgbgTm0Bg) and [this follow-up video](https://www.youtube.com/watch?v=zjMuIxRvygQ) from 3Blue1Brown). Unit Quaternions are one possible way of representing (\\text{SO}(3)) and they have some advantages, including the ability to compose transforms rapidly, compared to the matrix representation of (\\text{SO}(3)) (i.e. rotation matrices). However, rotation matrices are able to act on (i.e. rotate) points/vectors in fewer floating-point operations (FLOPs) than unit quaternions. The point is that one may have software design reasons for preferring one implementation of (\\text{SO}(3)) over another in certain contexts, and this is true for other groups as well. In this post, we explore an alternative way to represent (\\text{SE}(3)) using unit dual quaternions, which is different from our approach in the [open-core](https://github.com/resim-ai/open-core/blob/main/resim/transforms/se3.hh) repository. Rather than rigorously proving every claim or covering all possible operations, we aim to provide a high-level overview. We'll discuss how to compose transformations, express algebra elements as dual quaternions, and derive the exponential map. Our goal is to build intuition that may be useful for other representations of (\\text{SE}(3)) and beyond. ## Quaternions [Quaternions](https://en.wikipedia.org/wiki/Quaternion) are a number system introduced by William Hamilton that can be seen as an extension of the complex numbers. They have gained popularity in 3D graphics in the 21st Century due to their ability to store orientations efficiently (with just 4 floating point values) and their ability to be composed efficiently (16 multiplications vs 27 for composing rotation matrices), although they do have some disadvantages such as their inability to uniquely represent a given orientation (there are exactly two unit quaternions that represent a given orientation) and their lower efficiency at transforming points (15 multiplications vs 9 for rotation matrices). Here’s how they work in a nutshell. Analogous to how complex numbers are built off the definition of a constant (i) such that (i^2=-1), quaternions are defined based on three such values: (i), (j), and (k) such that (i^2=j^2=k^2=ijk=-1). From these rules, one can also infer: [ \\begin{aligned}jk&=i&kj&=-i\\ki&=j&ik&=-j\\ ij&=k&ji&=-k\\end{aligned} ] Note how these values *do not commute*. Similar to complex numbers, quaternions also have a real part, so in general, the set of all quaternions can be written as: [ \\mathbb{Q}=\\left{w+xi+yj+zk|(w,x,y,z)\\in\\mathbb{R}^4\\right} ] Where (w) is the real part of the quaternion. As a notational matter, we often refer to the non-real part of the quaternion (xi+yj+zk) as the “vector part” and write the whole quaternion as a tuple of it’s real part and vector part: \[ q=w+xi+yj+zk=(w,\\vec \\nu),\\quad\\text{where},,,\\vec \\nu=[x,y,z]^T \] Doing so allows us to express multiplication between two quaternions according to the above rules like so: [ q_1 q_2=(w_1,\\vec \\nu_1)(w_2,\\vec \\nu_2)=(w_1 w_2 - \\vec \\nu_1\\cdot \\vec \\nu_2, w_1\\vec \\nu_2 + w_2 \\vec \\nu_1 + \\vec \\nu_1 \\times \\vec \\nu_2) ] This product is referred to as the "Hamilton product". Verifying this formula based on the above rules is left as an exercise to the reader. From this expression, we can also note the fact that quaternion multiplication is **not** commutative due to the fact that (i), (j), and (k) do not commute. Like complex numbers, quaternions also have the notion of a *conjugate* defined such that: [ q^\* = (w,\\nu)^\*=(w,-\\nu) ] And the concept of a norm (a.k.a. modulus) defined by: [ |q|=\\sqrt{qq^\*}=\\sqrt{q^\*q} ] And the concept of a multiplicative inverse (q^{-1}) such that: [ qq^{-1}=q^{-1}q=1 ] Unit quaternions (i.e. those for which (|q| = 1)) can be used to [represent 3D rotations](https://en.wikipedia.org/wiki/Quaternions_and_spatial_rotation) as elucidated by the aforementioned videos. ## Dual numbers Just like quaternions, the [Dual Number](https://en.wikipedia.org/wiki/Dual_number) system is another number system that is similar to the complex numbers. Instead of defining a constant (i) such that (i^2=-1), dual numbers define a constant (\\varepsilon) such that (\\varepsilon^2 = 0). In other words, dual numbers look like (a + \\varepsilon b) for (a,b\\in\\mathbb{R}) and muliplying them works like so: [ (a + \\varepsilon b)(c + \\varepsilon d) = ac + \\varepsilon (bc + ad) ] Where the (\\varepsilon^2) drops out since it's zero. Doing this becomes useful for numerous applications including [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation#Automatic_differentiation_using_dual_numbers) where the rules of dual multiplication effectively enforce the product rule. This is not actually the way we leverage dual numbers when representing (\\text{SE}(3)), however. To see how we do, we'll have to take a deeper look at dual quaternions. ## Dual quaternions Even more interesting things begin to happen when we combine quaternions and dual numbers together. We do this by imagining a dual number whose real and dual parts (q) and (r) are themselves quaternions. [ Q = q + \\varepsilon r =(s,u)+\\varepsilon (t, \\nu) ] Multiplying such objects together works as one would expect. That is to say you multiply them using the distributive property and apply the rules outlined above regarding the quaternion constants (i), (j), and (k) and the dual constant (\\varepsilon). Importantly, (\\varepsilon) is allowed to commute with (i), (j), and (k) even though they don’t commute with each other. This ends up looking like so: [ \\begin{aligned}Q_1 Q_2 &= (q_1 + \\varepsilon r_1)(q_2 + \\varepsilon r_2) \\ &= q_1q_2 + \\varepsilon(r_1q_2 + q_1 r_2)\\end{aligned} ] Define the conjugate (Q^\*) such that: [ Q^\* = (s, -u) + \\varepsilon (t, -\\nu) ] And define the norm of a dual quaternion like so: \[ \\begin{aligned} ||Q||^2 &= Q Q^\* \\ &= \\left[(s, u) + \\varepsilon(t, \\nu)\\right]\\left[(s, -u) + \\varepsilon(t, -\\nu)\\right] \\ &= s^2 + u\\cdot u + 2\\varepsilon(ts + \\nu\\cdot u) \\end{aligned} \] With this definition we define unit dual quaternions as those with unit norm: [ ||Q||^2 = s^2 + u\\cdot u + 2\\varepsilon(ts + \\nu\\cdot u) = 1 ] If we define (q_r := (s, u)) as our real (i.e. non-dual) part, this implies that unit dual quaternions must have (|q_r| = 1). We also must satisfy (ts + \\nu\\cdot u = 0), and it turns out we can guarantee this by re-writing our dual part as ((t, \\nu) = (0, d)q_r) (with (d\\in\\mathbb{R}^3)), where we can compute (d) by using the fact that ((0, d) = (t, \\nu)q_r^\*). Therefore, we can write any unit dual quaternion in the form: [ Q = q_r + \\epsilon (0, d)q_r ] For some unit quaternion (q_r) and some three vector (d). We claim that the set of *unit dual quaternions* under the above product is a group. Notably it has the following properties: 1. It is closed under multiplication. One can relatively easily show that for dual quaternions (Q) and (U), ((QU)^\* = U^\*Q^\*). From here, it's easy to see that the norm of a product is a product of norms for dual quaternions by directly substituting (QU) into the definition of the norm. This implies closure for the set of unit dual quaternions under multiplication. 1. The product operation is associative. This follows from the associativity of the Hamilton and dual products, although we don't explicitly demonstrate this here. 1. Every unit dual quaternion has an inverse element, given by (Q^{-1} = Q^\*) and an identity element given by (1) such that (Q\\cdot 1 = Q). By definition, we see that (Q Q^{\*}=1) and since (Q = \\left(Q^\*\\right)^\*), we also know (Q^\*Q = 1) The unit dual quaternions are also a differentiable manifold. A rough argument for this could use the fact that we have a differentiable mapping (f) from (\\mathbb{Q}\_1\\times \\mathbb{R}^3) (where (\\mathbb{Q}\_1) is the set of unit quaternions) to the unit dual quaternions (\\mathbb{D}\_1). This mapping is given by (f(q_r, d) = q_r + \\varepsilon(0, d)q_r). Since (\\mathbb{Q}\_1) and (\\mathbb{R}^3) are differentiable manifolds, you can use their charts to define appropriate charts on (\\mathbb{D}\_1) to prove that it is as well. Since (\\mathbb{D}\_1) is a group and a differentiable manifold, we can say that it is a Lie group. ## Tangent vectors Let (\\mathbb{D}\_1) be the set of unit dual quaternions. Now, let's assume that we have a curve (\\gamma: \\mathbb{R} \\rightarrow \\mathbb{D}\_1) that has (\\gamma(0) = 1). Describe this curve by its components like so: [ \\gamma(\\tau) = (s(\\tau), u(\\tau)) + \\varepsilon(t(\\tau), \\nu(\\tau)) ] Where (s, t: \\mathbb{R}\\rightarrow\\mathbb{R}) are real curves and (u, \\nu: \\mathbb{R}\\rightarrow\\mathbb{R}^3) are vector curves. We know that the curve satisfies the unit constraint: \[ 1 = \\left[(s, u) + \\varepsilon(t, \\nu)\\right]\\left[(s, -u) + \\varepsilon(t, -\\nu)\\right] \] Differentiating this, we see: \[ 0 = \\left[(s', u') + \\varepsilon(t', \\nu')\\right]\\left[(s, -u) + \\varepsilon(t, -\\nu)\\right] - \\left[(s, u) + \\varepsilon(t, \\nu)\\right]\\left[(s', -u') + \\varepsilon(t', -\\nu')\\right] \] Evaluating this at the identity where ((s, u, t, \\nu) = (1, 0, 0, 0)) we see: \[ 0 = \\left[(s', u') + \\varepsilon(t', \\nu')\\right] + \\left[(s', -u') + \\varepsilon(t', -\\nu')\\right] \] By inspection of this expression, we can see that it is satisfied whenver (s' = t' = 0). These two constraints reduce the eight degrees of freedom inherent in the dual quaternion number system to six, which intuitively matches the number of degrees of freedom of (\\text{SE}(3)). We can therefore write our tangent vectors at the identity (i.e. Lie algebra elements) like so: [ \\gamma'(0) = \\frac{1}{2}(0, \\omega) + \\frac{1}{2}\\varepsilon(0, v) ] where (v) is unrelated to the (\\nu) used above. We've used (\\omega) and (v) here as these will end up corresponding to angular and linear velocities. The reason for the factor of (1/2) will be elucidated later. ## Correspondence to SE(3) So far, we've claimed that unit dual quaternions are a Lie group and shown what the derivatives at the identity should look like. We also claim that unit dual quaternions are related to (\\text{SE}(3)). To explore this, we first designate the tangent space at the identity as (T\\mathbb{D}\_1). The elements of this space are dual quaternions of the form ((1/2)(0, \\omega) + (1/2)\\varepsilon (0, v)). We claim that this space is isomorphic to (\\mathfrak{se}(3)) when equipped with an appropriate bracket operation. To do this, we define the bracket very simply like so. For [ \\begin{aligned} k_1 &= (1/2)(0, \\omega_1) + (1/2)\\varepsilon (0, v_1) \\ k_2 &= (1/2)(0, \\omega_2) + (1/2)\\varepsilon (0, v_2) \\end{aligned} ] Define the bracket as the commutator: \[ [k_1, k_2] = k_1 k_2 - k_2 k_1 \] One can relatively easily verify that this satisfies, closure, bilinearity, the alternating property, and the Jacobi identity, so it is a valid choice. This can be seen when one works out the algebra: \[ [k_1, k_2] = \\frac{1}{2}(0, \\omega_1 \\times \\omega_2) + \\frac{1}{2}\\varepsilon(0, \\omega_1 \\times v_2 - \\omega_2 \\times v_1) \] Note that the Lie algebra is not closed under the Hamilton/dual product, but is under the bracket. If we define (k_3) by (k_3 \\equiv [k_1, k_2]) and define (\\omega_3) and (v_3) as its angular and linear velocity respectively, then we can read from the above expression: [ \\begin{aligned} \\omega_3 &= \\omega_1 \\times \\omega_2 \\ v_3 &= \\omega_1 \\times v_2 - \\omega_2 \\times v_1 \\end{aligned} ] If we look at the common matrix form for (\\mathfrak{se}(3)), its members are usually expressed as matrices of the following form: [ X = \\begin{bmatrix} \\omega\_{\\times} & v \\ 0 & 0 \\end{bmatrix} ] And if one computes the bracket for (\\mathfrak{se}(3)) they'll get: \[ [X_1, X_2] = \\begin{bmatrix} (\\omega_1 \\times \\omega_2)\_\\times & [\\omega_1 \\times v_2 - \\omega_2 \\times v_1] \\ 0 & 0 \\end{bmatrix} \] By comparing this with the bracket we defined on (T\\mathbb{D}\_1), we see that both algebras have the same structure (i.e. you could compute the [structure constants](https://en.wikipedia.org/wiki/Structure_constants) for each and they would be identical). Note that the choice of ((1/2)) prefactors in the form of our tangent vectors allows the structure constants to match (rather than differing by a constant factor) and for us to interpret the vector components of our dual quaternion tangents as angular velocity and linear velocity directly. This all means there is a linear isomorphism (\\phi) that maps (\\mathfrak{se}(3)) to (T\\mathbb{D}\_1). The upshot of this is that we can apply the Lie group [homomorphisms theorem](https://en.wikipedia.org/wiki/Lie_group%E2%80%93Lie_algebra_correspondence#The_correspondence). Lie Homomorphisms Theorem (Hall Theorem 5.6) If (\\phi: Lie(G) \\rightarrow Lie(H)) is a Lie algebra homomorphism and if (G) is simply connected, then there exists a (unique) Lie group homomorphism (f: G\\rightarrow H) such that (\\phi = df). The proof for this is quite fascinating and I recommend having a look at *Lie Groups, Lie Algebras, and Representations: An Elementary Introduction* by Brian Hall. The Baker-Campbell-Hausdorff formula is used to establish a local homomorphism near the identity which is extended to the entirety of the domain employing the fact that it's simply connected. It turns out that the set of unit dual quaternions (\\mathbb{D}\_1) is simply connected. As we sketched above, the Lie algebra of (\\mathbb{D}\_1) is homomorphic to the Lie algebra of (\\text{SE}(3)), therefore there exists a homomorphism (\\phi: \\mathbb{D}\_1\\rightarrow \\text{SE}(3)). Recall the defining property of homomorphisms. For (Q_1, Q_2\\in\\mathbb{D}\_1): [ \\phi(Q_1 Q_2) = \\phi(Q_1)\\phi(Q_2) ] This means that we can work with dual quaternions in our code, composing them and such as we see fit, while always getting the same result we would have gotten using elements of (\\text{SE}(3)) Warning We do have to be careful when taking the unit dual quaternion logarithm as its codomain is larger than the (\\text{SE}(3)) logarithm. This could result in the logarithm giving us values that the (\\text{SE}(3)) logarithm could not. In practice, we just define it so it's equivalent to ((\\log \\circ, \\phi)). On a related note, it turns out that (\\text{SE}(3)) is *not* simply connected and this homomorphism is not invertible (i.e. we can't apply the theorem in the opposite direction). The reason for this is that, as we'll see below, (Q\\in \\mathbb{D}\_1) and (-Q) actually map to the same element of (\\text{SE}(3)) by (\\phi). To get a sense for what (\\phi) looks like, lets compute the quaternion exponential for tangent vectors of the above form. To do this, we simply plug them into the power series definition of the exponential and grind out the algebra (see the appendix). The expression you end up with looks like this: \[ \\begin{array}{rcl}X&:=&(0,\\omega/2)+\\varepsilon(0,v/2) \\ \\omega' &:=& \\omega/2 \\ v' &:=& v/2 \\ \\theta' &:=& ||\\omega'|| = ||\\omega||/2 \\ a\_\\theta &:=& \\frac{\\sin\\theta}{\\theta} \\ b\_\\theta &:=& \\frac{\\cos\\theta - a(\\theta)}{\\theta^2} \\ \\exp(X)&=& \\left(\\cos\\theta',\\omega'a\_{\\theta'}\\right) + \\varepsilon\\left[-(\\omega'\\cdot v')a\_{\\theta'}, v'a\_{\\theta'}+\\omega'(w'\\cdot v')b\_{\\theta'}\\right]\\end{array} \] If we plug a pure rotation ((v = 0)) into this, we get: [ Q_R = \\left(\\cos \\tfrac{\\theta}{2}, \\tfrac{\\omega}{\\theta}\\sin\\tfrac{\\theta}{2}\\right):= q_r ] Note that there's no dual part. You may recognize this as a unit quaternion that one might use for three dimensional rotation, which is very encouraging, and we define this as the non-dual quaternion (q_r) for the sake of brevity. Now if we plug in a pure translation ((\\omega = 0)) of (t = v \\cdot 1), we get: [ Q_t = 1 + \\frac{1}{2}\\varepsilon (0, t) ] Typically, we like to think of an (\\text{SE}(3)) as a translation composed with a rotation: [ Q = Q_tQ_R = (1 + \\varepsilon(0, t/2))q_r = q_r + \\varepsilon (0, t/2)q_r ] This expression tells us that we can get the rotation quaternion from our dual quaternion just by taking the real (i.e. non-dual) part of it. Then we can get the translation by taking the dual part, multiplying it by two, and right multiplying by (q_r^{-1} = q_r^\*). Quite encouragingly, it also matches the form we derived for unit dual quaternions above. Feel free to use the following calculator to familiarize yourself with the behavior of the dual quaternion exponential. It shows how the corresponding (\\text{SE}(3)) transform relates two frames. ## Seeing double Those familiar with (\\text{SO}(3)) may know that its exponential has the following property: \[ \\exp(\\omega) = \\exp\\left[\\frac{\\omega}{||\\omega||}(||\\omega|| + 2\\pi n )\\right] \\quad\\forall n\\in\\mathbb{Z} \] This is just a fancy way of stating that if you keep rotating around the same axis, you'll periodically end up back at the same orientation. Conventionally this multivaluedness leads to a branch cut at (||\\omega|| = \\pi) when defining the logarithm so that the bijective region of (\\exp)/(\\log) is (||\\omega|| < \\pi) However, if we plug such tangent vectors into our expression for (q_r), we'll see: [ \\begin{aligned} q\_{rn} &= \\left(\\cos\\tfrac{\\theta + 2\\pi n}{2}, \\tfrac{\\omega}{||\\omega||} \\sin\\tfrac{\\theta + 2\\pi n}{2}\\right) \\ &= \\left(\\cos\\left(\\tfrac{\\theta}{2} + \\pi n\\right), \\tfrac{\\omega}{||\\omega||} \\sin\\left(\\tfrac{\\theta}{2} + \\pi n\\right)\\right) \\ &= \\left{ \\begin{array}{rl} q_r & n ,,\\text{ even} \\ -q_r & n ,,\\text{ odd}\\end{array}\\right. \\end{aligned} ] What this tells us is that (q_r) and (-q_r) represent exactly the same element of (\\text{SO}(3)) making the unit quaternions a double cover of (\\text{SO}(3)). Since a factor of (q_r) appears in both terms of our expression above for (Q), it happens to be the case that (Q) and (-Q) represent exactly the same element of (\\text{SE}(3)) making the unit dual quaternions a double cover of (\\text{SE}(3)). This shows that there isn't an inverse of our homomorphism (\\phi) because there are two elements in (\\mathbb{D}\_1) which (\\phi) maps to each element of (\\text{SE}(3)) so (\\phi) is not injective. ## Conclusion In summary, dual quaternions with unit norm are a Lie group and a double cover of (\\text{SE}(3)) so we can use them to represent 3D rigid body transformations in our robotics software. In certain cases, this may be advantagous to save space or computation time. We showed that (\\mathbb{D}\_1) and (\\text{SE}(3)) share the same Lie algebra so we don't need to change our handling of tangent vectors in any way. In fact the dynamics simulation from our [prior SignalFlag U post](../lie_group_dynamics/) was formulated using unit dual quaternions. The only reason why these Lie groups aren't wholly isomorphic is due to the fact that (\\text{SE}(3)) is not simply connected whereas the unit dual quaternions are. This relationship is in many ways analogous to the relationshp between (\\text{SO}(3)) and the ordinary unit quaternions. ## Appendix: Dual quaternion exponential derivation To keep this from being more painful than it has to, we'll remove the factor of (2) from (\\omega) and (v), only re-introducing it in the final expressions. \[ \\begin{aligned}\\exp(X) &= \\sum\_{n=0}^\\infty \\frac{1}{n!} [(0, \\omega) + \\varepsilon(0, v)]^n \\&=\\sum\_{n=0}^\\infty \\frac{1}{n!}(0,\\omega)^n+\\varepsilon \\left((0, v) + \\frac{1}{2}((0, \\omega)(0, v) + (0,v)(0, \\omega)) + \\cdots\\right)\\end{aligned} \] [ \\begin{aligned}\\exp(X) &=\\sum\_{n=0}^\\infty \\frac{1}{n!}(0,\\omega)^n + \\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(n+k+1)!}(0,\\omega)^n(0,v)(0,\\omega)^k\\end{aligned} ] Let’s start by tackling the first term: [ \\begin{aligned}\\exp(\\omega) &= \\sum\_{n=0}^\\infty\\frac{1}{n!}(0,\\omega)^n\\ (0,\\omega)^0 &= (1,0) \\ (0, \\omega)^1 &= (0, \\omega) \\ (0, \\omega)^2 &= -||\\omega||^2(1, 0) \\\\end{aligned} ] So: [ \\begin{aligned}\\exp(\\omega) &= \\left(\\sum\_{n=0}^\\infty \\frac{(-1)^n}{(2n)!}||\\omega||^{2n},\\frac{\\omega }{||\\omega||}\\sum\_{n=0}^\\infty \\frac{(-1)^n}{(2n+1)!}||\\omega||^{2n+1}\\right) \\ &=(\\cos\\tfrac{\\theta}{2}, n\\sin\\tfrac \\theta 2)\\end{aligned} ] Where (n = \\omega/||\\omega||) and we’ve re-introduced the factor of (1/2) we removed from (\\omega) at the beginning of this derivation to show that we get exactly the form we expect for the rotation part (q_r=(\\cos \\tfrac \\theta 2,n\\sin \\tfrac \\theta 2)) of the dual quaternion. Now we look at the translation part: [ \\begin{aligned}T :=& \\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(n+k+1)!}(0,\\omega)^{n}(0,v)(0,\\omega)^{k} \\ =&\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+1)!}(0,\\omega)^{2n}(0,v)(0,\\omega)^{2k} \\ &+\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+2)!}(0,\\omega)^{2n + 1}(0,v)(0,\\omega)^{2k} \\ &+\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+2)!}(0,\\omega)^{2n}(0,v)(0,\\omega)^{2k+1} \\ &+ \\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+3)!}(0,\\omega)^{2n+1}(0,v)(0,\\omega)^{2k+1} \\ =&\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+1)!}(0,v)(0,\\omega)^{2(n+k)} \\ &+\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+2)!}(0,\\omega)(0,v)(0,\\omega)^{2(n+k)} \\ &+\\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+2)!}(0,v)(0,\\omega)^{2(n+k)+1} \\ &+ \\sum\_{n=0}^\\infty\\sum\_{k=0}^\\infty \\frac{1}{(2(n+k)+3)!}(0,\\omega)(0,v)(0,\\omega)^{2(n + k)+1} \\ =&\\sum\_{n=0}^\\infty\\frac{n+1}{(2n+1)!}(0,v)(0,\\omega)^{2n} \\ &+\\sum\_{n=0}^\\infty\\frac{n+1}{(2n+2)!}(0,\\omega)(0,v)(0,\\omega)^{2n} \\ &+\\sum\_{n=0}^\\infty \\frac{n+1}{(2n+2)!}(0,v)(0,\\omega)^{2n+1} \\ &+ \\sum\_{n=0}^\\infty\\frac{n+1}{(2n+3)!}(0,\\omega)(0,v)(0,\\omega)^{2n+1}\\end{aligned} ] Where we’ve used the fact that ((0,\\omega)^{2n}) is real and hence commutes with ((0,v)). Now break this down term-by-term: [ \\begin{aligned}\\text{I} :=&\\sum\_{n=0}^\\infty\\frac{n+1}{(2n+1)!}(0,v)(0,\\omega)^{2n} \\=& \\frac{1}{2}\\sum\_{n=0}^\\infty\\frac{1}{(2n)!}(0,v)(0,\\omega)^{2n} + \\frac{1}{2}\\sum\_{n=0}^\\infty\\frac{1}{(2n + 1)!}(0,v)(0,\\omega)^{2n} \\ =& \\frac{1}{2}(0,v)\\cosh(0,\\omega) - \\frac{1}{2||\\omega||^2}(0,v)(0,\\omega)\\sinh(0,\\omega) \\ =& \\frac 1 2 (0, v) \\cos||\\omega|| - \\frac 1 {2||\\omega||^3}(0,v)(0,\\omega)(0,\\omega)\\sin||\\omega|| \\ =&\\frac 1 2 (0, v) \\cos||\\omega|| + \\frac 1 {2||\\omega||}(0,v)\\sin||\\omega|| \\ \\text{II} :=& \\sum\_{n=0}^\\infty\\frac{n+1}{(2n+2)!}(0,\\omega)(0,v)(0,\\omega)^{2n} \\ =& -\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(0,\\omega)\\sinh(0,\\omega) \\ =& \\frac{1}{2||\\omega||}(0,\\omega)(0,v)\\sin ||\\omega|| \\ \\text{III} :=& \\sum\_{n=0}^\\infty \\frac{n+1}{(2n+2)!}(0,v)(0,\\omega)^{2n+1} \\=&\\frac{1}{2}(0,v)\\sinh(0,\\omega) \\ =&\\frac{1}{2||\\omega||}(0,v)(0,\\omega)\\sin||\\omega|| \\ \\text{IV}:=&\\sum\_{n=0}^\\infty\\frac{n+1}{(2n+3)!}(0,\\omega)(0,v)(0,\\omega)^{2n+1} \\ =&\\frac{1}{2}\\sum\_{n=0}^\\infty \\frac{1}{(2n + 2)!}(0,\\omega)(0,v)(0,\\omega)^{2n+1} - \\frac{1}{2}\\sum\_{n=0}^\\infty \\frac{1}{(2n + 3)!}(0,\\omega)(0,v)(0,\\omega)^{2n+1} \\ =& -\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(0,\\omega)(\\cosh(0,\\omega) - 1) \\ &+\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(\\sinh(0,\\omega) - (0,\\omega)) \\ =& -\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(0,\\omega)\\cosh(0,\\omega) \\ &+\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)\\sinh(0,\\omega)\\=& -\\frac{1}{2||\\omega||^2}(0,\\omega)(0,v)(0,\\omega)\\cos ||\\omega|| \\ &+\\frac{1}{2||\\omega||^3}(0,\\omega)(0,v)(0, \\omega)\\sin ||\\omega||\\ \\end{aligned} ] So: [ \\begin{aligned}T=&,,\\text{I}+\\text{II}+\\text{III}+\\text{IV}\\ =&\\left(-\\omega\\cdot v\\tfrac{\\sin||\\omega||}{||\\omega||}, v\\tfrac{\\sin||\\omega||}{||\\omega||} + \\tfrac{\\omega}{||\\omega||^2}(\\omega\\cdot v)(\\cos||\\omega|| - \\tfrac{\\sin||\\omega||}{||\\omega||})\\right)\\end{aligned} ] And: \[ \\begin{aligned}\\exp[(0,\\omega)+\\varepsilon(0,v)] =& \\left(\\cos||\\omega||, \\omega \\tfrac {\\sin||\\omega||}{||\\omega||}\\right) \\ &+ \\varepsilon\\left(-\\omega\\cdot v\\tfrac{\\sin||\\omega||}{||\\omega||}, v\\tfrac{\\sin||\\omega||}{||\\omega||} + \\tfrac{\\omega}{||\\omega||^2}(\\omega\\cdot v)(\\cos||\\omega|| - \\tfrac{\\sin||\\omega||}{||\\omega||})\\right)\\end{aligned} \] Re-introducing the factor of (1/2) into the basis we’re using for our algebra, we have: \[ \\boxed{\\begin{array}{rcl}X&:=&(0,\\omega/2)+\\varepsilon(0,v/2) \\ \\omega' &:=& \\omega/2 \\ v' &:=& v/2 \\ \\theta' &:=& ||\\omega'|| = ||\\omega||/2 \\ a\_\\theta &:=& \\frac{\\sin\\theta}{\\theta} \\ b\_\\theta &:=& \\frac{\\cos\\theta - a(\\theta)}{\\theta^2} \\ \\exp(X)&=& \\left(\\cos\\theta',\\omega'a\_{\\theta'}\\right) + \\varepsilon\\left[-(\\omega'\\cdot v')a\_{\\theta'}, v'a\_{\\theta'}+\\omega'(w'\\cdot v')b\_{\\theta'}\\right]\\end{array}} \] For small (\\theta'), we should use small angle approximations: [ \\begin{array}{rcrlrlrlrl}a\_\\theta &\\approx& &1 &-& x^2/6 &+& x^4/120 &+&O(\\theta^6)\\ b\_\\theta &\\approx& -& 1/3 &+& x^2/30 &-& x^4/840 &+&O(\\theta^6)\\end{array} ]