Skip to content

Metrics Images

You may not need this

If your emitted data from your experience build does not require post-processing you can skip this step and go straight to Metrics. A metrics build is only needed if your workflow requires post-processing of raw experience build outputs before metrics can be computed.

This document describes how the SignalFlag platform supports executing metrics builds.

Metrics in the SignalFlag platform

Similar to builds, customers can register a Metrics Build with SignalFlag, which will be executed after each test. A metrics build is also a container image, with a similar contract to builds.

After each test, the SignalFlag platform runs the metrics build with the /tmp/resim/inputs directory populated with:

  • a subdirectory, experience, which contains any files as part of the experience
  • a subdirectory, logs, which contains any files generated as logs from the execution of the simulation.

The metrics build post-processes these and writes its results as emissions with the SignalFlag SDK, to a file in /tmp/resim/outputs whose name ends in .resim.jsonl, such as emissions.resim.jsonl. SignalFlag ingests these emissions alongside those from the experience build, and the metrics config synced to the branch computes metrics from both.

Any additional files placed in the outputs directory will also be uploaded to the SignalFlag platform as logs.

Creating metrics builds

It is possible to create metrics builds in CI/CD or with the CLI tool:

Bash
resim metrics-builds create \
    --project "my-project" \
    --name "My Metrics Build" \
    --version <commit sha or version> \
    --image <Docker image URI> \
    --systems "perception" "localization"

Where, again, if one has a project selected, that parameter can be omitted. Similarly to builds, SignalFlag recommend the set-up of CI/CD workflows to generate metrics builds when the metrics generation code is updated.

Systems and metrics builds

Since metrics are closely tied to the test application which runs, the SignalFlag app enables metrics builds to be aligned to a given system, to enable compatibility checking at test creation time. One can register an existing metrics build against a given system:

Bash
resim metrics-builds add-system \
    --project "my-project" \
    --system "my-system" \
    --metrics-build-id "<uuid>"

and remove them with:

Bash
resim metrics-builds remove-system \
    --project "my-project" \
    --system "my-system" \
    --metrics-build-id "<uuid>"

Debugging metrics

To check how your metrics render without running a batch, generate an emissions file and pass it to signalflag metrics debug, which renders it against a metrics set in a temporary dashboard. See Adding your first metric.