Contributing a Benchmark for a Model¶

This guide walks you through adding a benchmark for an algorithm that you have already registered in Algorithm Nexus. If you have not yet registered your algorithm, start with Contributing a Python Algorithm Package to Algorithm Nexus.

There are four steps to add a benchmark for your algorithm:

Find or create a benchmark experiment
Register the experiment with your Nexus package
Define a benchmark submission for your algorithm
Run the benchmark

Prerequisites¶

Before you begin, ensure:

Your algorithm is registered in a Nexus package under packages/<package-name>/models/<model-name>/model.yaml. See Contributing a Python Algorithm Package if not.
uv is installed on your system.
You have a fork of the Algorithm Nexus repository checked out locally and your development environment is set up:

cd algorithm-nexus
uv sync --group dev --extra cli

Step 1: Find or create a benchmark experiment¶

A benchmark experiment is a Python package that defines how to evaluate an algorithm. First check whether a suitable experiment already exists in Algorithm Nexus before creating a new one.

Find an existing experiment¶

List all benchmark experiments registered across Algorithm Nexus:

uv run nexus list benchmark-experiments

If an experiment covers the evaluation you need, note its experiment ID and the benchmark package that provides it — you will reference both in the next step. Skip ahead to Step 2.

To inspect the inputs, outputs, and parameters of a specific experiment, install the benchmark package it belongs to and use the ado CLI:

uv pip install <benchmark-package>
ado describe experiment <experiment-id>

Create a new experiment¶

If no existing experiment fits your needs, create one following the ADO custom experiment template. A benchmark experiment is a standard ado custom experiment packaged as a Python package.

Place the package under your Nexus package directory:

packages/<package-name>/
└── benchmark_packages/
    └── <benchmark-package-name>/
        ├── pyproject.toml
        └── src/

Step 2: Register the experiment with your Nexus package¶

Edit packages/<package-name>/nexus.yaml to declare that it uses a the experiment from the given package

package:
    name: <package-name>

    benchmark_packages:
        # Local package stored in this repository
        - requirement_specifier: "./packages/<package-name>/benchmark_packages/<experiment-package-name>"
          experiments:
              - "<experiment-id>"

        # Remote package hosted on GitHub
        - requirement_specifier: "https://github.com/<org>/<experiment-package-repo>"
          experiments:
              - "<experiment-id>"

        # Package published on PyPI
        - requirement_specifier: "<experiment-package-name>"
          experiments:
              - "<experiment-id>"

Each requirement_specifier must resolve to a Python package that contains an ado custom experiment.

Validate the updated package configuration:

uv run nexus validate package packages/<package-name>

Fix any validation errors before proceeding.

Step 3: Define a benchmark submission for your model¶

A benchmark submission specifies executing a registered experiment on your model/algorithm with a specific set of parameter values (benchmark instance). Each benchmark submission is a folder under your model's benchmark_submissions/ directory, that contains an ado space.yaml

Create the directory,

mkdir -p packages/<package-name>/models/<model-name>/benchmark_submissions/<submission-name>

Create packages/<package-name>/models/<model-name>/benchmark_submissions/<submission-name>/space.yaml:

Example:

entitySpace:
    - identifier: dataset
      propertyDomain:
          values: ["<dataset-name>"]
    - identifier: split
      propertyDomain:
          values: ["test"]

experiments:
    - actuatorIdentifier: custom_experiments
      experimentIdentifier: <experiment-id>

The entitySpace defines the benchmark instance and any other parameters for this submission. The experimentIdentifier must match one of the experiment IDs you registered in nexus.yaml in the previous step.

For full details on space.yaml syntax, see the ADO discoveryspace documentation.

Validate the package again to confirm the instance is well-formed:

uv run nexus validate package packages/<package-name>

Step 4: Run the benchmark¶

Install the benchmark package and run the benchmark submission locally using the ado CLI. First save the following operation configuration to a file op.yaml. It will execute the experiment on all points in your space.

metadata:
    name: randomwalk-all
spaces:
    - dynamically_inserted
operation:
    module:
        operatorName: random_walk
        operationType: search
    parameters:
        numberEntities: all
        samplerConfig:
            samplerType: generator
            mode: random

Then,

uv pip install packages/<package-name>/benchmark_packages/<experiment-package-name>
ado create space -f <space-yaml-path>
ado create operation -f op.yaml --use-latest space

See the ADO documentation for the full set of execution options including parameter sweeps and remote execution.

Step 5: Commit Changes and Open a Pull Request¶

Add and commit your changes:

git add packages/<package-name>
git add benchmark_packages #required when also adding a benchmark package during Step 1
git commit -s -m "feat(benchmark): Add <submission-name> benchmark for <model-name>"
git push origin <your-branch>

Open a pull request from your fork to the Algorithm Nexus main branch.

Optional: Mapping to logical benchmarks¶

Once your benchmark is running, you can connect it to a logical benchmark so the results can be compared with results from other experiments targeting the same type of problem. This requires creating a benchmark biding that maps the experiments internal parameters and metrics to a shared, canonical vocabulary.

This is done in two parts:

Creating the logical benchmark definition (if it does not exist)
Adding the benchmark binding for your experiment to the logical benchmark

Check for an existing logical benchmark definition¶

Logical benchmark definitions live in the benchmarks/ directory at the root of the Algorithm Nexus repository. Browse that directory to see whether a definition already exists for your domain (e.g. benchmarks/llm-inference/problem.yaml, benchmarks/max-cut/problem.yaml).

If a definition exists — you add a binding for experiment to it. Continue to Add a benchmark binding for your experiment.
If no definition exists — you will create one first. Continue to Create a logical benchmark definition.

Create a logical benchmark definition¶

If no definition exists, create a directory in benchmarks/<logical-benchmark-id>/ with a problem.yaml file. This file establishes the canonical vocabulary for your benchmark — property names, valid values, and metric names.

A minimal example:

logicalBenchmark:
    benchmarkIdentifier: graph_coloring
    title: Graph Coloring
    description: >
        Evaluates graph-coloring algorithms on their ability to produce valid
        k-colorings with a small chromatic number. Instances span random
        Erdős–Rényi graphs and structured benchmark graphs at varying densities.
    instance:
        - identifier: graph_family
          metadata:
              description: Graph family (erdos_renyi, planar, random_regular).
          propertyDomain:
              variableType: CATEGORICAL_VARIABLE_TYPE
              values: [erdos_renyi, planar, random_regular]
        - identifier: num_vertices
          metadata:
              description: Number of vertices in the graph.
          propertyDomain:
              variableType: DISCRETE_VARIABLE_TYPE
              values: [50, 100, 250, 500]
        - identifier: edge_density
          metadata:
              description: Edge probability / density parameter.
          propertyDomain:
              variableType: CONTINUOUS_VARIABLE_TYPE
        - identifier: graph
          is_artifact: true
          metadata:
              description: Graph input files in various formats.
    metrics:
        - num_colors_used
        - is_valid_coloring
        - elapsed_ms
    ranking:
        metric: num_colors_used
        order: asc
    owner: "@graph-team"
bindings: [] # List of bindings to this benchmark

For the full schema and a complete worked example, see Section 2 of the Benchmark Metadata Convention.

Add logical benchmark instances¶

To define specific benchmark problem instances (e.g., individual graphs, routing problem instances, or datasets), add a subfolder per instance in benchmarks/<logical-benchmark-id>/instances/<instance-name>/ containing an instance.yaml file and any subfolders holding artifact files referenced by the instance.

Example instance YAML (benchmarks/<logical-benchmark-id>/instances/graph_01/instance.yaml):

identifier: graph_01
description: 50-node random graph
graph_family: random_regular
num_vertices: 50
edge_density: 0.2
graph:
    artifacts_location: graph_files # subfolder that exists inside instances/graph_01/

Add a benchmark binding for your experiment¶

The binding is added to the list of bindings in the logical benchmark yaml.

logicalBenchmark: ... #logical benchmark fields
bindings:
    - experiment:
          experimentIdentifer: myexperiment
          experimentVersion: 1.2
          actuatorIdentifier: customexperiments
      targetMapping: ...
    -  #next binding

A binding contains the following mapping sections:

targetMapping: Names the experiment property that carries the benchmark target (e.g. the algorithm or model identifier)
instanceMapping: Maps benchmark instance properties to experiment inputs
staticFilters: Sets static property values implicit in either the logical benchmark or the experiment. Contains two optional sub-keys:
- experimentFilters: pins experiment properties to values that are implicit in the logical benchmark (adds a constant WHERE clause to every query)
- benchmarkFilters: pins benchmark instance properties to values that are implicit in the experiment (injects a constant value into leaderboard rows without reading it from the experiment results)
metricMapping: Maps outputs of the logical benchmark to the outputs of the experiment

An example binding is

experiment:
    actuatorIdentifier: custom_experiments
    experimentIdentifier: rlx_coloring
    experimentVersion: 1.0.0 # The binding only uses the major version
targetMapping: solver # The experiment property that carries the benchmark target identifier
instanceMapping:
    - benchmark:
          identifier: num_vertices
      experiment:
          identifier: n_nodes # rlx_coloring's internal param name
    - benchmark:
          identifier: edge_density
      experiment:
          identifier: density # rlx_coloring's internal param name
metricMapping:
    - benchmark:
          identifier: num_colors_used
      experiment:
          identifier: colors # rlx_coloring's internal metric name
    - benchmark:
          identifier: elapsed_ms
      experiment:
          identifier: runtime_ms # rlx_coloring's internal metric name

For the full benchmark binding schema and worked examples, see the Benchmark Metadata Convention.

`benchmarks/` Directory Convention¶

The top-level benchmarks/ directory is the single source of truth for all logical benchmark definitions and their bindings.

Structure¶

Each logical benchmark has its own directory containing problem.yaml and an instances/ directory with per-instance subfolders:

benchmarks/
├── <benchmark-id>/
│   ├── problem.yaml
│   ├── README.md              # optional: full problem statement and context
│   └── instances/
│       ├── <instance-1-id>/
│       │   ├── instance.yaml
│       │   └── <artifacts-subfolder>/   # named by artifacts_location in instance.yaml
│       │       ├── graph.dimacs
│       │       └── graph.json
│       └── ...
└── ...

A README.md alongside problem.yaml is encouraged to provide a full description of the problem. For example to explain the mathematical formulation, provide references, or describe the instance structure in more detail than the description field in problem.yaml allows.

Each problem.yaml uses the following top-level structure:

logicalBenchmark:
    # Definition fields (see Section 2 of the Benchmark Metadata Convention)
bindings:
    # List of benchmark bindings (see Section 3 of the Benchmark Metadata Convention)

Ownership¶

The logical benchmark owner is given by the owner field in the logicalBenchmark block. If omitted, the author of the PR that introduced the file is treated as the owner.
Each benchmark binding is owned by the author of the PR that added it.
Ownership implies responsibility for keeping the definition and bindings consistent with the experiments that reference them.

Versioning and When a Binding May Change¶

A benchmark binding may only be updated when:

Experiment property names or values change — this should accompany a new experiment major version. The binding for the previous major version can be retained alongside the new one.
Logical benchmark property names or values change — if the existing mappings become invalid, update the definition and all affected bindings in place. If the original mappings are still valid, create a new logical benchmark instead.
New non-default experiment properties are added in a minor version bump — if those properties must be set to non-default values to reproduce the same measurement, the binding must be updated because a different subset of experiment data would be aggregated.

Validation¶

Validate all logical benchmark files before opening a pull request:

uv run nexus validate logical-benchmarks

To validate a single file or benchmark folder:

uv run nexus validate logical-benchmarks --file benchmarks/<logical-benchmark-id>

Getting Help¶

If you encounter issues:

Check the Benchmark Integration Design
Check the Benchmark Metadata Convention
Refer to the ADO documentation
Search existing issues on GitHub
Open a new issue with details about your problem