Taking a random walk

Note

This example illustrates:

Describing a set of points and how to measure them using a discoveryspace
Exploring the discoveryspace by creating an operation that samples and measures the points
Getting the results of an operation

The scenario¶

When you deploy a workload you have to choose values for workload parameters like the number of CPUs or the node type. To choose a combination of parameters that, for example, maximizes performance, a common strategy is to measure changes in performance by exploring the workload parameter space. This pattern applies to many domains where there is a parameter space to explore.

In this example ado is used to explore the workload parameter space for a cloud application. To explore a workload parameter space you have to:

define the values of the parameters to test - the parameter space
define what to test them with - the experiment
select points from the parameter space and perform the test - the sampling method

Here, we will use the simplest sampling method, random walk, where some number of points are randomly selected without replacement.

Caution

The commands below assume you are in the directory examples/ml-multi-cloud in the ado source repository. See here for how to get the source repository.

Using pre-existing data with `ado`¶

For this example we will use some pre-existing data. This makes the example simpler and quicker to execute but can also be useful in other situations. The data is in the file ml_export.csv and is consists of results of running a benchmark on different cloud hardware configurations from different providers.

In ado such configurations are called entities, and are stored, along with the results of measurements executed on them, in a samplestore. Let's start by copying the data in ml_export.csv into a new samplestore.

To do this execute,

ado create samplestore -f ml_multicloud_sample_store.yaml --set "copyFrom[0].storageLocation.path"=ml_export.csv

and it will report that a samplestore has been created:

Success! Created sample store with identifier $SAMPLE_STORE_IDENTIFIER

Note the samplestore resource identifier printed by this command for the next section.

Also try ado get samplestores and you will see an entry for the one you just created

Info

You only need to create this samplestore once. It can be reused in multiple discoveryspaces or examples that require the ml_export.csv data.

Creating a `discoveryspace` for the `ml-multi-cloud` data¶

A discoveryspace describes a set of points and how to measure them. Here we will create a discoveryspace to describe the space explored in ml_export.csv.

Execute:

ado create space -f ml_multicloud_space.yaml --set "sampleStoreIdentifier=$SAMPLE_STORE_IDENTIFIER"

where $SAMPLE_STORE_IDENTIFIER is the identifier you copied in last step.

This will confirm the creation of the discoveryspace with:

Success! Created space with identifier: $DISCOVERY_SPACE_IDENTIFIER

You can now describe the discoveryspace with:

ado describe space $DISCOVERY_SPACE_IDENTIFIER

where $DISCOVERY_SPACE_IDENTIFIER is the identifier of the discoveryspace resource that was just created. This will output:

Identifier: space-65cf33-a8df39

Entity Space: 
  
  Number entities: 48
  
  Categorical properties:
           name     values
    0  provider  [A, B, C]
  
  Discrete properties:
             name   range interval        values
    0  cpu_family  [0, 2]     None        [0, 1]
    1   vcpu_size  [0, 2]     None        [0, 1]
    2       nodes  [2, 6]     None  [2, 3, 4, 5]
  
   
Measurement Space:
                        experiment  supported
  0  replay.benchmark_performance       True 
  
  'replay.benchmark_performance' 
  
  Inputs:
      parameter      type value parameterized
  0  cpu_family  required  None            na
  1   vcpu_size  required  None            na
  2       nodes  required  None            na
  3    provider  required  None            na 
  
  Outputs:
       target property
  0  wallClockRuntime
  1            status  
  
Sample Store identifier: 'a8df39'

Note

The set of points is defined by the properties in the Entity Space - here 'cpu_family', 'provider', 'vcpu_size' and 'nodes' - and the values those properties can take.

Tip

Consider why the size of the entityspace is 48. Compare this to the number of rows in ml_export.csv.

Exploring the `discoveryspace`¶

Next we will run an operation that will "explore" the discoveryspace we just created. Since we already have the data, ado will transparently identify and reuse it. An example operation file is given in randomwalk_ml_multicloud_operation.yaml. The contents are:

# Copyright (c) IBM Corporation
# SPDX-License-Identifier: MIT

metadata:
  name: 'randomwalk-all'
  description: 'Perform a random walk on all points in a space'
spaces:
- 'space-630588-bfebfe'
operation:
  module:
    moduleClass: RandomWalk
  parameters:
    numberEntities: 48
    batchSize: 1
    mode: 'random'
    samplerType: 'generator'
    singleMeasurement: True

To run the operation execute (replacing $DISCOVERY_SPACE_IDENTIFIER with the identifier of the space you created):

ado create operation -f randomwalk_ml_multicloud_operation.yaml --set "spaces[0]=$DISCOVERY_SPACE_IDENTIFIER"

This will output a lot of information as it samples all the entities. Typically, you will see the following lines for each entity (point in the entity space) sampled and measured:

(RandomWalk pid=14797) Continuous batching: SUBMIT EXPERIMENT. Submitting experiment replay.benchmark_performance for provider.B-cpu_family.1-vcpu_size.1-nodes.4
(RandomWalk pid=14797) 
(RandomWalk pid=14797) Continuous batching: SUMMARY. Entities sampled and submitted: 2. Experiments completed: 1 Waiting on 1 active requests. There are 0 dependent experiments
(RandomWalk pid=14797) Continuous Batching: EXPERIMENT COMPLETION. Received finished notification for experiment in measurement request in group 1: request-randomwalk-0.9.6.dev91+884f713b.dirty-c5ed4b-579021-experiment-benchmark_performance-entities-provider.B-cpu_family.1-vcpu_size.1-nodes.4 (explicit_grid_sample_generator)-requester-randomwalk-0.9.6.dev91+884f713b.dirty-c5ed4b-time-2025-07-29 20:03:00.976809+01:00

The first line, "SUBMIT EXPERIMENT", indicates the entity - provider.B-cpu_family.1-vcpu_size.1-nodes.4 - and experiment - replay.benchmark_performance submitted. The next line gives a summary of what has happened so far: this is the second entity sampled and submitted; one experiment has completed; and the sampler is waiting on one active experiment before submitting a new one. Finally, the "EXPERIMENT COMPLETION" line indicates the experiment has finished.

The operation will end with information like:

config:
  operation:
    module:
      moduleClass: RandomWalk
      moduleName: orchestrator.modules.operators.randomwalk
      modulePath: .
      moduleType: operation
    parameters:
      batchSize: 1
      mode: sequential
      numberEntities: 48
      samplerType: generator
  spaces:
  - space-65cf33-a8df39
created: '2025-06-20T13:03:46.763154Z'
identifier: randomwalk-0.9.4.dev30+564196d4.dirty-b8a233
kind: operation
metadata:
  entities_submitted: 48
  experiments_requested: 74
operationType: search
operatorIdentifier: randomwalk-0.9.4.dev30+564196d4.dirty
status:
- event: created
  recorded_at: '2025-06-20T13:03:40.267005Z'
- event: added
  recorded_at: '2025-06-20T13:03:46.764750Z'
- event: started
  recorded_at: '2025-06-20T13:03:46.769169Z'
- event: finished
  exit_state: success
  recorded_at: '2025-06-20T13:03:48.369516Z'
- event: updated
  recorded_at: '2025-06-20T13:03:48.374765Z'
version: v1

Note the value of the identifier field: in above it is randomwalk-0.9.4.dev30+564196d4.dirty-b8a233

Note

The operation "reuses" existing measurements: this is an ado feature called memoization.

ado transparently executes experiments or memoizes data as appropriate - so the operator does not need to know if a measurement needs to be performed at the time it requests it, or if previous data can be reused.

Tip

Operations are domain agnostic. If you look in randomwalk_ml_multicloud_operation.yaml you will see there is no reference to characteristics of the discoveryspace we created. Indeed, this operation file could work on any discoveryspace.

This shows that operators, like randomwalk, don't have to know domain specific details. All information about what to explore and how to measure is captured in the discoveryspace.

Looking at the `operation` output¶

The command

request_index 0 0 1 0 2 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0 12 0 13 0 14 0 15 0 16 0 17 0 18 19 0 20 0 21 0 22 0 23 24 0 25 0 26 0 27 0 28 0 29 30 0 31 0 32 0 33 0 34 0 35 36 0 37 0 38 0 39 0 40 0 41 0 42 0 43 44 0 45 0 46 0 47 0

Taking a random walk

The scenario¶

Using pre-existing data with `ado`¶

Creating a `discoveryspace` for the `ml-multi-cloud` data¶

Exploring the `discoveryspace`¶

Looking at the `operation` output¶

Exploring Further¶

Viewing entities¶

Resource provenance¶

Operation timeseries¶

Resource templates¶

Rerun¶

Takeaways¶

What's next¶

Taking a random walk

The scenario¶

Using pre-existing data with ado¶

Creating a discoveryspace for the ml-multi-cloud data¶

Exploring the discoveryspace¶

Looking at the operation output¶

Exploring Further¶

Viewing entities¶

Resource provenance¶

Operation timeseries¶

Resource templates¶

Rerun¶

Takeaways¶

What's next¶

Using pre-existing data with `ado`¶

Creating a `discoveryspace` for the `ml-multi-cloud` data¶

Exploring the `discoveryspace`¶

Looking at the `operation` output¶