Search based on a custom objective function

Note

This example shows how to create and use a custom objective function, an experiment which requires the output of another experiment, with ado.

The scenario¶

Often experiments will not directly produce the value that you are interested in. For example, an experiment might measure the run time of an application, while the meaningful metric is the associated cost, which requires knowing information like the cost per hour of the GPUs used. Another common scenario involves aggregating data points from one or more experiments into a single value.

In this example we will install a custom objective function that calculates a cost for the application workload configurations used in the talking a random walk example. When the workload configuration space is explored using a random walk, both the wallClockRuntime and the cost, as defined by the custom function, will be measured.

Caution

The commands below assume you are in the directory examples/ml-multi-cloud in the ado source repository. See here for how to get the source repository.

Pre-requisites¶

Install the ray_tune ado operator¶

If you haven't already installed the ray_tune operator, run (assumes you are in examples/ml-multi-cloud/ ):

pip install ../../plugins/operators/ray_tune

then executing

ado get operators

should show an entry for ray_tune like below

Available operators by type:
      OPERATOR     TYPE
0  random_walk  explore
1     ray_tune  explore

Installing the custom experiment¶

The custom experiment is defined in a python package under custom_actuator_function/. To install it run:

pip install custom_experiment/

then

ado get actuators --details

will output something like

2          custom_experiments          CustomExperiments                        ml-multicloud-cost-v1.0       True
3         molecule-embeddings                 Embeddings                   calculate-morgan-fingerprint       True
4          molformer-toxicity         molformer-toxicity                               predict-toxicity       True
5                     mordred         Mordred Descriptor                  mordred-descriptor-calculator       True
6                       st4sd                      ST4SD                      toxicity-prediction-opera       True

You can see the custom experiment provided by the package, ml-multicloud-cost-v1.0 on the first line. Executing ado describe experiment ml-multicloud-cost-v1.0

outputs:

Identifier: custom_experiments.ml-multicloud-cost-v1.0

Required Inputs:
  Constitutive Properties:
      nodes
      Domain:
        Type: DISCRETE_VARIABLE_TYPE Interval: 1.0 Range: [0, 1000] 
      
      cpu_family
      Domain:
        Type: DISCRETE_VARIABLE_TYPE Values: [0, 1] Range: [0, 2] 
      
      
  Observed Properties:
      op-benchmark_performance-wallClockRuntime
      

Outputs: ml-multicloud-cost-v1.0-total_cost

From this you can see the ml-multicloud-cost-v1.0 requires an observed property, i.e. a property measured by another experiment, as input. From the observed property identifier, the experiment is called benchmark_performance and the property is wallClockRuntime.

Create a discoveryspace that uses the custom experiment¶

First create a samplestore with the ml-multi-cloud example data following these instructions. If you have already completed the talking a random walk example, reuse the samplestore you created there (use ado get samplestores if you cannot recall the identifier).

To use the custom experiment you must add it in the experiments list of a discoveryspace. The acutauatorIdentifer will be custom_experiments and the experimentIdentifier will be the name of your experiment. For this case the relevant section looks like:

experiments:
  - experimentIdentifier: 'benchmark_performance'
    actuatorIdentifier: 'replay'
  - experimentIdentifier: 'ml-multicloud-cost-v1.0'
    actuatorIdentifier: 'custom_experiments'

The complete discoveryspace for this example is given in ml_multicloud_space_with_custom.yaml To create it execute:

ado create space -f ml_multicloud_space_with_custom.yaml --set "sampleStoreIdentifier=$SAMPLE_STORE_IDENTIFIER"

Important

If an experiment takes the output of another experiment as input both experiments must be in the discoveryspace. In the above example if the entry benchmark_performance was omitted the ado create space command would fail with:

SpaceInconsistencyError: MeasurementSpace does not contain an experiment measuring an observed property required by another experiment in the space

Note the created discoveryspace resource identifier printed by this command for the next section. You can use this identifier in the following command to see a description of the space (replacing $DISCOVERY_SPACE_IDENTIFIER):

ado describe space $DISCOVERY_SPACE_IDENTIFIER

This will output:

Identifier: space-d5c150-0b762f

Entity Space: 
  
  Number entities: 48
  
  Categorical properties:
           name     values
    0  provider  [A, B, C]
  
  Discrete properties:
             name   range interval        values
    0  cpu_family  [0, 2]     None        [0, 1]
    1   vcpu_size  [0, 2]     None        [0, 1]
    2       nodes  [2, 6]     None  [2, 3, 4, 5]
  
   
Measurement Space:
                                       experiment  supported
  0                 replay.benchmark_performance       True
  1   custom_experiments.ml-multicloud-cost-v1.0       True 
  
  'replay.benchmark_performance' 
  
  Inputs:
      parameter      type value parameterized
  0  cpu_family  required  None            na
  1   vcpu_size  required  None            na
  2       nodes  required  None            na
  3    provider  required  None            na 
  
  Outputs:
       target property
  0  wallClockRuntime
  1            status  'custom_experiments.ml-multicloud-cost-v1.0' 
  
  Inputs:
                                  parameter      type value parameterized
  0                                   nodes  required  None            na
  1                              cpu_family  required  None            na
  2  benchmark_performance-wallClockRuntime  required  None            na 
  
  Outputs:
     target property
  0      total_cost  
  
Sample store identifier: '0b762f'

Exploring the `discoveryspace`¶

To run a randomwalk operation on the new space, execute ((replacing $DISCOVERY_SPACE_IDENTIFIER with the identifier of the space you created):

ado create operation -f randomwalk_ml_multicloud_operation.yaml --set "spaces[0]=$DISCOVERY_SPACE_IDENTIFIER"

This produces an output similar to that described in the talking a random walk example and will exit printing the operation identifier. However, in this case there is additional information related to the dependent experiment.

When it completes, execute the ado show entities operation command to see the results produced:

ado show entities operation $OPERATION_IDENTIFIER

You will see a table request_index 0 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 11 0 12 0 13 14 0 15 0 16 0 17 0 18 0 19 0 20 0 21 0 22 23 0 24 0 25 0 26 0 27 0 28 0 29 0 30 0 31 32 0 33 0 34 0 35 0 36 37 0 38 0 39 0 40 41 0 42 0 43 0 44 45 0 46 0 47 0

Search based on a custom objective function

The scenario¶

Pre-requisites¶

Install the ray_tune ado operator¶

Installing the custom experiment¶

Create a discoveryspace that uses the custom experiment¶

Exploring the `discoveryspace`¶

Explore Further¶

Takeaways¶

What's next¶

Search based on a custom objective function

The scenario¶

Pre-requisites¶

Install the ray_tune ado operator¶

Installing the custom experiment¶

Create a discoveryspace that uses the custom experiment¶

Exploring the discoveryspace¶

Explore Further¶

Takeaways¶

What's next¶

Exploring the `discoveryspace`¶