Skip to content

AutoPDL Tutorial

This tutorial describes how to use AutoPDL, PDL's prompt optimizer tool Spiess et al. (2025). AutoPDL can be used to optimize any part of a PDL program. This includes few-shots examples and textual prompts, but also prompting patterns. It outputs an optimized PDL program with optimal values.

Installing AutoPDL

Please ensure PDL was installed with extras e.g.

1
2
3
4
5
pip install 'prompt-declaration-language[all]'
# or from source
git clone git@github.com:IBM/prompt-declaration-language.git
cd prompt-declaration-language
pip install -e '.[all]'

Writing a PDL program to optimize

The first step in using AutoPDL is to write a PDL program that has free variables. Consider for example, the following PDL program, which queries an LLM to correct a sentence with grammatical errors (file):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
defs:
  max_tokens: 1024
lastOf:
  - "Here are examples of grammatically incorrect sentences and their corrected versions:\n\n"
  - for:
      example: ${ demonstrations }
    repeat:
      text: "${ example.input } -> ${ example.output }"
    join:
      with: "\n\n"
  - |+
    Correct the following sentence:

    ${ input }
    Here's the corrected sentence:

  - model: ${ model }
    def: response
    parameters:
      max_tokens: ${ max_tokens }
      temperature: 0

  - if: ${ verify }
    then:
      lastOf:
      - Do you think this was a correct answer? If not, generated a correct answer.
      - model: ${ model }
    else: ${ response }

This program starts with a definition section. Note that a defs section is necessary. This is followed by a lastOf sequence (a list of blocks to be executed where the result of the last block is returned as the result of the whole sequence). First, the program establishes some demonstrations obtained from a demonstrations variable. The for loop at lines 5 to 10 ensures that all demonstrations are formatted in a consistent way. On lines 11 to 16 the program formulates a prompt to correct a sentence stored in variable input. Lines 17 through 21 show a model call where the model id is given by variable model. Finally, lines 23 through 28 check if variable verify is set to true. If so, it makes another model to verify the previous response and to produce a new one if needed.

Notice that variables input, model, demonstrations, verify are not defined. The first of these is an instance variable that will help in holding different instances when the optimizer is running. The rest of them are parameters to be optimized. We can pick among different models, different demonstrations, and especially different prompting patterns. PDL supports first-class functions, so the program could be made to pick the optimal function to be used, thereby choosing the prompting pattern. In this example, finding an optimal value for verify will determine whether it's best to call the model once or twice.

Dataset

In addition to the PDL program, AutoPDL also needs a dataset. These will be used to perform the optimization, and as a source of demonstrations. The train split will be used to draw instances and demonstrations, the validation for checking during the optimization, and test to evaluate and obtain a final score at the end of the optimization run.

In this example, we need a dataset containing sentences with mistakes and the corrected version. We can use process_grammar_correction.py to obtain a dataset split into train/validation/test. Simply run:

python process_grammar_correction.py

Loss function

The lost function is use to guide the optimizer towards the best solution and evaluate the final program. The loss function can must be a PDL function named score that takes as input the result of the program, the ground truth, and returns a floating point number. In our example, we are using the Levenshtein distance that we import from the textdistance Python module. The score function is defined in the eval_levenshtein.pdl file:

defs:
  score:
    function:
      document: string
      ground_truth: string
    return:
      lang: python
      fallback: 0
      code: |
        import textdistance
        result = textdistance.levenshtein.normalized_similarity(document, ground_truth)

The final ingredient needed is a configuration file as explained in the next section.

Writing a configuration file

An AutoPDL configuration file describes the state-space and parameters for the search. In this example, the configuration is given in the following file:

pdl_path: grammar_correction.pdl # Path to the PDL file to optimize
dataset: 
  train: grammar_correction_jsonl/train.jsonl # Path to the training split in JSONL format
  test: grammar_correction_jsonl/test.jsonl # Path to the test split in JSONL format
  validation: grammar_correction_jsonl/validation.jsonl # Path to the validation split in JSONL format

demonstrations_variable_name: demonstrations # variable name to insert demonstrations into
demonstration_columns:
  - input # column name for the question in the dataset
  - output # column name for the answer in the dataset

instance_columns:
  - input # column name for the question in the dataset

groundtruth_column: output # column name for the ground truth in the dataset

eval_pdl: eval_levenshtein.pdl # Path to the PDL file for evaluation

#budget: 2h # Set a budget, can be number of iterations, or a duration string e.g. "2h"
#budget_growth: double # double validation set size each iteration. ## 
# or to_max: reach max_test_set_size by final iteration
initial_validation_set_size: 2 # size of test set in first iteration
max_validation_set_size: 10 # maximum test set size. 
max_test_set_size: 10
num_candidates: 10 # how many candidates to evaluate
parallelism: 5 # how many threads to run evaluations across
#shuffle_test: false # shuffling of test set
#test_set_name: test # name of test set
#train_set_name: train # name of train set
#validation_set_name: validation # name of validation set
variables: # define discrete options to sample from
  model: # set ${ model } variable
    - ollama_chat/granite3.3:8b
    - ollama_chat/gpt-oss:20b
  num_demonstrations: # overrides num demonstrations above
    - 0
    - 3
    - 5
  verify:
    - true 
    - false

Field pdl_path is the path to the PDL program to optimize. dataset points to the dataset to be used. In this case, it's an object with paths for train/validation/test splits. demonstrations_variable_name gives the name of the PDL variable that will hold the demonstrations in the optimized program. demonstration_columns indicates the field names in the dataset that will be used to create demonstrations, and instance_columns are those fields that will be used to formulate an instance query (see the query in the PDL program above, which uses input). The groundtruth_column holds the field with the ground truth (in this case output). eval_pdl is the path of the PDL program that encapsulates the loss function.

initial_validation_set_size is the initial size of the validation set (i.e., the number of tests used initially to validate candidates). max_validation_set_size indicates the maximum to which this validation set will grow. For more details on the successive halving algorithm used in AutoPDL see here. max_test_set_size: 10 is the maximum of the test set used to evaluate at the end of the evaluation run. num_candidates indicates the number of candidates to consider (sampled randomly). parallelism indicates the level of parallelism used by the optimizer.

Last but not least, variables indicates the domain of each variable that needs to be tuned. In this case, model can be either an Ollama Granite model or gpt-oss. num_demonstration is a special variable that the user can set to indicate how many demonstrations to consider. In this case, zero-shot is also included. Finally, the domain of variable verify can be true or false.

Notice that variable input in the PDL program is not given a domain. This is because it will hold the different instances that will be evaluated (it was included in the instance_columns field).

For a complete list of available fields in the configuration file is given in the configuration parser file.

We are ready to run the optimizer!

Running AutoPDL

To run the optimizer, execute the following command:

pdl-optimize -c grammar_correction_example.yml

After a while, AutoPDL creates a new file optimized_grammar_correction.pdl with definitions for all the free variables. It determined that gtp-oss is the better model for the task at hand, and that verify is best set to False. The optimized program contains the selection of demonstrations. To run this program add a definition for input:

defs:
  ...
  input: This sentence have an error.

To run the optimized program, execute the command:

pdl optimized_grammar_correction.pdl

A log of the optimization process is written to experiments/ by default.