Tune Experiment#

TuneExperiment#

class ibm_watson_machine_learning.experiment.fm_tune.TuneExperiment(credentials, project_id=None, space_id=None, verify=None)[source]#

Bases: BaseExperiment

TuneExperiment class for tuning models with prompts.

Parameters:

credentials (dict) – credentials to Watson Machine Learning instance
project_id (str, optional) – ID of the Watson Studio project
space_id (str, optional) – ID of the Watson Studio Space
verify (bool or str, optional) –
user can pass as verify one of following:
- the path to a CA_BUNDLE file
- the path of directory with certificates of trusted CAs
- True - default path to truststore will be taken
- False - no verification will be made

Example

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(
    credentials={
        "apikey": "...",
        "iam_apikey_description": "...",
        "iam_apikey_name": "...",
        "iam_role_crn": "...",
        "iam_serviceid_crn": "...",
        "instance_id": "...",
        "url": "https://us-south.ml.cloud.ibm.com"
    },
    project_id="...",
    space_id="...")

prompt_tuner(name, task_id, description=None, base_model=None, accumulate_steps=None, batch_size=None, init_method=None, init_text=None, learning_rate=None, max_input_tokens=None, max_output_tokens=None, num_epochs=None, verbalizer=None, tuning_type=None, auto_update_model=True, group_by_name=False)[source]#

Initialize a PromptTuner module.

Parameters:

name (str) – name for the PromptTuner
task_id (str) –
task that is targeted for this model. Example: experiment.Tasks.CLASSIFICATION

Possible values:
- experiment.Tasks.CLASSIFICATION: ‘classification’ (default)
- experiment.Tasks.QUESTION_ANSWERING: ‘question_answering’
- experiment.Tasks.SUMMARIZATION: ‘summarization’
- experiment.Tasks.RETRIEVAL_AUGMENTED_GENERATION: ‘retrieval_augmented_generation’
- experiment.Tasks.GENERATION: ‘generation’
- experiment.Tasks.CODE_GENERATION_AND_CONVERSION: ‘code’
- experiment.Tasks.EXTRACTION: ‘extraction
description (str, optional) – description
base_model (str, optional) – model id of the base model for this prompt tuning. Example: google/flan-t5-xl
accumulate_steps (int, optional) – Number of steps to be used for gradient accumulation. Gradient accumulation refers to a method of collecting gradient for configured number of steps instead of updating the model variables at every step and then applying the update to model variables. This can be used as a tool to overcome smaller batch size limitation. Often also referred in conjunction with “effective batch size”. Possible values: 1 ≤ value ≤ 128, default value: 16
batch_size (int, optional) – The batch size is a number of samples processed before the model is updated. Possible values: 1 ≤ value ≤ 16, default value: 16
init_method (str, optional) – text method requires init_text to be set. Allowable values: [random, text], default value: random
init_text (str, optional) – initialization text to be used if init_method is set to text otherwise this will be ignored.
learning_rate (float, optional) – learning rate to be used while tuning prompt vectors. Possible values: 0.01 ≤ value ≤ 0.5, default value: 0.3
max_input_tokens (int, optional) – maximum length of input tokens being considered. Possible values: 1 ≤ value ≤ 256, default value: 256
max_output_tokens (int, optional) – maximum length of output tokens being predicted. Possible values: 1 ≤ value ≤ 128 default value: 128
num_epochs (int, optional) – number of epochs to tune the prompt vectors, this affects the quality of the trained model. Possible values: 1 ≤ value ≤ 50, default value: 20
verbalizer (str, optional) – verbalizer template to be used for formatting data at train and inference time. This template may use brackets to indicate where fields from the data model must be rendered. The default value is “{{input}}” which means use the raw text, default value: Input: {{input}} Output:
tuning_type (str, optional) – type of Peft (Parameter-Efficient Fine-Tuning) config to build. Allowable values: [experiment.PromptTuningTypes.PT], default value: experiment.PromptTuningTypes.PT
auto_update_model (bool, optional) – define if model should be automatically updated, default value: True
group_by_name (bool, optional) – define if tunings should be grouped by name, default value: False

Examples

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(...)

prompt_tuner = experiment.prompt_tuner(
    name="prompt tuning name",
    task_id=experiment.Tasks.CLASSIFICATION,
    base_model='google/flan-t5-xl',
    accumulate_steps=32,
    batch_size=16,
    learning_rate=0.2,
    max_input_tokens=256,
    max_output_tokens=2,
    num_epochs=6,
    tuning_type=experiment.PromptTuningTypes.PT,
    verbalizer="Extract the satisfaction from the comment. Return simple '1' for satisfied customer or '0' for unsatisfied. Input: {{input}} Output: ",
    auto_update_model=True)

runs(*, filter)[source]#

Get the historical tuning runs but with name filter.

Parameters:: filter (str) – filter, user can choose which runs to fetch specifying tuning name

Examples

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(...)
experiment.runs(filter='prompt tuning name').list()

Tune Runs#

class ibm_watson_machine_learning.experiment.fm_tune.TuneRuns(client, filter=None, limit=50)[source]#

Bases: object

TuneRuns class is used to work with historical PromptTuner runs.

Parameters:

client (APIClient) – APIClient to handle service operations
filter (str, optional) – filter, user can choose which runs to fetch specifying tuning name
limit (int) – int number of records to be returned

get_run_details(run_id=None, include_metrics=False)[source]#

Get run details. If run_id is not supplied, last run will be taken.

Parameters:

run_id (str, optional) – ID of the run
include_metrics (bool, optional) – indicates to include metrics in the training details output

Returns:

run configuration parameters

Return type:

dict

Example

from ibm_watson_machine_learning.experiment import TuneExperiment
experiment = TuneExperiment(credentials, ...)

experiment.runs.get_run_details(run_id='02bab973-ae83-4283-9d73-87b9fd462d35')
experiment.runs.get_run_details()

get_tuner(run_id)[source]#

Create instance of PromptTuner based on tuning run with specific run_id.

Parameters:: run_id (str) – ID of the run
Returns:: prompt tuner object
Return type:: PromptTuner class instance

Example

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(credentials, ...)
historical_tuner = experiment.runs.get_tuner(run_id='02bab973-ae83-4283-9d73-87b9fd462d35')

list()[source]#

Lists historical runs with status. If user has a lot of runs stored in the service, it may take long time to fetch all the information. If there is no limit set, get last 50 records.

Returns:: Pandas DataFrame with runs IDs and state
Return type:: pandas.DataFrame

Examples

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(...)
df = experiment.runs.list()

Prompt Tuner#

class ibm_watson_machine_learning.foundation_models.PromptTuner(name, task_id, *, description=None, base_model=None, accumulate_steps=None, batch_size=None, init_method=None, init_text=None, learning_rate=None, max_input_tokens=None, max_output_tokens=None, num_epochs=None, verbalizer=None, tuning_type=None, auto_update_model=True, group_by_name=None)[source]#

Bases: object

cancel_run(hard_delete=False)[source]#

Cancels or deletes a Prompt Tuning run.

Parameters:: hard_delete (bool, optional) – When True then the completed or cancelled prompt tuning run is deleted, if False then the current run is canceled. Default: False

get_data_connections()[source]#

Create DataConnection objects for further user usage: (eg. to handle data storage connection).

Returns:: list of DataConnections
Return type:: list[‘DataConnection’]

Example

from ibm_watson_machine_learning.experiment import TuneExperiment
experiment = TuneExperiment(credentials, ...)
prompt_tuner = experiment.prompt_tuner(...)
prompt_tuner.run(...)

data_connections = prompt_tuner.get_data_connections()

get_model_id()[source]#

Get model id.

Example

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(credentials, ...)
prompt_tuner = experiment.prompt_tuner(...)
prompt_tuner.run(...)

prompt_tuner.get_model_id()

get_params()[source]#

Get configuration parameters of PromptTuner.

Returns:: PromptTuner parameters
Return type:: dict

Example

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(credentials, ...)
prompt_tuner = experiment.prompt_tuner(...)

prompt_tuner.get_params()

# Result:
#
# {'base_model': {'name': 'google/flan-t5-xl'},
#  'task_id': 'summarization',
#  'name': 'Prompt Tuning of Flan T5 model',
#  'auto_update_model': False,
#  'group_by_name': False}

get_run_details(include_metrics=False)[source]#

Get prompt tuning run details.

Parameters:: include_metrics (bool, optional) – indicates to include metrics in the training details output
Returns:: Prompt tuning details
Return type:: dict

Example

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(credentials, ...)
prompt_tuner = experiment.prompt_tuner(...)
prompt_tuner.run(...)

prompt_tuner.get_run_details()

get_run_status()[source]#

Check status/state of initialized Prompt Tuning run if ran in background mode.

Returns:: Prompt tuning run status
Return type:: str

Example

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(credentials, ...)
prompt_tuner = experiment.prompt_tuner(...)
prompt_tuner.run(...)

prompt_tuner.get_run_details()

# Result:
# 'completed'

plot_learning_curve()[source]#

Plot learning curves.

Note

Available only for Jupyter notebooks.

Example

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(credentials, ...)
prompt_tuner = experiment.prompt_tuner(...)
prompt_tuner.run(...)

prompt_tuner.plot_learning_curve()

run(training_data_references, training_results_reference=None, background_mode=False)[source]#

Run a prompt tuning process of foundation model on top of the training data referenced by DataConnection.

Parameters:

training_data_references (list[DataConnection]) – data storage connection details to inform where training data is stored
training_results_reference (DataConnection, optional) – data storage connection details to store pipeline training results
background_mode (bool, optional) – indicator if fit() method will run in background (async) or (sync)

Returns:

run details

Return type:

dict

Example

from ibm_watson_machine_learning.experiment import TuneExperiment
from ibm_watson_machine_learning.helpers import DataConnection, S3Location

experiment = TuneExperiment(credentials, ...)
prompt_tuner = experiment.prompt_tuner(...)

prompt_tuner.run(
    training_data_connection=[DataConnection(
        connection_asset_id=connection_id,
        location=S3Location(
            bucket='prompt_tuning_data',
            path='pt_train_data.json')
        )
    )]
    background_mode=False)

summary(scoring='loss')[source]#

Print PromptTuner models details (prompt-tuned models).

Parameters:: scoring (string, optional) – scoring metric which user wants to use to sort pipelines by, when not provided use loss one
Returns:: computed models and metrics
Return type:: pandas.DataFrame

Example

from ibm_watson_machine_learning.experiment import TuneExperiment

experiment = TuneExperiment(credentials, ...)
prompt_tuner = experiment.prompt_tuner(...)
prompt_tuner.run(...)

prompt_tuner.summary()

# Result:
#                          Enhancements            Base model  ...         loss
#       Model Name
# Prompt_tuned_M_1      [prompt_tuning]     google/flan-t5-xl  ...     0.449197

Enums#

class ibm_watson_machine_learning.foundation_models.utils.enums.PromptTuningTypes[source]#

Bases: object

PT = 'prompt_tuning'#

class ibm_watson_machine_learning.foundation_models.utils.enums.PromptTuningInitMethods[source]#

Bases: object

Supported methods for prompt initialization in prompt tuning.

RANDOM = 'random'#

TEXT = 'text'#

class ibm_watson_machine_learning.foundation_models.utils.enums.TuneExperimentTasks(value)[source]#

Bases: Enum

An enumeration.

CLASSIFICATION = 'classification'#

CODE_GENERATION_AND_CONVERSION = 'code'#

EXTRACTION = 'extraction'#

GENERATION = 'generation'#

QUESTION_ANSWERING = 'question_answering'#

RETRIEVAL_AUGMENTED_GENERATION = 'retrieval_augmented_generation'#

SUMMARIZATION = 'summarization'#