ModelInference¶

class ibm_watsonx_ai.foundation_models.inference.ModelInference(*, model_id=None, deployment_id=None, params=None, credentials=None, project_id=None, space_id=None, verify=None, api_client=None, validate=True, persistent_connection=True)[source]¶

Bases: BaseModelInference

Instantiate the model interface.

Hint

To use the ModelInference class with LangChain, use the WatsonxLLM wrapper.

Parameters:

model_id (str, optional) – the type of model to use
deployment_id (str, optional) – ID of tuned model’s deployment
credentials (ibm_watsonx_ai.Credentials or dict, optional) – credentials to Watson Machine Learning instance
params (dict, optional) – parameters to use during generate requests
project_id (str, optional) – ID of the Watson Studio project
space_id (str, optional) – ID of the Watson Studio space
verify (bool or str, optional) –
user can pass as verify one of following:
- the path to a CA_BUNDLE file
- the path of directory with certificates of trusted CAs
- True - default path to truststore will be taken
- False - no verification will be made
api_client (APIClient, optional) – Initialized APIClient object with set project or space ID. If passed, credentials and project_id/space_id are not required.
validate (bool, optional) – Model id validation, defaults to True
persistent_connection (bool, optional) – Whether to keep persistent connection when evaluating generate, generate_text or tokenize methods. Note, this parameter is only applicable for mentioned methods when prompt is of type str. To close connection run model.close_persistent_connection(), defaults to True. Added in 1.1.2.

Note

One of these parameters is required: [model_id, deployment_id]

Note

One of these parameters is required: [project_id, space_id] when credentials parameter passed.

Hint

You can copy the project_id from the Project’s Manage tab (Project -> Manage -> General -> Details).

Example

from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import ModelInference
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes, DecodingMethods

# To display example params enter
GenParams().get_example_values()

generate_params = {
    GenParams.MAX_NEW_TOKENS: 25
}

model_inference = ModelInference(
    model_id=ModelTypes.FLAN_UL2,
    params=generate_params,
    credentials=Credentials(
                        api_key = "***",
                        url = "https://us-south.ml.cloud.ibm.com"),
    project_id="*****"
    )

from ibm_watsonx_ai.foundation_models import ModelInference
from ibm_watsonx_ai import Credentials

deployment_inference = ModelInference(
    deployment_id="<ID of deployed model>",
    credentials=Credentials(
                    api_key = "***",
                    url = "https://us-south.ml.cloud.ibm.com"),
    project_id="*****"
    )

close_persistent_connection()[source]¶: Only applicable if persistent_connection was set to True in ModelInference initialization.

generate(prompt=None, params=None, guardrails=False, guardrails_hap_params=None, guardrails_pii_params=None, concurrency_limit=10, async_mode=False, validate_prompt_variables=True)[source]¶

Given a text prompt as input, and parameters the selected model (model_id) or deployment (deployment_id) will generate a completion text as generated_text. For prompt template deployment prompt should be None.

Parameters:

params (dict) – meta props for text generation, use ibm_watsonx_ai.metanames.GenTextParamsMetaNames().show() to view the list of MetaNames
concurrency_limit (int) – number of requests that will be sent in parallel, max is 10
prompt ((str | list | None), optional) – the prompt string or list of strings. If list of strings is passed requests will be managed in parallel with the rate of concurency_limit, defaults to None
guardrails (bool) – If True then potentially hateful, abusive, and/or profane language (HAP) detection filter is toggle on for both prompt and generated text, defaults to False
guardrails_hap_params (dict) – meta props for HAP moderations, use ibm_watsonx_ai.metanames.GenTextModerationsMetaNames().show() to view the list of MetaNames
async_mode (bool) – If True then yield results asynchronously (using generator). In this case both prompt and generated text will be concatenated in the final response - under generated_text, defaults to False
validate_prompt_variables (bool, optional) – If True, prompt variables provided in params are validated with the ones in Prompt Template Asset. This parameter is only applicable in a Prompt Template Asset deployment scenario and should not be changed for different cases, defaults to True

Returns:

scoring result containing generated content

Return type:

dict

Example

q = "What is 1 + 1?"
generated_response = model_inference.generate(prompt=q)
print(generated_response['results'][0]['generated_text'])

generate_text(prompt=None, params=None, raw_response=False, guardrails=False, guardrails_hap_params=None, guardrails_pii_params=None, concurrency_limit=10, validate_prompt_variables=True)[source]¶

Given a text prompt as input, and parameters the selected model (model_id) will generate a completion text as generated_text. For prompt template deployment prompt should be None.

Parameters:

params (dict) – meta props for text generation, use ibm_watsonx_ai.metanames.GenTextParamsMetaNames().show() to view the list of MetaNames
concurrency_limit (int) – number of requests that will be sent in parallel, max is 10
prompt ((str | list | None), optional) – the prompt string or list of strings. If list of strings is passed requests will be managed in parallel with the rate of concurency_limit, defaults to None
guardrails (bool) – If True then potentially hateful, abusive, and/or profane language (HAP) detection filter is toggle on for both prompt and generated text, defaults to False If HAP is detected the HAPDetectionWarning is issued
guardrails_hap_params (dict) – meta props for HAP moderations, use ibm_watsonx_ai.metanames.GenTextModerationsMetaNames().show() to view the list of MetaNames
raw_response (bool, optional) – return the whole response object
validate_prompt_variables (bool) – If True, prompt variables provided in params are validated with the ones in Prompt Template Asset. This parameter is only applicable in a Prompt Template Asset deployment scenario and should not be changed for different cases, defaults to True

Returns:

generated content

Return type:

str | list | dict

Note

By default only the first occurrence of HAPDetectionWarning is displayed. To enable printing all warnings of this category, use:

import warnings
from ibm_watsonx_ai.foundation_models.utils import HAPDetectionWarning

warnings.filterwarnings("always", category=HAPDetectionWarning)

Example

q = "What is 1 + 1?"
generated_text = model_inference.generate_text(prompt=q)
print(generated_text)

generate_text_stream(prompt=None, params=None, raw_response=False, guardrails=False, guardrails_hap_params=None, guardrails_pii_params=None, validate_prompt_variables=True)[source]¶

Given a text prompt as input, and parameters the selected model (model_id) will generate a streamed text as generate_text_stream. For prompt template deployment prompt should be None.

Parameters:

params (dict) – meta props for text generation, use ibm_watsonx_ai.metanames.GenTextParamsMetaNames().show() to view the list of MetaNames
prompt (str, optional) – the prompt string, defaults to None
raw_response (bool, optional) – yields the whole response object
guardrails (bool) – If True then potentially hateful, abusive, and/or profane language (HAP) detection filter is toggle on for both prompt and generated text, defaults to False If HAP is detected the HAPDetectionWarning is issued
guardrails_hap_params (dict) – meta props for HAP moderations, use ibm_watsonx_ai.metanames.GenTextModerationsMetaNames().show() to view the list of MetaNames
validate_prompt_variables (bool) – If True, prompt variables provided in params are validated with the ones in Prompt Template Asset. This parameter is only applicable in a Prompt Template Asset deployment scenario and should not be changed for different cases, defaults to True

Returns:

scoring result containing generated content

Return type:

generator

Note

By default only the first occurrence of HAPDetectionWarning is displayed. To enable printing all warnings of this category, use:

import warnings
from ibm_watsonx_ai.foundation_models.utils import HAPDetectionWarning

warnings.filterwarnings("always", category=HAPDetectionWarning)

Example

q = "Write an epigram about the sun"
generated_response = model_inference.generate_text_stream(prompt=q)

for chunk in generated_response:
    print(chunk, end='', flush=True)

get_details()[source]¶

Get model interface’s details

Returns:: details of model or deployment
Return type:: dict

Example

model_inference.get_details()

get_identifying_params()[source]¶: Represent Model Inference’s setup in dictionary

to_langchain()[source]¶

Returns:: WatsonxLLM wrapper for watsonx foundation models
Return type:: WatsonxLLM

Example

from langchain import PromptTemplate
from langchain.chains import LLMChain
from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import ModelInference
from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes

flan_ul2_model = ModelInference(
    model_id=ModelTypes.FLAN_UL2,
    credentials=Credentials(
                        api_key = "***",
                        url = "https://us-south.ml.cloud.ibm.com"),
    project_id="*****"
    )

prompt_template = "What color is the {flower}?"

llm_chain = LLMChain(llm=flan_ul2_model.to_langchain(), prompt=PromptTemplate.from_template(prompt_template))
llm_chain('sunflower')

from langchain import PromptTemplate
from langchain.chains import LLMChain
from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.foundation_models import ModelInference
from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes

deployed_model = ModelInference(
    deployment_id="<ID of deployed model>",
    credentials=Credentials(
                        api_key = "***",
                        url = "https://us-south.ml.cloud.ibm.com"),
    space_id="*****"
    )

prompt_template = "What color is the {car}?"

llm_chain = LLMChain(llm=deployed_model.to_langchain(), prompt=PromptTemplate.from_template(prompt_template))
llm_chain('sunflower')

tokenize(prompt, return_tokens=False)[source]¶

The text tokenize operation allows you to check the conversion of provided input to tokens for a given model. It splits text into words or sub-words, which then are converted to ids through a look-up table (vocabulary). Tokenization allows the model to have a reasonable vocabulary size.

Note

Method is not supported for deployments, available only for base models.

Parameters:

prompt (str, optional) – the prompt string, defaults to None
return_tokens (bool) – the parameter for text tokenization, defaults to False

Returns:

the result of tokenizing the input string.

Return type:

dict

Example

q = "Write an epigram about the moon"
tokenized_response = model_inference.tokenize(prompt=q, return_tokens=True)
print(tokenized_response["result"])