ModelInference¶
- class ibm_watsonx_ai.foundation_models.inference.ModelInference(*, model_id=None, deployment_id=None, params=None, credentials=None, project_id=None, space_id=None, verify=None, api_client=None, validate=True, persistent_connection=True)[source]¶
Bases:
BaseModelInference
Instantiate the model interface.
Hint
To use the ModelInference class with LangChain, use the
WatsonxLLM
wrapper.- Parameters:
model_id (str, optional) – the type of model to use
deployment_id (str, optional) – ID of tuned model’s deployment
credentials (ibm_watsonx_ai.Credentials or dict, optional) – credentials to Watson Machine Learning instance
params (dict, optional) – parameters to use during generate requests
project_id (str, optional) – ID of the Watson Studio project
space_id (str, optional) – ID of the Watson Studio space
verify (bool or str, optional) –
user can pass as verify one of following:
the path to a CA_BUNDLE file
the path of directory with certificates of trusted CAs
True - default path to truststore will be taken
False - no verification will be made
api_client (APIClient, optional) – Initialized APIClient object with set project or space ID. If passed,
credentials
andproject_id
/space_id
are not required.validate (bool, optional) – Model id validation, defaults to True
persistent_connection (bool, optional) – Whether to keep persistent connection when evaluating generate, generate_text or tokenize methods. Note, this parameter is only applicable for mentioned methods when prompt is of type str. To close connection run model.close_persistent_connection(), defaults to True. Added in 1.1.2.
Note
One of these parameters is required: [
model_id
,deployment_id
]Note
One of these parameters is required: [
project_id
,space_id
] whencredentials
parameter passed.Hint
You can copy the project_id from the Project’s Manage tab (Project -> Manage -> General -> Details).
Example
from ibm_watsonx_ai import Credentials from ibm_watsonx_ai.foundation_models import ModelInference from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes, DecodingMethods # To display example params enter GenParams().get_example_values() generate_params = { GenParams.MAX_NEW_TOKENS: 25 } model_inference = ModelInference( model_id=ModelTypes.FLAN_UL2, params=generate_params, credentials=Credentials( api_key = "***", url = "https://us-south.ml.cloud.ibm.com"), project_id="*****" )
from ibm_watsonx_ai.foundation_models import ModelInference from ibm_watsonx_ai import Credentials deployment_inference = ModelInference( deployment_id="<ID of deployed model>", credentials=Credentials( api_key = "***", url = "https://us-south.ml.cloud.ibm.com"), project_id="*****" )
- close_persistent_connection()[source]¶
Only applicable if persistent_connection was set to True in ModelInference initialization.
- generate(prompt=None, params=None, guardrails=False, guardrails_hap_params=None, guardrails_pii_params=None, concurrency_limit=10, async_mode=False, validate_prompt_variables=True)[source]¶
Given a text prompt as input, and parameters the selected model (model_id) or deployment (deployment_id) will generate a completion text as generated_text. For prompt template deployment prompt should be None.
- Parameters:
params (dict) – meta props for text generation, use
ibm_watsonx_ai.metanames.GenTextParamsMetaNames().show()
to view the list of MetaNamesconcurrency_limit (int) – number of requests that will be sent in parallel, max is 10
prompt ((str | list | None), optional) – the prompt string or list of strings. If list of strings is passed requests will be managed in parallel with the rate of concurency_limit, defaults to None
guardrails (bool) – If True then potentially hateful, abusive, and/or profane language (HAP) detection filter is toggle on for both prompt and generated text, defaults to False
guardrails_hap_params (dict) – meta props for HAP moderations, use
ibm_watsonx_ai.metanames.GenTextModerationsMetaNames().show()
to view the list of MetaNamesasync_mode (bool) – If True then yield results asynchronously (using generator). In this case both prompt and generated text will be concatenated in the final response - under generated_text, defaults to False
validate_prompt_variables (bool, optional) – If True, prompt variables provided in params are validated with the ones in Prompt Template Asset. This parameter is only applicable in a Prompt Template Asset deployment scenario and should not be changed for different cases, defaults to True
- Returns:
scoring result containing generated content
- Return type:
dict
Example
q = "What is 1 + 1?" generated_response = model_inference.generate(prompt=q) print(generated_response['results'][0]['generated_text'])
- generate_text(prompt=None, params=None, raw_response=False, guardrails=False, guardrails_hap_params=None, guardrails_pii_params=None, concurrency_limit=10, validate_prompt_variables=True)[source]¶
Given a text prompt as input, and parameters the selected model (model_id) will generate a completion text as generated_text. For prompt template deployment prompt should be None.
- Parameters:
params (dict) – meta props for text generation, use
ibm_watsonx_ai.metanames.GenTextParamsMetaNames().show()
to view the list of MetaNamesconcurrency_limit (int) – number of requests that will be sent in parallel, max is 10
prompt ((str | list | None), optional) – the prompt string or list of strings. If list of strings is passed requests will be managed in parallel with the rate of concurency_limit, defaults to None
guardrails (bool) – If True then potentially hateful, abusive, and/or profane language (HAP) detection filter is toggle on for both prompt and generated text, defaults to False If HAP is detected the HAPDetectionWarning is issued
guardrails_hap_params (dict) – meta props for HAP moderations, use
ibm_watsonx_ai.metanames.GenTextModerationsMetaNames().show()
to view the list of MetaNamesraw_response (bool, optional) – return the whole response object
validate_prompt_variables (bool) – If True, prompt variables provided in params are validated with the ones in Prompt Template Asset. This parameter is only applicable in a Prompt Template Asset deployment scenario and should not be changed for different cases, defaults to True
- Returns:
generated content
- Return type:
str | list | dict
Note
By default only the first occurrence of HAPDetectionWarning is displayed. To enable printing all warnings of this category, use:
import warnings from ibm_watsonx_ai.foundation_models.utils import HAPDetectionWarning warnings.filterwarnings("always", category=HAPDetectionWarning)
Example
q = "What is 1 + 1?" generated_text = model_inference.generate_text(prompt=q) print(generated_text)
- generate_text_stream(prompt=None, params=None, raw_response=False, guardrails=False, guardrails_hap_params=None, guardrails_pii_params=None, validate_prompt_variables=True)[source]¶
Given a text prompt as input, and parameters the selected model (model_id) will generate a streamed text as generate_text_stream. For prompt template deployment prompt should be None.
- Parameters:
params (dict) – meta props for text generation, use
ibm_watsonx_ai.metanames.GenTextParamsMetaNames().show()
to view the list of MetaNamesprompt (str, optional) – the prompt string, defaults to None
raw_response (bool, optional) – yields the whole response object
guardrails (bool) – If True then potentially hateful, abusive, and/or profane language (HAP) detection filter is toggle on for both prompt and generated text, defaults to False If HAP is detected the HAPDetectionWarning is issued
guardrails_hap_params (dict) – meta props for HAP moderations, use
ibm_watsonx_ai.metanames.GenTextModerationsMetaNames().show()
to view the list of MetaNamesvalidate_prompt_variables (bool) – If True, prompt variables provided in params are validated with the ones in Prompt Template Asset. This parameter is only applicable in a Prompt Template Asset deployment scenario and should not be changed for different cases, defaults to True
- Returns:
scoring result containing generated content
- Return type:
generator
Note
By default only the first occurrence of HAPDetectionWarning is displayed. To enable printing all warnings of this category, use:
import warnings from ibm_watsonx_ai.foundation_models.utils import HAPDetectionWarning warnings.filterwarnings("always", category=HAPDetectionWarning)
Example
q = "Write an epigram about the sun" generated_response = model_inference.generate_text_stream(prompt=q) for chunk in generated_response: print(chunk, end='', flush=True)
- get_details()[source]¶
Get model interface’s details
- Returns:
details of model or deployment
- Return type:
dict
Example
model_inference.get_details()
- to_langchain()[source]¶
- Returns:
WatsonxLLM wrapper for watsonx foundation models
- Return type:
Example
from langchain import PromptTemplate from langchain.chains import LLMChain from ibm_watsonx_ai import Credentials from ibm_watsonx_ai.foundation_models import ModelInference from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes flan_ul2_model = ModelInference( model_id=ModelTypes.FLAN_UL2, credentials=Credentials( api_key = "***", url = "https://us-south.ml.cloud.ibm.com"), project_id="*****" ) prompt_template = "What color is the {flower}?" llm_chain = LLMChain(llm=flan_ul2_model.to_langchain(), prompt=PromptTemplate.from_template(prompt_template)) llm_chain('sunflower')
from langchain import PromptTemplate from langchain.chains import LLMChain from ibm_watsonx_ai import Credentials from ibm_watsonx_ai.foundation_models import ModelInference from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes deployed_model = ModelInference( deployment_id="<ID of deployed model>", credentials=Credentials( api_key = "***", url = "https://us-south.ml.cloud.ibm.com"), space_id="*****" ) prompt_template = "What color is the {car}?" llm_chain = LLMChain(llm=deployed_model.to_langchain(), prompt=PromptTemplate.from_template(prompt_template)) llm_chain('sunflower')
- tokenize(prompt, return_tokens=False)[source]¶
The text tokenize operation allows you to check the conversion of provided input to tokens for a given model. It splits text into words or sub-words, which then are converted to ids through a look-up table (vocabulary). Tokenization allows the model to have a reasonable vocabulary size.
Note
Method is not supported for deployments, available only for base models.
- Parameters:
prompt (str, optional) – the prompt string, defaults to None
return_tokens (bool) – the parameter for text tokenization, defaults to False
- Returns:
the result of tokenizing the input string.
- Return type:
dict
Example
q = "Write an epigram about the moon" tokenized_response = model_inference.tokenize(prompt=q, return_tokens=True) print(tokenized_response["result"])