Model¶
- class ibm_watsonx_ai.foundation_models.Model(model_id, credentials, params=None, project_id=None, space_id=None, verify=None, validate=True)[source]¶
Bases:
ModelInference
Instantiate the model interface.
Hint
To use the Model class with LangChain, use the
to_langchain()
function.- Parameters:
model_id (str) – the type of model to use
credentials (ibm_watsonx_ai.Credentials or dict) – credentials to Watson Machine Learning instance
params (dict, optional) – parameters to use during generate requests
project_id (str, optional) – ID of the Watson Studio project
space_id (str, optional) – ID of the Watson Studio space
verify (bool or str, optional) –
user can pass as verify one of following:
the path to a CA_BUNDLE file
the path of directory with certificates of trusted CAs
True - default path to truststore will be taken
False - no verification will be made
validate (bool, optional) – Model id validation, defaults to True
Note
One of these parameters is required: [‘project_id ‘, ‘space_id’]
Hint
You can copy the project_id from Project’s Manage tab (Project -> Manage -> General -> Details).
Example
from ibm_watsonx_ai.foundation_models import Model from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes, DecodingMethods # To display example params enter GenParams().get_example_values() generate_params = { GenParams.MAX_NEW_TOKENS: 25 } model = Model( model_id=ModelTypes.FLAN_UL2, params=generate_params, credentials=Credentials( api_key = "***", url = "https://us-south.ml.cloud.ibm.com"), project_id="*****" )
- generate(prompt=None, params=None, guardrails=False, guardrails_hap_params=None, guardrails_pii_params=None, concurrency_limit=10, async_mode=False)[source]¶
Given a text prompt as input, and parameters the selected model (model_id) will generate a completion text as generated_text.
- Parameters:
params (dict) – meta props for text generation, use
ibm_watsonx_ai.metanames.GenTextParamsMetaNames().show()
to view the list of MetaNamesconcurrency_limit (int) – number of requests that will be sent in parallel, max is 10
prompt (str, list) – the prompt string or list of strings. If list of strings is passed requests will be managed in parallel with the rate of concurency_limit
guardrails (bool) – If True then potentially hateful, abusive, and/or profane language (HAP) detection filter is toggle on for both prompt and generated text, defaults to False
guardrails_hap_params – meta props for HAP moderations, use
ibm_watsonx_ai.metanames.GenTextModerationsMetaNames().show()
to view the list of MetaNamesasync_mode (bool) – If True then yield results asynchronously (using generator). In this case both prompt and generated text will be concatenated in the final response - under generated_text, defaults to False
- Returns:
scoring result containing generated content
- Return type:
dict
Example
q = "What is 1 + 1?" generated_response = model.generate(prompt=q) print(generated_response['results'][0]['generated_text'])
- generate_text(prompt=None, params=None, raw_response=False, guardrails=False, guardrails_hap_params=None, guardrails_pii_params=None, concurrency_limit=10)[source]¶
Given a text prompt as input, and parameters the selected model (model_id) will generate a completion text as generated_text.
- Parameters:
params (dict) – meta props for text generation, use
ibm_watsonx_ai.metanames.GenTextParamsMetaNames().show()
to view the list of MetaNamesconcurrency_limit (int) – number of requests that will be sent in parallel, max is 10
prompt (str, list) – the prompt string or list of strings. If list of strings is passed requests will be managed in parallel with the rate of concurency_limit
raw_response (bool, optional) – return the whole response object
guardrails (bool) – If True then potentially hateful, abusive, and/or profane language (HAP) detection filter is toggle on for both prompt and generated text, defaults to False
guardrails_hap_params – meta props for HAP moderations, use
ibm_watsonx_ai.metanames.GenTextModerationsMetaNames().show()
to view the list of MetaNames
- Returns:
generated content
- Return type:
str or dict
Example
q = "What is 1 + 1?" generated_text = model.generate_text(prompt=q) print(generated_text)
- generate_text_stream(prompt=None, params=None, raw_response=False, guardrails=False, guardrails_hap_params=None, guardrails_pii_params=None)[source]¶
Given a text prompt as input, and parameters the selected model (model_id) will generate a streamed text as generate_text_stream.
- Parameters:
params (dict) – meta props for text generation, use
ibm_watsonx_ai.metanames.GenTextParamsMetaNames().show()
to view the list of MetaNamesprompt (str,) – the prompt string
raw_response (bool, optional) – yields the whole response object
guardrails (bool) – If True then potentially hateful, abusive, and/or profane language (HAP) detection filter is toggle on for both prompt and generated text, defaults to False
guardrails_hap_params – meta props for HAP moderations, use
ibm_watsonx_ai.metanames.GenTextModerationsMetaNames().show()
to view the list of MetaNames
- Returns:
scoring result containing generated content
- Return type:
generator
Example
q = "Write an epigram about the sun" generated_response = model.generate_text_stream(prompt=q) for chunk in generated_response: print(chunk, end='')
- get_details()[source]¶
Get model’s details
- Returns:
model’s details
- Return type:
dict
Example
model.get_details()
- to_langchain()[source]¶
- Returns:
WatsonxLLM wrapper for watsonx foundation models
- Return type:
Example
from langchain import PromptTemplate from langchain.chains import LLMChain from ibm_watsonx_ai.foundation_models import Model from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes flan_ul2_model = Model( model_id=ModelTypes.FLAN_UL2, credentials=Credentials( api_key = "***", url = "https://us-south.ml.cloud.ibm.com"), project_id="*****" ) prompt_template = "What color is the {flower}?" llm_chain = LLMChain(llm=flan_ul2_model.to_langchain(), prompt=PromptTemplate.from_template(prompt_template)) llm_chain('sunflower')
- tokenize(prompt, return_tokens=False)[source]¶
The text tokenize operation allows you to check the conversion of provided input to tokens for a given model. It splits text into words or sub-words, which then are converted to ids through a look-up table (vocabulary). Tokenization allows the model to have a reasonable vocabulary size.
- Parameters:
prompt (str) – the prompt string
return_tokens (bool) – the parameter for text tokenization, defaults to False
- Returns:
the result of tokenizing the input string.
- Return type:
dict
Example
q = "Write an epigram about the moon" tokenized_response = model.tokenize(prompt=q, return_tokens=True) print(tokenized_response["result"])
Enums¶
- class metanames.GenTextParamsMetaNames[source]¶
Set of MetaNames for Foundation Model Parameters.
Available MetaNames:
MetaName
Type
Required
Example value
DECODING_METHOD
str
N
sample
LENGTH_PENALTY
dict
N
{'decay_factor': 2.5, 'start_index': 5}
TEMPERATURE
float
N
0.5
TOP_P
float
N
0.2
TOP_K
int
N
1
RANDOM_SEED
int
N
33
REPETITION_PENALTY
float
N
2
MIN_NEW_TOKENS
int
N
50
MAX_NEW_TOKENS
int
N
200
STOP_SEQUENCES
list
N
['fail']
TIME_LIMIT
int
N
600000
TRUNCATE_INPUT_TOKENS
int
N
200
RETURN_OPTIONS
dict
N
{'input_text': True, 'generated_tokens': True, 'input_tokens': True, 'token_logprobs': True, 'token_ranks': False, 'top_n_tokens': False}
- class metanames.GenTextReturnOptMetaNames[source]¶
Set of MetaNames for Foundation Model Parameters.
Available MetaNames:
MetaName
Type
Required
Example value
INPUT_TEXT
bool
Y
True
GENERATED_TOKENS
bool
N
True
INPUT_TOKENS
bool
Y
True
TOKEN_LOGPROBS
bool
N
True
TOKEN_RANKS
bool
N
True
TOP_N_TOKENS
int
N
True
Note
One of these parameters is required: [‘INPUT_TEXT’, ‘INPUT_TOKENS’]
- class ibm_watsonx_ai.foundation_models.utils.enums.DecodingMethods(value)[source]¶
Bases:
Enum
Supported decoding methods for text generation.
- GREEDY = 'greedy'¶
- SAMPLE = 'sample'¶
- class ibm_watsonx_ai.foundation_models.utils.enums.ModelTypes(value)[source]¶
Bases:
Enum
Supported foundation models.
Note
Current list of supported models types of various environments, user can check with
get_model_specs()
- CODELLAMA_34B_INSTRUCT_HF = 'codellama/codellama-34b-instruct-hf'¶
- ELYZA_JAPANESE_LLAMA_2_7B_INSTRUCT = 'elyza/elyza-japanese-llama-2-7b-instruct'¶
- FLAN_T5_XL = 'google/flan-t5-xl'¶
- FLAN_T5_XXL = 'google/flan-t5-xxl'¶
- FLAN_UL2 = 'google/flan-ul2'¶
- GPT_NEOX = 'eleutherai/gpt-neox-20b'¶
- GRANITE_13B_CHAT = 'ibm/granite-13b-chat-v1'¶
- GRANITE_13B_CHAT_V2 = 'ibm/granite-13b-chat-v2'¶
- GRANITE_13B_INSTRUCT = 'ibm/granite-13b-instruct-v1'¶
- GRANITE_13B_INSTRUCT_V2 = 'ibm/granite-13b-instruct-v2'¶
- GRANITE_20B_CODE_INSTRUCT = 'ibm/granite-20b-code-instruct'¶
- GRANITE_20B_MULTILINGUAL = 'ibm/granite-20b-multilingual'¶
- GRANITE_34B_CODE_INSTRUCT = 'ibm/granite-34b-code-instruct'¶
- GRANITE_3B_CODE_INSTRUCT = 'ibm/granite-3b-code-instruct'¶
- GRANITE_7B_LAB = 'ibm/granite-7b-lab'¶
- GRANITE_8B_CODE_INSTRUCT = 'ibm/granite-8b-code-instruct'¶
- LLAMA_2_13B_CHAT = 'meta-llama/llama-2-13b-chat'¶
- LLAMA_2_70B_CHAT = 'meta-llama/llama-2-70b-chat'¶
- LLAMA_3_70B_INSTRUCT = 'meta-llama/llama-3-70b-instruct'¶
- LLAMA_3_8B_INSTRUCT = 'meta-llama/llama-3-8b-instruct'¶
- MERLINITE_7B = 'ibm-mistralai/merlinite-7b'¶
- MIXTRAL_8X7B_INSTRUCT_V01 = 'mistralai/mixtral-8x7b-instruct-v01'¶
- MIXTRAL_8X7B_INSTRUCT_V01_Q = 'ibm-mistralai/mixtral-8x7b-instruct-v01-q'¶
- MPT_7B_INSTRUCT2 = 'ibm/mpt-7b-instruct2'¶
- MT0_XXL = 'bigscience/mt0-xxl'¶
- STARCODER = 'bigcode/starcoder'¶