Gateway Inference¶

GatewayInference¶

class ibm_watsonx_ai.gateway.GatewayInference(*, model, temperature=None, max_tokens=None, top_p=None, n=None, stop=None, frequency_penalty=None, presence_penalty=None, logit_bias=None, seed=None, stream_options=None, metadata=None, user=None, router=None, cache=None, logprobs=None, max_completion_tokens=None, top_logprobs=None, reasoning_effort=None, tools=None, tool_choice=None, parallel_tool_calls=None, function_call=None, functions=None, response_format=None, modalities=None, audio=None, store=None, service_tier=None, prediction=None, best_of=None, echo=None, suffix=None, credentials=None, api_client=None, project_id=None, space_id=None, verify=None, max_retries=None, delay_time=None, retry_status_codes=None, **kwargs)[source]¶

Bases: WMLResource

Instantiate the AI Gateway model interface.

Parameters:: model (str) – type of model to use

Note

The parameters below set default values applied to every request made through this instance. They can be overridden per-call by passing the same keyword argument directly to chat / generate (and their async / streaming variants).

Shared parameters (chat + generate)

Parameters:

temperature (float, optional) – sampling temperature between 0 and 2; higher values make output more random, lower values make it more focused and deterministic
max_tokens (int, optional) – maximum number of tokens that can be generated in the completion
top_p (float, optional) – nucleus sampling probability mass; alternative to temperature
n (int, optional) – how many completion choices to generate for each input
stop (dict or list[str], optional) – stop sequence(s) — the API will stop generating further tokens when any of these sequences is encountered
frequency_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on their existing frequency in the text so far
presence_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on whether they appear in the text so far
logit_bias (dict, optional) – modifies the likelihood of specified tokens appearing in the completion
seed (int, optional) – seed for deterministic sampling
stream_options (dict, optional) – options for streaming responses
metadata (dict, optional) – metadata to store with the request
user (str, optional) – unique identifier representing your end-user
router (dict, optional) – model routing configuration for the request
cache (dict, optional) – caching configuration for the request
logprobs (bool or int, optional) – log probabilities of the output tokens; pass True/False for chat endpoints or an integer (0–5) for generate endpoints

Chat-only parameters (chat / chat_stream / achat / achat_stream)

Parameters:

max_completion_tokens (int, optional) – upper bound for the number of tokens that can be generated, including reasoning tokens; preferred over max_tokens for o-series models
top_logprobs (int, optional) – integer between 0 and 20 specifying the number of most likely tokens to return at each token position; requires logprobs=True
reasoning_effort (dict, optional) – reasoning effort configuration for the request
tools (list[dict], optional) – list of tools the model may call; currently only functions are supported as tools; use this to provide a list of functions the model may generate JSON inputs for
tool_choice (dict, optional) – controls which tool is called by the model
parallel_tool_calls (bool, optional) – whether to enable parallel function calling during tool use
function_call (dict, optional) – controls which (if any) function is called by the model (deprecated in favor of tool_choice)
functions (dict, optional) – list of functions the model may generate JSON inputs for (deprecated in favor of tools)
response_format (dict, optional) – object specifying the format that the model must output (e.g. JSON mode or structured outputs)
modalities (list[str], optional) – output types that the model should generate, e.g. ["text"] or ["text", "audio"]
audio (dict, optional) – parameters for audio output, required when audio output is requested with modalities ["audio"]
store (bool, optional) – whether to store the output of this chat completion for use in model distillation or evals
service_tier (dict, optional) – service tier configuration for the request
prediction (dict, optional) – prediction configuration for the request

Generate-only parameters (generate / generate_stream / agenerate / agenerate_stream)

Parameters:

best_of (int, optional) – generates best_of completions server-side and returns the best one; must be greater than n when used together; incompatible with streaming
echo (bool, optional) – whether to echo back the prompt in addition to the completion
suffix (str, optional) – text that comes after a completion of inserted text

SDK / transport parameters

Parameters:

credentials (Credentials or dict, optional) – credentials for the watsonx.ai instance
api_client (APIClient, optional) – initialized APIClient object with a set project ID or space ID. If passed, credentials and project_id/space_id are not required.
project_id (str, optional) – ID of the Watson Studio project
space_id (str, optional) – ID of the Watson Studio space
verify (bool or str or Path, optional) –
You can pass one of the following as verify:
- the path to a CA_BUNDLE file
- the path of directory with certificates of trusted CAs
- True — default path to truststore will be taken
- False — no verification will be made
max_retries (int, optional) – number of retries performed when request was not successful and status code is in retry_status_codes, defaults to 10
delay_time (float, optional) – delay time to retry request, factor in exponential backoff formula: wx_delay_time * pow(2.0, attempt), defaults to 0.5s
retry_status_codes (list[int], optional) – list of status codes which will be considered for retry mechanism, defaults to [429, 503, 504, 520]

async achat(messages, *, temperature=None, max_completion_tokens=None, max_tokens=None, top_p=None, n=None, stop=None, frequency_penalty=None, presence_penalty=None, logprobs=None, top_logprobs=None, logit_bias=None, seed=None, reasoning_effort=None, tools=None, tool_choice=None, parallel_tool_calls=None, function_call=None, functions=None, response_format=None, modalities=None, audio=None, stream_options=None, store=None, metadata=None, user=None, service_tier=None, prediction=None, router=None, cache=None, **kwargs)[source]¶

Generate chat completions asynchronously.

Parameters:

messages (list[dict]) – messages to be processed during call
temperature (float, optional) – sampling temperature between 0 and 2; higher values make output more random
max_completion_tokens (int, optional) – upper bound for the number of tokens that can be generated for a completion, including reasoning tokens
max_tokens (int, optional) – maximum number of tokens that can be generated in the chat completion (deprecated in favor of max_completion_tokens)
top_p (float, optional) – nucleus sampling probability mass; alternative to temperature
n (int, optional) – how many chat completion choices to generate for each input message
stop (dict, optional) – stop sequence configuration for the request
frequency_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on their existing frequency in the text so far
presence_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on whether they appear in the text so far
logprobs (bool, optional) – whether to return log probabilities of the output tokens
top_logprobs (int, optional) – integer between 0 and 20 specifying the number of most likely tokens to return at each token position
logit_bias (dict, optional) – modifies the likelihood of specified tokens appearing in the completion
seed (int, optional) – seed for deterministic sampling (Beta for OpenAI)
reasoning_effort (dict, optional) – reasoning effort configuration for the request
tools (list[dict], optional) – list of tools the model may call; currently only functions are supported as tools; use this to provide a list of functions the model may generate JSON inputs for
tool_choice (dict, optional) – controls which tool is called by the model
parallel_tool_calls (bool, optional) – whether to enable parallel function calling during tool use
function_call (dict, optional) – controls which (if any) function is called by the model (deprecated in favor of tool_choice)
functions (dict, optional) – list of functions the model may generate JSON inputs for (deprecated in favor of tools)
response_format (dict, optional) – object specifying the format that the model must output (e.g. JSON mode or structured outputs)
modalities (list[str], optional) – output types that the model should generate, e.g. ["text"] or ["text", "audio"]
audio (dict, optional) – parameters for audio output, required when audio output is requested with modalities ["audio"]
stream_options (dict, optional) – options for streaming responses
store (bool, optional) – whether to store the output of this chat completion for use in model distillation or evals
metadata (dict, optional) – metadata for the request
user (str, optional) – unique identifier representing your end-user
service_tier (dict, optional) – service tier configuration for the request
prediction (dict, optional) – prediction configuration for the request
router (dict, optional) – model routing configurations for the request
cache (dict, optional) – caching configuration for the request
kwargs (Any) – additional keyword arguments passed directly to the gateway

Returns:

model answer

Return type:

dict

async achat_stream(messages, *, temperature=None, max_completion_tokens=None, max_tokens=None, top_p=None, n=None, stop=None, frequency_penalty=None, presence_penalty=None, logprobs=None, top_logprobs=None, logit_bias=None, seed=None, reasoning_effort=None, tools=None, tool_choice=None, parallel_tool_calls=None, function_call=None, functions=None, response_format=None, modalities=None, audio=None, stream_options=None, store=None, metadata=None, user=None, service_tier=None, prediction=None, router=None, cache=None, **kwargs)[source]¶

Generate chat completions asynchronously with streaming.

Parameters:

messages (list[dict]) – messages to be processed during call
temperature (float, optional) – sampling temperature between 0 and 2; higher values make output more random
max_completion_tokens (int, optional) – upper bound for the number of tokens that can be generated for a completion, including reasoning tokens
max_tokens (int, optional) – maximum number of tokens that can be generated in the chat completion (deprecated in favor of max_completion_tokens)
top_p (float, optional) – nucleus sampling probability mass; alternative to temperature
n (int, optional) – how many chat completion choices to generate for each input message
stop (dict, optional) – stop sequence configuration for the request
frequency_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on their existing frequency in the text so far
presence_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on whether they appear in the text so far
logprobs (bool, optional) – whether to return log probabilities of the output tokens
top_logprobs (int, optional) – integer between 0 and 20 specifying the number of most likely tokens to return at each token position
logit_bias (dict, optional) – modifies the likelihood of specified tokens appearing in the completion
seed (int, optional) – seed for deterministic sampling (Beta for OpenAI)
reasoning_effort (dict, optional) – reasoning effort configuration for the request
tools (list[dict], optional) – list of tools the model may call; currently only functions are supported as tools; use this to provide a list of functions the model may generate JSON inputs for
tool_choice (dict, optional) – controls which tool is called by the model
parallel_tool_calls (bool, optional) – whether to enable parallel function calling during tool use
function_call (dict, optional) – controls which (if any) function is called by the model (deprecated in favor of tool_choice)
functions (dict, optional) – list of functions the model may generate JSON inputs for (deprecated in favor of tools)
response_format (dict, optional) – object specifying the format that the model must output (e.g. JSON mode or structured outputs)
modalities (list[str], optional) – output types that the model should generate, e.g. ["text"] or ["text", "audio"]
audio (dict, optional) – parameters for audio output, required when audio output is requested with modalities ["audio"]
stream_options (dict, optional) – options for streaming responses
store (bool, optional) – whether to store the output of this chat completion for use in model distillation or evals
metadata (dict, optional) – metadata for the request
user (str, optional) – unique identifier representing your end-user
service_tier (dict, optional) – service tier configuration for the request
prediction (dict, optional) – prediction configuration for the request
router (dict, optional) – model routing configurations for the request
cache (dict, optional) – caching configuration for the request
kwargs (Any) – additional keyword arguments passed directly to the gateway

Returns:

async iterator of model response chunks

Return type:

AsyncIterator

async agenerate(prompt, *, temperature=None, max_tokens=None, top_p=None, n=None, best_of=None, stop=None, frequency_penalty=None, presence_penalty=None, logprobs=None, logit_bias=None, seed=None, echo=None, suffix=None, stream_options=None, metadata=None, user=None, router=None, cache=None, **kwargs)[source]¶

Generate text completions asynchronously.

Parameters:

prompt (str or list[str] or list[int]) – prompt to be processed during call
temperature (float, optional) – sampling temperature between 0 and 2; higher values make output more random, lower values make it more focused and deterministic
max_tokens (int, optional) – maximum number of tokens that can be generated in the completion; token count of prompt plus max_tokens cannot exceed the model’s context length
top_p (float, optional) – nucleus sampling probability mass; alternative to temperature
n (int, optional) – how many completions to generate for each prompt
best_of (int, optional) – generates best_of completions server-side and returns the best one; must be greater than n when used together; cannot be used with streaming
stop (list[str], optional) – up to 4 sequences where the API will stop generating further tokens; the returned text will not contain the stop sequence
frequency_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on their existing frequency in the text so far
presence_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on whether they appear in the text so far
logprobs (int, optional) – number of most likely output tokens (0–5) to include log probabilities for
logit_bias (dict, optional) – modifies the likelihood of specified tokens appearing in the completion; maps token IDs to a bias value from -100 to 100
seed (int, optional) – seed for deterministic sampling; repeated requests with the same seed and parameters should return the same result
echo (bool, optional) – whether to echo back the prompt in addition to the completion
suffix (str, optional) – text that comes after a completion of inserted text
stream_options (dict, optional) – options for streaming responses; only applicable when stream=True
metadata (dict, optional) – metadata to store with the completion
user (str, optional) – unique identifier representing your end-user
router (dict, optional) – model routing configuration for the request
cache (dict, optional) – caching configuration for the request; only supported for non-streaming requests
kwargs (Any) – additional keyword arguments passed directly to the gateway

Returns:

model answer

Return type:

dict

async agenerate_stream(prompt, *, temperature=None, max_tokens=None, top_p=None, n=None, stop=None, frequency_penalty=None, presence_penalty=None, logprobs=None, logit_bias=None, seed=None, echo=None, suffix=None, stream_options=None, metadata=None, user=None, router=None, **kwargs)[source]¶

Generate text completions asynchronously with streaming.

Parameters:

prompt (str or list[str] or list[int]) – prompt to be processed during call
temperature (float, optional) – sampling temperature between 0 and 2; higher values make output more random, lower values make it more focused and deterministic
max_tokens (int, optional) – maximum number of tokens that can be generated in the completion; token count of prompt plus max_tokens cannot exceed the model’s context length
top_p (float, optional) – nucleus sampling probability mass; alternative to temperature
n (int, optional) – how many completions to generate for each prompt
stop (list[str], optional) – up to 4 sequences where the API will stop generating further tokens; the returned text will not contain the stop sequence
frequency_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on their existing frequency in the text so far
presence_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on whether they appear in the text so far
logprobs (int, optional) – number of most likely output tokens (0–5) to include log probabilities for
logit_bias (dict, optional) – modifies the likelihood of specified tokens appearing in the completion; maps token IDs to a bias value from -100 to 100
seed (int, optional) – seed for deterministic sampling; repeated requests with the same seed and parameters should return the same result
echo (bool, optional) – whether to echo back the prompt in addition to the completion
suffix (str, optional) – text that comes after a completion of inserted text
stream_options (dict, optional) – options for streaming responses
metadata (dict, optional) – metadata to store with the completion
user (str, optional) – unique identifier representing your end-user
router (dict, optional) – model routing configuration for the request
kwargs (Any) – additional keyword arguments passed directly to the gateway

Returns:

async iterator of model response chunks

Return type:

AsyncIterator

chat(messages, *, temperature=None, max_completion_tokens=None, max_tokens=None, top_p=None, n=None, stop=None, frequency_penalty=None, presence_penalty=None, logprobs=None, top_logprobs=None, logit_bias=None, seed=None, reasoning_effort=None, tools=None, tool_choice=None, parallel_tool_calls=None, function_call=None, functions=None, response_format=None, modalities=None, audio=None, stream_options=None, store=None, metadata=None, user=None, service_tier=None, prediction=None, router=None, cache=None, **kwargs)[source]¶

Generate chat completions.

Parameters:

messages (list[dict]) – messages to be processed during call
temperature (float, optional) – sampling temperature between 0 and 2; higher values make output more random
max_completion_tokens (int, optional) – upper bound for the number of tokens that can be generated for a completion, including reasoning tokens
max_tokens (int, optional) – maximum number of tokens that can be generated in the chat completion (deprecated in favor of max_completion_tokens)
top_p (float, optional) – nucleus sampling probability mass; alternative to temperature
n (int, optional) – how many chat completion choices to generate for each input message
stop (dict, optional) – stop sequence configuration for the request
frequency_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on their existing frequency in the text so far
presence_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on whether they appear in the text so far
logprobs (bool, optional) – whether to return log probabilities of the output tokens
top_logprobs (int, optional) – integer between 0 and 20 specifying the number of most likely tokens to return at each token position
logit_bias (dict, optional) – modifies the likelihood of specified tokens appearing in the completion
seed (int, optional) – seed for deterministic sampling (Beta for OpenAI)
reasoning_effort (dict, optional) – reasoning effort configuration for the request
tools (list[dict], optional) – list of tools the model may call; currently only functions are supported as tools; use this to provide a list of functions the model may generate JSON inputs for
tool_choice (dict, optional) – controls which tool is called by the model
parallel_tool_calls (bool, optional) – whether to enable parallel function calling during tool use
function_call (dict, optional) – controls which (if any) function is called by the model (deprecated in favor of tool_choice)
functions (dict, optional) – list of functions the model may generate JSON inputs for (deprecated in favor of tools)
response_format (dict, optional) – object specifying the format that the model must output (e.g. JSON mode or structured outputs)
modalities (list[str], optional) – output types that the model should generate, e.g. ["text"] or ["text", "audio"]
audio (dict, optional) – parameters for audio output, required when audio output is requested with modalities ["audio"]
stream_options (dict, optional) – options for streaming responses
store (bool, optional) – whether to store the output of this chat completion for use in model distillation or evals
metadata (dict, optional) – metadata for the request
user (str, optional) – unique identifier representing your end-user
service_tier (dict, optional) – service tier configuration for the request
prediction (dict, optional) – prediction configuration for the request
router (dict, optional) – model routing configurations for the request
cache (dict, optional) – caching configuration for the request
kwargs (Any) – additional keyword arguments passed directly to the gateway

Returns:

model answer

Return type:

dict

chat_stream(messages, *, temperature=None, max_completion_tokens=None, max_tokens=None, top_p=None, n=None, stop=None, frequency_penalty=None, presence_penalty=None, logprobs=None, top_logprobs=None, logit_bias=None, seed=None, reasoning_effort=None, tools=None, tool_choice=None, parallel_tool_calls=None, function_call=None, functions=None, response_format=None, modalities=None, audio=None, stream_options=None, store=None, metadata=None, user=None, service_tier=None, prediction=None, router=None, cache=None, **kwargs)[source]¶

Generate chat completions with streaming.

Parameters:

messages (list[dict]) – messages to be processed during call
temperature (float, optional) – sampling temperature between 0 and 2; higher values make output more random
max_completion_tokens (int, optional) – upper bound for the number of tokens that can be generated for a completion, including reasoning tokens
max_tokens (int, optional) – maximum number of tokens that can be generated in the chat completion (deprecated in favor of max_completion_tokens)
top_p (float, optional) – nucleus sampling probability mass; alternative to temperature
n (int, optional) – how many chat completion choices to generate for each input message
stop (dict, optional) – stop sequence configuration for the request
frequency_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on their existing frequency in the text so far
presence_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on whether they appear in the text so far
logprobs (bool, optional) – whether to return log probabilities of the output tokens
top_logprobs (int, optional) – integer between 0 and 20 specifying the number of most likely tokens to return at each token position
logit_bias (dict, optional) – modifies the likelihood of specified tokens appearing in the completion
seed (int, optional) – seed for deterministic sampling (Beta for OpenAI)
reasoning_effort (dict, optional) – reasoning effort configuration for the request
tools (list[dict], optional) – list of tools the model may call; currently only functions are supported as tools; use this to provide a list of functions the model may generate JSON inputs for
tool_choice (dict, optional) – controls which tool is called by the model
parallel_tool_calls (bool, optional) – whether to enable parallel function calling during tool use
function_call (dict, optional) – controls which (if any) function is called by the model (deprecated in favor of tool_choice)
functions (dict, optional) – list of functions the model may generate JSON inputs for (deprecated in favor of tools)
response_format (dict, optional) – object specifying the format that the model must output (e.g. JSON mode or structured outputs)
modalities (list[str], optional) – output types that the model should generate, e.g. ["text"] or ["text", "audio"]
audio (dict, optional) – parameters for audio output, required when audio output is requested with modalities ["audio"]
stream_options (dict, optional) – options for streaming responses
store (bool, optional) – whether to store the output of this chat completion for use in model distillation or evals
metadata (dict, optional) – metadata for the request
user (str, optional) – unique identifier representing your end-user
service_tier (dict, optional) – service tier configuration for the request
prediction (dict, optional) – prediction configuration for the request
router (dict, optional) – model routing configurations for the request
cache (dict, optional) – caching configuration for the request
kwargs (Any) – additional keyword arguments passed directly to the gateway

Returns:

iterator of model response chunks

Return type:

Iterator

generate(prompt, *, temperature=None, max_tokens=None, top_p=None, n=None, best_of=None, stop=None, frequency_penalty=None, presence_penalty=None, logprobs=None, logit_bias=None, seed=None, echo=None, suffix=None, stream_options=None, metadata=None, user=None, router=None, cache=None, **kwargs)[source]¶

Generate text completions.

Parameters:

prompt (str or list[str] or list[int]) – prompt to be processed during call
temperature (float, optional) – sampling temperature between 0 and 2; higher values make output more random, lower values make it more focused and deterministic
max_tokens (int, optional) – maximum number of tokens that can be generated in the completion; token count of prompt plus max_tokens cannot exceed the model’s context length
top_p (float, optional) – nucleus sampling probability mass; alternative to temperature
n (int, optional) – how many completions to generate for each prompt
best_of (int, optional) – generates best_of completions server-side and returns the best one; must be greater than n when used together; cannot be used with streaming
stop (list[str], optional) – up to 4 sequences where the API will stop generating further tokens; the returned text will not contain the stop sequence
frequency_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on their existing frequency in the text so far
presence_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on whether they appear in the text so far
logprobs (int, optional) – number of most likely output tokens (0–5) to include log probabilities for
logit_bias (dict, optional) – modifies the likelihood of specified tokens appearing in the completion; maps token IDs to a bias value from -100 to 100
seed (int, optional) – seed for deterministic sampling; repeated requests with the same seed and parameters should return the same result
echo (bool, optional) – whether to echo back the prompt in addition to the completion
suffix (str, optional) – text that comes after a completion of inserted text
stream_options (dict, optional) – options for streaming responses; only applicable when stream=True
metadata (dict, optional) – metadata to store with the completion
user (str, optional) – unique identifier representing your end-user
router (dict, optional) – model routing configuration for the request
cache (dict, optional) – caching configuration for the request; only supported for non-streaming requests
kwargs (Any) – additional keyword arguments passed directly to the gateway

Returns:

model answer

Return type:

dict

generate_stream(prompt, *, temperature=None, max_tokens=None, top_p=None, n=None, stop=None, frequency_penalty=None, presence_penalty=None, logprobs=None, logit_bias=None, seed=None, echo=None, suffix=None, stream_options=None, metadata=None, user=None, router=None, **kwargs)[source]¶

Generate text completions with streaming.

Parameters:

prompt (str or list[str] or list[int]) – prompt to be processed during call
temperature (float, optional) – sampling temperature between 0 and 2; higher values make output more random, lower values make it more focused and deterministic
max_tokens (int, optional) – maximum number of tokens that can be generated in the completion; token count of prompt plus max_tokens cannot exceed the model’s context length
top_p (float, optional) – nucleus sampling probability mass; alternative to temperature
n (int, optional) – how many completions to generate for each prompt
stop (list[str], optional) – up to 4 sequences where the API will stop generating further tokens; the returned text will not contain the stop sequence
frequency_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on their existing frequency in the text so far
presence_penalty (float, optional) – number between -2.0 and 2.0; positive values penalize new tokens based on whether they appear in the text so far
logprobs (int, optional) – number of most likely output tokens (0–5) to include log probabilities for
logit_bias (dict, optional) – modifies the likelihood of specified tokens appearing in the completion; maps token IDs to a bias value from -100 to 100
seed (int, optional) – seed for deterministic sampling; repeated requests with the same seed and parameters should return the same result
echo (bool, optional) – whether to echo back the prompt in addition to the completion
suffix (str, optional) – text that comes after a completion of inserted text
stream_options (dict, optional) – options for streaming responses
metadata (dict, optional) – metadata to store with the completion
user (str, optional) – unique identifier representing your end-user
router (dict, optional) – model routing configuration for the request
kwargs (Any) – additional keyword arguments passed directly to the gateway

Returns:

iterator of model response chunks

Return type:

Iterator

get_details()[source]¶

Get the details of the model(s) registered under the current model ID.

Calls get_details() and filters the results to entries whose id matches self._model.

Returns:: a single model detail dict when exactly one match is found, or a list of dicts when multiple providers expose the same model ID
Return type:: dict | list[dict]

Example:

gateway_inference.get_details()