Gateway API Reference¶

Quick start¶

The steps below walk through the full lifecycle: connecting to Model Gateway, registering an IBM watsonx.ai provider and IBM models, then running inference through both the high-level and low-level interfaces.

Note

The way a provider is authenticated differs between deployment targets:

IBM Cloud — pass secret_crn_id pointing to a secret in IBM Secrets Manager. Credentials never appear in your code.
Cloud Pak for Data (CPD) — pass data inline with base_url, apikey, and project_id, because CPD does not have IBM Secrets Manager.

This difference applies to every provider type ("watsonxai", "nim", etc.). The data dict keys vary per provider — "watsonxai" requires base_url, apikey, and project_id; "nim" requires only apikey.

from ibm_watsonx_ai import Credentials
from ibm_watsonx_ai.gateway import Gateway, GatewayInference

# Step 1 — Create a Gateway client
credentials = Credentials(url="https://us-south.ml.cloud.ibm.com", api_key="...")
gateway = Gateway(credentials=credentials)

# Step 2 — Register an IBM watsonx.ai provider
#
# IBM Cloud: use secret_crn_id (recommended — keeps credentials out of your code)
provider_details = gateway.providers.create(
    provider="watsonxai",
    name="My watsonx.ai connection",
    secret_crn_id="crn:v1:bluemix:public:secrets-manager:us-south:a/<account-id>::",
)
#
# CPD: use data with inline credentials (Secrets Manager is not available on CPD)
# provider_details = gateway.providers.create(
#     provider="watsonxai",
#     name="My watsonx.ai connection",
#     data={
#         "base_url": "https://<cpd-host>/",
#         "apikey": "<cpd-api-key>",
#         "project_id": "<project-id>",
#     },
# )
provider_id = gateway.providers.get_id(provider_details)

# Step 3 — Register models under the provider
# `alias` is optional but lets you reference the model by a stable, short name
# regardless of the underlying provider model ID.
#
# Chat model (supports chat completions)
chat_model_details = gateway.models.create(
    provider_id=provider_id,
    model="meta-llama/llama-3-3-70b-instruct",
    alias="llama",
)
#
# Text generation model (supports text completions)
text_model_details = gateway.models.create(
    provider_id=provider_id,
    model="ibm/granite-4-h-small",
    alias="granite",
)

# Step 4 — Run inference via the high-level GatewayInference interface
# The model is bound once at construction time; all sampling parameters set
# here become instance-level defaults that can be overridden per call.
gateway_inference = GatewayInference(
    model="llama",
    credentials=credentials,
    temperature=0.3,
    max_tokens=512,
)

response = gateway_inference.chat(messages=[{"role": "user", "content": "Hello!"}])
print(response["choices"][0]["message"]["content"])

# Streaming variant — yields Server-Sent Event chunks as they arrive
for chunk in gateway_inference.chat_stream(
    messages=[{"role": "user", "content": "Tell me a joke."}]
):
    print(chunk, end="", flush=True)

# Step 5 — Run inference via the low-level Gateway interface
# Stateless — model is passed per call, no instance-level defaults.
response = gateway.chat.completions.create(
    model="llama",
    messages=[{"role": "user", "content": "What is 2 + 2?"}],
)
print(response["choices"][0]["message"]["content"])

# Text completions (generate-style) — use a text generation model
response = gateway.completions.create(
    model="granite",
    prompt="The capital of France is",
)
print(response["choices"][0]["text"])

Gateway¶

class ibm_watsonx_ai.gateway.Gateway(*, credentials=None, verify=None, api_client=None, max_retries=None, delay_time=None, retry_status_codes=None)[source]¶

Model Gateway client.

Provides access to providers, models, policies, rate limits, and inference endpoints (chat completions, text completions, and embeddings) registered in IBM watsonx.ai Model Gateway.

Parameters:

credentials (Credentials, optional) – credentials for the watsonx.ai instance; mutually exclusive with api_client
verify (bool or str, optional) –
SSL certificate verification setting:
- path to a CA_BUNDLE file or directory of trusted CA certificates
- True — use the default truststore
- False — disable verification (not recommended for production)
api_client (APIClient, optional) – pre-initialised APIClient with a project or space ID already set; if provided, credentials is not required
max_retries (int, optional) – maximum number of retries when a request fails with a status code in retry_status_codes; defaults to 10
delay_time (float, optional) – base delay (in seconds) for the exponential back-off formula delay_time * 2 ** attempt; defaults to 0.5
retry_status_codes (list[int], optional) – HTTP status codes that trigger the retry mechanism; defaults to [429, 503, 504, 520]

Raises:

InvalidMultipleArguments – if neither credentials nor api_client is provided
WMLClientError – if the connected platform release does not support Model Gateway (CPD < 5.2)

Providers¶

class ibm_watsonx_ai.gateway.providers.Providers(api_client)[source]¶

Model Gateway providers class.

create(provider, name, data=None, secret_crn_id=None)[source]¶

Create provider in Model Gateway.

Parameters:

provider (str) – provider name
name (str) – name of provider for display
data (dict, optional) – data required to connect to provider api
secret_crn_id (str, optional) – crn of secret for given provider in the Secrets Manager

Returns:

provider details

Return type:

dict

delete(provider_id)[source]¶

Delete provider.

Parameters:: provider_id (str) – unique provider ID
Returns:: status (“SUCCESS” if succeeded)
Return type:: str

get_available_models_details(provider_id)[source]¶

Get available models details for given provider.

Parameters:: provider_id (str) – unique provider ID
Returns:: details of available models for provider
Return type:: dict

get_details(provider_id=None)[source]¶

Get provider/providers details:

provider_id is set - details for given provider are returned
provider_id is None - details for all providers are returned

Parameters:: provider_id (str, optional) – unique provider ID
Returns:: provider/providers details
Return type:: dict

static get_id(provider_details)[source]¶

Get provider ID from provider details.

Parameters:: provider_details (dict) – details of the provider in Model Gateway
Returns:: unique provider ID
Return type:: str

list()[source]¶

List providers.

Returns:: dataframe with providers details
Return type:: pandas.DataFrame

list_available_models(provider_id)[source]¶

List available models for provider.

Parameters:: provider_id (str) – unique provider ID
Returns:: dataframe with available models details
Return type:: pandas.DataFrame

Models¶

class ibm_watsonx_ai.gateway.models.Models(api_client)[source]¶

Model Gateway models class.

create(provider_id, model, alias=None, metadata=None)[source]¶

Parameters:

provider_id (str) – unique provider ID obtained from provider details
model (str) – model name as supported by provider
alias (str, optional) – alias for registered model, can be used later as model name during embeddings or text/chat completions calls
metadata (dict, optional) –
additional metadata for the model.

Note

For AutoAI RAG experiments, "functions" (see GatewayModelFunctions) and "context_window" must be set.

Returns:

model details

Return type:

dict

delete(model_id)[source]¶

Unregister model from Model Gateway.

Parameters:: model_id (str) – unique model ID obtained from model details
Returns:: status (“SUCCESS” if succeeded)
Return type:: str

get_details(*, model_id=None, provider_id=None)[source]¶

Get details of model or models:

model_id is set - details for single model are returned, provider_id if set is ignored
provider_id is set, model_id is None - details for all models for given provider are returned
both model_id and provider_id are None - all models details are returned

Parameters:

model_id (str, optional) – unique model ID
provider_id (str, optional) – unique provider ID, ignored if model_id is set

Returns:

details of model/models

Return type:

dict

static get_id(model_details)[source]¶

Get model ID from model details.

Parameters:: model_details (dict) – details of the model registered in Model Gateway
Returns:: unique model ID
Return type:: str

list(provider_id=None)[source]¶

List models registered in Model Gateway. List can be filtered by provider_id.

The returned DataFrame includes a FUNCTIONS column containing the capability tags stored in each model’s metadata (see GatewayModelFunctions). This column is required for AutoAI RAG to identify models that carry the AUTOAI_RAG function tag.

Parameters:: provider_id (str, optional) – ID of provider added into Model Gateway
Returns:: dataframe containing list results with columns ID, MODEL, CREATED, TYPE, and FUNCTIONS
Return type:: pandas.DataFrame

class ibm_watsonx_ai.gateway.enums.GatewayModelFunctions(value)[source]¶

Allowable values for model functions passed in the metadata parameter of Gateway.models.create().

Model functions describe the capabilities of a model registered in Model Gateway. They are required for AutoAI RAG and other feature-specific workflows that need to filter available models by capability.

Pass one or more values as the "functions" key inside the metadata dict:

from ibm_watsonx_ai.gateway import GatewayModelFunctions

gateway.models.create(
    provider_id="...",
    model="my-model",
    metadata={
        "functions": [
            GatewayModelFunctions.AUTOAI_RAG,
            GatewayModelFunctions.TEXT_CHAT,
        ],
        "context_window": 8192,
    },
)

Note

AutoAI RAG requires at least AUTOAI_RAG (or AUTOAI_SQL_RAG) to be present in functions, and a valid context_window value, so that the framework can calculate the maximum number of input tokens for a given model.

AUTOAI_RAG = 'autoai_rag'¶: Model supports AutoAI RAG pipelines.

AUTOAI_SQL_RAG = 'autoai_sql_rag'¶: Model supports AutoAI SQL RAG pipelines.

BASE_FOUNDATION_MODEL_DEPLOYABLE = 'base_foundation_model_deployable'¶: Model can be deployed as a base foundation model.

EMBEDDING = 'embedding'¶: Model produces text embeddings.

IMAGE_CHAT = 'image_chat'¶: Model supports image-based chat (multimodal).

LORA_FINE_TUNE_TRAINABLE = 'lora_fine_tune_trainable'¶: Model can be fine-tuned with LoRA.

MULTILINGUAL = 'multilingual'¶: Model supports multiple languages.

RERANK = 'rerank'¶: Model supports document re-ranking.

SIMILARITY = 'similarity'¶: Model supports semantic similarity scoring.

TEXT_CHAT = 'text_chat'¶: Model supports chat (text) completions.

TEXT_GENERATION = 'text_generation'¶: Model supports text generation.

TIME_SERIES_FORECAST = 'time_series_forecast'¶: Model supports time-series forecasting.

VIDEO_CHAT = 'video_chat'¶: Model supports video-based chat (multimodal).

Policies¶

class ibm_watsonx_ai.gateway.policies.Policies(api_client)[source]¶

Model Gateway policies class.

create(action, resource, subject, effect=None)[source]¶

Create policy.

Parameters:

action (str) – action for policy
resource (str) – resource for policy
subject (str) – subject for policy
effect (str, optional) – effect for policy

delete(policy_id)[source]¶

Delete policy.

Parameters:: policy_id (str) – ID of policy
Returns:: status (“SUCCESS” if succeeded)
Return type:: str

get_details()[source]¶

Get policies details.

Returns:: policies details
Return type:: dict

static get_id(policy_details)[source]¶

Get policy ID from policy details.

Parameters:: policy_details (dict) – details of the policy for asset registered in Model Gateway
Returns:: unique policy ID
Return type:: str

list()[source]¶

List policies.

Returns:: dataframe with policies details
Return type:: pandas.DataFrame

RateLimits¶

class ibm_watsonx_ai.gateway.rate_limits.RateLimitSettings[source]¶

Model Gateway rate limit settings.

Parameters:

amount (int) – amount is the number of tokens refilled into the bucket each interval
capacity (int) – capacity is the maximum number of tokens (requests) the bucket can hold
duration (str) – duration is the refill interval, formatted as a Go duration string (for more information please see: https://pkg.go.dev/time#ParseDuration)

class ibm_watsonx_ai.gateway.rate_limits.RateLimits(api_client)[source]¶

Model Gateway rate limits class.

create_for_model(model_id, *, request=None, token=None)[source]¶

Create rate limit for model in Model Gateway.

Parameters:

model_id (str) – ID of the Model Gateway model
request (RateLimitSettings, optional) – request rate limiting settings
token (RateLimitSettings, optional) – token rate limiting settings

Returns:

rate limit details

Return type:

dict

create_for_provider(provider_id, *, request=None, token=None)[source]¶

Create rate limit for provider in Model Gateway.

Parameters:

provider_id (str) – ID of the Model Gateway provider
request (RateLimitSettings, optional) – request rate limiting settings
token (RateLimitSettings, optional) – token rate limiting settings

Returns:

rate limit details

Return type:

dict

create_for_tenant(*, request=None, token=None)[source]¶

Create rate limit for tenant in Model Gateway.

Parameters:

request (RateLimitSettings, optional) – request rate limiting settings
token (RateLimitSettings, optional) – token rate limiting settings

Returns:

rate limit details

Return type:

dict

delete(rate_limit_id)[source]¶

Delete rate limit from Model Gateway.

Parameters:: rate_limit_id (str) – ID of the rate limit
Returns:: status “SUCCESS” if deletion is successful
Return type:: Literal[“SUCCESS”]
Raises:: WMLClientError – if deletion failed

get_details(*, rate_limit_id=None)[source]¶

Get details of rate limits. If rate_limit_id is specified, returns details of that rate limit.

Parameters:: rate_limit_id (str, optional) – ID of the rate limit
Returns:: details of rate limits or rate limit if rate_limit_id is specified
Return type:: dict

static get_id(rate_limit_details)[source]¶

Get rate limit ID from rate limit details.

Parameters:: rate_limit_details (dict) – details of the rate limit
Returns:: ID of the rate limit
Return type:: str

list()[source]¶

List rate limits registered in Model Gateway.

Returns:: dataframe containing list results
Return type:: pandas.DataFrame

update_for_model(rate_limit_id, model_id, *, request=None, token=None)[source]¶

Update rate limit for model in Model Gateway.

Parameters:

rate_limit_id (str) – ID of the rate limit
model_id (str) – ID of the Model Gateway model
request (RateLimitSettings, optional) – request rate limiting settings
token (RateLimitSettings, optional) – token rate limiting settings

Returns:

rate limit details

Return type:

dict

update_for_provider(rate_limit_id, provider_id, *, request=None, token=None)[source]¶

Update rate limit for provider in Model Gateway.

Parameters:

rate_limit_id (str) – ID of the rate limit
provider_id (str) – ID of the Model Gateway provider
request (RateLimitSettings, optional) – request rate limiting settings
token (RateLimitSettings, optional) – token rate limiting settings

Returns:

rate limit details

Return type:

dict

update_for_tenant(rate_limit_id, *, request=None, token=None)[source]¶

Update rate limit for tenant in Model Gateway.

Parameters:

rate_limit_id (str) – ID of the rate limit
request (RateLimitSettings, optional) – request rate limiting settings
token (RateLimitSettings, optional) – token rate limiting settings

Returns:

rate limit details

Return type:

dict

Get rate limit details for model requests¶

In order to get details of a request, which returned an error because of rate limits, you should use try-except to catch the APIRequestFailure exception. The caught exception has the response property, which is the underlying httpx.Response instance. Using that instance, you can retrieve the response headers, which contain information about the rate limit.

try:
    response = gateway.completions.create(
        model_id, "The default voltage provided in USB is "
    )
except APIRequestFailure as exc:
    error_response = exc.response
    rate_limit_headers = {
        name: value
        for name, value in error_response.headers
        if name.startswith("x-ratelimit-")
    }