Model Risk Configuration¶

pydantic model ibm_watsonx_gov.config.model_risk_configuration.ModelRiskConfiguration¶

Bases: BaseModel

Defines the ModelRiskConfiguration class.

This configuration class encapsulates all parameters required to perform model risk evaluation, including model metadata, evaluation scope, thresholds, and output/reporting preferences.

Examples

Create a basic configuration:

model_details = WxAIFoundationModel(
    model_name="mymodel_flan",
    model_id="google/flan-ul2",
    project_id="project_id")

model_config = ModelRiskConfiguration(
    model_details=model_details,
    risk_dimensions=["hallucination"],
    max_sample_size=500,
    thresholds=(20, 80),
    pdf_report_output_path="/reports"
)

Include watsonx Governance Console integration:

model_details = WxAIFoundationModel(
    model_name="mymodel_flan",
    model_id="google/flan-ul2",
    project_id="project_id")

wx_gc_credentials = WxGovConsoleCredentials(
    url="https://governance.example.com",
    username="admin",
    password="securepass"
    api_key="console API key"
)

wx_config = WxGovConsoleConfiguration(
    model_id="model-abc123",
    credentials=wx_gc_credentials
)

model_config = ModelRiskConfiguration(
    model_details=model_details,
    risk_dimensions=["hallucination"],
    max_sample_size=500,
    thresholds=(20, 80),
    wx_gc_configuration=wx_config,
    pdf_report_output_path="/reports"
)

Validators:

thresholds: Ensures that the threshold values are between 0 and 100, and that the lower value is less than the upper value.

Show JSON schema

{
   "title": "ModelRiskConfiguration",
   "description": "Defines the ModelRiskConfiguration class.\n\nThis configuration class encapsulates all parameters required to perform model risk evaluation,\nincluding model metadata, evaluation scope, thresholds, and output/reporting preferences.\n\nExamples:\n    1. Create a basic configuration:\n        .. code-block:: python\n\n            model_details = WxAIFoundationModel(\n                model_name=\"mymodel_flan\",\n                model_id=\"google/flan-ul2\",\n                project_id=\"project_id\")\n\n            model_config = ModelRiskConfiguration(\n                model_details=model_details,\n                risk_dimensions=[\"hallucination\"],\n                max_sample_size=500,\n                thresholds=(20, 80),\n                pdf_report_output_path=\"/reports\"                    \n            )\n\n    2. Include watsonx Governance Console integration:\n        .. code-block:: python\n\n            model_details = WxAIFoundationModel(\n                model_name=\"mymodel_flan\",\n                model_id=\"google/flan-ul2\",\n                project_id=\"project_id\")\n\n            wx_gc_credentials = WxGovConsoleCredentials(\n                url=\"https://governance.example.com\",\n                username=\"admin\",\n                password=\"securepass\"\n                api_key=\"console API key\"\n            )        \n                    \n            wx_config = WxGovConsoleConfiguration(\n                model_id=\"model-abc123\",\n                credentials=wx_gc_credentials\n            )\n\n            model_config = ModelRiskConfiguration(\n                model_details=model_details,\n                risk_dimensions=[\"hallucination\"],\n                max_sample_size=500,\n                thresholds=(20, 80),\n                wx_gc_configuration=wx_config,\n                pdf_report_output_path=\"/reports\"                    \n            )                \n\nValidators:\n    - `thresholds`: Ensures that the threshold values are between 0 and 100, and that the lower value is less than the upper value.",
   "type": "object",
   "properties": {
      "model_details": {
         "$ref": "#/$defs/FoundationModel",
         "description": "The details of the foundation model being evaluated.",
         "title": "Foundation Model Details"
      },
      "risk_dimensions": {
         "anyOf": [
            {
               "items": {
                  "type": "string"
               },
               "type": "array"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "A list of risk categories to be evaluated for the model. These could include hallucination, jailbreaking etc.",
         "examples": [
            [
               "hallucination",
               "jailbreaking",
               "harmful-code-generation"
            ]
         ],
         "title": "Risk Dimensions"
      },
      "max_sample_size": {
         "anyOf": [
            {
               "exclusiveMinimum": 0,
               "type": "integer"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The maximum number of samples to be used during the evaluation process. Must be a positive integer.",
         "examples": [
            50
         ],
         "title": "Maximum Sample Size"
      },
      "wx_gc_configuration": {
         "anyOf": [
            {
               "$ref": "#/$defs/WxGovConsoleConfiguration"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "Optional configuration for storing results in watsonx Governance Console.",
         "title": "watsonx Governance Console Configuration"
      },
      "pdf_report_output_path": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The output file path to store the model risk evaluation PDF report.",
         "examples": [
            "/reports/"
         ],
         "title": "PDF Report Output Path"
      },
      "thresholds": {
         "anyOf": [
            {
               "maxItems": 2,
               "minItems": 2,
               "prefixItems": [
                  {
                     "type": "integer"
                  },
                  {
                     "type": "integer"
                  }
               ],
               "type": "array"
            },
            {
               "type": "null"
            }
         ],
         "default": [
            25,
            75
         ],
         "description": "A tuple representing the percentile-based threshold configuration used for categorizing LLM performance. The first element is the lower percentile threshold, and the second is the upper percentile threshold",
         "examples": [
            [
               25,
               75
            ]
         ],
         "title": "Performance Thresholds"
      }
   },
   "$defs": {
      "FoundationModel": {
         "description": "Defines the base FoundationModel class.",
         "properties": {
            "model_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The name of the foundation model.",
               "title": "Model Name"
            },
            "provider": {
               "$ref": "#/$defs/ModelProvider",
               "description": "The provider of the foundation model."
            }
         },
         "required": [
            "provider"
         ],
         "title": "FoundationModel",
         "type": "object"
      },
      "ModelProvider": {
         "properties": {
            "type": {
               "$ref": "#/$defs/ModelProviderType",
               "description": "The type of model provider."
            }
         },
         "required": [
            "type"
         ],
         "title": "ModelProvider",
         "type": "object"
      },
      "ModelProviderType": {
         "description": "Supported model provider types for Generative AI",
         "enum": [
            "ibm_watsonx.ai",
            "azure_openai",
            "rits",
            "openai",
            "custom"
         ],
         "title": "ModelProviderType",
         "type": "string"
      },
      "WxGovConsoleConfiguration": {
         "description": "Defines the WxGovConsoleConfiguration class.\n\nThis configuration is used to integrate with the watsonx Governance Console for storing model risk evaluation results.\nIt includes the model identifier and the credentials required for authentication.\n\nExamples:\n    1. Create configuration with explicit credentials:\n        .. code-block:: python\n\n            credentials = WxGovConsoleCredentials(\n                url=\"https://governance-console.example.com\",\n                username=\"admin\",\n                password=\"securepassword\",\n                api_key=\"optional-api-key\"\n            )\n            configuration = WxGovConsoleConfiguration(\n                model_id=\"model-12345\",\n                credentials=credentials\n            )",
         "properties": {
            "model_id": {
               "description": "The watsonx Governance Console identifier of the model to store the model risk result.",
               "title": "Model Id",
               "type": "string"
            },
            "credentials": {
               "$ref": "#/$defs/WxGovConsoleCredentials",
               "description": "The watsonx Governance Console credentials."
            }
         },
         "required": [
            "model_id",
            "credentials"
         ],
         "title": "WxGovConsoleConfiguration",
         "type": "object"
      },
      "WxGovConsoleCredentials": {
         "description": "This class holds the authentication credentials required to connect to the watsonx Governance Console.\n\nExamples:\n    1. Create credentials manually:\n        .. code-block:: python\n\n            credentials = WxGovConsoleCredentials(\n                url=\"https://governance-console.example.com\",\n                username=\"admin\",\n                password=\"securepassword\",\n                api_key=\"optional-api-key\"\n            )\n\n    2. Create credentials using environment variables:\n        .. code-block:: python\n\n            import os\n\n            os.environ['WXGC_URL'] = \"https://governance-console.example.com\"\n            os.environ['WXGC_USERNAME'] = \"admin\"\n            os.environ['WXGC_PASSWORD'] = \"securepassword\"\n            os.environ['WXGC_API_KEY'] = \"optional-api-key\"  # Optional\n\n            credentials = WxGovConsoleCredentials.create_from_env()",
         "properties": {
            "url": {
               "description": "The base URL of the watsonx Governance Console.",
               "title": "Url",
               "type": "string"
            },
            "username": {
               "description": "The username used for authentication.",
               "title": "Username",
               "type": "string"
            },
            "password": {
               "description": "The password used for authentication.",
               "title": "Password",
               "type": "string"
            },
            "api_key": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "Optional API key for token-based authentication.",
               "title": "Api Key"
            }
         },
         "required": [
            "url",
            "username",
            "password"
         ],
         "title": "WxGovConsoleCredentials",
         "type": "object"
      }
   },
   "required": [
      "model_details"
   ]
}

Config:

protected_namespaces: tuple = ()

Fields:

max_sample_size (int | None)
model_details (ibm_watsonx_gov.entities.foundation_model.FoundationModel)
pdf_report_output_path (str | None)
risk_dimensions (List[str] | None)
thresholds (Tuple[int, int] | None)
wx_gc_configuration (ibm_watsonx_gov.config.model_risk_configuration.WxGovConsoleConfiguration | None)

Validators:

validate_thresholds » thresholds

field max_sample_size: Annotated[Annotated[int, Gt(gt=0)] | None, FieldInfo(annotation=NoneType, required=False, default=None, title='Maximum Sample Size', description='The maximum number of samples to be used during the evaluation process. Must be a positive integer.', examples=[50])] = None¶: The maximum number of samples to be used during the evaluation process. Must be a positive integer.

field model_details: Annotated[FoundationModel, FieldInfo(annotation=NoneType, required=True, title='Foundation Model Details', description='The details of the foundation model being evaluated.')] [Required]¶: The details of the foundation model being evaluated.

field pdf_report_output_path: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, title='PDF Report Output Path', description='The output file path to store the model risk evaluation PDF report.', examples=['/reports/'])] = None¶: The output file path to store the model risk evaluation PDF report.

field risk_dimensions: Annotated[List[str] | None, FieldInfo(annotation=NoneType, required=False, default=None, title='Risk Dimensions', description='A list of risk categories to be evaluated for the model. These could include hallucination, jailbreaking etc.', examples=[['hallucination', 'jailbreaking', 'harmful-code-generation']])] = None¶: A list of risk categories to be evaluated for the model. These could include hallucination, jailbreaking etc.

field thresholds: Annotated[Tuple[int, int] | None, FieldInfo(annotation=NoneType, required=False, default=25, 75, title='Performance Thresholds', description='A tuple representing the percentile-based threshold configuration used for categorizing LLM performance. The first element is the lower percentile threshold, and the second is the upper percentile threshold', examples=[25, 75])] = (25, 75)¶

A tuple representing the percentile-based threshold configuration used for categorizing LLM performance. The first element is the lower percentile threshold, and the second is the upper percentile threshold

Validated by:

validate_thresholds

field wx_gc_configuration: Annotated[WxGovConsoleConfiguration | None, FieldInfo(annotation=NoneType, required=False, default=None, title='watsonx Governance Console Configuration', description='Optional configuration for storing results in watsonx Governance Console.')] = None¶: Optional configuration for storing results in watsonx Governance Console.

validator validate_thresholds » thresholds¶

pydantic model ibm_watsonx_gov.config.model_risk_configuration.WxGovConsoleConfiguration¶

Bases: BaseModel

Defines the WxGovConsoleConfiguration class.

This configuration is used to integrate with the watsonx Governance Console for storing model risk evaluation results. It includes the model identifier and the credentials required for authentication.

Examples

Create configuration with explicit credentials:

credentials = WxGovConsoleCredentials(
    url="https://governance-console.example.com",
    username="admin",
    password="securepassword",
    api_key="optional-api-key"
)
configuration = WxGovConsoleConfiguration(
    model_id="model-12345",
    credentials=credentials
)

Show JSON schema

{
   "title": "WxGovConsoleConfiguration",
   "description": "Defines the WxGovConsoleConfiguration class.\n\nThis configuration is used to integrate with the watsonx Governance Console for storing model risk evaluation results.\nIt includes the model identifier and the credentials required for authentication.\n\nExamples:\n    1. Create configuration with explicit credentials:\n        .. code-block:: python\n\n            credentials = WxGovConsoleCredentials(\n                url=\"https://governance-console.example.com\",\n                username=\"admin\",\n                password=\"securepassword\",\n                api_key=\"optional-api-key\"\n            )\n            configuration = WxGovConsoleConfiguration(\n                model_id=\"model-12345\",\n                credentials=credentials\n            )",
   "type": "object",
   "properties": {
      "model_id": {
         "description": "The watsonx Governance Console identifier of the model to store the model risk result.",
         "title": "Model Id",
         "type": "string"
      },
      "credentials": {
         "$ref": "#/$defs/WxGovConsoleCredentials",
         "description": "The watsonx Governance Console credentials."
      }
   },
   "$defs": {
      "WxGovConsoleCredentials": {
         "description": "This class holds the authentication credentials required to connect to the watsonx Governance Console.\n\nExamples:\n    1. Create credentials manually:\n        .. code-block:: python\n\n            credentials = WxGovConsoleCredentials(\n                url=\"https://governance-console.example.com\",\n                username=\"admin\",\n                password=\"securepassword\",\n                api_key=\"optional-api-key\"\n            )\n\n    2. Create credentials using environment variables:\n        .. code-block:: python\n\n            import os\n\n            os.environ['WXGC_URL'] = \"https://governance-console.example.com\"\n            os.environ['WXGC_USERNAME'] = \"admin\"\n            os.environ['WXGC_PASSWORD'] = \"securepassword\"\n            os.environ['WXGC_API_KEY'] = \"optional-api-key\"  # Optional\n\n            credentials = WxGovConsoleCredentials.create_from_env()",
         "properties": {
            "url": {
               "description": "The base URL of the watsonx Governance Console.",
               "title": "Url",
               "type": "string"
            },
            "username": {
               "description": "The username used for authentication.",
               "title": "Username",
               "type": "string"
            },
            "password": {
               "description": "The password used for authentication.",
               "title": "Password",
               "type": "string"
            },
            "api_key": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "Optional API key for token-based authentication.",
               "title": "Api Key"
            }
         },
         "required": [
            "url",
            "username",
            "password"
         ],
         "title": "WxGovConsoleCredentials",
         "type": "object"
      }
   },
   "required": [
      "model_id",
      "credentials"
   ]
}

Config:

protected_namespaces: tuple = ()

Fields:

credentials (ibm_watsonx_gov.entities.credentials.WxGovConsoleCredentials)
model_id (str)

field credentials: Annotated[WxGovConsoleCredentials, FieldInfo(annotation=NoneType, required=True, description='The watsonx Governance Console credentials.')] [Required]¶: The watsonx Governance Console credentials.

field model_id: Annotated[str, FieldInfo(annotation=NoneType, required=True, description='The watsonx Governance Console identifier of the model to store the model risk result.')] [Required]¶: The watsonx Governance Console identifier of the model to store the model risk result.