Model Risk Evaluator¶

pydantic model ibm_watsonx_gov.evaluators.model_risk_evaluator.ModelRiskEvaluator¶

Bases: BaseEvaluator

The class to evaluate the foundational model risk and display the results.

Example

Basic usage

configuration = ModelRiskConfiguration(
    model_details=model_details,
    risk_dimensions=risk_dimensions,
    max_sample_size=max_sample_size,
    pdf_report_output_path=pdf_report_output_path
)
wxgov_client = APIClient(credentials=Credentials(api_key=""))
evaluator = ModelRiskEvaluator(
    configuration=config, api_client=wxgov_client)

result = evaluator.evaluate()

# Get the results in the required format
result.to_json()

# Display the results
evaluator.display_table()
evaluator.download_model_risk_report()

Show JSON schema

{
   "title": "ModelRiskEvaluator",
   "type": "object",
   "properties": {
      "api_client": {
         "default": null,
         "title": "Api Client"
      },
      "configuration": {
         "$ref": "#/$defs/ModelRiskConfiguration",
         "name": "The configuration for model risk evaluation."
      }
   },
   "$defs": {
      "FoundationModel": {
         "description": "Defines the base FoundationModel class.",
         "properties": {
            "model_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The name of the foundation model.",
               "title": "Model Name"
            },
            "provider": {
               "$ref": "#/$defs/ModelProvider",
               "description": "The provider of the foundation model."
            }
         },
         "required": [
            "provider"
         ],
         "title": "FoundationModel",
         "type": "object"
      },
      "ModelProvider": {
         "properties": {
            "type": {
               "$ref": "#/$defs/ModelProviderType",
               "description": "The type of model provider."
            }
         },
         "required": [
            "type"
         ],
         "title": "ModelProvider",
         "type": "object"
      },
      "ModelProviderType": {
         "description": "Supported model provider types for Generative AI",
         "enum": [
            "ibm_watsonx.ai",
            "azure_openai",
            "rits",
            "openai",
            "custom"
         ],
         "title": "ModelProviderType",
         "type": "string"
      },
      "ModelRiskConfiguration": {
         "description": "Defines the ModelRiskConfiguration class.\n\nThis configuration class encapsulates all parameters required to perform model risk evaluation,\nincluding model metadata, evaluation scope, thresholds, and output/reporting preferences.\n\nExamples:\n    1. Create a basic configuration:\n        .. code-block:: python\n\n            model_details = WxAIFoundationModel(\n                model_name=\"mymodel_flan\",\n                model_id=\"google/flan-ul2\",\n                project_id=\"project_id\")\n\n            model_config = ModelRiskConfiguration(\n                model_details=model_details,\n                risk_dimensions=[\"hallucination\"],\n                max_sample_size=500,\n                thresholds=(20, 80),\n                pdf_report_output_path=\"/reports\"                    \n            )\n\n    2. Include watsonx Governance Console integration:\n        .. code-block:: python\n\n            model_details = WxAIFoundationModel(\n                model_name=\"mymodel_flan\",\n                model_id=\"google/flan-ul2\",\n                project_id=\"project_id\")\n\n            wx_gc_credentials = WxGovConsoleCredentials(\n                url=\"https://governance.example.com\",\n                username=\"admin\",\n                password=\"securepass\"\n                api_key=\"console API key\"\n            )        \n                    \n            wx_config = WxGovConsoleConfiguration(\n                model_id=\"model-abc123\",\n                credentials=wx_gc_credentials\n            )\n\n            model_config = ModelRiskConfiguration(\n                model_details=model_details,\n                risk_dimensions=[\"hallucination\"],\n                max_sample_size=500,\n                thresholds=(20, 80),\n                wx_gc_configuration=wx_config,\n                pdf_report_output_path=\"/reports\"                    \n            )                \n\nValidators:\n    - `thresholds`: Ensures that the threshold values are between 0 and 100, and that the lower value is less than the upper value.",
         "properties": {
            "model_details": {
               "$ref": "#/$defs/FoundationModel",
               "description": "The details of the foundation model being evaluated.",
               "title": "Foundation Model Details"
            },
            "risk_dimensions": {
               "anyOf": [
                  {
                     "items": {
                        "type": "string"
                     },
                     "type": "array"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "A list of risk categories to be evaluated for the model. These could include hallucination, jailbreaking etc.",
               "examples": [
                  [
                     "hallucination",
                     "jailbreaking",
                     "harmful-code-generation"
                  ]
               ],
               "title": "Risk Dimensions"
            },
            "max_sample_size": {
               "anyOf": [
                  {
                     "exclusiveMinimum": 0,
                     "type": "integer"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The maximum number of samples to be used during the evaluation process. Must be a positive integer.",
               "examples": [
                  50
               ],
               "title": "Maximum Sample Size"
            },
            "wx_gc_configuration": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/WxGovConsoleConfiguration"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "Optional configuration for storing results in watsonx Governance Console.",
               "title": "watsonx Governance Console Configuration"
            },
            "pdf_report_output_path": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The output file path to store the model risk evaluation PDF report.",
               "examples": [
                  "/reports/"
               ],
               "title": "PDF Report Output Path"
            },
            "thresholds": {
               "anyOf": [
                  {
                     "maxItems": 2,
                     "minItems": 2,
                     "prefixItems": [
                        {
                           "type": "integer"
                        },
                        {
                           "type": "integer"
                        }
                     ],
                     "type": "array"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": [
                  25,
                  75
               ],
               "description": "A tuple representing the percentile-based threshold configuration used for categorizing LLM performance. The first element is the lower percentile threshold, and the second is the upper percentile threshold",
               "examples": [
                  [
                     25,
                     75
                  ]
               ],
               "title": "Performance Thresholds"
            }
         },
         "required": [
            "model_details"
         ],
         "title": "ModelRiskConfiguration",
         "type": "object"
      },
      "WxGovConsoleConfiguration": {
         "description": "Defines the WxGovConsoleConfiguration class.\n\nThis configuration is used to integrate with the watsonx Governance Console for storing model risk evaluation results.\nIt includes the model identifier and the credentials required for authentication.\n\nExamples:\n    1. Create configuration with explicit credentials:\n        .. code-block:: python\n\n            credentials = WxGovConsoleCredentials(\n                url=\"https://governance-console.example.com\",\n                username=\"admin\",\n                password=\"securepassword\",\n                api_key=\"optional-api-key\"\n            )\n            configuration = WxGovConsoleConfiguration(\n                model_id=\"model-12345\",\n                credentials=credentials\n            )",
         "properties": {
            "model_id": {
               "description": "The watsonx Governance Console identifier of the model to store the model risk result.",
               "title": "Model Id",
               "type": "string"
            },
            "credentials": {
               "$ref": "#/$defs/WxGovConsoleCredentials",
               "description": "The watsonx Governance Console credentials."
            }
         },
         "required": [
            "model_id",
            "credentials"
         ],
         "title": "WxGovConsoleConfiguration",
         "type": "object"
      },
      "WxGovConsoleCredentials": {
         "description": "This class holds the authentication credentials required to connect to the watsonx Governance Console.\n\nExamples:\n    1. Create credentials manually:\n        .. code-block:: python\n\n            credentials = WxGovConsoleCredentials(\n                url=\"https://governance-console.example.com\",\n                username=\"admin\",\n                password=\"securepassword\",\n                api_key=\"optional-api-key\"\n            )\n\n    2. Create credentials using environment variables:\n        .. code-block:: python\n\n            import os\n\n            os.environ['WXGC_URL'] = \"https://governance-console.example.com\"\n            os.environ['WXGC_USERNAME'] = \"admin\"\n            os.environ['WXGC_PASSWORD'] = \"securepassword\"\n            os.environ['WXGC_API_KEY'] = \"optional-api-key\"  # Optional\n\n            credentials = WxGovConsoleCredentials.create_from_env()",
         "properties": {
            "url": {
               "description": "The base URL of the watsonx Governance Console.",
               "title": "Url",
               "type": "string"
            },
            "username": {
               "description": "The username used for authentication.",
               "title": "Username",
               "type": "string"
            },
            "password": {
               "description": "The password used for authentication.",
               "title": "Password",
               "type": "string"
            },
            "api_key": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "Optional API key for token-based authentication.",
               "title": "Api Key"
            }
         },
         "required": [
            "url",
            "username",
            "password"
         ],
         "title": "WxGovConsoleCredentials",
         "type": "object"
      }
   },
   "required": [
      "configuration"
   ]
}

Config:

arbitrary_types_allowed: bool = True

Fields:

api_client (Annotated[ibm_watsonx_gov.clients.api_client.APIClient | None, FieldInfo(annotation=NoneType, required=False, default=None, json_schema_extra={'name': 'The IBM watsonx.governance client.'})])
configuration (Annotated[ibm_watsonx_gov.config.model_risk_configuration.ModelRiskConfiguration, FieldInfo(annotation=NoneType, required=True, json_schema_extra={'name': 'The configuration for model risk evaluation.'})])

field api_client: '})] = None¶

field configuration: '})] [Required]¶

display_table()¶

download_model_risk_report()¶: Downloads the model risk report and returns the download link.

evaluate() → ModelRiskResult¶

Evaluates the risk of a Foundation model.

Returns:: The result of the model risk evaluation.
Return type:: ModelRiskResult

model_post_init(context: Any, /) → None¶

This function is meant to behave like a BaseModel method to initialise private attributes.

It takes context as an argument since that’s what pydantic-core passes when calling it.

Parameters:

self – The BaseModel instance.
context – The context.