AI Evaluation¶
- pydantic model ibm_watsonx_gov.entities.ai_evaluation.AIEvaluationAsset¶
Bases:
BaseModelThe class for AIEvaluationAsset.
Examples
- Create AIEvaluationAsset instance:
# Define evaluation configuration evaluation_config = EvaluationConfig( monitors={ "agentic_ai_quality": { "parameters": { "metrics_configuration": {} } } } ) # Create the evaluation asset ai_evaluation_asset = AIEvaluationAsset( name="AI Evaluation for agent", evaluation_configuration=evaluation_config ) # Compare two or more AI experiments using the evaluation asset response = ai_experiment_client.compare_ai_experiments( ai_experiment_ids=["experiment_id_1", "experiment_id_2"], ai_evaluation_asset=ai_evaluation_asset )
Show JSON schema
{ "title": "AIEvaluationAsset", "description": "The class for AIEvaluationAsset.\n\nExamples\n--------\nCreate AIEvaluationAsset instance:\n .. code-block:: python\n\n # Define evaluation configuration\n evaluation_config = EvaluationConfig(\n monitors={\n \"agentic_ai_quality\": {\n \"parameters\": {\n \"metrics_configuration\": {}\n }\n }\n }\n )\n\n # Create the evaluation asset\n ai_evaluation_asset = AIEvaluationAsset(\n name=\"AI Evaluation for agent\",\n evaluation_configuration=evaluation_config\n )\n\n # Compare two or more AI experiments using the evaluation asset\n response = ai_experiment_client.compare_ai_experiments(\n ai_experiment_ids=[\"experiment_id_1\", \"experiment_id_2\"],\n ai_evaluation_asset=ai_evaluation_asset\n )", "type": "object", "properties": { "container_id": { "default": "", "description": "The project or space id for the AI Evaluation.", "examples": [ "proj--1" ], "title": "Container Id", "type": "string" }, "container_type": { "default": "", "description": "The container type for the AI Evaluation.", "examples": [ "project", "space" ], "title": "Container Type", "type": "string" }, "container_name": { "default": "", "description": "The name of the project or the space.", "examples": [ "Project_1" ], "title": "Container Name", "type": "string" }, "name": { "default": "", "description": "The name of the AI Evaluation asset.", "examples": [ "AI agents evaluation" ], "title": "Name", "type": "string" }, "description": { "default": "", "description": "The description of the AI Evaluation asset.", "examples": [ "AI agents evaluation" ], "title": "Description", "type": "string" }, "asset_type": { "default": "ai_evaluation", "description": "The asset type of the AI Evaluation.", "examples": [ "ai_evaluation" ], "title": "Asset Type", "type": "string" }, "created_at": { "default": "", "description": "The timestamp of creation of AI Evaluation asset.", "examples": [ "2025-04-01T12:00:00Z" ], "title": "Created At", "type": "string" }, "owner_id": { "default": "", "description": "The owner of the AI Evaluation.", "examples": [ "user-123" ], "title": "Owner Id", "type": "string" }, "asset_id": { "default": "", "description": "The asset id of the AI Evaluation.", "examples": [ "43676d70-1ecc-412e-832f-8762aa899247" ], "title": "Asset Id", "type": "string" }, "creator_id": { "default": "", "description": "The creator id of the AI Evaluation.", "examples": [ "user-123" ], "title": "Creator Id", "type": "string" }, "asset_details": { "default": {}, "description": "The asset details of the AI Evluation asset.", "examples": [ { "evaluation_asset_type": "ai_experiment|prompt", "input_data_type": "unstructured_text", "job_id": "", "label_column": "", "operational_space_id": "development", "service_instance_id": "", "task_ids": [] } ], "title": "Asset Details", "type": "object" }, "evaluation_configuration": { "$ref": "#/$defs/EvaluationConfig", "default": [], "description": "The list of the evaluation configuration", "examples": [ { "evaluation_assets": [ { "attachment_id": "ab914f9b-9475-4c10-88d8-480b6c9f4963", "container_id": "b76d2ebb-4e05-496e-b377-557d409e8c45", "container_type": "project", "id": "d4d6ac43-0bec-47f9-8924-0b74ea1b8ec3", "name": "AI_Experiment asset for Agent governence", "nodes": [], "run_id": "fa7629e9-e1bb-4779-9198-9a6343dab1ad", "run_name": "Experiment run 1", "test_data": { "total_rows": 0 } } ], "monitors": { "agentic_ai_quality": { "parameters": { "metrics_configuration": {} } } } } ] }, "href": { "default": "", "description": "The link of the AI Evaluation Asset", "title": "Href", "type": "string" } }, "$defs": { "EvaluationAsset": { "properties": { "id": { "default": "", "description": "The id of the AI Experiment asset", "examples": [ "asset-001" ], "title": "Id", "type": "string" }, "container_id": { "default": "", "description": "The project id or space id.", "examples": [ "proj-01" ], "title": "Container Id", "type": "string" }, "container_type": { "default": "", "description": "The container type of AI Experiment", "examples": [ "project", "spcae" ], "title": "Container Type", "type": "string" }, "name": { "default": "", "description": "The name of the AI Experiment.", "examples": [ "AI_experiment_1" ], "title": "Name", "type": "string" }, "run_id": { "default": "", "description": "The experiment run id of the AI Experiment.", "examples": [ "run-01" ], "title": "Run Id", "type": "string" }, "run_name": { "default": "", "description": "The experiment run name of the AI Experiment.", "examples": [ "Test run 1" ], "title": "Run Name", "type": "string" }, "attachment_id": { "default": "", "description": "the attachment id for the evaluation result for that experiment run.", "examples": [ "att-01" ], "title": "Attachment Id", "type": "string" }, "test_data": { "default": {}, "description": "The test data of that experiment run.", "examples": [], "title": "Test Data", "type": "object" }, "nodes": { "default": "", "description": "List of the node for that experiment run.", "examples": [ { "id": "node-001", "name": "Node_1", "type": "tool" } ], "items": { "$ref": "#/$defs/Node" }, "title": "Nodes", "type": "array" } }, "title": "EvaluationAsset", "type": "object" }, "EvaluationConfig": { "properties": { "monitors": { "default": {}, "description": "The monitors configuration of for that AI Evaluation.", "examples": [ { "agentic_ai_quality": { "parameters": { "metrics_configuration": {} } } } ], "title": "Monitors", "type": "object" }, "evaluation_assets": { "default": [], "description": "The evaluation asset details.", "examples": [ [ { "attachment_id": "ab914f9b-9475-4c10-88d8-480b6c9f4963", "container_id": "b76d2ebb-4e05-496e-b377-557d409e8c45", "container_type": "project", "id": "d4d6ac43-0bec-47f9-8924-0b74ea1b8ec3", "name": "AI_Experiment asset", "nodes": [], "run_id": "fa7629e9-e1bb-4779-9198-9a6343dab1ad", "run_name": "Experiment run 1", "test_data": { "total_rows": 0 } } ] ], "items": { "$ref": "#/$defs/EvaluationAsset" }, "title": "Evaluation Assets", "type": "array" } }, "title": "EvaluationConfig", "type": "object" }, "FoundationModelInfo": { "description": "Represents a foundation model used in an experiment.", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "model_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The id of the foundation model.", "title": "Model Id" }, "provider": { "description": "The provider of the foundation model.", "title": "Provider", "type": "string" }, "type": { "description": "The type of foundation model.", "example": [ "chat", "embedding", "text-generation" ], "title": "Type", "type": "string" } }, "required": [ "provider", "type" ], "title": "FoundationModelInfo", "type": "object" }, "Node": { "properties": { "id": { "default": "", "description": "The ID of node for AI Experiemnt.", "examples": [ "node-001" ], "title": "Id", "type": "string" }, "name": { "default": "", "description": "The name of node for AI Experiment.", "examples": [ "Node_1" ], "title": "Name", "type": "string" }, "type": { "default": "tool", "description": "The type of node for AI Experiment.", "examples": [ "tool", "agent" ], "title": "Type", "type": "string" }, "foundation_models": { "default": [], "description": "The Foundation models invoked by the node", "items": { "$ref": "#/$defs/FoundationModelInfo" }, "title": "Foundation Models", "type": "array" } }, "title": "Node", "type": "object" } } }
- Fields:
- field asset_details: ', examples=[{'task_ids': [], 'label_column': '', 'operational_space_id': 'development', 'input_data_type': 'unstructured_text', 'job_id': '', 'service_instance_id': '', 'evaluation_asset_type': 'ai_experiment|prompt'}])] = {}¶
The asset details of the AI Evluation asset.
- field asset_id: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The asset id of the AI Evaluation.', examples=['43676d70-1ecc-412e-832f-8762aa899247'])] = ''¶
The asset id of the AI Evaluation.
- field asset_type: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='ai_evaluation', description='The asset type of the AI Evaluation.', examples=['ai_evaluation'])] = 'ai_evaluation'¶
The asset type of the AI Evaluation.
- field container_id: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The project or space id for the AI Evaluation.', examples=['proj--1'])] = ''¶
The project or space id for the AI Evaluation.
- field container_name: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The name of the project or the space.', examples=['Project_1'])] = ''¶
The name of the project or the space.
- field container_type: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The container type for the AI Evaluation.', examples=['project', 'space'])] = ''¶
The container type for the AI Evaluation.
- field created_at: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The timestamp of creation of AI Evaluation asset.', examples=['2025-04-01T12:00:00Z'])] = ''¶
The timestamp of creation of AI Evaluation asset.
- field creator_id: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The creator id of the AI Evaluation.', examples=['user-123'])] = ''¶
The creator id of the AI Evaluation.
- field description: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The description of the AI Evaluation asset.', examples=['AI agents evaluation'])] = ''¶
The description of the AI Evaluation asset.
- field evaluation_configuration: EvaluationConfig, FieldInfo(annotation=NoneType, required=False, default=[], description='The list of the evaluation configuration', examples=[{'monitors': {'agentic_ai_quality': {'parameters': {'metrics_configuration': {}}}}, 'evaluation_assets': [{'id': 'd4d6ac43-0bec-47f9-8924-0b74ea1b8ec3', 'container_id': 'b76d2ebb-4e05-496e-b377-557d409e8c45', 'container_type': 'project', 'name': 'AI_Experiment asset for Agent governence', 'run_id': 'fa7629e9-e1bb-4779-9198-9a6343dab1ad', 'run_name': 'Experiment run 1', 'attachment_id': 'ab914f9b-9475-4c10-88d8-480b6c9f4963', 'test_data': {'total_rows': 0}, 'nodes': []}]}])] = []¶
The list of the evaluation configuration
- field href: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The link of the AI Evaluation Asset')] = ''¶
The link of the AI Evaluation Asset
- field name: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The name of the AI Evaluation asset.', examples=['AI agents evaluation'])] = ''¶
The name of the AI Evaluation asset.
- field owner_id: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The owner of the AI Evaluation.', examples=['user-123'])] = ''¶
The owner of the AI Evaluation.
- to_json()¶
Transform the AIEvaluationAsset instance to json
- pydantic model ibm_watsonx_gov.entities.ai_evaluation.EvaluationAsset¶
Bases:
BaseModelShow JSON schema
{ "title": "EvaluationAsset", "type": "object", "properties": { "id": { "default": "", "description": "The id of the AI Experiment asset", "examples": [ "asset-001" ], "title": "Id", "type": "string" }, "container_id": { "default": "", "description": "The project id or space id.", "examples": [ "proj-01" ], "title": "Container Id", "type": "string" }, "container_type": { "default": "", "description": "The container type of AI Experiment", "examples": [ "project", "spcae" ], "title": "Container Type", "type": "string" }, "name": { "default": "", "description": "The name of the AI Experiment.", "examples": [ "AI_experiment_1" ], "title": "Name", "type": "string" }, "run_id": { "default": "", "description": "The experiment run id of the AI Experiment.", "examples": [ "run-01" ], "title": "Run Id", "type": "string" }, "run_name": { "default": "", "description": "The experiment run name of the AI Experiment.", "examples": [ "Test run 1" ], "title": "Run Name", "type": "string" }, "attachment_id": { "default": "", "description": "the attachment id for the evaluation result for that experiment run.", "examples": [ "att-01" ], "title": "Attachment Id", "type": "string" }, "test_data": { "default": {}, "description": "The test data of that experiment run.", "examples": [], "title": "Test Data", "type": "object" }, "nodes": { "default": "", "description": "List of the node for that experiment run.", "examples": [ { "id": "node-001", "name": "Node_1", "type": "tool" } ], "items": { "$ref": "#/$defs/Node" }, "title": "Nodes", "type": "array" } }, "$defs": { "FoundationModelInfo": { "description": "Represents a foundation model used in an experiment.", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "model_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The id of the foundation model.", "title": "Model Id" }, "provider": { "description": "The provider of the foundation model.", "title": "Provider", "type": "string" }, "type": { "description": "The type of foundation model.", "example": [ "chat", "embedding", "text-generation" ], "title": "Type", "type": "string" } }, "required": [ "provider", "type" ], "title": "FoundationModelInfo", "type": "object" }, "Node": { "properties": { "id": { "default": "", "description": "The ID of node for AI Experiemnt.", "examples": [ "node-001" ], "title": "Id", "type": "string" }, "name": { "default": "", "description": "The name of node for AI Experiment.", "examples": [ "Node_1" ], "title": "Name", "type": "string" }, "type": { "default": "tool", "description": "The type of node for AI Experiment.", "examples": [ "tool", "agent" ], "title": "Type", "type": "string" }, "foundation_models": { "default": [], "description": "The Foundation models invoked by the node", "items": { "$ref": "#/$defs/FoundationModelInfo" }, "title": "Foundation Models", "type": "array" } }, "title": "Node", "type": "object" } } }
- Fields:
- field attachment_id: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='the attachment id for the evaluation result for that experiment run.', examples=['att-01'])] = ''¶
the attachment id for the evaluation result for that experiment run.
- field container_id: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The project id or space id.', examples=['proj-01'])] = ''¶
The project id or space id.
- field container_type: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The container type of AI Experiment', examples=['project', 'spcae'])] = ''¶
The container type of AI Experiment
- field id: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The id of the AI Experiment asset', examples=['asset-001'])] = ''¶
The id of the AI Experiment asset
- field name: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The name of the AI Experiment.', examples=['AI_experiment_1'])] = ''¶
The name of the AI Experiment.
- field nodes: ', examples=[{'id': 'node-001', 'name': 'Node_1', 'type': 'tool'}])] = ''¶
List of the node for that experiment run.
- field run_id: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The experiment run id of the AI Experiment.', examples=['run-01'])] = ''¶
The experiment run id of the AI Experiment.
- field run_name: Annotated[str, FieldInfo(annotation=NoneType, required=False, default='', description='The experiment run name of the AI Experiment.', examples=['Test run 1'])] = ''¶
The experiment run name of the AI Experiment.
- field test_data: ', examples=[])] = {}¶
The test data of that experiment run.
- pydantic model ibm_watsonx_gov.entities.ai_evaluation.EvaluationConfig¶
Bases:
BaseModelShow JSON schema
{ "title": "EvaluationConfig", "type": "object", "properties": { "monitors": { "default": {}, "description": "The monitors configuration of for that AI Evaluation.", "examples": [ { "agentic_ai_quality": { "parameters": { "metrics_configuration": {} } } } ], "title": "Monitors", "type": "object" }, "evaluation_assets": { "default": [], "description": "The evaluation asset details.", "examples": [ [ { "attachment_id": "ab914f9b-9475-4c10-88d8-480b6c9f4963", "container_id": "b76d2ebb-4e05-496e-b377-557d409e8c45", "container_type": "project", "id": "d4d6ac43-0bec-47f9-8924-0b74ea1b8ec3", "name": "AI_Experiment asset", "nodes": [], "run_id": "fa7629e9-e1bb-4779-9198-9a6343dab1ad", "run_name": "Experiment run 1", "test_data": { "total_rows": 0 } } ] ], "items": { "$ref": "#/$defs/EvaluationAsset" }, "title": "Evaluation Assets", "type": "array" } }, "$defs": { "EvaluationAsset": { "properties": { "id": { "default": "", "description": "The id of the AI Experiment asset", "examples": [ "asset-001" ], "title": "Id", "type": "string" }, "container_id": { "default": "", "description": "The project id or space id.", "examples": [ "proj-01" ], "title": "Container Id", "type": "string" }, "container_type": { "default": "", "description": "The container type of AI Experiment", "examples": [ "project", "spcae" ], "title": "Container Type", "type": "string" }, "name": { "default": "", "description": "The name of the AI Experiment.", "examples": [ "AI_experiment_1" ], "title": "Name", "type": "string" }, "run_id": { "default": "", "description": "The experiment run id of the AI Experiment.", "examples": [ "run-01" ], "title": "Run Id", "type": "string" }, "run_name": { "default": "", "description": "The experiment run name of the AI Experiment.", "examples": [ "Test run 1" ], "title": "Run Name", "type": "string" }, "attachment_id": { "default": "", "description": "the attachment id for the evaluation result for that experiment run.", "examples": [ "att-01" ], "title": "Attachment Id", "type": "string" }, "test_data": { "default": {}, "description": "The test data of that experiment run.", "examples": [], "title": "Test Data", "type": "object" }, "nodes": { "default": "", "description": "List of the node for that experiment run.", "examples": [ { "id": "node-001", "name": "Node_1", "type": "tool" } ], "items": { "$ref": "#/$defs/Node" }, "title": "Nodes", "type": "array" } }, "title": "EvaluationAsset", "type": "object" }, "FoundationModelInfo": { "description": "Represents a foundation model used in an experiment.", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "model_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The id of the foundation model.", "title": "Model Id" }, "provider": { "description": "The provider of the foundation model.", "title": "Provider", "type": "string" }, "type": { "description": "The type of foundation model.", "example": [ "chat", "embedding", "text-generation" ], "title": "Type", "type": "string" } }, "required": [ "provider", "type" ], "title": "FoundationModelInfo", "type": "object" }, "Node": { "properties": { "id": { "default": "", "description": "The ID of node for AI Experiemnt.", "examples": [ "node-001" ], "title": "Id", "type": "string" }, "name": { "default": "", "description": "The name of node for AI Experiment.", "examples": [ "Node_1" ], "title": "Name", "type": "string" }, "type": { "default": "tool", "description": "The type of node for AI Experiment.", "examples": [ "tool", "agent" ], "title": "Type", "type": "string" }, "foundation_models": { "default": [], "description": "The Foundation models invoked by the node", "items": { "$ref": "#/$defs/FoundationModelInfo" }, "title": "Foundation Models", "type": "array" } }, "title": "Node", "type": "object" } } }
- Fields:
- field evaluation_assets: ', examples=[[{'id': 'd4d6ac43-0bec-47f9-8924-0b74ea1b8ec3', 'container_id': 'b76d2ebb-4e05-496e-b377-557d409e8c45', 'container_type': 'project', 'name': 'AI_Experiment asset', 'run_id': 'fa7629e9-e1bb-4779-9198-9a6343dab1ad', 'run_name': 'Experiment run 1', 'attachment_id': 'ab914f9b-9475-4c10-88d8-480b6c9f4963', 'test_data': {'total_rows': 0}, 'nodes': []}]])] = []¶
The evaluation asset details.
- field monitors: ', examples=[{'agentic_ai_quality': {'parameters': {'metrics_configuration': {}}}}])] = {}¶
The monitors configuration of for that AI Evaluation.