Agentic Evaluation Result¶
- pydantic model ibm_watsonx_gov.entities.agentic_evaluation_result.AgenticEvaluationResult¶
Bases:
BaseModelShow JSON schema
{ "title": "AgenticEvaluationResult", "type": "object", "properties": { "metrics_results": { "description": "The list of metrics result.", "items": { "$ref": "#/$defs/AgentMetricResult" }, "title": "Metrics result", "type": "array" }, "aggregated_metrics_results": { "description": "The list of aggregated metrics result. The metrics are aggregated for each node in the agent.", "items": { "$ref": "#/$defs/AggregateAgentMetricResult" }, "title": "Aggregated metrics result", "type": "array" }, "messages_data": { "default": [], "description": "The list of agent messages data.", "items": { "$ref": "#/$defs/MessageData" }, "title": "Messages", "type": "array" }, "nodes_data": { "default": [], "description": "The list of nodes data.", "items": { "$ref": "#/$defs/NodeData" }, "title": "Node messages", "type": "array" }, "metrics_mapping_data": { "default": [], "description": "The mapping data used to compute the metric.", "items": { "$ref": "#/$defs/MetricsMappingData" }, "title": "Metrics mapping data", "type": "array" }, "nodes": { "default": [], "description": "The list of nodes details", "items": { "$ref": "#/$defs/Node" }, "title": "Nodes", "type": "array" }, "edges": { "default": [], "description": "The list of nodes details", "items": { "type": "object" }, "title": "Nodes", "type": "array" } }, "$defs": { "AWSBedrockCredentials": { "description": "Defines the AWSBedrockCredentials class for accessing AWS Bedrock using environment variables or manual input.\n\nExamples:\n 1. Create credentials manually:\n .. code-block:: python\n\n credentials = AWSBedrockCredentials(\n aws_access_key_id=\"your-access-key-id\",\n aws_secret_access_key=\"your-secret-access-key\",\n aws_region_name=\"us-east-1\",\n aws_session_token=\"optional-session-token\"\n )\n\n 2. Create credentials from environment:\n .. code-block:: python\n\n os.environ[\"AWS_ACCESS_KEY_ID\"] = \"your-access-key-id\"\n os.environ[\"AWS_DEFAULT_REGION\"] = \"us-east-1\"\n os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"your-secret-access-key\"\n\n credentials = AWSBedrockCredentials.create_from_env()", "properties": { "aws_access_key_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The AWS access key id. This attribute value will be read from AWS_ACCESS_KEY_ID environment variable when creating AWSBedrockCredentials from environment.", "title": "AWS Access Key ID" }, "aws_secret_access_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The AWS secret access key. This attribute value will be read from AWS_SECRET_ACCESS_KEY environment variable when creating AWSBedrockCredentials from environment.", "title": "AWS Secret Access Key" }, "aws_region_name": { "default": "us-east-1", "description": "AWS region. This attribute value will be read from AWS_DEFAULT_REGION environment variable when creating AWSBedrockCredentials from environment.", "examples": [ "us-east-1", "eu-west-1" ], "title": "AWS Region", "type": "string" }, "aws_session_token": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Optional AWS session token for temporary credentials.", "title": "AWS Session Token" } }, "required": [ "aws_access_key_id", "aws_secret_access_key", "aws_session_token" ], "title": "AWSBedrockCredentials", "type": "object" }, "AWSBedrockFoundationModel": { "description": " The Amazon Bedrock foundation model details.\n\n Examples:\n 1. Create AWS Bedrock foundation model by passing credentials manually:\n .. code-block:: python\n\n bedrock_model = AWSBedrockFoundationModel(\n model_id=\"anthropic.claude-v2\",\n provider=AWSBedrockModelProvider(\n credentials=AWSBedrockCredentials(\n aws_access_key_id=\"your-access-key-id\",\n aws_secret_access_key=\"your-secret-access-key\",\n aws_region_name=\"us-east-1\",\n aws_session_token=\"optional-session-token\"\n )\n ),\n parameters={\n \"temperature\": 0.7,\n \"top_p\": 0.9,\n \"max_tokens\": 200,\n \"stop_sequences\": [\"\n\"],\n \"system\": \"You are a concise assistant.\",\n \"reasoning_effort\": \"high\",\n \"tool_choice\": \"auto\"\n }\n )\n\n 2. Create AWS Bedrock foundation model using environment variables:\n os.environ[\"AWS_ACCESS_KEY_ID\"] = \"your-access-key-id\"\n os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"your-secret-access-key\"\n os.environ[\"AWS_DEFAULT_REGION\"] = \"us-east-1\"\n\n .. code-block:: python\n\n bedrock_model = AWSBedrockFoundationModel(\n model_id=\"anthropic.claude-v2\"\n )\n ", "properties": { "model_id": { "description": "The AWS Bedrock model name. It must be a valid AWS Bedrock model identifier.", "examples": [ "anthropic.claude-v2" ], "title": "Model ID", "type": "string" }, "provider": { "$ref": "#/$defs/AWSBedrockModelProvider", "description": "The AWS Bedrock provider details.", "title": "Provider" }, "parameters": { "anyOf": [ { "type": "object" }, { "type": "null" } ], "description": "The model parameters to be used when invoking the model. The parameters may include temperature, top_p, max_tokens, etc..", "title": "Parameters" } }, "required": [ "model_id" ], "title": "AWSBedrockFoundationModel", "type": "object" }, "AWSBedrockModelProvider": { "description": "Represents a model provider using Amazon Bedrock.\n\nExamples:\n 1. Create provider using credentials object:\n .. code-block:: python\n\n provider = AWSBedrockModelProvider(\n credentials=AWSBedrockCredentials(\n aws_access_key_id=\"your-access-key-id\",\n aws_secret_access_key=\"your-secret-access-key\",\n aws_region_name=\"us-east-1\",\n aws_session_token=\"optional-session-token\"\n )\n )\n\n 2. Create provider using environment variables:\n .. code-block:: python\n\n os.environ['AWS_ACCESS_KEY_ID'] = \"your-access-key-id\"\n os.environ['AWS_SECRET_ACCESS_KEY'] = \"your-secret-access-key\"\n os.environ['AWS_SESSION_TOKEN'] = \"optional-session-token\" # Optional\n os.environ['AWS_DEFAULT_REGION'] = \"us-east-1\"\n provider = AWSBedrockModelProvider()", "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "aws_bedrock", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/AWSBedrockCredentials" }, { "type": "null" } ], "default": null, "description": "AWS Bedrock credentials." } }, "title": "AWSBedrockModelProvider", "type": "object" }, "AgentMetricResult": { "description": "This is the data model for metric results in the agentic app.\nIt stores evaluation results for conversations, messages and nodes.", "properties": { "name": { "description": "The name of the metric.", "examples": [ "answer_relevance", "context_relevance" ], "title": "Metric Name", "type": "string" }, "display_name": { "description": "The display name of the metric.", "examples": [ "Answer Relevance", "Context Relevance" ], "title": "Metric display name", "type": "string" }, "value_type": { "default": "numeric", "description": "The type of the metric value. Indicates whether the metric value is numeric or categorical.", "examples": [ "numeric", "categorical" ], "title": "Metric value type", "type": "string" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute this metric result.", "examples": [ "token_recall" ], "title": "Method" }, "provider": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The provider used to compute this metric result.", "title": "Provider" }, "value": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "additionalProperties": { "type": "integer" }, "type": "object" }, { "type": "null" } ], "description": "The metric value.", "title": "Value" }, "label": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The string equivalent of the metric value. This is used for metrics with categorical value type.", "title": "Label" }, "errors": { "anyOf": [ { "items": { "$ref": "#/$defs/Error" }, "type": "array" }, { "type": "null" } ], "default": null, "description": "The list of error messages", "title": "Errors" }, "additional_info": { "anyOf": [ { "type": "object" }, { "type": "null" } ], "default": null, "description": "The additional information about the metric result.", "title": "Additional Info" }, "explanation": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The explanation about the metric result.", "title": "Explanation" }, "group": { "anyOf": [ { "$ref": "#/$defs/MetricGroup" }, { "type": "null" } ], "default": null, "description": "The metric group", "title": "Group" }, "thresholds": { "default": [], "description": "The metric thresholds", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "id": { "description": "The unique identifier for the metric result record. UUID.", "title": "Id", "type": "string" }, "ts": { "description": "The timestamp when the metric was recorded.", "format": "date-time", "title": "Ts", "type": "string" }, "applies_to": { "description": "The type of component the metric result applies to.", "examples": [ "conversation", "message", "node" ], "title": "Applies To", "type": "string" }, "message_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the message being evaluated.", "title": "Message Id" }, "message_ts": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "default": null, "description": "The timestamp of the message being evaluated.", "title": "Message Ts" }, "conversation_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The ID of the conversation containing the message.", "title": "Conversation Id" }, "node_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the node being evaluated.", "title": "Node Name" }, "execution_count": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "The execution count of the node in a message.", "title": "Execution count" }, "execution_order": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "The execution order number in the sequence of nodes executed in a message.", "title": "Execution order" }, "is_violated": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "Indicates whether the metric threshold is violated or not. For numeric metric, its set to 1 if the metric value violates the defined threshold lower or upper limit and 0 otherwise. For categorical metric, its set to 1 if the metric value belongs to unfavourable category and 0 otherwise.", "title": "Is Violated" } }, "required": [ "name", "display_name", "value", "applies_to", "message_id" ], "title": "AgentMetricResult", "type": "object" }, "AgenticAIConfiguration": { "description": "Defines the AgenticAIConfiguration class.\n\nThe configuration interface for Agentic AI tools and applications.\nThis is used to specify the fields mapping details in the data and other configuration parameters needed for evaluation.\n\nExamples:\n 1. Create configuration with default parameters\n .. code-block:: python\n\n configuration = AgenticAIConfiguration()\n\n 2. Create configuration with parameters\n .. code-block:: python\n\n configuration = AgenticAIConfiguration(input_fields=[\"input\"], \n output_fields=[\"output\"])\n\n 2. Create configuration with dict parameters\n .. code-block:: python\n\n config = {\"input_fields\": [\"input\"],\n \"output_fields\": [\"output\"],\n \"context_fields\": [\"contexts\"],\n \"reference_fields\": [\"reference\"]}\n configuration = AgenticAIConfiguration(**config)", "properties": { "record_id_field": { "default": "record_id", "description": "The record identifier field name.", "examples": [ "record_id" ], "title": "Record id field", "type": "string" }, "record_timestamp_field": { "default": "record_timestamp", "description": "The record timestamp field name.", "examples": [ "record_timestamp" ], "title": "Record timestamp field", "type": "string" }, "task_type": { "anyOf": [ { "$ref": "#/$defs/TaskType" }, { "type": "null" } ], "default": null, "description": "The generative task type. Default value is None.", "examples": [ "retrieval_augmented_generation" ], "title": "Task Type" }, "input_fields": { "default": [ "input_text" ], "description": "The list of model input fields in the data. Default value is ['input_text'].", "examples": [ [ "question" ] ], "items": { "type": "string" }, "title": "Input Fields", "type": "array" }, "context_fields": { "default": [ "context" ], "description": "The list of context fields in the input fields. Default value is ['context'].", "examples": [ [ "context1", "context2" ] ], "items": { "type": "string" }, "title": "Context Fields", "type": "array" }, "output_fields": { "default": [ "generated_text" ], "description": "The list of model output fields in the data. Default value is ['generated_text'].", "examples": [ [ "output" ] ], "items": { "type": "string" }, "title": "Output Fields", "type": "array" }, "reference_fields": { "default": [ "ground_truth" ], "description": "The list of reference fields in the data. Default value is ['ground_truth'].", "examples": [ [ "reference" ] ], "items": { "type": "string" }, "title": "Reference Fields", "type": "array" }, "locale": { "anyOf": [ { "$ref": "#/$defs/Locale" }, { "type": "null" } ], "default": null, "description": "The language locale of the input, output and reference fields in the data.", "title": "Locale" }, "tools": { "default": [], "description": "The list of tools used by the LLM.", "examples": [ [ "function1", "function2" ] ], "items": { "type": "object" }, "title": "Tools", "type": "array" }, "tool_calls_field": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "tool_calls", "description": "The tool calls field in the input fields. Default value is 'tool_calls'.", "examples": [ "tool_calls" ], "title": "Tool Calls Field" }, "available_tools_field": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "available_tools", "description": "The tool inventory field in the data. Default value is 'available_tools'.", "examples": [ "available_tools" ], "title": "Available Tools Field" }, "llm_judge": { "anyOf": [ { "$ref": "#/$defs/LLMJudge" }, { "type": "null" } ], "default": null, "description": "LLM as Judge Model details.", "title": "LLM Judge" }, "prompt_field": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "model_prompt", "description": "The prompt field in the input fields. Default value is 'model_prompt'.", "examples": [ "model_prompt" ], "title": "Model Prompt Field" }, "message_id_field": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "message_id", "description": "The message identifier field name. Default value is 'message_id'.", "examples": [ "message_id" ], "title": "Message id field" }, "conversation_id_field": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "conversation_id", "description": "The conversation identifier field name. Default value is 'conversation_id'.", "examples": [ "conversation_id" ], "title": "Conversation id field" } }, "title": "AgenticAIConfiguration", "type": "object" }, "AggregateAgentMetricResult": { "properties": { "name": { "description": "The name of the metric.", "examples": [ "answer_relevance", "context_relevance" ], "title": "Metric Name", "type": "string" }, "display_name": { "description": "The display name of the metric.", "examples": [ "Answer Relevance", "Context Relevance" ], "title": "Metric display name", "type": "string" }, "value_type": { "default": "numeric", "description": "The type of the metric value. Indicates whether the metric value is numeric or categorical.", "examples": [ "numeric", "categorical" ], "title": "Metric value type", "type": "string" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute this metric result.", "examples": [ "token_recall" ], "title": "Method" }, "provider": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The provider used to compute this metric result.", "title": "Provider" }, "value": { "anyOf": [ { "type": "number" }, { "additionalProperties": { "type": "integer" }, "type": "object" }, { "type": "null" } ], "default": null, "description": "The value of the metric. Defaults to mean for numeric metric types. For categorical metric types, this has the frequency distribution of non-null categories.", "title": "Value" }, "label": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The string equivalent of the metric value. This is used for metrics with categorical value type.", "title": "Label" }, "errors": { "anyOf": [ { "items": { "$ref": "#/$defs/Error" }, "type": "array" }, { "type": "null" } ], "default": null, "description": "The list of error messages", "title": "Errors" }, "additional_info": { "anyOf": [ { "type": "object" }, { "type": "null" } ], "default": null, "description": "The additional information about the metric result.", "title": "Additional Info" }, "explanation": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The explanation about the metric result.", "title": "Explanation" }, "group": { "anyOf": [ { "$ref": "#/$defs/MetricGroup" }, { "type": "null" } ], "default": null, "description": "The metric group", "title": "Group" }, "thresholds": { "default": [], "description": "The metric thresholds", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "min": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "default": null, "description": "The minimum value of the metric. Applicable for numeric metric types.", "title": "Min" }, "max": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "default": null, "description": "The maximum value of the metric. Applicable for numeric metric types.", "title": "Max" }, "mean": { "anyOf": [ { "type": "number" }, { "type": "null" } ], "default": null, "description": "The mean value of the metric. Applicable for numeric metric types.", "title": "Mean" }, "percentiles": { "anyOf": [ { "additionalProperties": { "type": "number" }, "type": "object" }, { "type": "null" } ], "default": null, "description": "Dictionary of percentile values (25th, 50th, 75th, 90th, 95th, 99th) of the metric. Applicable for numeric metric types.", "title": "Percentiles" }, "unique": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "The distinct count of the string values found. Applicable for categorical metric types.", "title": "Unique" }, "count": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "The count for metric results used for aggregation.", "title": "Count" }, "node_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the node being evaluated.", "title": "Node Name" }, "applies_to": { "description": "The type of component the metric result applies to.", "examples": [ "conversation", "message", "node" ], "title": "Applies To", "type": "string" }, "individual_results": { "default": [], "description": "The list individual metric results.", "items": { "$ref": "#/$defs/AgentMetricResult" }, "title": "Individual Results", "type": "array" }, "violations_count": { "anyOf": [ { "type": "integer" }, { "type": "null" } ], "default": null, "description": "The count of records that violated the defined thresholds.", "title": "Violations Count" } }, "required": [ "name", "display_name", "applies_to" ], "title": "AggregateAgentMetricResult", "type": "object" }, "AzureOpenAICredentials": { "properties": { "url": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Azure OpenAI url. This attribute can be read from `AZURE_OPENAI_HOST` environment variable.", "title": "Url" }, "api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "API key for Azure OpenAI. This attribute can be read from `AZURE_OPENAI_API_KEY` environment variable.", "title": "Api Key" }, "api_version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The model API version from Azure OpenAI. This attribute can be read from `AZURE_OPENAI_API_VERSION` environment variable.", "title": "Api Version" } }, "required": [ "url", "api_key", "api_version" ], "title": "AzureOpenAICredentials", "type": "object" }, "AzureOpenAIFoundationModel": { "description": "The Azure OpenAI foundation model details\n\nExamples:\n 1. Create Azure OpenAI foundation model by passing the credentials during object creation.\n .. code-block:: python\n\n azure_openai_foundation_model = AzureOpenAIFoundationModel(\n model_id=\"gpt-4o-mini\",\n provider=AzureOpenAIModelProvider(\n credentials=AzureOpenAICredentials(\n api_key=azure_api_key,\n url=azure_host_url,\n api_version=azure_api_model_version,\n )\n )\n )\n\n2. Create Azure OpenAI foundation model by setting the credentials in environment variables:\n * ``AZURE_OPENAI_API_KEY`` is used to set the api key for OpenAI.\n * ``AZURE_OPENAI_HOST`` is used to set the url for Azure OpenAI.\n * ``AZURE_OPENAI_API_VERSION`` is uses to set the the api version for Azure OpenAI.\n\n .. code-block:: python\n\n openai_foundation_model = AzureOpenAIFoundationModel(\n model_id=\"gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/AzureOpenAIModelProvider", "description": "Azure OpenAI provider" }, "model_id": { "description": "Model deployment name from Azure OpenAI", "title": "Model Id", "type": "string" } }, "required": [ "model_id" ], "title": "AzureOpenAIFoundationModel", "type": "object" }, "AzureOpenAIModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "azure_openai", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/AzureOpenAICredentials" }, { "type": "null" } ], "default": null, "description": "Azure OpenAI credentials." } }, "title": "AzureOpenAIModelProvider", "type": "object" }, "CustomFoundationModel": { "description": "Defines the CustomFoundationModel class.\n\nThis class extends the base `FoundationModel` to support custom inference logic through a user-defined scoring function.\nIt is intended for use cases where the model is externally hosted and not in the list of supported frameworks.\nExamples:\n 1. Define a custom scoring function and create a model:\n .. code-block:: python\n\n import pandas as pd\n\n def scoring_fn(data: pd.DataFrame):\n predictions_list = []\n # Custom logic to call an external LLM\n return pd.DataFrame({\"generated_text\": predictions_list}) \n\n model = CustomFoundationModel(\n scoring_fn=scoring_fn\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/ModelProvider", "description": "The provider of the model." } }, "title": "CustomFoundationModel", "type": "object" }, "Error": { "properties": { "code": { "description": "The error code", "title": "Code", "type": "string" }, "message_en": { "description": "The error message in English.", "title": "Message En", "type": "string" }, "parameters": { "default": [], "description": "The list of parameters to construct the message in a different locale.", "items": {}, "title": "Parameters", "type": "array" } }, "required": [ "code", "message_en" ], "title": "Error", "type": "object" }, "FoundationModelInfo": { "description": "Represents a foundation model used in an experiment.", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "model_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The id of the foundation model.", "title": "Model Id" }, "provider": { "description": "The provider of the foundation model.", "title": "Provider", "type": "string" }, "type": { "description": "The type of foundation model.", "example": [ "chat", "embedding", "text-generation" ], "title": "Type", "type": "string" } }, "required": [ "provider", "type" ], "title": "FoundationModelInfo", "type": "object" }, "GenAIMetric": { "description": "Defines the Generative AI metric interface", "properties": { "name": { "description": "The name of the metric.", "examples": [ "answer_relevance", "context_relevance" ], "title": "Metric Name", "type": "string" }, "display_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The display name of the metric.", "examples": [ "Answer Relevance", "Context Relevance" ], "title": "Metric display name" }, "type_": { "default": "ootb", "description": "The type of the metric. Indicates whether the metric is ootb or custom.", "examples": [ "ootb", "custom" ], "title": "Metric type", "type": "string" }, "value_type": { "default": "numeric", "description": "The type of the metric value. Indicates whether the metric value is numeric or categorical.", "examples": [ "numeric", "categorical" ], "title": "Metric value type", "type": "string" }, "thresholds": { "default": [], "description": "The list of thresholds", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "tasks": { "default": [], "description": "The task types this metric is associated with.", "items": { "$ref": "#/$defs/TaskType" }, "title": "Tasks", "type": "array" }, "group": { "anyOf": [ { "$ref": "#/$defs/MetricGroup" }, { "type": "null" } ], "default": null, "description": "The metric group this metric belongs to." }, "is_reference_free": { "default": true, "description": "Decides whether this metric needs a reference for computation", "title": "Is Reference Free", "type": "boolean" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute the metric.", "title": "Method" }, "metric_dependencies": { "default": [], "description": "Metrics that needs to be evaluated first", "items": { "$ref": "#/$defs/GenAIMetric" }, "title": "Metric Dependencies", "type": "array" }, "applies_to": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "message", "description": "The tag to indicate for which the metric is applied to. Used for agentic application metric computation.", "examples": [ "message", "conversation", "sub_agent" ], "title": "Applies to" }, "mapping": { "anyOf": [ { "$ref": "#/$defs/Mapping" }, { "type": "null" } ], "default": null, "description": "The data mapping details for the metric which are used to read the values needed to compute the metric.", "examples": { "items": [ { "attribute_name": "traceloop.entity.input", "column_name": null, "json_path": "$.inputs.input_text", "lookup_child_spans": false, "name": "input_text", "span_name": "LangGraph.workflow", "type": "input" }, { "attribute_name": "traceloop.entity.output", "column_name": null, "json_path": "$.outputs.generated_text", "lookup_child_spans": false, "name": "generated_text", "span_name": "LangGraph.workflow", "type": "output" } ], "source": "trace" }, "title": "Mapping" } }, "required": [ "name" ], "title": "GenAIMetric", "type": "object" }, "GoogleAIStudioCredentials": { "description": "Defines the GoogleAIStudioCredentials class for accessing Google AI Studio using an API key.\n\nExamples:\n 1. Create credentials manually:\n .. code-block:: python\n\n google_credentials = GoogleAIStudioCredentials(api_key=\"your-api-key\")\n\n 2. Create credentials from environment:\n .. code-block:: python\n\n os.environ[\"GOOGLE_API_KEY\"] = \"your-api-key\"\n google_credentials = GoogleAIStudioCredentials.create_from_env()", "properties": { "api_key": { "description": "The Google AI Studio key. This attribute can be read from GOOGLE_API_KEY environment variable when creating GoogleAIStudioCredentials from environment.", "title": "Api Key", "type": "string" } }, "required": [ "api_key" ], "title": "GoogleAIStudioCredentials", "type": "object" }, "GoogleAIStudioFoundationModel": { "description": "Represents a foundation model served via Google AI Studio.\n\nExamples:\n 1. Create Google AI Studio foundation model by passing the credentials during object creation.\n .. code-block:: python\n\n model = GoogleAIStudioFoundationModel(\n model_id=\"gemini-1.5-pro-002\",\n provider=GoogleAIStudioModelProvider(\n credentials=GoogleAIStudioCredentials(api_key=\"your_api_key\")\n )\n )\n 2. Create Google AI Studio foundation model by setting the credentials in environment variables:\n * ``GOOGLE_API_KEY`` OR ``GEMINI_API_KEY`` is used to set the Credentials path for Vertex AI.\n .. code-block:: python\n\n model = GoogleAIStudioFoundationModel(\n model_id=\"gemini/gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/GoogleAIStudioModelProvider", "description": "Google AI Studio provider.", "title": "Provider" }, "model_id": { "description": "Model name for Google AI Studio. Must be a valid Google AI model identifier or a fully-qualified publisher path", "examples": [ "gemini-1.5-pro-002" ], "title": "Model id", "type": "string" } }, "required": [ "model_id" ], "title": "GoogleAIStudioFoundationModel", "type": "object" }, "GoogleAIStudioModelProvider": { "description": "Represents a model provider using Google AI Studio.\n\nExamples:\n 1. Create provider using credentials object:\n .. code-block:: python\n\n provider = GoogleAIStudioModelProvider(\n credentials=GoogleAIStudioCredentials(api_key=\"api-key\")\n )\n\n 2. Create provider using environment variables:\n .. code-block:: python\n\n os.environ['GOOGLE_API_KEY'] = \"your_api_key\"\n\n provider = GoogleAIStudioModelProvider()", "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "google_ai_studio", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/GoogleAIStudioCredentials" }, { "type": "null" } ], "default": null, "description": "Google AI Studio credentials." } }, "title": "GoogleAIStudioModelProvider", "type": "object" }, "LLMJudge": { "description": "Defines the LLMJudge.\n\nThe LLMJudge class contains the details of the llm judge model to be used for computing the metric.\n\nExamples:\n 1. Create LLMJudge using watsonx.ai foundation model:\n .. code-block:: python\n\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=PROJECT_ID,\n provider=WxAIModelProvider(\n credentials=WxAICredentials(api_key=wx_apikey)\n )\n )\n llm_judge = LLMJudge(model=wx_ai_foundation_model)", "properties": { "model": { "anyOf": [ { "$ref": "#/$defs/WxAIFoundationModel" }, { "$ref": "#/$defs/OpenAIFoundationModel" }, { "$ref": "#/$defs/AzureOpenAIFoundationModel" }, { "$ref": "#/$defs/PortKeyGateway" }, { "$ref": "#/$defs/RITSFoundationModel" }, { "$ref": "#/$defs/VertexAIFoundationModel" }, { "$ref": "#/$defs/GoogleAIStudioFoundationModel" }, { "$ref": "#/$defs/AWSBedrockFoundationModel" }, { "$ref": "#/$defs/CustomFoundationModel" } ], "description": "The foundation model to be used as judge", "title": "Model" } }, "required": [ "model" ], "title": "LLMJudge", "type": "object" }, "Locale": { "properties": { "input": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "additionalProperties": { "type": "string" }, "type": "object" }, { "type": "string" }, { "type": "null" } ], "default": null, "title": "Input" }, "output": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "type": "null" } ], "default": null, "title": "Output" }, "reference": { "anyOf": [ { "items": { "type": "string" }, "type": "array" }, { "additionalProperties": { "type": "string" }, "type": "object" }, { "type": "string" }, { "type": "null" } ], "default": null, "title": "Reference" } }, "title": "Locale", "type": "object" }, "Mapping": { "description": "Defines the field mapping details to be used for computing a metric.", "properties": { "source": { "default": "trace", "description": "The source type of the data. Use trace if the data should be read from span in trace. Use tabular if the data is passed as a dataframe.", "enum": [ "trace", "tabular" ], "examples": [ "trace", "tabular" ], "title": "Source", "type": "string" }, "items": { "description": "The list of mapping items for the field. They are used to read the data from trace or tabular data for computing the metric.", "items": { "$ref": "#/$defs/MappingItem" }, "title": "Mapping Items", "type": "array" } }, "required": [ "items" ], "title": "Mapping", "type": "object" }, "MappingItem": { "description": "The mapping details to be used for reading the values from the data.", "properties": { "name": { "description": "The name of the item.", "examples": [ "input_text", "generated_text", "context", "ground_truth" ], "title": "Name", "type": "string" }, "type": { "description": "The type of the item.", "enum": [ "input", "output", "reference", "context", "tool_call" ], "examples": [ "input" ], "title": "Type", "type": "string" }, "column_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The column name in the tabular data to be used for reading the field value. Applicable for tabular source.", "title": "Column Name" }, "span_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The span name in the trace data to be used for reading the field value. Applicable for trace source.", "title": "Span Name" }, "attribute_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The attribute name in the trace to be used for reading the field value. Applicable for trace source.", "title": "Attribute Name" }, "json_path": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The json path to be used for reading the field value from the attribute value. Applicable for trace source. If not provided, the span attribute value is read as the field value.", "title": "Json Path" }, "lookup_child_spans": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "default": false, "description": "The flag to indicate if all the child spans should be searched for the attribute value. Applicable for trace source.", "title": "Look up child spans" } }, "required": [ "name", "type" ], "title": "MappingItem", "type": "object" }, "MessageData": { "description": "The model class to capture the message input output data for an agent.", "properties": { "message_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the message.", "title": "Message ID" }, "message_ts": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The timestamp of the message in ISO format. The end timestamp of the message processing is considered as the message timestamp.", "title": "Message timestamp" }, "conversation_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the conversation containing the message.", "title": "Conversation ID" }, "start_time": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The message execution start time in ISO format.", "title": "Start time" }, "end_time": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The message excution end time in ISO format.", "title": "End time" }, "input": { "anyOf": [ { "type": "object" }, { "type": "string" }, { "type": "null" } ], "description": "The message input data.", "title": "Input" }, "output": { "anyOf": [ { "type": "object" }, { "type": "string" }, { "type": "null" } ], "description": "The message output data.", "title": "Input" }, "num_loops": { "default": 0, "description": "The number of loops occurred in the agent while generating the output.", "title": "Number of Loops", "type": "integer" } }, "required": [ "message_id", "message_ts", "conversation_id", "start_time", "end_time", "input", "output" ], "title": "MessageData", "type": "object" }, "MetricGroup": { "enum": [ "retrieval_quality", "answer_quality", "content_safety", "performance", "usage", "message_completion", "tool_call_quality", "readability", "custom" ], "title": "MetricGroup", "type": "string" }, "MetricMapping": { "description": "The metric mapping data", "properties": { "name": { "description": "The name of the metric.", "title": "Name", "type": "string" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute the metric.", "title": "Method" }, "applies_to": { "title": "Applies To", "type": "string" }, "mapping": { "anyOf": [ { "$ref": "#/$defs/Mapping" }, { "type": "null" } ], "default": null, "description": "The data mapping details for the metric which are used to read the values needed to compute the metric.", "title": "Mapping" } }, "required": [ "name" ], "title": "MetricMapping", "type": "object" }, "MetricThreshold": { "description": "The class that defines the threshold for a metric.", "properties": { "type": { "description": "Threshold type. One of 'lower_limit', 'upper_limit'", "enum": [ "lower_limit", "upper_limit" ], "title": "Type", "type": "string" }, "value": { "default": 0, "description": "The value of metric threshold", "title": "Threshold value", "type": "number" } }, "required": [ "type" ], "title": "MetricThreshold", "type": "object" }, "MetricsConfiguration": { "description": "The class representing the metrics to be computed and the configuration details required for them.\n\nExamples:\n 1. Create MetricsConfiguration with default agentic ai configuration\n .. code-block:: python\n\n metrics_configuration = MetricsConfiguration(metrics=[ContextRelevanceMetric()],\n metric_groups=[MetricGroup.RETRIEVAL_QUALITY])])\n\n 2. Create MetricsConfiguration by specifying agentic ai configuration\n .. code-block:: python\n\n config = {\n \"input_fields\": [\"input\"],\n \"context_fields\": [\"contexts\"]\n }\n metrics_configuration = MetricsConfiguration(configuration=AgenticAIConfiguration(**config)\n metrics=[ContextRelevanceMetric()],\n metric_groups=[MetricGroup.RETRIEVAL_QUALITY])])", "properties": { "configuration": { "$ref": "#/$defs/AgenticAIConfiguration", "default": { "record_id_field": "record_id", "record_timestamp_field": "record_timestamp", "task_type": null, "input_fields": [ "input_text" ], "context_fields": [ "context" ], "output_fields": [ "generated_text" ], "reference_fields": [ "ground_truth" ], "locale": null, "tools": [], "tool_calls_field": "tool_calls", "available_tools_field": "available_tools", "llm_judge": null, "prompt_field": "model_prompt", "message_id_field": "message_id", "conversation_id_field": "conversation_id" }, "description": "The configuration of the metrics to compute. The configuration contains the fields names to be read when computing the metrics.", "title": "Metrics configuration" }, "metrics": { "anyOf": [ { "items": { "$ref": "#/$defs/GenAIMetric" }, "type": "array" }, { "type": "null" } ], "default": [], "description": "The list of metrics to compute.", "title": "Metrics" }, "metric_groups": { "anyOf": [ { "items": { "$ref": "#/$defs/MetricGroup" }, "type": "array" }, { "type": "null" } ], "default": [], "description": "The list of metric groups to compute.", "title": "Metric Groups" } }, "title": "MetricsConfiguration", "type": "object" }, "MetricsMappingData": { "description": "The model class to capture the metrics mappings and the span data.", "properties": { "message_id": { "description": "The ID of the message.", "title": "Message ID", "type": "string" }, "metric_mappings": { "description": "The list of metric mappings.", "items": { "$ref": "#/$defs/MetricMapping" }, "title": "Metric Mapping", "type": "array" }, "data": { "description": "The span data used for metrics computation.", "examples": [ { "LangGraph.workflow": { "traceloop.entity.output": { "$.outputs.generated_text": "The response" } } } ], "title": "Data", "type": "object" } }, "required": [ "message_id", "metric_mappings", "data" ], "title": "MetricsMappingData", "type": "object" }, "ModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "description": "The type of model provider." } }, "required": [ "type" ], "title": "ModelProvider", "type": "object" }, "ModelProviderType": { "description": "Supported model provider types for Generative AI", "enum": [ "ibm_watsonx.ai", "azure_openai", "rits", "openai", "vertex_ai", "google_ai_studio", "aws_bedrock", "custom", "portkey" ], "title": "ModelProviderType", "type": "string" }, "Node": { "description": "The class representing a node in an agentic application.\n\nExamples:\n 1. Create Node with metrics configuration and default agentic ai configuration\n .. code-block:: python\n\n metrics_configurations = [MetricsConfiguration(metrics=[ContextRelevanceMetric()],\n metric_groups=[MetricGroup.RETRIEVAL_QUALITY])])]\n node = Node(name=\"Retrieval Node\",\n metrics_configurations=metrics_configurations)\n\n 2. Create Node with metrics configuration and specifying agentic ai configuration\n .. code-block:: python\n\n node_config = {\"input_fields\": [\"input\"],\n \"output_fields\": [\"output\"],\n \"context_fields\": [\"contexts\"],\n \"reference_fields\": [\"reference\"]}\n metrics_configurations = [MetricsConfiguration(configuration=AgenticAIConfiguration(**node_config)\n metrics=[ContextRelevanceMetric()],\n metric_groups=[MetricGroup.RETRIEVAL_QUALITY])])]\n node = Node(name=\"Retrieval Node\",\n metrics_configurations=metrics_configurations)", "properties": { "name": { "description": "The name of the node.", "title": "Name", "type": "string" }, "func_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the node function.", "title": "Node function name" }, "metrics_configurations": { "default": [], "description": "The list of metrics and their configuration details.", "items": { "$ref": "#/$defs/MetricsConfiguration" }, "title": "Metrics configuration", "type": "array" }, "foundation_models": { "default": [], "description": "The Foundation models invoked by the node", "items": { "$ref": "#/$defs/FoundationModelInfo" }, "title": "Foundation Models", "type": "array" } }, "required": [ "name" ], "title": "Node", "type": "object" }, "NodeData": { "description": "The model class to capture the node input output data of a langgraph agent.", "properties": { "message_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the message.", "title": "Message ID" }, "message_ts": { "anyOf": [ { "format": "date-time", "type": "string" }, { "type": "null" } ], "description": "The timestamp of the message in ISO format. The end timestamp of the message processing is considered as the message timestamp.", "title": "Message timestamp" }, "conversation_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The ID of the conversation containing the message.", "title": "Conversation ID" }, "node_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The name of the node.", "title": "Node name" }, "start_time": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The node execution start time in ISO format.", "title": "Start time" }, "end_time": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The node execution end time in ISO format.", "title": "End time" }, "input": { "anyOf": [ { "type": "object" }, { "type": "string" }, { "type": "null" } ], "description": "The node input data.", "title": "Input" }, "output": { "anyOf": [ { "type": "object" }, { "type": "string" }, { "type": "null" } ], "description": "The node output data.", "title": "Input" }, "execution_order": { "default": 0, "description": "The execution order of the node in the langgraph.", "title": "Execution Order", "type": "integer" }, "execution_count": { "default": 0, "description": "The execution count of the node in the langgraph.", "title": "Execution Count", "type": "integer" }, "node_txn_id": { "default": "94a5377c-af46-45bd-b4ea-1a61b1d7e126", "description": "Unique identifier of the object.", "title": "Node transaction id", "type": "string" }, "node_txn_timestamp": { "description": "The node transaction timestamp. The end timestamp of the node execution is considered as the node transaction timestamp.", "title": "Node transaction timestamp", "type": "string" } }, "required": [ "message_id", "message_ts", "conversation_id", "node_name", "start_time", "end_time", "input", "output", "node_txn_timestamp" ], "title": "NodeData", "type": "object" }, "OpenAICredentials": { "description": "Defines the OpenAICredentials class to specify the OpenAI server details.\n\nExamples:\n 1. Create OpenAICredentials with default parameters. By default Dallas region is used.\n .. code-block:: python\n\n openai_credentials = OpenAICredentials(api_key=api_key,\n url=openai_url)\n\n 2. Create OpenAICredentials by reading from environment variables.\n .. code-block:: python\n\n os.environ[\"OPENAI_API_KEY\"] = \"...\"\n os.environ[\"OPENAI_URL\"] = \"...\"\n openai_credentials = OpenAICredentials.create_from_env()", "properties": { "url": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Url" }, "api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Api Key" } }, "required": [ "url", "api_key" ], "title": "OpenAICredentials", "type": "object" }, "OpenAIFoundationModel": { "description": "The OpenAI foundation model details\n\nExamples:\n 1. Create OpenAI foundation model by passing the credentials during object creation. Note that the url is optional and will be set to the default value for OpenAI. To change the default value, the url should be passed to ``OpenAICredentials`` object.\n .. code-block:: python\n\n openai_foundation_model = OpenAIFoundationModel(\n model_id=\"gpt-4o-mini\",\n provider=OpenAIModelProvider(\n credentials=OpenAICredentials(\n api_key=api_key,\n url=openai_url,\n )\n )\n )\n\n 2. Create OpenAI foundation model by setting the credentials in environment variables:\n * ``OPENAI_API_KEY`` is used to set the api key for OpenAI.\n * ``OPENAI_URL`` is used to set the url for OpenAI\n\n .. code-block:: python\n\n openai_foundation_model = OpenAIFoundationModel(\n model_id=\"gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/OpenAIModelProvider", "description": "OpenAI provider" }, "model_id": { "description": "Model name from OpenAI", "title": "Model Id", "type": "string" } }, "required": [ "model_id" ], "title": "OpenAIFoundationModel", "type": "object" }, "OpenAIModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "openai", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/OpenAICredentials" }, { "type": "null" } ], "default": null, "description": "OpenAI credentials. This can also be set by using `OPENAI_API_KEY` environment variable." } }, "title": "OpenAIModelProvider", "type": "object" }, "PortKeyCredentials": { "description": "Defines the PortKeyCredentials class to specify the PortKey Gateway details.\n\nExamples:\n 1. Create PortKeyCredentials with default parameters.\n .. code-block:: python\n\n portkey_credentials = PortKeyCredentials(api_key=api_key,\n url=portkey_url,\n provider_api_key=provider_api_key,\n provider=provider_name)\n\n 2. Create PortKeyCredentials by reading from environment variables.\n .. code-block:: python\n\n os.environ[\"PORTKEY_API_KEY\"] = \"...\"\n os.environ[\"PORTKEY_URL\"] = \"...\"\n os.environ[\"PORTKEY_PROVIDER_API_KEY\"] = \"...\"\n os.environ[\"PORTKEY_PROVIDER_NAME\"] = \"...\"\n portkey_credentials = PortKeyCredentials.create_from_env()", "properties": { "url": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "PortKey url. This attribute can be read from `PORTKEY_URL` environment variable.", "title": "Url" }, "api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "API key for PortKey. This attribute can be read from `PORTKEY_API_KEY` environment variable.", "title": "Api Key" }, "provider_api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "API key for the provider. This attribute can be read from `PORTKEY_PROVIDER_API_KEY` environment variable.", "title": "Provider Api Key" }, "provider": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The provider name. This attribute can be read from `PORTKEY_PROVIDER_NAME` environment variable.", "title": "Provider" } }, "required": [ "url", "api_key", "provider_api_key", "provider" ], "title": "PortKeyCredentials", "type": "object" }, "PortKeyGateway": { "description": "The PortKey gateway details\n\nExamples:\n 1. Create PortKeyGateway by passing the credentials during object creation. Note that the url is optional and will be set to the default value for PortKey. To change the default value, the url should be passed to ``PortKeyCredentials`` object.\n .. code-block:: python\n\n port_key_gateway = PortKeyGateway(\n model_id=\"gpt-4o-mini\",\n provider=PortKeyModelProvider(\n credentials=PortKeyCredentials(\n api_key=api_key,\n url=openai_url,\n provider_api_key=provider_api_key,\n provider_name=provider_name\n )\n )\n )\n\n 2. Create PortKeyGateway by setting the credentials in environment variables:\n * ``PORTKEY_API_KEY`` is used to set the api key for PortKey.\n * ``PORTKEY_URL`` is used to set the url for PortKey.\n * ``PORTKEY_PROVIDER_API_KEY`` is used to set the provider api key for PortKey.\n * ``PORTKEY_PROVIDER_NAME`` is used to set the provider name for PortKey\n\n .. code-block:: python\n\n port_key_gateway = PortKeyGateway(\n model_id=\"gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/PortKeyModelProvider", "description": "PortKey Provider" }, "model_id": { "description": "Model name from the Provider", "title": "Model Id", "type": "string" } }, "required": [ "model_id" ], "title": "PortKeyGateway", "type": "object" }, "PortKeyModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "portkey", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/PortKeyCredentials" }, { "type": "null" } ], "default": null, "description": "PortKey credentials." } }, "title": "PortKeyModelProvider", "type": "object" }, "RITSCredentials": { "properties": { "hostname": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "https://inference-3scale-apicast-production.apps.rits.fmaas.res.ibm.com", "description": "The rits hostname", "title": "Hostname" }, "api_key": { "title": "Api Key", "type": "string" } }, "required": [ "api_key" ], "title": "RITSCredentials", "type": "object" }, "RITSFoundationModel": { "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/RITSModelProvider", "description": "The provider of the model." } }, "title": "RITSFoundationModel", "type": "object" }, "RITSModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "rits", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/RITSCredentials" }, { "type": "null" } ], "default": null, "description": "RITS credentials." } }, "title": "RITSModelProvider", "type": "object" }, "TaskType": { "description": "Supported task types for generative AI models", "enum": [ "question_answering", "classification", "summarization", "generation", "extraction", "retrieval_augmented_generation" ], "title": "TaskType", "type": "string" }, "VertexAICredentials": { "description": "Defines the VertexAICredentials class for accessing Vertex AI using service account credentials.\n\nExamples:\n 1. Create credentials manually:\n .. code-block:: python\n\n vertex_credentials = VertexAICredentials(\n credentials_path=\"path/to/service_account.json\",\n project_id=\"my-gcp-project\",\n location=\"us-central1\"\n )\n\n 2. Create credentials from environment:\n .. code-block:: python\n\n os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = \"path/to/service_account.json\"\n os.environ[\"GOOGLE_CLOUD_PROJECT\"] = \"my-gcp-project\"\n os.environ[\"GOOGLE_CLOUD_LOCATION\"] = \"us-central1\"\n\n vertex_ai_credentials = VertexAICredentials.create_from_env()", "properties": { "credentials_path": { "description": "Path to service-account JSON. This attribute can be read from GOOGLE_APPLICATION_CREDENTIALS environment variable when creating VertexAICredentials from environment.", "title": "Credentials Path", "type": "string" }, "project_id": { "description": "The Google Cloud project id. This attribute can be read from GOOGLE_CLOUD_PROJECT or GCLOUD_PROJECT environment variable when creating VertexAICredentials from environment.", "title": "Project ID", "type": "string" }, "location": { "default": "us-central1", "description": "Vertex AI region. This attribute can be read from GOOGLE_CLOUD_LOCATION environment variable when creating VertexAICredentials from environment. By default us-central1 location is used.", "examples": [ "us-central1", "europe-west4" ], "title": "Location", "type": "string" } }, "required": [ "credentials_path", "project_id" ], "title": "VertexAICredentials", "type": "object" }, "VertexAIFoundationModel": { "description": "Represents a foundation model served via Vertex AI.\n\nExamples:\n 1. Create Vertex AI foundation model by passing the credentials during object creation.\n .. code-block:: python\n\n model = VertexAIFoundationModel(\n model_id=\"gemini-1.5-pro-002\",\n provider=VertexAIModelProvider(\n credentials=VertexAICredentials(\n project_id=\"your-project\",\n location=\"us-central1\", # This is optional field, by default us-central1 location is selected\n credentials_path=\"/path/to/service_account.json\"\n )\n )\n )\n 2. Create Vertex AI foundation model by setting the credentials in environment variables:\n * ``GOOGLE_APPLICATION_CREDENTIALS`` is used to set the Credentials path for Vertex AI.\n * ``GOOGLE_CLOUD_PROJECT`` is used to set the Project id for Vertex AI.\n * ``GOOGLE_CLOUD_LOCATION`` is uses to set the Location for Vertex AI. By default us-central1 location is used when GOOGLE_CLOUD_LOCATION is not provided .\n\n .. code-block:: python\n\n os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = \"path/to/service_account.json\"\n os.environ[\"GOOGLE_CLOUD_PROJECT\"] = \"my-gcp-project\"\n os.environ[\"GOOGLE_CLOUD_LOCATION\"] = \"us-central1\"\n\n model = VertexAIFoundationModel(\n model_id=\"gemini/gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/VertexAIModelProvider", "description": "Vertex AI provider.", "title": "Provider" }, "model_id": { "description": "Model name for Vertex AI. Must be a valid Vertex AI model identifier or a fully-qualified publisher path", "examples": [ "gemini-1.5-pro-002" ], "title": "Model id", "type": "string" } }, "required": [ "model_id" ], "title": "VertexAIFoundationModel", "type": "object" }, "VertexAIModelProvider": { "description": "Represents a model provider using Vertex AI.\n\nExamples:\n 1. Create provider using credentials object:\n .. code-block:: python\n\n provider = VertexAIModelProvider(\n credentials=VertexAICredentials(\n credentials_path=\"path/to/key.json\",\n project_id=\"your-project\",\n location=\"us-central1\" \n )\n )\n\n 2. Create provider using environment variables:\n .. code-block:: python\n\n os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = \"/path/to/service_account.json\"\n os.environ['GOOGLE_CLOUD_PROJECT'] = \"your-project\"\n os.environ['GOOGLE_CLOUD_LOCATION'] = \"us-central1\" # This is optional field, by default us-central1 location is selected\n\n provider = VertexAIModelProvider()", "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "vertex_ai", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/VertexAICredentials" }, { "type": "null" } ], "default": null, "description": "Vertex AI credentials." } }, "title": "VertexAIModelProvider", "type": "object" }, "WxAICredentials": { "description": "Defines the WxAICredentials class to specify the watsonx.ai server details.\n\nExamples:\n 1. Create WxAICredentials with default parameters. By default Dallas region is used.\n .. code-block:: python\n\n wxai_credentials = WxAICredentials(api_key=\"...\")\n\n 2. Create WxAICredentials by specifying region url.\n .. code-block:: python\n\n wxai_credentials = WxAICredentials(api_key=\"...\",\n url=\"https://au-syd.ml.cloud.ibm.com\")\n\n 3. Create WxAICredentials by reading from environment variables.\n .. code-block:: python\n\n os.environ[\"WATSONX_APIKEY\"] = \"...\"\n # [Optional] Specify watsonx region specific url. Default is https://us-south.ml.cloud.ibm.com .\n os.environ[\"WATSONX_URL\"] = \"https://eu-gb.ml.cloud.ibm.com\"\n wxai_credentials = WxAICredentials.create_from_env()\n\n 4. Create WxAICredentials for on-prem.\n .. code-block:: python\n\n wxai_credentials = WxAICredentials(url=\"https://<hostname>\",\n username=\"...\"\n api_key=\"...\",\n version=\"5.2\")\n\n 5. Create WxAICredentials by reading from environment variables for on-prem.\n .. code-block:: python\n\n os.environ[\"WATSONX_URL\"] = \"https://<hostname>\"\n os.environ[\"WATSONX_VERSION\"] = \"5.2\"\n os.environ[\"WATSONX_USERNAME\"] = \"...\"\n os.environ[\"WATSONX_APIKEY\"] = \"...\"\n # Only one of api_key or password is needed\n #os.environ[\"WATSONX_PASSWORD\"] = \"...\"\n wxai_credentials = WxAICredentials.create_from_env()", "properties": { "url": { "default": "https://us-south.ml.cloud.ibm.com", "description": "The url for watsonx ai service", "examples": [ "https://us-south.ml.cloud.ibm.com", "https://eu-de.ml.cloud.ibm.com", "https://eu-gb.ml.cloud.ibm.com", "https://jp-tok.ml.cloud.ibm.com", "https://au-syd.ml.cloud.ibm.com" ], "title": "watsonx.ai url", "type": "string" }, "api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The user api key. Required for using watsonx as a service and one of api_key or password is required for using watsonx on-prem software.", "strip_whitespace": true, "title": "Api Key" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The watsonx on-prem software version. Required for using watsonx on-prem software.", "title": "Version" }, "username": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The user name. Required for using watsonx on-prem software.", "title": "User name" }, "password": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The user password. One of api_key or password is required for using watsonx on-prem software.", "title": "Password" }, "instance_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "openshift", "description": "The watsonx.ai instance id. Default value is openshift.", "title": "Instance id" } }, "title": "WxAICredentials", "type": "object" }, "WxAIFoundationModel": { "description": "The IBM watsonx.ai foundation model details\n\nTo initialize the foundation model, you can either pass in the credentials directly or set the environment.\nYou can follow these examples to create the provider.\n\nExamples:\n 1. Create foundation model by specifying the credentials during object creation:\n .. code-block:: python\n\n # Specify the credentials during object creation\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=<PROJECT_ID>,\n provider=WxAIModelProvider(\n credentials=WxAICredentials(\n url=wx_url, # This is optional field, by default US-Dallas region is selected\n api_key=wx_apikey,\n )\n )\n )\n\n 2. Create foundation model by setting the credentials environment variables:\n * The api key can be set using one of the environment variables ``WXAI_API_KEY``, ``WATSONX_APIKEY``, or ``WXG_API_KEY``. These will be read in the order of precedence.\n * The url is optional and will be set to US-Dallas region by default. It can be set using one of the environment variables ``WXAI_URL``, ``WATSONX_URL``, or ``WXG_URL``. These will be read in the order of precedence.\n\n .. code-block:: python\n\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=<PROJECT_ID>,\n )\n\n 3. Create foundation model by specifying watsonx.governance software credentials during object creation:\n .. code-block:: python\n\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=project_id,\n provider=WxAIModelProvider(\n credentials=WxAICredentials(\n url=wx_url,\n api_key=wx_apikey,\n username=wx_username,\n version=wx_version,\n )\n )\n )\n\n 4. Create foundation model by setting watsonx.governance software credentials environment variables:\n * The api key can be set using one of the environment variables ``WXAI_API_KEY``, ``WATSONX_APIKEY``, or ``WXG_API_KEY``. These will be read in the order of precedence.\n * The url can be set using one of these environment variable ``WXAI_URL``, ``WATSONX_URL``, or ``WXG_URL``. These will be read in the order of precedence.\n * The username can be set using one of these environment variable ``WXAI_USERNAME``, ``WATSONX_USERNAME``, or ``WXG_USERNAME``. These will be read in the order of precedence.\n * The version of watsonx.governance software can be set using one of these environment variable ``WXAI_VERSION``, ``WATSONX_VERSION``, or ``WXG_VERSION``. These will be read in the order of precedence.\n\n .. code-block:: python\n\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=project_id,\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/WxAIModelProvider", "description": "The provider of the model." }, "model_id": { "description": "The unique identifier for the watsonx.ai model.", "title": "Model Id", "type": "string" }, "project_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The project ID associated with the model.", "title": "Project Id" }, "space_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The space ID associated with the model.", "title": "Space Id" } }, "required": [ "model_id" ], "title": "WxAIFoundationModel", "type": "object" }, "WxAIModelProvider": { "description": "This class represents a model provider configuration for IBM watsonx.ai. It includes the provider type and\ncredentials required to authenticate and interact with the watsonx.ai platform. If credentials are not explicitly\nprovided, it attempts to load them from environment variables.\n\nExamples:\n 1. Create provider using credentials object:\n .. code-block:: python\n\n credentials = WxAICredentials(\n url=\"https://us-south.ml.cloud.ibm.com\",\n api_key=\"your-api-key\"\n )\n provider = WxAIModelProvider(credentials=credentials)\n\n 2. Create provider using environment variables:\n .. code-block:: python\n\n import os\n\n os.environ['WATSONX_URL'] = \"https://us-south.ml.cloud.ibm.com\"\n os.environ['WATSONX_APIKEY'] = \"your_api_key\"\n\n provider = WxAIModelProvider()", "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "ibm_watsonx.ai", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/WxAICredentials" }, { "type": "null" } ], "default": null, "description": "The credentials used to authenticate with watsonx.ai. If not provided, they will be loaded from environment variables." } }, "title": "WxAIModelProvider", "type": "object" } }, "required": [ "metrics_results", "aggregated_metrics_results" ] }
- Fields:
messages_data (List[ibm_watsonx_gov.entities.evaluation_result.MessageData])metrics_mapping_data (List[ibm_watsonx_gov.entities.evaluation_result.MetricsMappingData])metrics_results (List[ibm_watsonx_gov.entities.evaluation_result.AgentMetricResult])nodes_data (List[ibm_watsonx_gov.entities.evaluation_result.NodeData])
- field aggregated_metrics_results: Annotated[List[AggregateAgentMetricResult], FieldInfo(annotation=NoneType, required=True, title='Aggregated metrics result', description='The list of aggregated metrics result. The metrics are aggregated for each node in the agent.')] [Required]¶
The list of aggregated metrics result. The metrics are aggregated for each node in the agent.
- field edges: Annotated[list[dict], FieldInfo(annotation=NoneType, required=False, default=[], title='Nodes', description='The list of nodes details')] = []¶
The list of nodes details
- field messages_data: Annotated[List[MessageData], FieldInfo(annotation=NoneType, required=False, default=[], title='Messages', description='The list of agent messages data.')] = []¶
The list of agent messages data.
- field metrics_mapping_data: Annotated[List[MetricsMappingData], FieldInfo(annotation=NoneType, required=False, default=[], title='Metrics mapping data', description='The mapping data used to compute the metric.')] = []¶
The mapping data used to compute the metric.
- field metrics_results: Annotated[List[AgentMetricResult], FieldInfo(annotation=NoneType, required=True, title='Metrics result', description='The list of metrics result.')] [Required]¶
The list of metrics result.
- field nodes: Annotated[list[Node], FieldInfo(annotation=NoneType, required=False, default=[], title='Nodes', description='The list of nodes details')] = []¶
The list of nodes details
- field nodes_data: Annotated[List[NodeData], FieldInfo(annotation=NoneType, required=False, default=[], title='Node messages', description='The list of nodes data.')] = []¶
The list of nodes data.
- get_aggregated_metrics_results(applies_to: list[str] = ['conversation', 'message', 'node'], node_name: str | None = None, include_individual_results: bool = True, format: Literal['json', 'object'] = 'json', **kwargs) list[AggregateAgentMetricResult] | list[dict]¶
Get the aggregated agentic metrics results based on the specified arguments.
- Parameters:
applies_to (AGENTIC_RESULT_COMPONENTS, optional) – The type of component the metric result applies to. Defaults to [“conversation”, “message”, “node”].
node_name (str, optional) – The name of the node to get the aggregated results for. Defaults to None.
include_individual_results (bool, optional) – Whether to return the individual metrics results. Defaults to False.
format (Literal["json", "object"], optional) – The format of the output. Defaults to “json”.
- Returns:
list[AggregateAgentMetricResult] | list [dict]
- Return type:
returns
- get_metrics_results(applies_to: list[str] = ['conversation', 'message', 'node'], node_name: str | None = None, format: Literal['json', 'object'] = 'json', **kwargs) list[AgentMetricResult] | list[dict]¶
Get the agentic metrics results based on the specified arguments.
- Parameters:
applies_to (AGENTIC_RESULT_COMPONENTS, optional) – The type of component the metrics results applies to. Defaults to [“conversation”, “message”, “node”].
node_name (str, optional) – The name of the node to get the metrics results for. Defaults to None.
format (Literal["json", "object"], optional) – The format of the output. Defaults to “json”.
- Returns:
list[AgentMetricResult] | list [dict]
- Return type:
returns
- to_df(input_data: DataFrame | None = None, message_id_field: str = 'message_id', wide_format: bool = True) DataFrame¶
Get individual metrics dataframe.
If the input dataframe is provided, it will be merged with the metrics dataframe.
- Parameters:
input_data (Optional[pd.DataFrame], optional) – Input data to merge with metrics dataframe. Defaults to None.
message_id_field (str, optional) – Field to use for merging input data and metrics dataframe. Defaults to “message_id”.
wide_format (bool) – Determines whether to display the results in a pivot table format. Defaults to True
- Returns:
Metrics dataframe.
- Return type:
pd.DataFrame
- to_json(**kwargs) dict¶
Get the AgenticEvaluationResult as json
- Returns:
The AgenticEvaluationResult
- Return type:
dict