Tool Call Parameter Accuracy Metric¶
- pydantic model ibm_watsonx_gov.metrics.tool_call_parameter_accuracy.tool_call_parameter_accuracy_metric.ToolCallParameterAccuracyMetric¶
Bases:
GenAIMetricToolCallParameterAccuracyMetric assesses whether ALL parameter values in a function call are directly supported by conversation history or API specifications. Identifies hallucinated values, missing information, format errors, and contradictory values.
The ToolCallParameterAccuracyMetric will be computed using llm_as_judge.
Examples
- Create ToolCallParameterAccuracyMetric by passing the basic configuration.
config = GenAIConfiguration(tools = [get_weather,fetch_stock_price]) evaluator = MetricsEvaluator(configuration=config) df = pd.read_csv("") llm_judge = LLMJudge( model=WxAIFoundationModel( model_id="meta-llama/llama-3-3-70b-instruct", project_id=os.getenv("WATSONX_PROJECT_ID"), ) ) metrics = [ToolCallParameterAccuracyMetric(llm_judge=llm_judge)] result = evaluator.evaluate(data=df, metrics=metrics)
- Create ToolCallParameterAccuracyMetric by passing custom tool calls field in configuration.
config = GenAIConfiguration(tools = [get_weather,fetch_stock_price], tool_calls_field="tools_used") evaluator = MetricsEvaluator(configuration=config) df = pd.read_csv("") llm_judge = LLMJudge( model=WxAIFoundationModel( model_id="meta-llama/llama-3-3-70b-instruct", project_id=os.getenv("WATSONX_PROJECT_ID"), ) ) metrics = [ToolCallParameterAccuracyMetric(llm_judge=llm_judge)] result = evaluator.evaluate(data=df, metrics=metrics)
- Create ToolCallParameterAccuracyMetric with a custom threshold.
llm_judge = LLMJudge( model=WxAIFoundationModel( model_id="meta-llama/llama-3-3-70b-instruct", project_id=os.getenv("WATSONX_PROJECT_ID"), ) ) threshold = MetricThreshold(type="upper_limit", value=0.8) metric = ToolCallParameterAccuracyMetric(llm_judge=llm_judge, threshold=threshold)
Show JSON schema
{ "title": "ToolCallParameterAccuracyMetric", "description": "ToolCallParameterAccuracyMetric assesses whether ALL parameter values \nin a function call are directly supported by conversation history \nor API specifications. Identifies hallucinated values, missing information, \nformat errors, and contradictory values.\n\nThe ToolCallParameterAccuracyMetric will be computed using llm_as_judge.\n\nExamples:\n 1. Create ToolCallParameterAccuracyMetric by passing the basic configuration.\n .. code-block:: python\n\n config = GenAIConfiguration(tools = [get_weather,fetch_stock_price])\n evaluator = MetricsEvaluator(configuration=config)\n df = pd.read_csv(\"\")\n llm_judge = LLMJudge(\n model=WxAIFoundationModel(\n model_id=\"meta-llama/llama-3-3-70b-instruct\",\n project_id=os.getenv(\"WATSONX_PROJECT_ID\"),\n )\n )\n metrics = [ToolCallParameterAccuracyMetric(llm_judge=llm_judge)]\n result = evaluator.evaluate(data=df, metrics=metrics)\n\n 2. Create ToolCallParameterAccuracyMetric by passing custom tool calls field in configuration.\n .. code-block:: python\n\n config = GenAIConfiguration(tools = [get_weather,fetch_stock_price],\n tool_calls_field=\"tools_used\")\n evaluator = MetricsEvaluator(configuration=config)\n df = pd.read_csv(\"\")\n llm_judge = LLMJudge(\n model=WxAIFoundationModel(\n model_id=\"meta-llama/llama-3-3-70b-instruct\",\n project_id=os.getenv(\"WATSONX_PROJECT_ID\"),\n )\n )\n metrics = [ToolCallParameterAccuracyMetric(llm_judge=llm_judge)]\n result = evaluator.evaluate(data=df, metrics=metrics)\n\n 3. Create ToolCallParameterAccuracyMetric with a custom threshold.\n .. code-block:: python\n\n llm_judge = LLMJudge(\n model=WxAIFoundationModel(\n model_id=\"meta-llama/llama-3-3-70b-instruct\",\n project_id=os.getenv(\"WATSONX_PROJECT_ID\"),\n )\n )\n threshold = MetricThreshold(type=\"upper_limit\", value=0.8)\n metric = ToolCallParameterAccuracyMetric(llm_judge=llm_judge, threshold=threshold)", "type": "object", "properties": { "name": { "const": "tool_call_parameter_accuracy", "default": "tool_call_parameter_accuracy", "description": "The name of metric.", "title": "Metric Name", "type": "string" }, "display_name": { "const": "Tool Call Parameter Accuracy", "default": "Tool Call Parameter Accuracy", "description": "The tool call parameter accuracy metric display name.", "title": "Display Name", "type": "string" }, "type_": { "default": "ootb", "description": "The type of the metric. Indicates whether the metric is ootb or custom.", "examples": [ "ootb", "custom" ], "title": "Metric type", "type": "string" }, "value_type": { "default": "numeric", "description": "The type of the metric value. Indicates whether the metric value is numeric or categorical.", "examples": [ "numeric", "categorical" ], "title": "Metric value type", "type": "string" }, "thresholds": { "default": [ { "type": "lower_limit", "value": 0.8 } ], "description": "Value that defines the violation limit for the metric", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Metric threshold", "type": "array" }, "tasks": { "default": [ "retrieval_augmented_generation" ], "description": "The generative task type.", "items": { "$ref": "#/$defs/TaskType" }, "title": "Task Type", "type": "array" }, "group": { "$ref": "#/$defs/MetricGroup", "default": "tool_call_quality" }, "is_reference_free": { "default": true, "description": "Decides whether this metric needs a reference for computation", "title": "Is Reference Free", "type": "boolean" }, "method": { "const": "llm_as_judge", "default": "llm_as_judge", "description": "The method used to compute the metric.", "title": "Computation Method", "type": "string" }, "metric_dependencies": { "default": [], "description": "Metrics that needs to be evaluated first", "items": { "$ref": "#/$defs/GenAIMetric" }, "title": "Metric Dependencies", "type": "array" }, "applies_to": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "message", "description": "The tag to indicate for which the metric is applied to. Used for agentic application metric computation.", "examples": [ "message", "conversation", "sub_agent" ], "title": "Applies to" }, "mapping": { "anyOf": [ { "$ref": "#/$defs/Mapping" }, { "type": "null" } ], "default": null, "description": "The data mapping details for the metric which are used to read the values needed to compute the metric.", "examples": { "items": [ { "attribute_name": "traceloop.entity.input", "column_name": null, "json_path": "$.inputs.input_text", "lookup_child_spans": false, "name": "input_text", "span_name": "LangGraph.workflow", "type": "input" }, { "attribute_name": "traceloop.entity.output", "column_name": null, "json_path": "$.outputs.generated_text", "lookup_child_spans": false, "name": "generated_text", "span_name": "LangGraph.workflow", "type": "output" } ], "source": "trace" }, "title": "Mapping" }, "llm_judge": { "anyOf": [ { "$ref": "#/$defs/LLMJudge" }, { "type": "null" } ], "default": null, "description": "The LLM judge used to compute the metric." }, "metric_mapping_name": { "const": "general_hallucination_check", "default": "general_hallucination_check", "description": "The mapping name of metric with llmevalkit.", "title": "Metric Mapping Name", "type": "string" } }, "$defs": { "AWSBedrockCredentials": { "description": "Defines the AWSBedrockCredentials class for accessing AWS Bedrock using environment variables or manual input.\n\nExamples:\n 1. Create credentials manually:\n .. code-block:: python\n\n credentials = AWSBedrockCredentials(\n aws_access_key_id=\"your-access-key-id\",\n aws_secret_access_key=\"your-secret-access-key\",\n aws_region_name=\"us-east-1\",\n aws_session_token=\"optional-session-token\"\n )\n\n 2. Create credentials from environment:\n .. code-block:: python\n\n os.environ[\"AWS_ACCESS_KEY_ID\"] = \"your-access-key-id\"\n os.environ[\"AWS_DEFAULT_REGION\"] = \"us-east-1\"\n os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"your-secret-access-key\"\n\n credentials = AWSBedrockCredentials.create_from_env()", "properties": { "aws_access_key_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The AWS access key id. This attribute value will be read from AWS_ACCESS_KEY_ID environment variable when creating AWSBedrockCredentials from environment.", "title": "AWS Access Key ID" }, "aws_secret_access_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The AWS secret access key. This attribute value will be read from AWS_SECRET_ACCESS_KEY environment variable when creating AWSBedrockCredentials from environment.", "title": "AWS Secret Access Key" }, "aws_region_name": { "default": "us-east-1", "description": "AWS region. This attribute value will be read from AWS_DEFAULT_REGION environment variable when creating AWSBedrockCredentials from environment.", "examples": [ "us-east-1", "eu-west-1" ], "title": "AWS Region", "type": "string" }, "aws_session_token": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Optional AWS session token for temporary credentials.", "title": "AWS Session Token" } }, "required": [ "aws_access_key_id", "aws_secret_access_key", "aws_session_token" ], "title": "AWSBedrockCredentials", "type": "object" }, "AWSBedrockFoundationModel": { "description": " The Amazon Bedrock foundation model details.\n\n Examples:\n 1. Create AWS Bedrock foundation model by passing credentials manually:\n .. code-block:: python\n\n bedrock_model = AWSBedrockFoundationModel(\n model_id=\"anthropic.claude-v2\",\n provider=AWSBedrockModelProvider(\n credentials=AWSBedrockCredentials(\n aws_access_key_id=\"your-access-key-id\",\n aws_secret_access_key=\"your-secret-access-key\",\n aws_region_name=\"us-east-1\",\n aws_session_token=\"optional-session-token\"\n )\n ),\n parameters={\n \"temperature\": 0.7,\n \"top_p\": 0.9,\n \"max_tokens\": 200,\n \"stop_sequences\": [\"\n\"],\n \"system\": \"You are a concise assistant.\",\n \"reasoning_effort\": \"high\",\n \"tool_choice\": \"auto\"\n }\n )\n\n 2. Create AWS Bedrock foundation model using environment variables:\n os.environ[\"AWS_ACCESS_KEY_ID\"] = \"your-access-key-id\"\n os.environ[\"AWS_SECRET_ACCESS_KEY\"] = \"your-secret-access-key\"\n os.environ[\"AWS_DEFAULT_REGION\"] = \"us-east-1\"\n\n .. code-block:: python\n\n bedrock_model = AWSBedrockFoundationModel(\n model_id=\"anthropic.claude-v2\"\n )\n ", "properties": { "model_id": { "description": "The AWS Bedrock model name. It must be a valid AWS Bedrock model identifier.", "examples": [ "anthropic.claude-v2" ], "title": "Model ID", "type": "string" }, "provider": { "$ref": "#/$defs/AWSBedrockModelProvider", "description": "The AWS Bedrock provider details.", "title": "Provider" }, "parameters": { "anyOf": [ { "type": "object" }, { "type": "null" } ], "description": "The model parameters to be used when invoking the model. The parameters may include temperature, top_p, max_tokens, etc..", "title": "Parameters" } }, "required": [ "model_id" ], "title": "AWSBedrockFoundationModel", "type": "object" }, "AWSBedrockModelProvider": { "description": "Represents a model provider using Amazon Bedrock.\n\nExamples:\n 1. Create provider using credentials object:\n .. code-block:: python\n\n provider = AWSBedrockModelProvider(\n credentials=AWSBedrockCredentials(\n aws_access_key_id=\"your-access-key-id\",\n aws_secret_access_key=\"your-secret-access-key\",\n aws_region_name=\"us-east-1\",\n aws_session_token=\"optional-session-token\"\n )\n )\n\n 2. Create provider using environment variables:\n .. code-block:: python\n\n os.environ['AWS_ACCESS_KEY_ID'] = \"your-access-key-id\"\n os.environ['AWS_SECRET_ACCESS_KEY'] = \"your-secret-access-key\"\n os.environ['AWS_SESSION_TOKEN'] = \"optional-session-token\" # Optional\n os.environ['AWS_DEFAULT_REGION'] = \"us-east-1\"\n provider = AWSBedrockModelProvider()", "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "aws_bedrock", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/AWSBedrockCredentials" }, { "type": "null" } ], "default": null, "description": "AWS Bedrock credentials." } }, "title": "AWSBedrockModelProvider", "type": "object" }, "AzureOpenAICredentials": { "properties": { "url": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "Azure OpenAI url. This attribute can be read from `AZURE_OPENAI_HOST` environment variable.", "title": "Url" }, "api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "API key for Azure OpenAI. This attribute can be read from `AZURE_OPENAI_API_KEY` environment variable.", "title": "Api Key" }, "api_version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The model API version from Azure OpenAI. This attribute can be read from `AZURE_OPENAI_API_VERSION` environment variable.", "title": "Api Version" } }, "required": [ "url", "api_key", "api_version" ], "title": "AzureOpenAICredentials", "type": "object" }, "AzureOpenAIFoundationModel": { "description": "The Azure OpenAI foundation model details\n\nExamples:\n 1. Create Azure OpenAI foundation model by passing the credentials during object creation.\n .. code-block:: python\n\n azure_openai_foundation_model = AzureOpenAIFoundationModel(\n model_id=\"gpt-4o-mini\",\n provider=AzureOpenAIModelProvider(\n credentials=AzureOpenAICredentials(\n api_key=azure_api_key,\n url=azure_host_url,\n api_version=azure_api_model_version,\n )\n )\n )\n\n2. Create Azure OpenAI foundation model by setting the credentials in environment variables:\n * ``AZURE_OPENAI_API_KEY`` is used to set the api key for OpenAI.\n * ``AZURE_OPENAI_HOST`` is used to set the url for Azure OpenAI.\n * ``AZURE_OPENAI_API_VERSION`` is uses to set the the api version for Azure OpenAI.\n\n .. code-block:: python\n\n openai_foundation_model = AzureOpenAIFoundationModel(\n model_id=\"gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/AzureOpenAIModelProvider", "description": "Azure OpenAI provider" }, "model_id": { "description": "Model deployment name from Azure OpenAI", "title": "Model Id", "type": "string" } }, "required": [ "model_id" ], "title": "AzureOpenAIFoundationModel", "type": "object" }, "AzureOpenAIModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "azure_openai", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/AzureOpenAICredentials" }, { "type": "null" } ], "default": null, "description": "Azure OpenAI credentials." } }, "title": "AzureOpenAIModelProvider", "type": "object" }, "CustomFoundationModel": { "description": "Defines the CustomFoundationModel class.\n\nThis class extends the base `FoundationModel` to support custom inference logic through a user-defined scoring function.\nIt is intended for use cases where the model is externally hosted and not in the list of supported frameworks.\nExamples:\n 1. Define a custom scoring function and create a model:\n .. code-block:: python\n\n import pandas as pd\n\n def scoring_fn(data: pd.DataFrame):\n predictions_list = []\n # Custom logic to call an external LLM\n return pd.DataFrame({\"generated_text\": predictions_list}) \n\n model = CustomFoundationModel(\n scoring_fn=scoring_fn\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/ModelProvider", "description": "The provider of the model." } }, "title": "CustomFoundationModel", "type": "object" }, "GenAIMetric": { "description": "Defines the Generative AI metric interface", "properties": { "name": { "description": "The name of the metric.", "examples": [ "answer_relevance", "context_relevance" ], "title": "Metric Name", "type": "string" }, "display_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The display name of the metric.", "examples": [ "Answer Relevance", "Context Relevance" ], "title": "Metric display name" }, "type_": { "default": "ootb", "description": "The type of the metric. Indicates whether the metric is ootb or custom.", "examples": [ "ootb", "custom" ], "title": "Metric type", "type": "string" }, "value_type": { "default": "numeric", "description": "The type of the metric value. Indicates whether the metric value is numeric or categorical.", "examples": [ "numeric", "categorical" ], "title": "Metric value type", "type": "string" }, "thresholds": { "default": [], "description": "The list of thresholds", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "tasks": { "default": [], "description": "The task types this metric is associated with.", "items": { "$ref": "#/$defs/TaskType" }, "title": "Tasks", "type": "array" }, "group": { "anyOf": [ { "$ref": "#/$defs/MetricGroup" }, { "type": "null" } ], "default": null, "description": "The metric group this metric belongs to." }, "is_reference_free": { "default": true, "description": "Decides whether this metric needs a reference for computation", "title": "Is Reference Free", "type": "boolean" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute the metric.", "title": "Method" }, "metric_dependencies": { "default": [], "description": "Metrics that needs to be evaluated first", "items": { "$ref": "#/$defs/GenAIMetric" }, "title": "Metric Dependencies", "type": "array" }, "applies_to": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "message", "description": "The tag to indicate for which the metric is applied to. Used for agentic application metric computation.", "examples": [ "message", "conversation", "sub_agent" ], "title": "Applies to" }, "mapping": { "anyOf": [ { "$ref": "#/$defs/Mapping" }, { "type": "null" } ], "default": null, "description": "The data mapping details for the metric which are used to read the values needed to compute the metric.", "examples": { "items": [ { "attribute_name": "traceloop.entity.input", "column_name": null, "json_path": "$.inputs.input_text", "lookup_child_spans": false, "name": "input_text", "span_name": "LangGraph.workflow", "type": "input" }, { "attribute_name": "traceloop.entity.output", "column_name": null, "json_path": "$.outputs.generated_text", "lookup_child_spans": false, "name": "generated_text", "span_name": "LangGraph.workflow", "type": "output" } ], "source": "trace" }, "title": "Mapping" } }, "required": [ "name" ], "title": "GenAIMetric", "type": "object" }, "GoogleAIStudioCredentials": { "description": "Defines the GoogleAIStudioCredentials class for accessing Google AI Studio using an API key.\n\nExamples:\n 1. Create credentials manually:\n .. code-block:: python\n\n google_credentials = GoogleAIStudioCredentials(api_key=\"your-api-key\")\n\n 2. Create credentials from environment:\n .. code-block:: python\n\n os.environ[\"GOOGLE_API_KEY\"] = \"your-api-key\"\n google_credentials = GoogleAIStudioCredentials.create_from_env()", "properties": { "api_key": { "description": "The Google AI Studio key. This attribute can be read from GOOGLE_API_KEY environment variable when creating GoogleAIStudioCredentials from environment.", "title": "Api Key", "type": "string" } }, "required": [ "api_key" ], "title": "GoogleAIStudioCredentials", "type": "object" }, "GoogleAIStudioFoundationModel": { "description": "Represents a foundation model served via Google AI Studio.\n\nExamples:\n 1. Create Google AI Studio foundation model by passing the credentials during object creation.\n .. code-block:: python\n\n model = GoogleAIStudioFoundationModel(\n model_id=\"gemini-1.5-pro-002\",\n provider=GoogleAIStudioModelProvider(\n credentials=GoogleAIStudioCredentials(api_key=\"your_api_key\")\n )\n )\n 2. Create Google AI Studio foundation model by setting the credentials in environment variables:\n * ``GOOGLE_API_KEY`` OR ``GEMINI_API_KEY`` is used to set the Credentials path for Vertex AI.\n .. code-block:: python\n\n model = GoogleAIStudioFoundationModel(\n model_id=\"gemini/gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/GoogleAIStudioModelProvider", "description": "Google AI Studio provider.", "title": "Provider" }, "model_id": { "description": "Model name for Google AI Studio. Must be a valid Google AI model identifier or a fully-qualified publisher path", "examples": [ "gemini-1.5-pro-002" ], "title": "Model id", "type": "string" } }, "required": [ "model_id" ], "title": "GoogleAIStudioFoundationModel", "type": "object" }, "GoogleAIStudioModelProvider": { "description": "Represents a model provider using Google AI Studio.\n\nExamples:\n 1. Create provider using credentials object:\n .. code-block:: python\n\n provider = GoogleAIStudioModelProvider(\n credentials=GoogleAIStudioCredentials(api_key=\"api-key\")\n )\n\n 2. Create provider using environment variables:\n .. code-block:: python\n\n os.environ['GOOGLE_API_KEY'] = \"your_api_key\"\n\n provider = GoogleAIStudioModelProvider()", "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "google_ai_studio", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/GoogleAIStudioCredentials" }, { "type": "null" } ], "default": null, "description": "Google AI Studio credentials." } }, "title": "GoogleAIStudioModelProvider", "type": "object" }, "LLMJudge": { "description": "Defines the LLMJudge.\n\nThe LLMJudge class contains the details of the llm judge model to be used for computing the metric.\n\nExamples:\n 1. Create LLMJudge using watsonx.ai foundation model:\n .. code-block:: python\n\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=PROJECT_ID,\n provider=WxAIModelProvider(\n credentials=WxAICredentials(api_key=wx_apikey)\n )\n )\n llm_judge = LLMJudge(model=wx_ai_foundation_model)", "properties": { "model": { "anyOf": [ { "$ref": "#/$defs/WxAIFoundationModel" }, { "$ref": "#/$defs/OpenAIFoundationModel" }, { "$ref": "#/$defs/AzureOpenAIFoundationModel" }, { "$ref": "#/$defs/PortKeyGateway" }, { "$ref": "#/$defs/RITSFoundationModel" }, { "$ref": "#/$defs/VertexAIFoundationModel" }, { "$ref": "#/$defs/GoogleAIStudioFoundationModel" }, { "$ref": "#/$defs/AWSBedrockFoundationModel" }, { "$ref": "#/$defs/CustomFoundationModel" } ], "description": "The foundation model to be used as judge", "title": "Model" } }, "required": [ "model" ], "title": "LLMJudge", "type": "object" }, "Mapping": { "description": "Defines the field mapping details to be used for computing a metric.", "properties": { "source": { "default": "trace", "description": "The source type of the data. Use trace if the data should be read from span in trace. Use tabular if the data is passed as a dataframe.", "enum": [ "trace", "tabular" ], "examples": [ "trace", "tabular" ], "title": "Source", "type": "string" }, "items": { "description": "The list of mapping items for the field. They are used to read the data from trace or tabular data for computing the metric.", "items": { "$ref": "#/$defs/MappingItem" }, "title": "Mapping Items", "type": "array" } }, "required": [ "items" ], "title": "Mapping", "type": "object" }, "MappingItem": { "description": "The mapping details to be used for reading the values from the data.", "properties": { "name": { "description": "The name of the item.", "examples": [ "input_text", "generated_text", "context", "ground_truth" ], "title": "Name", "type": "string" }, "type": { "description": "The type of the item.", "enum": [ "input", "output", "reference", "context", "tool_call" ], "examples": [ "input" ], "title": "Type", "type": "string" }, "column_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The column name in the tabular data to be used for reading the field value. Applicable for tabular source.", "title": "Column Name" }, "span_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The span name in the trace data to be used for reading the field value. Applicable for trace source.", "title": "Span Name" }, "attribute_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The attribute name in the trace to be used for reading the field value. Applicable for trace source.", "title": "Attribute Name" }, "json_path": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The json path to be used for reading the field value from the attribute value. Applicable for trace source. If not provided, the span attribute value is read as the field value.", "title": "Json Path" }, "lookup_child_spans": { "anyOf": [ { "type": "boolean" }, { "type": "null" } ], "default": false, "description": "The flag to indicate if all the child spans should be searched for the attribute value. Applicable for trace source.", "title": "Look up child spans" } }, "required": [ "name", "type" ], "title": "MappingItem", "type": "object" }, "MetricGroup": { "enum": [ "retrieval_quality", "answer_quality", "content_safety", "performance", "usage", "message_completion", "tool_call_quality", "readability", "custom" ], "title": "MetricGroup", "type": "string" }, "MetricThreshold": { "description": "The class that defines the threshold for a metric.", "properties": { "type": { "description": "Threshold type. One of 'lower_limit', 'upper_limit'", "enum": [ "lower_limit", "upper_limit" ], "title": "Type", "type": "string" }, "value": { "default": 0, "description": "The value of metric threshold", "title": "Threshold value", "type": "number" } }, "required": [ "type" ], "title": "MetricThreshold", "type": "object" }, "ModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "description": "The type of model provider." } }, "required": [ "type" ], "title": "ModelProvider", "type": "object" }, "ModelProviderType": { "description": "Supported model provider types for Generative AI", "enum": [ "ibm_watsonx.ai", "azure_openai", "rits", "openai", "vertex_ai", "google_ai_studio", "aws_bedrock", "custom", "portkey" ], "title": "ModelProviderType", "type": "string" }, "OpenAICredentials": { "description": "Defines the OpenAICredentials class to specify the OpenAI server details.\n\nExamples:\n 1. Create OpenAICredentials with default parameters. By default Dallas region is used.\n .. code-block:: python\n\n openai_credentials = OpenAICredentials(api_key=api_key,\n url=openai_url)\n\n 2. Create OpenAICredentials by reading from environment variables.\n .. code-block:: python\n\n os.environ[\"OPENAI_API_KEY\"] = \"...\"\n os.environ[\"OPENAI_URL\"] = \"...\"\n openai_credentials = OpenAICredentials.create_from_env()", "properties": { "url": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Url" }, "api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "title": "Api Key" } }, "required": [ "url", "api_key" ], "title": "OpenAICredentials", "type": "object" }, "OpenAIFoundationModel": { "description": "The OpenAI foundation model details\n\nExamples:\n 1. Create OpenAI foundation model by passing the credentials during object creation. Note that the url is optional and will be set to the default value for OpenAI. To change the default value, the url should be passed to ``OpenAICredentials`` object.\n .. code-block:: python\n\n openai_foundation_model = OpenAIFoundationModel(\n model_id=\"gpt-4o-mini\",\n provider=OpenAIModelProvider(\n credentials=OpenAICredentials(\n api_key=api_key,\n url=openai_url,\n )\n )\n )\n\n 2. Create OpenAI foundation model by setting the credentials in environment variables:\n * ``OPENAI_API_KEY`` is used to set the api key for OpenAI.\n * ``OPENAI_URL`` is used to set the url for OpenAI\n\n .. code-block:: python\n\n openai_foundation_model = OpenAIFoundationModel(\n model_id=\"gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/OpenAIModelProvider", "description": "OpenAI provider" }, "model_id": { "description": "Model name from OpenAI", "title": "Model Id", "type": "string" } }, "required": [ "model_id" ], "title": "OpenAIFoundationModel", "type": "object" }, "OpenAIModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "openai", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/OpenAICredentials" }, { "type": "null" } ], "default": null, "description": "OpenAI credentials. This can also be set by using `OPENAI_API_KEY` environment variable." } }, "title": "OpenAIModelProvider", "type": "object" }, "PortKeyCredentials": { "description": "Defines the PortKeyCredentials class to specify the PortKey Gateway details.\n\nExamples:\n 1. Create PortKeyCredentials with default parameters.\n .. code-block:: python\n\n portkey_credentials = PortKeyCredentials(api_key=api_key,\n url=portkey_url,\n provider_api_key=provider_api_key,\n provider=provider_name)\n\n 2. Create PortKeyCredentials by reading from environment variables.\n .. code-block:: python\n\n os.environ[\"PORTKEY_API_KEY\"] = \"...\"\n os.environ[\"PORTKEY_URL\"] = \"...\"\n os.environ[\"PORTKEY_PROVIDER_API_KEY\"] = \"...\"\n os.environ[\"PORTKEY_PROVIDER_NAME\"] = \"...\"\n portkey_credentials = PortKeyCredentials.create_from_env()", "properties": { "url": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "PortKey url. This attribute can be read from `PORTKEY_URL` environment variable.", "title": "Url" }, "api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "API key for PortKey. This attribute can be read from `PORTKEY_API_KEY` environment variable.", "title": "Api Key" }, "provider_api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "API key for the provider. This attribute can be read from `PORTKEY_PROVIDER_API_KEY` environment variable.", "title": "Provider Api Key" }, "provider": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "description": "The provider name. This attribute can be read from `PORTKEY_PROVIDER_NAME` environment variable.", "title": "Provider" } }, "required": [ "url", "api_key", "provider_api_key", "provider" ], "title": "PortKeyCredentials", "type": "object" }, "PortKeyGateway": { "description": "The PortKey gateway details\n\nExamples:\n 1. Create PortKeyGateway by passing the credentials during object creation. Note that the url is optional and will be set to the default value for PortKey. To change the default value, the url should be passed to ``PortKeyCredentials`` object.\n .. code-block:: python\n\n port_key_gateway = PortKeyGateway(\n model_id=\"gpt-4o-mini\",\n provider=PortKeyModelProvider(\n credentials=PortKeyCredentials(\n api_key=api_key,\n url=openai_url,\n provider_api_key=provider_api_key,\n provider_name=provider_name\n )\n )\n )\n\n 2. Create PortKeyGateway by setting the credentials in environment variables:\n * ``PORTKEY_API_KEY`` is used to set the api key for PortKey.\n * ``PORTKEY_URL`` is used to set the url for PortKey.\n * ``PORTKEY_PROVIDER_API_KEY`` is used to set the provider api key for PortKey.\n * ``PORTKEY_PROVIDER_NAME`` is used to set the provider name for PortKey\n\n .. code-block:: python\n\n port_key_gateway = PortKeyGateway(\n model_id=\"gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/PortKeyModelProvider", "description": "PortKey Provider" }, "model_id": { "description": "Model name from the Provider", "title": "Model Id", "type": "string" } }, "required": [ "model_id" ], "title": "PortKeyGateway", "type": "object" }, "PortKeyModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "portkey", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/PortKeyCredentials" }, { "type": "null" } ], "default": null, "description": "PortKey credentials." } }, "title": "PortKeyModelProvider", "type": "object" }, "RITSCredentials": { "properties": { "hostname": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "https://inference-3scale-apicast-production.apps.rits.fmaas.res.ibm.com", "description": "The rits hostname", "title": "Hostname" }, "api_key": { "title": "Api Key", "type": "string" } }, "required": [ "api_key" ], "title": "RITSCredentials", "type": "object" }, "RITSFoundationModel": { "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/RITSModelProvider", "description": "The provider of the model." } }, "title": "RITSFoundationModel", "type": "object" }, "RITSModelProvider": { "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "rits", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/RITSCredentials" }, { "type": "null" } ], "default": null, "description": "RITS credentials." } }, "title": "RITSModelProvider", "type": "object" }, "TaskType": { "description": "Supported task types for generative AI models", "enum": [ "question_answering", "classification", "summarization", "generation", "extraction", "retrieval_augmented_generation" ], "title": "TaskType", "type": "string" }, "VertexAICredentials": { "description": "Defines the VertexAICredentials class for accessing Vertex AI using service account credentials.\n\nExamples:\n 1. Create credentials manually:\n .. code-block:: python\n\n vertex_credentials = VertexAICredentials(\n credentials_path=\"path/to/service_account.json\",\n project_id=\"my-gcp-project\",\n location=\"us-central1\"\n )\n\n 2. Create credentials from environment:\n .. code-block:: python\n\n os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = \"path/to/service_account.json\"\n os.environ[\"GOOGLE_CLOUD_PROJECT\"] = \"my-gcp-project\"\n os.environ[\"GOOGLE_CLOUD_LOCATION\"] = \"us-central1\"\n\n vertex_ai_credentials = VertexAICredentials.create_from_env()", "properties": { "credentials_path": { "description": "Path to service-account JSON. This attribute can be read from GOOGLE_APPLICATION_CREDENTIALS environment variable when creating VertexAICredentials from environment.", "title": "Credentials Path", "type": "string" }, "project_id": { "description": "The Google Cloud project id. This attribute can be read from GOOGLE_CLOUD_PROJECT or GCLOUD_PROJECT environment variable when creating VertexAICredentials from environment.", "title": "Project ID", "type": "string" }, "location": { "default": "us-central1", "description": "Vertex AI region. This attribute can be read from GOOGLE_CLOUD_LOCATION environment variable when creating VertexAICredentials from environment. By default us-central1 location is used.", "examples": [ "us-central1", "europe-west4" ], "title": "Location", "type": "string" } }, "required": [ "credentials_path", "project_id" ], "title": "VertexAICredentials", "type": "object" }, "VertexAIFoundationModel": { "description": "Represents a foundation model served via Vertex AI.\n\nExamples:\n 1. Create Vertex AI foundation model by passing the credentials during object creation.\n .. code-block:: python\n\n model = VertexAIFoundationModel(\n model_id=\"gemini-1.5-pro-002\",\n provider=VertexAIModelProvider(\n credentials=VertexAICredentials(\n project_id=\"your-project\",\n location=\"us-central1\", # This is optional field, by default us-central1 location is selected\n credentials_path=\"/path/to/service_account.json\"\n )\n )\n )\n 2. Create Vertex AI foundation model by setting the credentials in environment variables:\n * ``GOOGLE_APPLICATION_CREDENTIALS`` is used to set the Credentials path for Vertex AI.\n * ``GOOGLE_CLOUD_PROJECT`` is used to set the Project id for Vertex AI.\n * ``GOOGLE_CLOUD_LOCATION`` is uses to set the Location for Vertex AI. By default us-central1 location is used when GOOGLE_CLOUD_LOCATION is not provided .\n\n .. code-block:: python\n\n os.environ[\"GOOGLE_APPLICATION_CREDENTIALS\"] = \"path/to/service_account.json\"\n os.environ[\"GOOGLE_CLOUD_PROJECT\"] = \"my-gcp-project\"\n os.environ[\"GOOGLE_CLOUD_LOCATION\"] = \"us-central1\"\n\n model = VertexAIFoundationModel(\n model_id=\"gemini/gpt-4o-mini\",\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/VertexAIModelProvider", "description": "Vertex AI provider.", "title": "Provider" }, "model_id": { "description": "Model name for Vertex AI. Must be a valid Vertex AI model identifier or a fully-qualified publisher path", "examples": [ "gemini-1.5-pro-002" ], "title": "Model id", "type": "string" } }, "required": [ "model_id" ], "title": "VertexAIFoundationModel", "type": "object" }, "VertexAIModelProvider": { "description": "Represents a model provider using Vertex AI.\n\nExamples:\n 1. Create provider using credentials object:\n .. code-block:: python\n\n provider = VertexAIModelProvider(\n credentials=VertexAICredentials(\n credentials_path=\"path/to/key.json\",\n project_id=\"your-project\",\n location=\"us-central1\" \n )\n )\n\n 2. Create provider using environment variables:\n .. code-block:: python\n\n os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = \"/path/to/service_account.json\"\n os.environ['GOOGLE_CLOUD_PROJECT'] = \"your-project\"\n os.environ['GOOGLE_CLOUD_LOCATION'] = \"us-central1\" # This is optional field, by default us-central1 location is selected\n\n provider = VertexAIModelProvider()", "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "vertex_ai", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/VertexAICredentials" }, { "type": "null" } ], "default": null, "description": "Vertex AI credentials." } }, "title": "VertexAIModelProvider", "type": "object" }, "WxAICredentials": { "description": "Defines the WxAICredentials class to specify the watsonx.ai server details.\n\nExamples:\n 1. Create WxAICredentials with default parameters. By default Dallas region is used.\n .. code-block:: python\n\n wxai_credentials = WxAICredentials(api_key=\"...\")\n\n 2. Create WxAICredentials by specifying region url.\n .. code-block:: python\n\n wxai_credentials = WxAICredentials(api_key=\"...\",\n url=\"https://au-syd.ml.cloud.ibm.com\")\n\n 3. Create WxAICredentials by reading from environment variables.\n .. code-block:: python\n\n os.environ[\"WATSONX_APIKEY\"] = \"...\"\n # [Optional] Specify watsonx region specific url. Default is https://us-south.ml.cloud.ibm.com .\n os.environ[\"WATSONX_URL\"] = \"https://eu-gb.ml.cloud.ibm.com\"\n wxai_credentials = WxAICredentials.create_from_env()\n\n 4. Create WxAICredentials for on-prem.\n .. code-block:: python\n\n wxai_credentials = WxAICredentials(url=\"https://<hostname>\",\n username=\"...\"\n api_key=\"...\",\n version=\"5.2\")\n\n 5. Create WxAICredentials by reading from environment variables for on-prem.\n .. code-block:: python\n\n os.environ[\"WATSONX_URL\"] = \"https://<hostname>\"\n os.environ[\"WATSONX_VERSION\"] = \"5.2\"\n os.environ[\"WATSONX_USERNAME\"] = \"...\"\n os.environ[\"WATSONX_APIKEY\"] = \"...\"\n # Only one of api_key or password is needed\n #os.environ[\"WATSONX_PASSWORD\"] = \"...\"\n wxai_credentials = WxAICredentials.create_from_env()", "properties": { "url": { "default": "https://us-south.ml.cloud.ibm.com", "description": "The url for watsonx ai service", "examples": [ "https://us-south.ml.cloud.ibm.com", "https://eu-de.ml.cloud.ibm.com", "https://eu-gb.ml.cloud.ibm.com", "https://jp-tok.ml.cloud.ibm.com", "https://au-syd.ml.cloud.ibm.com" ], "title": "watsonx.ai url", "type": "string" }, "api_key": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The user api key. Required for using watsonx as a service and one of api_key or password is required for using watsonx on-prem software.", "strip_whitespace": true, "title": "Api Key" }, "version": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The watsonx on-prem software version. Required for using watsonx on-prem software.", "title": "Version" }, "username": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The user name. Required for using watsonx on-prem software.", "title": "User name" }, "password": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The user password. One of api_key or password is required for using watsonx on-prem software.", "title": "Password" }, "instance_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": "openshift", "description": "The watsonx.ai instance id. Default value is openshift.", "title": "Instance id" } }, "title": "WxAICredentials", "type": "object" }, "WxAIFoundationModel": { "description": "The IBM watsonx.ai foundation model details\n\nTo initialize the foundation model, you can either pass in the credentials directly or set the environment.\nYou can follow these examples to create the provider.\n\nExamples:\n 1. Create foundation model by specifying the credentials during object creation:\n .. code-block:: python\n\n # Specify the credentials during object creation\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=<PROJECT_ID>,\n provider=WxAIModelProvider(\n credentials=WxAICredentials(\n url=wx_url, # This is optional field, by default US-Dallas region is selected\n api_key=wx_apikey,\n )\n )\n )\n\n 2. Create foundation model by setting the credentials environment variables:\n * The api key can be set using one of the environment variables ``WXAI_API_KEY``, ``WATSONX_APIKEY``, or ``WXG_API_KEY``. These will be read in the order of precedence.\n * The url is optional and will be set to US-Dallas region by default. It can be set using one of the environment variables ``WXAI_URL``, ``WATSONX_URL``, or ``WXG_URL``. These will be read in the order of precedence.\n\n .. code-block:: python\n\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=<PROJECT_ID>,\n )\n\n 3. Create foundation model by specifying watsonx.governance software credentials during object creation:\n .. code-block:: python\n\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=project_id,\n provider=WxAIModelProvider(\n credentials=WxAICredentials(\n url=wx_url,\n api_key=wx_apikey,\n username=wx_username,\n version=wx_version,\n )\n )\n )\n\n 4. Create foundation model by setting watsonx.governance software credentials environment variables:\n * The api key can be set using one of the environment variables ``WXAI_API_KEY``, ``WATSONX_APIKEY``, or ``WXG_API_KEY``. These will be read in the order of precedence.\n * The url can be set using one of these environment variable ``WXAI_URL``, ``WATSONX_URL``, or ``WXG_URL``. These will be read in the order of precedence.\n * The username can be set using one of these environment variable ``WXAI_USERNAME``, ``WATSONX_USERNAME``, or ``WXG_USERNAME``. These will be read in the order of precedence.\n * The version of watsonx.governance software can be set using one of these environment variable ``WXAI_VERSION``, ``WATSONX_VERSION``, or ``WXG_VERSION``. These will be read in the order of precedence.\n\n .. code-block:: python\n\n wx_ai_foundation_model = WxAIFoundationModel(\n model_id=\"ibm/granite-3-3-8b-instruct\",\n project_id=project_id,\n )", "properties": { "model_name": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The name of the foundation model.", "title": "Model Name" }, "provider": { "$ref": "#/$defs/WxAIModelProvider", "description": "The provider of the model." }, "model_id": { "description": "The unique identifier for the watsonx.ai model.", "title": "Model Id", "type": "string" }, "project_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The project ID associated with the model.", "title": "Project Id" }, "space_id": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The space ID associated with the model.", "title": "Space Id" } }, "required": [ "model_id" ], "title": "WxAIFoundationModel", "type": "object" }, "WxAIModelProvider": { "description": "This class represents a model provider configuration for IBM watsonx.ai. It includes the provider type and\ncredentials required to authenticate and interact with the watsonx.ai platform. If credentials are not explicitly\nprovided, it attempts to load them from environment variables.\n\nExamples:\n 1. Create provider using credentials object:\n .. code-block:: python\n\n credentials = WxAICredentials(\n url=\"https://us-south.ml.cloud.ibm.com\",\n api_key=\"your-api-key\"\n )\n provider = WxAIModelProvider(credentials=credentials)\n\n 2. Create provider using environment variables:\n .. code-block:: python\n\n import os\n\n os.environ['WATSONX_URL'] = \"https://us-south.ml.cloud.ibm.com\"\n os.environ['WATSONX_APIKEY'] = \"your_api_key\"\n\n provider = WxAIModelProvider()", "properties": { "type": { "$ref": "#/$defs/ModelProviderType", "default": "ibm_watsonx.ai", "description": "The type of model provider." }, "credentials": { "anyOf": [ { "$ref": "#/$defs/WxAICredentials" }, { "type": "null" } ], "default": null, "description": "The credentials used to authenticate with watsonx.ai. If not provided, they will be loaded from environment variables." } }, "title": "WxAIModelProvider", "type": "object" } } }
- Fields:
group (Annotated[ibm_watsonx_gov.entities.enums.MetricGroup, FieldInfo(annotation=NoneType, required=False, default=tasks (Annotated[list[ibm_watsonx_gov.entities.enums.TaskType], FieldInfo(annotation=NoneType, required=False, default=[
- Validators:
- field display_name: Annotated[Literal['Tool Call Parameter Accuracy'], FieldInfo(annotation=NoneType, required=False, default='Tool Call Parameter Accuracy', title='Display Name', description='The tool call parameter accuracy metric display name.', frozen=True)] = 'Tool Call Parameter Accuracy'¶
The tool call parameter accuracy metric display name.
- Validated by:
- field group: TOOL_CALL_QUALITY: 'tool_call_quality'>, frozen=True)] = MetricGroup.TOOL_CALL_QUALITY¶
- Validated by:
- field llm_judge: Annotated[LLMJudge | None, FieldInfo(annotation=NoneType, required=False, default=None, description='The LLM judge used to compute the metric.')] = None¶
The LLM judge used to compute the metric.
- Validated by:
- field method: Annotated[Literal['llm_as_judge'], FieldInfo(annotation=NoneType, required=False, default='llm_as_judge', title='Computation Method', description='The method used to compute the metric.')] = 'llm_as_judge'¶
The method used to compute the metric.
- Validated by:
- field metric_mapping_name: Annotated[Literal['general_hallucination_check'], FieldInfo(annotation=NoneType, required=False, default='general_hallucination_check', title='Metric Mapping Name', description='The mapping name of metric with llmevalkit.')] = 'general_hallucination_check'¶
The mapping name of metric with llmevalkit.
- Validated by:
- field name: Annotated[Literal['tool_call_parameter_accuracy'], FieldInfo(annotation=NoneType, required=False, default='tool_call_parameter_accuracy', title='Metric Name', description='The name of metric.')] = 'tool_call_parameter_accuracy'¶
The name of metric.
- Validated by:
- field thresholds: Annotated[list[MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[MetricThreshold(type='lower_limit', value=0.8)], title='Metric threshold', description='Value that defines the violation limit for the metric')] = [MetricThreshold(type='lower_limit', value=0.8)]¶
Value that defines the violation limit for the metric
- Validated by:
- evaluate(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs)¶
- async evaluate_async(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs) AggregateMetricResult¶
Evaluate the data for ToolCallParameterAccuracyMetric :param data: Data to be evaluated :type data: pd.DataFrame | dict :param configuration: Metrics configuration :type configuration: GenAIConfiguration | AgenticAIConfiguration :param **kwargs: Additional keyword arguments
- Returns:
The computed metrics
- Return type:
- model_post_init(context: Any, /) None¶
We need to both initialize private attributes and call the user-defined model_post_init method.