Tool Call Syntactic Accuracy Metric¶

pydantic model ibm_watsonx_gov.metrics.tool_call_syntactic_accuracy.tool_call_syntactic_accuracy_metric.ToolCallSyntacticAccuracyMetric¶

Bases: GenAIMetric

ToolCallSyntacticAccuracyMetric compute the tool call syntactic correctness by validating tool call against the schema of the list of available tools.

The ToolCallSyntacticAccuracy metric will be computed by performing the syntactic checks.

Examples

Create ToolCallSyntacticAccuracy metric by passing the basic configuration.

config = GenAIConfiguration(tools = [get_weather,fetch_stock_price])
evaluator = MetricsEvaluator(configuration=config)
df = pd.read_csv("")
metrics = [ToolCallSyntacticAccuracyMetric()]
result = evaluator.evaluate(data=df, metrics=metrics)

Create ToolCallSyntacticAccuracy metric by passing custom tool calls field in configuration.

config = GenAIConfiguration(tools = [get_weather,fetch_stock_price],
                            tool_calls_field="tools_used")
evaluator = MetricsEvaluator(configuration=config)
df = pd.read_csv("")
metrics = [ToolCallSyntacticAccuracyMetric()]
result = evaluator.evaluate(data=df, metrics=metrics)

Create ToolCallSyntacticAccuracy metric with a custom threshold.

threshold  = MetricThreshold(type="upper_limit", value=0.8)
metric = ToolCallSyntacticAccuracyMetric(threshold=threshold)

Show JSON schema

{
   "title": "ToolCallSyntacticAccuracyMetric",
   "description": "ToolCallSyntacticAccuracyMetric compute the tool call syntactic correctness \nby validating tool call against the schema of the list of available tools.\n\nThe ToolCallSyntacticAccuracy metric will be computed by performing the syntactic checks.\n\nExamples:\n    1. Create ToolCallSyntacticAccuracy metric by passing the basic configuration.\n        .. code-block:: python\n\n            config = GenAIConfiguration(tools = [get_weather,fetch_stock_price])\n            evaluator = MetricsEvaluator(configuration=config)\n            df = pd.read_csv(\"\")\n            metrics = [ToolCallSyntacticAccuracyMetric()]\n            result = evaluator.evaluate(data=df, metrics=metrics)\n\n    2. Create ToolCallSyntacticAccuracy metric by passing custom tool calls field in configuration.\n        .. code-block:: python\n\n            config = GenAIConfiguration(tools = [get_weather,fetch_stock_price],\n                                        tool_calls_field=\"tools_used\")\n            evaluator = MetricsEvaluator(configuration=config)\n            df = pd.read_csv(\"\")\n            metrics = [ToolCallSyntacticAccuracyMetric()]\n            result = evaluator.evaluate(data=df, metrics=metrics)\n\n    3. Create ToolCallSyntacticAccuracy metric with a custom threshold.\n        .. code-block:: python\n\n            threshold  = MetricThreshold(type=\"upper_limit\", value=0.8)\n            metric = ToolCallSyntacticAccuracyMetric(threshold=threshold)",
   "type": "object",
   "properties": {
      "name": {
         "const": "tool_call_syntactic_accuracy",
         "default": "tool_call_syntactic_accuracy",
         "description": "The name of metric.",
         "title": "Metric Name",
         "type": "string"
      },
      "thresholds": {
         "default": [
            {
               "type": "upper_limit",
               "value": 0.7
            }
         ],
         "description": "Value that defines the violation limit for the metric",
         "items": {
            "$ref": "#/$defs/MetricThreshold"
         },
         "title": "Metric threshold",
         "type": "array"
      },
      "tasks": {
         "default": [
            "retrieval_augmented_generation"
         ],
         "description": "The generative task type.",
         "items": {
            "$ref": "#/$defs/TaskType"
         },
         "title": "Task Type",
         "type": "array"
      },
      "group": {
         "$ref": "#/$defs/MetricGroup",
         "default": "tool_call_quality",
         "description": "The metric group.",
         "title": "Group"
      },
      "is_reference_free": {
         "default": true,
         "description": "Decides whether this metric needs a reference for computation",
         "title": "Is Reference Free",
         "type": "boolean"
      },
      "method": {
         "const": "syntactic_check",
         "default": "syntactic_check",
         "description": "The method used to compute the metric.",
         "title": "Computation Method",
         "type": "string"
      },
      "metric_dependencies": {
         "default": [],
         "description": "Metrics that needs to be evaluated first",
         "items": {
            "$ref": "#/$defs/GenAIMetric"
         },
         "title": "Metric Dependencies",
         "type": "array"
      }
   },
   "$defs": {
      "GenAIMetric": {
         "description": "Defines the Generative AI metric interface",
         "properties": {
            "name": {
               "description": "The name of the metric",
               "title": "Metric Name",
               "type": "string"
            },
            "thresholds": {
               "default": [],
               "description": "The list of thresholds",
               "items": {
                  "$ref": "#/$defs/MetricThreshold"
               },
               "title": "Thresholds",
               "type": "array"
            },
            "tasks": {
               "description": "The task types this metric is associated with.",
               "items": {
                  "$ref": "#/$defs/TaskType"
               },
               "title": "Tasks",
               "type": "array"
            },
            "group": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/MetricGroup"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The metric group this metric belongs to."
            },
            "is_reference_free": {
               "default": true,
               "description": "Decides whether this metric needs a reference for computation",
               "title": "Is Reference Free",
               "type": "boolean"
            },
            "method": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The method used to compute the metric.",
               "title": "Method"
            },
            "metric_dependencies": {
               "default": [],
               "description": "Metrics that needs to be evaluated first",
               "items": {
                  "$ref": "#/$defs/GenAIMetric"
               },
               "title": "Metric Dependencies",
               "type": "array"
            }
         },
         "required": [
            "name",
            "tasks"
         ],
         "title": "GenAIMetric",
         "type": "object"
      },
      "MetricGroup": {
         "enum": [
            "retrieval_quality",
            "answer_quality",
            "content_safety",
            "performance",
            "usage",
            "tool_call_quality",
            "readability"
         ],
         "title": "MetricGroup",
         "type": "string"
      },
      "MetricThreshold": {
         "description": "The class that defines the threshold for a metric.",
         "properties": {
            "type": {
               "description": "Threshold type. One of 'lower_limit', 'upper_limit'",
               "enum": [
                  "lower_limit",
                  "upper_limit"
               ],
               "title": "Type",
               "type": "string"
            },
            "value": {
               "default": 0,
               "description": "The value of metric threshold",
               "title": "Threshold value",
               "type": "number"
            }
         },
         "required": [
            "type"
         ],
         "title": "MetricThreshold",
         "type": "object"
      },
      "TaskType": {
         "description": "Supported task types for generative AI models",
         "enum": [
            "question_answering",
            "classification",
            "summarization",
            "generation",
            "extraction",
            "retrieval_augmented_generation"
         ],
         "title": "TaskType",
         "type": "string"
      }
   }
}

Fields:

group (Annotated[ibm_watsonx_gov.entities.enums.MetricGroup, FieldInfo(annotation=NoneType, required=False, default=
method (Annotated[Literal['syntactic_check'], FieldInfo(annotation=NoneType, required=False, default='syntactic_check', title='Computation Method', description='The method used to compute the metric.')])
name (Annotated[Literal['tool_call_syntactic_accuracy'], FieldInfo(annotation=NoneType, required=False, default='tool_call_syntactic_accuracy', title='Metric Name', description='The name of metric.')])
tasks (Annotated[list[ibm_watsonx_gov.entities.enums.TaskType], FieldInfo(annotation=NoneType, required=False, default=[
thresholds (Annotated[list[ibm_watsonx_gov.entities.metric_threshold.MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[MetricThreshold(type='upper_limit', value=0.7)], title='Metric threshold', description='Value that defines the violation limit for the metric')])

field group: ', frozen=True)] = MetricGroup.TOOL_CALL_QUALITY¶: The metric group.

field method: Annotated[Literal['syntactic_check'], FieldInfo(annotation=NoneType, required=False, default='syntactic_check', title='Computation Method', description='The method used to compute the metric.')] = 'syntactic_check'¶: The method used to compute the metric.

field name: Annotated[Literal['tool_call_syntactic_accuracy'], FieldInfo(annotation=NoneType, required=False, default='tool_call_syntactic_accuracy', title='Metric Name', description='The name of metric.')] = 'tool_call_syntactic_accuracy'¶: The name of metric.

field tasks: ')] = [TaskType.RAG]¶: The generative task type.

field thresholds: Annotated[list[MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[MetricThreshold(type='upper_limit', value=0.7)], title='Metric threshold', description='Value that defines the violation limit for the metric')] = [MetricThreshold(type='upper_limit', value=0.7)]¶: Value that defines the violation limit for the metric

evaluate(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs)¶

async evaluate_async(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs) → AggregateMetricResult¶

Evaluate the data for ToolCallSyntacticAccuracyMetric :param data: Data to be evaluated :type data: pd.DataFrame | dict :param configuration: Metrics configuration :type configuration: GenAIConfiguration | AgenticAIConfiguration

Returns:: The computed metrics
Return type:: AggregateMetricResult

model_post_init(context: Any, /) → None¶: We need to both initialize private attributes and call the user-defined model_post_init method.