Metric

pydantic model ibm_watsonx_gov.entities.metric.GenAIMetric

Bases: BaseMetric

Defines the Generative AI metric interface

Show JSON schema

{
   "$defs": {
      "GenAIMetric": {
         "description": "Defines the Generative AI metric interface",
         "properties": {
            "name": {
               "description": "The name of the metric.",
               "examples": [
                  "answer_relevance",
                  "context_relevance"
               ],
               "title": "Metric Name",
               "type": "string"
            },
            "display_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The display name of the metric.",
               "examples": [
                  "Answer Relevance",
                  "Context Relevance"
               ],
               "title": "Metric display name"
            },
            "type_": {
               "default": "ootb",
               "description": "The type of the metric. Indicates whether the metric is ootb or custom.",
               "examples": [
                  "ootb",
                  "custom"
               ],
               "title": "Metric type",
               "type": "string"
            },
            "value_type": {
               "default": "numeric",
               "description": "The type of the metric value. Indicates whether the metric value is numeric or categorical.",
               "examples": [
                  "numeric",
                  "categorical"
               ],
               "title": "Metric value type",
               "type": "string"
            },
            "thresholds": {
               "default": [],
               "description": "The list of thresholds",
               "items": {
                  "$ref": "#/$defs/MetricThreshold"
               },
               "title": "Thresholds",
               "type": "array"
            },
            "tasks": {
               "default": [],
               "description": "The task types this metric is associated with.",
               "items": {
                  "$ref": "#/$defs/TaskType"
               },
               "title": "Tasks",
               "type": "array"
            },
            "group": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/MetricGroup"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The metric group this metric belongs to."
            },
            "is_reference_free": {
               "default": true,
               "description": "Decides whether this metric needs a reference for computation",
               "title": "Is Reference Free",
               "type": "boolean"
            },
            "method": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The method used to compute the metric.",
               "title": "Method"
            },
            "metric_dependencies": {
               "default": [],
               "description": "Metrics that needs to be evaluated first",
               "items": {
                  "$ref": "#/$defs/GenAIMetric"
               },
               "title": "Metric Dependencies",
               "type": "array"
            },
            "applies_to": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": "message",
               "description": "The tag to indicate for which the metric is applied to. Used for agentic application metric computation.",
               "examples": [
                  "message",
                  "conversation",
                  "sub_agent"
               ],
               "title": "Applies to"
            },
            "mapping": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/Mapping"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The data mapping details for the metric which are used to read the values needed to compute the metric.",
               "examples": {
                  "items": [
                     {
                        "attribute_name": "traceloop.entity.input",
                        "column_name": null,
                        "json_path": "$.inputs.input_text",
                        "lookup_child_spans": false,
                        "name": "input_text",
                        "span_name": "LangGraph.workflow",
                        "type": "input"
                     },
                     {
                        "attribute_name": "traceloop.entity.output",
                        "column_name": null,
                        "json_path": "$.outputs.generated_text",
                        "lookup_child_spans": false,
                        "name": "generated_text",
                        "span_name": "LangGraph.workflow",
                        "type": "output"
                     }
                  ],
                  "source": "trace"
               },
               "title": "Mapping"
            }
         },
         "required": [
            "name"
         ],
         "title": "GenAIMetric",
         "type": "object"
      },
      "Mapping": {
         "description": "Defines the field mapping details to be used for computing a metric.",
         "properties": {
            "source": {
               "default": "trace",
               "description": "The source type of the data. Use trace if the data should be read from span in trace. Use tabular if the data is passed as a dataframe.",
               "enum": [
                  "trace",
                  "tabular"
               ],
               "examples": [
                  "trace",
                  "tabular"
               ],
               "title": "Source",
               "type": "string"
            },
            "items": {
               "description": "The list of mapping items for the field. They are used to read the data from trace or tabular data for computing the metric.",
               "items": {
                  "$ref": "#/$defs/MappingItem"
               },
               "title": "Mapping Items",
               "type": "array"
            }
         },
         "required": [
            "items"
         ],
         "title": "Mapping",
         "type": "object"
      },
      "MappingItem": {
         "description": "The mapping details to be used for reading the values from the data.",
         "properties": {
            "name": {
               "description": "The name of the item.",
               "examples": [
                  "input_text",
                  "generated_text",
                  "context",
                  "ground_truth"
               ],
               "title": "Name",
               "type": "string"
            },
            "type": {
               "description": "The type of the item.",
               "enum": [
                  "input",
                  "output",
                  "reference",
                  "context",
                  "tool_call"
               ],
               "examples": [
                  "input"
               ],
               "title": "Type",
               "type": "string"
            },
            "column_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The column name in the tabular data to be used for reading the field value. Applicable for tabular source.",
               "title": "Column Name"
            },
            "span_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The span name in the trace data to be used for reading the field value. Applicable for trace source.",
               "title": "Span Name"
            },
            "attribute_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The attribute name in the trace to be used for reading the field value. Applicable for trace source.",
               "title": "Attribute Name"
            },
            "json_path": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The json path to be used for reading the field value from the attribute value. Applicable for trace source. If not provided, the span attribute value is read as the field value.",
               "title": "Json Path"
            },
            "lookup_child_spans": {
               "anyOf": [
                  {
                     "type": "boolean"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": false,
               "description": "The flag to indicate if all the child spans should be searched for the attribute value. Applicable for trace source.",
               "title": "Look up child spans"
            }
         },
         "required": [
            "name",
            "type"
         ],
         "title": "MappingItem",
         "type": "object"
      },
      "MetricGroup": {
         "enum": [
            "retrieval_quality",
            "answer_quality",
            "content_safety",
            "performance",
            "usage",
            "message_completion",
            "tool_call_quality",
            "readability",
            "custom"
         ],
         "title": "MetricGroup",
         "type": "string"
      },
      "MetricThreshold": {
         "description": "The class that defines the threshold for a metric.",
         "properties": {
            "type": {
               "description": "Threshold type. One of 'lower_limit', 'upper_limit'",
               "enum": [
                  "lower_limit",
                  "upper_limit"
               ],
               "title": "Type",
               "type": "string"
            },
            "value": {
               "default": 0,
               "description": "The value of metric threshold",
               "title": "Threshold value",
               "type": "number"
            }
         },
         "required": [
            "type"
         ],
         "title": "MetricThreshold",
         "type": "object"
      },
      "TaskType": {
         "description": "Supported task types for generative AI models",
         "enum": [
            "question_answering",
            "classification",
            "summarization",
            "generation",
            "extraction",
            "retrieval_augmented_generation"
         ],
         "title": "TaskType",
         "type": "string"
      }
   },
   "$ref": "#/$defs/GenAIMetric"
}

Fields:

• applies_to (Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default='message', title='Applies to', description='The tag to indicate for which the metric is applied to. Used for agentic application metric computation.', examples=['message', 'conversation', 'sub_agent'])])

• group (Annotated[ibm_watsonx_gov.entities.enums.MetricGroup | None, FieldInfo(annotation=NoneType, required=False, default=None, description='The metric group this metric belongs to.', frozen=True)])

• is_reference_free (Annotated[bool, FieldInfo(annotation=NoneType, required=False, default=True, description='Decides whether this metric needs a reference for computation', frozen=True)])

• mapping (Annotated[ibm_watsonx_gov.entities.metric.Mapping | None, FieldInfo(annotation=NoneType, required=False, default=None, title='Mapping', description='The data mapping details for the metric which are used to read the values needed to compute the metric.', examples=Mapping(source='trace', items=[MappingItem(name='input_text', type_='input', column_name=None, span_name='LangGraph.workflow', attribute_name='traceloop.entity.input', json_path='$.inputs.input_text', lookup_child_spans=False), MappingItem(name='generated_text', type_='output', column_name=None, span_name='LangGraph.workflow', attribute_name='traceloop.entity.output', json_path='$.outputs.generated_text', lookup_child_spans=False)]))])

• method (Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='The method used to compute the metric.')])

• metric_dependencies (Annotated[list[GenAIMetric], FieldInfo(annotation=NoneType, required=False, default=[], description='Metrics that needs to be evaluated first')])

• tasks (Annotated[list[ibm_watsonx_gov.entities.enums.TaskType], FieldInfo(annotation=NoneType, required=False, default=[], description='The task types this metric is associated with.', frozen=True)])

• thresholds (Annotated[list[ibm_watsonx_gov.entities.metric_threshold.MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[], description='The list of thresholds')])

Validators:

• validate » all fields

field applies_to: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default='message', title='Applies to', description='The tag to indicate for which the metric is applied to. Used for agentic application metric computation.', examples=['message', 'conversation', 'sub_agent'])] = 'message'

The tag to indicate for which the metric is applied to. Used for agentic application metric computation.

Validated by:

• validate

field group: Annotated[MetricGroup | None, FieldInfo(annotation=NoneType, required=False, default=None, description='The metric group this metric belongs to.', frozen=True)] = None

The metric group this metric belongs to.

Validated by:

• validate

field is_reference_free: Annotated[bool, FieldInfo(annotation=NoneType, required=False, default=True, description='Decides whether this metric needs a reference for computation', frozen=True)] = True

Decides whether this metric needs a reference for computation

Validated by:

• validate

field mapping: Annotated[Mapping | None, FieldInfo(annotation=NoneType, required=False, default=None, title='Mapping', description='The data mapping details for the metric which are used to read the values needed to compute the metric.', examples=Mapping(source='trace', items=[MappingItem(name='input_text', type_='input', column_name=None, span_name='LangGraph.workflow', attribute_name='traceloop.entity.input', json_path='$.inputs.input_text', lookup_child_spans=False), MappingItem(name='generated_text', type_='output', column_name=None, span_name='LangGraph.workflow', attribute_name='traceloop.entity.output', json_path='$.outputs.generated_text', lookup_child_spans=False)]))] = None

The data mapping details for the metric which are used to read the values needed to compute the metric.

Validated by:

• validate

field method: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='The method used to compute the metric.')] = None

The method used to compute the metric.

Validated by:

• validate

field metric_dependencies: Annotated[list[GenAIMetric], FieldInfo(annotation=NoneType, required=False, default=[], description='Metrics that needs to be evaluated first')] = []

Metrics that needs to be evaluated first

Validated by:

• validate

field tasks: Annotated[list[TaskType], FieldInfo(annotation=NoneType, required=False, default=[], description='The task types this metric is associated with.', frozen=True)] = []

The task types this metric is associated with.

Validated by:

• validate

field thresholds: Annotated[list[MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[], description='The list of thresholds')] = []

The list of thresholds

Validated by:

• validate

abstract evaluate(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs) → AggregateMetricResult

async evaluate_async(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs) → AggregateMetricResult

get_aggregated_results_from_individual_results(record_results: List[RecordMetricResult])

info()

metric_dependencies_serializer(metric_dependencies: list[GenAIMetric])

model_post_init(context: Any, /) → None

We need to both initialize private attributes and call the user-defined model_post_init method.

validator validate  »  all fields

property id

pydantic model ibm_watsonx_gov.entities.metric.Mapping

Bases: BaseModel

Defines the field mapping details to be used for computing a metric.

Show JSON schema

{
   "title": "Mapping",
   "description": "Defines the field mapping details to be used for computing a metric.",
   "type": "object",
   "properties": {
      "source": {
         "default": "trace",
         "description": "The source type of the data. Use trace if the data should be read from span in trace. Use tabular if the data is passed as a dataframe.",
         "enum": [
            "trace",
            "tabular"
         ],
         "examples": [
            "trace",
            "tabular"
         ],
         "title": "Source",
         "type": "string"
      },
      "items": {
         "description": "The list of mapping items for the field. They are used to read the data from trace or tabular data for computing the metric.",
         "items": {
            "$ref": "#/$defs/MappingItem"
         },
         "title": "Mapping Items",
         "type": "array"
      }
   },
   "$defs": {
      "MappingItem": {
         "description": "The mapping details to be used for reading the values from the data.",
         "properties": {
            "name": {
               "description": "The name of the item.",
               "examples": [
                  "input_text",
                  "generated_text",
                  "context",
                  "ground_truth"
               ],
               "title": "Name",
               "type": "string"
            },
            "type": {
               "description": "The type of the item.",
               "enum": [
                  "input",
                  "output",
                  "reference",
                  "context",
                  "tool_call"
               ],
               "examples": [
                  "input"
               ],
               "title": "Type",
               "type": "string"
            },
            "column_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The column name in the tabular data to be used for reading the field value. Applicable for tabular source.",
               "title": "Column Name"
            },
            "span_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The span name in the trace data to be used for reading the field value. Applicable for trace source.",
               "title": "Span Name"
            },
            "attribute_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The attribute name in the trace to be used for reading the field value. Applicable for trace source.",
               "title": "Attribute Name"
            },
            "json_path": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The json path to be used for reading the field value from the attribute value. Applicable for trace source. If not provided, the span attribute value is read as the field value.",
               "title": "Json Path"
            },
            "lookup_child_spans": {
               "anyOf": [
                  {
                     "type": "boolean"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": false,
               "description": "The flag to indicate if all the child spans should be searched for the attribute value. Applicable for trace source.",
               "title": "Look up child spans"
            }
         },
         "required": [
            "name",
            "type"
         ],
         "title": "MappingItem",
         "type": "object"
      }
   },
   "required": [
      "items"
   ]
}

Fields:

• items (list[ibm_watsonx_gov.entities.metric.MappingItem])

• source (Literal['trace', 'tabular'])

field items: Annotated[list[MappingItem], FieldInfo(annotation=NoneType, required=True, title='Mapping Items', description='The list of mapping items for the field. They are used to read the data from trace or tabular data for computing the metric.')] [Required]

The list of mapping items for the field. They are used to read the data from trace or tabular data for computing the metric.

field source: Annotated[Literal['trace', 'tabular'], FieldInfo(annotation=NoneType, required=False, default='trace', title='Source', description='The source type of the data. Use trace if the data should be read from span in trace. Use tabular if the data is passed as a dataframe.', examples=['trace', 'tabular'])] = 'trace'

The source type of the data. Use trace if the data should be read from span in trace. Use tabular if the data is passed as a dataframe.

pydantic model ibm_watsonx_gov.entities.metric.MappingItem

Bases: BaseModel

The mapping details to be used for reading the values from the data.

Show JSON schema

{
   "title": "MappingItem",
   "description": "The mapping details to be used for reading the values from the data.",
   "type": "object",
   "properties": {
      "name": {
         "description": "The name of the item.",
         "examples": [
            "input_text",
            "generated_text",
            "context",
            "ground_truth"
         ],
         "title": "Name",
         "type": "string"
      },
      "type": {
         "description": "The type of the item.",
         "enum": [
            "input",
            "output",
            "reference",
            "context",
            "tool_call"
         ],
         "examples": [
            "input"
         ],
         "title": "Type",
         "type": "string"
      },
      "column_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The column name in the tabular data to be used for reading the field value. Applicable for tabular source.",
         "title": "Column Name"
      },
      "span_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The span name in the trace data to be used for reading the field value. Applicable for trace source.",
         "title": "Span Name"
      },
      "attribute_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The attribute name in the trace to be used for reading the field value. Applicable for trace source.",
         "title": "Attribute Name"
      },
      "json_path": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The json path to be used for reading the field value from the attribute value. Applicable for trace source. If not provided, the span attribute value is read as the field value.",
         "title": "Json Path"
      },
      "lookup_child_spans": {
         "anyOf": [
            {
               "type": "boolean"
            },
            {
               "type": "null"
            }
         ],
         "default": false,
         "description": "The flag to indicate if all the child spans should be searched for the attribute value. Applicable for trace source.",
         "title": "Look up child spans"
      }
   },
   "required": [
      "name",
      "type"
   ]
}

Fields:

• attribute_name (str | None)

• column_name (str | None)

• json_path (str | None)

• lookup_child_spans (bool | None)

• name (str)

• span_name (str | None)

• type_ (Literal['input', 'output', 'reference', 'context', 'tool_call'])

field attribute_name: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, title='Attribute Name', description='The attribute name in the trace to be used for reading the field value. Applicable for trace source.')] = None

The attribute name in the trace to be used for reading the field value. Applicable for trace source.

field column_name: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, title='Column Name', description='The column name in the tabular data to be used for reading the field value. Applicable for tabular source.')] = None

The column name in the tabular data to be used for reading the field value. Applicable for tabular source.

field json_path: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, title='Json Path', description='The json path to be used for reading the field value from the attribute value. Applicable for trace source. If not provided, the span attribute value is read as the field value.')] = None

The json path to be used for reading the field value from the attribute value. Applicable for trace source. If not provided, the span attribute value is read as the field value.

field lookup_child_spans: Annotated[bool | None, FieldInfo(annotation=NoneType, required=False, default=False, title='Look up child spans', description='The flag to indicate if all the child spans should be searched for the attribute value. Applicable for trace source.')] = False

The flag to indicate if all the child spans should be searched for the attribute value. Applicable for trace source.

field name: Annotated[str, FieldInfo(annotation=NoneType, required=True, title='Name', description='The name of the item.', examples=['input_text', 'generated_text', 'context', 'ground_truth'])] [Required]

The name of the item.

field span_name: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, title='Span Name', description='The span name in the trace data to be used for reading the field value. Applicable for trace source.')] = None

The span name in the trace data to be used for reading the field value. Applicable for trace source.

field type_: Annotated[Literal['input', 'output', 'reference', 'context', 'tool_call'], FieldInfo(annotation=NoneType, required=True, alias='type', alias_priority=2, title='Type', description='The type of the item.', examples=['input'])] [Required] (alias 'type')

The type of the item.

pydantic model ibm_watsonx_gov.entities.metric.PredictiveAIMetric

Bases: BaseMetric

Show JSON schema

{
   "title": "PredictiveAIMetric",
   "type": "object",
   "properties": {
      "name": {
         "description": "The name of the metric.",
         "examples": [
            "answer_relevance",
            "context_relevance"
         ],
         "title": "Metric Name",
         "type": "string"
      },
      "display_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The display name of the metric.",
         "examples": [
            "Answer Relevance",
            "Context Relevance"
         ],
         "title": "Metric display name"
      },
      "type_": {
         "default": "ootb",
         "description": "The type of the metric. Indicates whether the metric is ootb or custom.",
         "examples": [
            "ootb",
            "custom"
         ],
         "title": "Metric type",
         "type": "string"
      },
      "value_type": {
         "default": "numeric",
         "description": "The type of the metric value. Indicates whether the metric value is numeric or categorical.",
         "examples": [
            "numeric",
            "categorical"
         ],
         "title": "Metric value type",
         "type": "string"
      }
   },
   "required": [
      "name"
   ]
}

Fields:

model_post_init(context: Any, /) → None

We need to both initialize private attributes and call the user-defined model_post_init method.