Metric¶

pydantic model ibm_watsonx_gov.entities.metric.GenAIMetric¶

Bases: BaseMetric

Defines the Generative AI metric interface

Show JSON schema

{
   "$defs": {
      "GenAIMetric": {
         "description": "Defines the Generative AI metric interface",
         "properties": {
            "name": {
               "description": "The name of the metric",
               "title": "Metric Name",
               "type": "string"
            },
            "thresholds": {
               "default": [],
               "description": "The list of thresholds",
               "items": {
                  "$ref": "#/$defs/MetricThreshold"
               },
               "title": "Thresholds",
               "type": "array"
            },
            "tasks": {
               "description": "The task types this metric is associated with.",
               "items": {
                  "$ref": "#/$defs/TaskType"
               },
               "title": "Tasks",
               "type": "array"
            },
            "group": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/MetricGroup"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The metric group this metric belongs to."
            },
            "is_reference_free": {
               "default": true,
               "description": "Decides whether this metric needs a reference for computation",
               "title": "Is Reference Free",
               "type": "boolean"
            },
            "method": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The method used to compute the metric.",
               "title": "Method"
            },
            "metric_dependencies": {
               "default": [],
               "description": "Metrics that needs to be evaluated first",
               "items": {
                  "$ref": "#/$defs/GenAIMetric"
               },
               "title": "Metric Dependencies",
               "type": "array"
            }
         },
         "required": [
            "name",
            "tasks"
         ],
         "title": "GenAIMetric",
         "type": "object"
      },
      "MetricGroup": {
         "enum": [
            "retrieval_quality",
            "answer_quality",
            "content_safety",
            "performance",
            "usage",
            "tool_call_quality",
            "readability"
         ],
         "title": "MetricGroup",
         "type": "string"
      },
      "MetricThreshold": {
         "description": "The class that defines the threshold for a metric.",
         "properties": {
            "type": {
               "description": "Threshold type. One of 'lower_limit', 'upper_limit'",
               "enum": [
                  "lower_limit",
                  "upper_limit"
               ],
               "title": "Type",
               "type": "string"
            },
            "value": {
               "default": 0,
               "description": "The value of metric threshold",
               "title": "Threshold value",
               "type": "number"
            }
         },
         "required": [
            "type"
         ],
         "title": "MetricThreshold",
         "type": "object"
      },
      "TaskType": {
         "description": "Supported task types for generative AI models",
         "enum": [
            "question_answering",
            "classification",
            "summarization",
            "generation",
            "extraction",
            "retrieval_augmented_generation"
         ],
         "title": "TaskType",
         "type": "string"
      }
   },
   "$ref": "#/$defs/GenAIMetric"
}

Fields:

group (Annotated[ibm_watsonx_gov.entities.enums.MetricGroup | None, FieldInfo(annotation=NoneType, required=False, default=None, description='The metric group this metric belongs to.', frozen=True)])
is_reference_free (Annotated[bool, FieldInfo(annotation=NoneType, required=False, default=True, description='Decides whether this metric needs a reference for computation', frozen=True)])
method (Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='The method used to compute the metric.')])
metric_dependencies (Annotated[list[GenAIMetric], FieldInfo(annotation=NoneType, required=False, default=[], description='Metrics that needs to be evaluated first')])
tasks (Annotated[list[ibm_watsonx_gov.entities.enums.TaskType], FieldInfo(annotation=NoneType, required=True, description='The task types this metric is associated with.', frozen=True)])
thresholds (Annotated[list[ibm_watsonx_gov.entities.metric_threshold.MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[], description='The list of thresholds')])

field group: Annotated[MetricGroup | None, FieldInfo(annotation=NoneType, required=False, default=None, description='The metric group this metric belongs to.', frozen=True)] = None¶: The metric group this metric belongs to.

field is_reference_free: Annotated[bool, FieldInfo(annotation=NoneType, required=False, default=True, description='Decides whether this metric needs a reference for computation', frozen=True)] = True¶: Decides whether this metric needs a reference for computation

field method: Annotated[str | None, FieldInfo(annotation=NoneType, required=False, default=None, description='The method used to compute the metric.')] = None¶: The method used to compute the metric.

field metric_dependencies: Annotated[list[GenAIMetric], FieldInfo(annotation=NoneType, required=False, default=[], description='Metrics that needs to be evaluated first')] = []¶: Metrics that needs to be evaluated first

field tasks: Annotated[list[TaskType], FieldInfo(annotation=NoneType, required=True, description='The task types this metric is associated with.', frozen=True)] [Required]¶: The task types this metric is associated with.

field thresholds: Annotated[list[MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[], description='The list of thresholds')] = []¶: The list of thresholds

abstract evaluate(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs) → AggregateMetricResult¶

async evaluate_async(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs) → AggregateMetricResult¶

get_aggregated_results_from_individual_results(record_results: List[RecordMetricResult])¶

info()¶

metric_dependencies_serializer(metric_dependencies: list[GenAIMetric])¶

model_post_init(context: Any, /) → None¶: We need to both initialize private attributes and call the user-defined model_post_init method.

property id¶

pydantic model ibm_watsonx_gov.entities.metric.Locale¶

Bases: BaseModel

Show JSON schema

{
   "title": "Locale",
   "type": "object",
   "properties": {
      "input": {
         "anyOf": [
            {
               "items": {
                  "type": "string"
               },
               "type": "array"
            },
            {
               "additionalProperties": {
                  "type": "string"
               },
               "type": "object"
            },
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Input"
      },
      "output": {
         "anyOf": [
            {
               "items": {
                  "type": "string"
               },
               "type": "array"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Output"
      },
      "reference": {
         "anyOf": [
            {
               "items": {
                  "type": "string"
               },
               "type": "array"
            },
            {
               "additionalProperties": {
                  "type": "string"
               },
               "type": "object"
            },
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Reference"
      }
   }
}

Fields:

input (list[str] | dict[str, str] | str | None)
output (list[str] | None)
reference (list[str] | dict[str, str] | str | None)

field input: list[str] | dict[str, str] | str | None = None¶

field output: list[str] | None = None¶

field reference: list[str] | dict[str, str] | str | None = None¶

pydantic model ibm_watsonx_gov.entities.metric.PredictiveAIMetric¶

Bases: BaseMetric

Show JSON schema

{
   "title": "PredictiveAIMetric",
   "type": "object",
   "properties": {
      "name": {
         "description": "The name of the metric",
         "title": "Metric Name",
         "type": "string"
      }
   },
   "required": [
      "name"
   ]
}

Fields:

model_post_init(context: Any, /) → None¶: We need to both initialize private attributes and call the user-defined model_post_init method.