Hap Metric¶
- pydantic model ibm_watsonx_gov.metrics.hap.hap_metric.HAPMetric¶
Bases:
GenAIMetricDefines the HAP metric class.
The HAP metric measures if there is any toxic content that contains hate, abuse, or profanity in the input. It is computed using the hap model.
Examples
- Create HAP metric with default parameters and compute using metrics evaluator.
metric = HAPMetric() result = MetricsEvaluator().evaluate(data={"input_text": "...", metrics=[metric])
- Create HAP metric with a custom threshold.
threshold = MetricThreshold(type="lower_limit", value=0.5) metric = HAPMetric(threshold=threshold)
Show JSON schema
{ "title": "HAPMetric", "description": "Defines the HAP metric class.\n\nThe HAP metric measures if there is any toxic content that contains hate, abuse, or profanity in the input.\nIt is computed using the hap model.\n\nExamples:\n 1. Create HAP metric with default parameters and compute using metrics evaluator.\n .. code-block:: python\n\n metric = HAPMetric()\n result = MetricsEvaluator().evaluate(data={\"input_text\": \"...\", metrics=[metric])\n\n 2. Create HAP metric with a custom threshold.\n .. code-block:: python\n\n threshold = MetricThreshold(type=\"lower_limit\", value=0.5)\n metric = HAPMetric(threshold=threshold)", "type": "object", "properties": { "name": { "const": "hap", "default": "hap", "description": "The hap metric name.", "title": "Name", "type": "string" }, "thresholds": { "default": [ { "type": "lower_limit", "value": 0.7 } ], "description": "The metric thresholds.", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "tasks": { "default": [ "question_answering", "classification", "summarization", "generation", "extraction", "retrieval_augmented_generation" ], "description": "The list of supported tasks.", "items": { "$ref": "#/$defs/TaskType" }, "title": "Tasks", "type": "array" }, "group": { "$ref": "#/$defs/MetricGroup", "default": "content_safety", "description": "The metric group.", "title": "Group" }, "is_reference_free": { "default": true, "description": "Decides whether this metric needs a reference for computation", "title": "Is Reference Free", "type": "boolean" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute the metric.", "title": "Method" }, "metric_dependencies": { "default": [], "description": "Metrics that needs to be evaluated first", "items": { "$ref": "#/$defs/GenAIMetric" }, "title": "Metric Dependencies", "type": "array" } }, "$defs": { "GenAIMetric": { "description": "Defines the Generative AI metric interface", "properties": { "name": { "description": "The name of the metric", "title": "Metric Name", "type": "string" }, "thresholds": { "default": [], "description": "The list of thresholds", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "tasks": { "description": "The task types this metric is associated with.", "items": { "$ref": "#/$defs/TaskType" }, "title": "Tasks", "type": "array" }, "group": { "anyOf": [ { "$ref": "#/$defs/MetricGroup" }, { "type": "null" } ], "default": null, "description": "The metric group this metric belongs to." }, "is_reference_free": { "default": true, "description": "Decides whether this metric needs a reference for computation", "title": "Is Reference Free", "type": "boolean" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute the metric.", "title": "Method" }, "metric_dependencies": { "default": [], "description": "Metrics that needs to be evaluated first", "items": { "$ref": "#/$defs/GenAIMetric" }, "title": "Metric Dependencies", "type": "array" } }, "required": [ "name", "tasks" ], "title": "GenAIMetric", "type": "object" }, "MetricGroup": { "enum": [ "retrieval_quality", "answer_quality", "content_safety", "performance", "usage", "tool_call_quality", "readability" ], "title": "MetricGroup", "type": "string" }, "MetricThreshold": { "description": "The class that defines the threshold for a metric.", "properties": { "type": { "description": "Threshold type. One of 'lower_limit', 'upper_limit'", "enum": [ "lower_limit", "upper_limit" ], "title": "Type", "type": "string" }, "value": { "default": 0, "description": "The value of metric threshold", "title": "Threshold value", "type": "number" } }, "required": [ "type" ], "title": "MetricThreshold", "type": "object" }, "TaskType": { "description": "Supported task types for generative AI models", "enum": [ "question_answering", "classification", "summarization", "generation", "extraction", "retrieval_augmented_generation" ], "title": "TaskType", "type": "string" } } }
- Fields:
group (Annotated[ibm_watsonx_gov.entities.enums.MetricGroup, FieldInfo(annotation=NoneType, required=False, default=
- field group: ', frozen=True)] = MetricGroup.CONTENT_SAFETY¶
The metric group.
- field name: Annotated[Literal['hap'], FieldInfo(annotation=NoneType, required=False, default='hap', title='Name', description='The hap metric name.', frozen=True)] = 'hap'¶
The hap metric name.
- field tasks: Annotated[list[TaskType], FieldInfo(annotation=NoneType, required=False, default=['question_answering', 'classification', 'summarization', 'generation', 'extraction', 'retrieval_augmented_generation'], title='Tasks', description='The list of supported tasks.', frozen=True)] = ['question_answering', 'classification', 'summarization', 'generation', 'extraction', 'retrieval_augmented_generation']¶
The list of supported tasks.
- field thresholds: Annotated[list[MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[MetricThreshold(type='lower_limit', value=0.7)], title='Thresholds', description='The metric thresholds.')] = [MetricThreshold(type='lower_limit', value=0.7)]¶
The metric thresholds.
- evaluate(data: DataFrame | dict, configuration: GenAIConfiguration, **kwargs)¶
- async evaluate_async(data: DataFrame, configuration: GenAIConfiguration, **kwargs) list[AggregateMetricResult]¶
- model_post_init(context: Any, /) None¶
We need to both initialize private attributes and call the user-defined model_post_init method.
- pydantic model ibm_watsonx_gov.metrics.hap.hap_metric.HAPResult¶
Bases:
RecordMetricResultShow JSON schema
{ "title": "HAPResult", "type": "object", "properties": { "name": { "default": "hap", "title": "Name", "type": "string" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute this metric result.", "examples": [ "token_recall" ], "title": "Method" }, "provider": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The provider used to compute this metric result.", "title": "Provider" }, "value": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "type": "null" } ], "description": "The metric value.", "title": "Value" }, "errors": { "anyOf": [ { "items": { "$ref": "#/$defs/Error" }, "type": "array" }, { "type": "null" } ], "default": null, "description": "The list of error messages", "title": "Errors" }, "additional_info": { "anyOf": [ { "type": "object" }, { "type": "null" } ], "default": null, "description": "The additional information about the metric result.", "title": "Additional Info" }, "group": { "$ref": "#/$defs/MetricGroup", "default": "content_safety" }, "thresholds": { "default": [], "description": "The metric thresholds", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "record_id": { "description": "The record identifier.", "examples": [ "record1" ], "title": "Record Id", "type": "string" }, "record_timestamp": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The record timestamp.", "examples": [ "2025-01-01T00:00:00.000000Z" ], "title": "Record Timestamp" } }, "$defs": { "Error": { "properties": { "code": { "description": "The error code", "title": "Code", "type": "string" }, "message_en": { "description": "The error message in English.", "title": "Message En", "type": "string" }, "parameters": { "default": [], "description": "The list of parameters to construct the message in a different locale.", "items": {}, "title": "Parameters", "type": "array" } }, "required": [ "code", "message_en" ], "title": "Error", "type": "object" }, "MetricGroup": { "enum": [ "retrieval_quality", "answer_quality", "content_safety", "performance", "usage", "tool_call_quality", "readability" ], "title": "MetricGroup", "type": "string" }, "MetricThreshold": { "description": "The class that defines the threshold for a metric.", "properties": { "type": { "description": "Threshold type. One of 'lower_limit', 'upper_limit'", "enum": [ "lower_limit", "upper_limit" ], "title": "Type", "type": "string" }, "value": { "default": 0, "description": "The value of metric threshold", "title": "Threshold value", "type": "number" } }, "required": [ "type" ], "title": "MetricThreshold", "type": "object" } }, "required": [ "value", "record_id" ] }
- Config:
arbitrary_types_allowed: bool = True
use_enum_values: bool = True
- Fields:
- field group: MetricGroup = MetricGroup.CONTENT_SAFETY¶
- field name: str = 'hap'¶