Social Bias Metric¶
- pydantic model ibm_watsonx_gov.metrics.social_bias.social_bias_metric.SocialBiasMetric¶
Bases:
GenAIMetricDefines the Social Bias metric class.
The Social Bias metric measures the risk of systemic prejudice against groups based on shared identity or characteristics, often stemming from stereotypes or cultural influences. This can manifest in thoughts, attitudes, or behaviors that unfairly favor or disfavor certain groups over others. It is computed using the granite guardian model.
Examples
- Create Social Bias metric with default parameters and compute using metrics evaluator.
metric = SocialBiasMetric() result = MetricsEvaluator().evaluate(data={"input_text": "...", metrics=[metric])
- Create Social Bias metric with a custom threshold.
threshold = MetricThreshold(type="lower_limit", value=0.5) metric = SocialBiasMetric(threshold=threshold)
Show JSON schema
{ "title": "SocialBiasMetric", "description": "Defines the Social Bias metric class.\n\nThe Social Bias metric measures the risk of systemic prejudice against groups based on shared identity or characteristics, often stemming from stereotypes or cultural influences. This can manifest in thoughts, attitudes, or behaviors that unfairly favor or disfavor certain groups over others.\nIt is computed using the granite guardian model.\n\nExamples:\n 1. Create Social Bias metric with default parameters and compute using metrics evaluator.\n .. code-block:: python\n\n metric = SocialBiasMetric()\n result = MetricsEvaluator().evaluate(data={\"input_text\": \"...\", metrics=[metric])\n\n 2. Create Social Bias metric with a custom threshold.\n .. code-block:: python\n\n threshold = MetricThreshold(type=\"lower_limit\", value=0.5)\n metric = SocialBiasMetric(threshold=threshold)", "type": "object", "properties": { "name": { "const": "social_bias", "default": "social_bias", "description": "The social bias metric name.", "title": "Name", "type": "string" }, "thresholds": { "default": [ { "type": "lower_limit", "value": 0.7 } ], "description": "The metric thresholds.", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "tasks": { "default": [ "question_answering", "classification", "summarization", "generation", "extraction", "retrieval_augmented_generation" ], "description": "The list of supported tasks.", "items": { "$ref": "#/$defs/TaskType" }, "title": "Tasks", "type": "array" }, "group": { "$ref": "#/$defs/MetricGroup", "default": "content_safety", "description": "The metric group.", "title": "Group" }, "is_reference_free": { "default": true, "description": "Decides whether this metric needs a reference for computation", "title": "Is Reference Free", "type": "boolean" }, "method": { "const": "granite_guardian", "default": "granite_guardian", "description": "The method used to compute harm metric.", "title": "Method", "type": "string" }, "metric_dependencies": { "default": [], "description": "Metrics that needs to be evaluated first", "items": { "$ref": "#/$defs/GenAIMetric" }, "title": "Metric Dependencies", "type": "array" } }, "$defs": { "GenAIMetric": { "description": "Defines the Generative AI metric interface", "properties": { "name": { "description": "The name of the metric", "title": "Metric Name", "type": "string" }, "thresholds": { "default": [], "description": "The list of thresholds", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "tasks": { "description": "The task types this metric is associated with.", "items": { "$ref": "#/$defs/TaskType" }, "title": "Tasks", "type": "array" }, "group": { "anyOf": [ { "$ref": "#/$defs/MetricGroup" }, { "type": "null" } ], "default": null, "description": "The metric group this metric belongs to." }, "is_reference_free": { "default": true, "description": "Decides whether this metric needs a reference for computation", "title": "Is Reference Free", "type": "boolean" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute the metric.", "title": "Method" }, "metric_dependencies": { "default": [], "description": "Metrics that needs to be evaluated first", "items": { "$ref": "#/$defs/GenAIMetric" }, "title": "Metric Dependencies", "type": "array" } }, "required": [ "name", "tasks" ], "title": "GenAIMetric", "type": "object" }, "MetricGroup": { "enum": [ "retrieval_quality", "answer_quality", "content_safety", "performance", "usage", "tool_call_quality", "readability" ], "title": "MetricGroup", "type": "string" }, "MetricThreshold": { "description": "The class that defines the threshold for a metric.", "properties": { "type": { "description": "Threshold type. One of 'lower_limit', 'upper_limit'", "enum": [ "lower_limit", "upper_limit" ], "title": "Type", "type": "string" }, "value": { "default": 0, "description": "The value of metric threshold", "title": "Threshold value", "type": "number" } }, "required": [ "type" ], "title": "MetricThreshold", "type": "object" }, "TaskType": { "description": "Supported task types for generative AI models", "enum": [ "question_answering", "classification", "summarization", "generation", "extraction", "retrieval_augmented_generation" ], "title": "TaskType", "type": "string" } } }
- Fields:
group (Annotated[ibm_watsonx_gov.entities.enums.MetricGroup, FieldInfo(annotation=NoneType, required=False, default=
- field group: ', frozen=True)] = MetricGroup.CONTENT_SAFETY¶
The metric group.
- field method: Annotated[Literal['granite_guardian'], FieldInfo(annotation=NoneType, required=False, default='granite_guardian', title='Method', description='The method used to compute harm metric.')] = 'granite_guardian'¶
The method used to compute harm metric.
- field name: Annotated[Literal['social_bias'], FieldInfo(annotation=NoneType, required=False, default='social_bias', title='Name', description='The social bias metric name.', frozen=True)] = 'social_bias'¶
The social bias metric name.
- field tasks: Annotated[list[TaskType], FieldInfo(annotation=NoneType, required=False, default=['question_answering', 'classification', 'summarization', 'generation', 'extraction', 'retrieval_augmented_generation'], title='Tasks', description='The list of supported tasks.', frozen=True)] = ['question_answering', 'classification', 'summarization', 'generation', 'extraction', 'retrieval_augmented_generation']¶
The list of supported tasks.
- field thresholds: Annotated[list[MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[MetricThreshold(type='lower_limit', value=0.7)], title='Thresholds', description='The metric thresholds.')] = [MetricThreshold(type='lower_limit', value=0.7)]¶
The metric thresholds.
- evaluate(data: DataFrame | dict, configuration: GenAIConfiguration, **kwargs)¶
- async evaluate_async(data: DataFrame | dict, configuration: GenAIConfiguration, **kwargs) → list[AggregateMetricResult]¶
- model_post_init(context: Any, /) → None¶
We need to both initialize private attributes and call the user-defined model_post_init method.
- pydantic model ibm_watsonx_gov.metrics.social_bias.social_bias_metric.SocialBiasResult¶
Bases:
RecordMetricResultShow JSON schema
{ "title": "SocialBiasResult", "type": "object", "properties": { "name": { "default": "social_bias", "title": "Name", "type": "string" }, "method": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The method used to compute this metric result.", "examples": [ "token_recall" ], "title": "Method" }, "provider": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The provider used to compute this metric result.", "title": "Provider" }, "value": { "anyOf": [ { "type": "number" }, { "type": "string" }, { "type": "boolean" }, { "type": "null" } ], "description": "The metric value.", "title": "Value" }, "errors": { "anyOf": [ { "items": { "$ref": "#/$defs/Error" }, "type": "array" }, { "type": "null" } ], "default": null, "description": "The list of error messages", "title": "Errors" }, "additional_info": { "anyOf": [ { "type": "object" }, { "type": "null" } ], "default": null, "description": "The additional information about the metric result.", "title": "Additional Info" }, "group": { "$ref": "#/$defs/MetricGroup", "default": "content_safety" }, "thresholds": { "default": [], "description": "The metric thresholds", "items": { "$ref": "#/$defs/MetricThreshold" }, "title": "Thresholds", "type": "array" }, "record_id": { "description": "The record identifier.", "examples": [ "record1" ], "title": "Record Id", "type": "string" }, "record_timestamp": { "anyOf": [ { "type": "string" }, { "type": "null" } ], "default": null, "description": "The record timestamp.", "examples": [ "2025-01-01T00:00:00.000000Z" ], "title": "Record Timestamp" } }, "$defs": { "Error": { "properties": { "code": { "description": "The error code", "title": "Code", "type": "string" }, "message_en": { "description": "The error message in English.", "title": "Message En", "type": "string" }, "parameters": { "default": [], "description": "The list of parameters to construct the message in a different locale.", "items": {}, "title": "Parameters", "type": "array" } }, "required": [ "code", "message_en" ], "title": "Error", "type": "object" }, "MetricGroup": { "enum": [ "retrieval_quality", "answer_quality", "content_safety", "performance", "usage", "tool_call_quality", "readability" ], "title": "MetricGroup", "type": "string" }, "MetricThreshold": { "description": "The class that defines the threshold for a metric.", "properties": { "type": { "description": "Threshold type. One of 'lower_limit', 'upper_limit'", "enum": [ "lower_limit", "upper_limit" ], "title": "Type", "type": "string" }, "value": { "default": 0, "description": "The value of metric threshold", "title": "Threshold value", "type": "number" } }, "required": [ "type" ], "title": "MetricThreshold", "type": "object" } }, "required": [ "value", "record_id" ] }
- Config:
arbitrary_types_allowed: bool = True
use_enum_values: bool = True
- Fields:
- field group: MetricGroup = MetricGroup.CONTENT_SAFETY¶
- field name: str = 'social_bias'¶