Context Relevance Metric

pydantic model ibm_watsonx_gov.metrics.context_relevance.context_relevance_metric.ContextRelevanceMetric

Bases: GenAIMetric

Defines the Context Relevance metric class.

The Context Relevance metric measures the relevance of the contexts to the given input query. It can be computed using the below methods:

  1. token_precision (default)

  2. sentence_bert_bge

  3. sentence_bert_mini_lm

  4. llm_as_judge

  5. granite_guardian

If there are multiple context fields, the context relevance score is computed by combining all the contexts. To compute the individual context relevance scores, set the compute_per_context flag to True. The default value is False. When compute_per_context is set to True, the metric value is taken as the maximum of the combined context relevance score and the context relevance scores for each context. The other retrieval quality metrics use per context scores for computation. Its recommended to set the compute_per_context flag to True when computing the retrieval quality metrics for better accuracy.

Examples

  1. Create Context Relevance metric with default parameters and compute using metrics evaluator.
    metric = ContextRelevanceMetric()
result = MetricsEvaluator().evaluate(data={"input_text": "...", "context": "..."},
                                    metrics=[metric])
# A list of contexts can also be passed as shown below
result = MetricsEvaluator().evaluate(data={"input_text": "...", "context": ["...", "..."]},
                                    metrics=[metric])
  2. Create Context Relevance metric with a custom thresholds and method.
    thresholds  = [MetricThreshold(type="lower_limit", value=0.5)]
method = "sentence_bert_bge"
metric = ContextRelevanceMetric(
    method=method, thresholds=thresholds)
  3. Create Context Relevance metric with llm_as_judge method.
    # Define LLM Judge using watsonx.ai
# To use other frameworks and models as llm_judge, see :module:`ibm_watsonx_gov.entities.foundation_model`
llm_judge = LLMJudge(model=WxAIFoundationModel(
                            model_id="google/flan-ul2",
                            project_id="<PROJECT_ID>"))
metric = ContextRelevanceMetric(llm_judge=llm_judge)
  4. Create Context Relevance metric with granite_guardian method.
    metric = ContextRelevanceMetric(method="granite_guardian")

Show JSON schema
{
   "title": "ContextRelevanceMetric",
   "description": "Defines the Context Relevance metric class.\n\nThe Context Relevance metric measures the relevance of the contexts to the given input query.\nIt can be computed using the below methods:\n    1. token_precision (default)\n    2. sentence_bert_bge\n    3. sentence_bert_mini_lm\n    4. llm_as_judge\n    5. granite_guardian\nIf there are multiple context fields, the context relevance score is computed by combining all the contexts.\nTo compute the individual context relevance scores, set the `compute_per_context` flag to True. The default value is False.\nWhen `compute_per_context` is set to True, the metric value is taken as the maximum of the combined context relevance score and the context relevance scores for each context.\nThe other retrieval quality metrics use per context scores for computation. Its recommended to set the `compute_per_context` flag to True when computing the retrieval quality metrics for better accuracy.\n\nExamples:\n    1. Create Context Relevance metric with default parameters and compute using metrics evaluator.\n        .. code-block:: python\n\n            metric = ContextRelevanceMetric()\n            result = MetricsEvaluator().evaluate(data={\"input_text\": \"...\", \"context\": \"...\"},\n                                                metrics=[metric])\n            # A list of contexts can also be passed as shown below\n            result = MetricsEvaluator().evaluate(data={\"input_text\": \"...\", \"context\": [\"...\", \"...\"]},\n                                                metrics=[metric])\n\n    2. Create Context Relevance metric with a custom thresholds and method.\n        .. code-block:: python\n\n            thresholds  = [MetricThreshold(type=\"lower_limit\", value=0.5)]\n            method = \"sentence_bert_bge\"\n            metric = ContextRelevanceMetric(\n                method=method, thresholds=thresholds)\n\n    3. Create Context Relevance metric with llm_as_judge method.\n        .. code-block:: python\n\n            # Define LLM Judge using watsonx.ai\n            # To use other frameworks and models as llm_judge, see :module:`ibm_watsonx_gov.entities.foundation_model`\n            llm_judge = LLMJudge(model=WxAIFoundationModel(\n                                        model_id=\"google/flan-ul2\",\n                                        project_id=\"<PROJECT_ID>\"))\n            metric = ContextRelevanceMetric(llm_judge=llm_judge)\n\n    4. Create Context Relevance metric with granite_guardian method.\n        .. code-block:: python\n\n            metric = ContextRelevanceMetric(method=\"granite_guardian\")",
   "type": "object",
   "properties": {
      "name": {
         "const": "context_relevance",
         "default": "context_relevance",
         "description": "The context relevance metric name.",
         "title": "Name",
         "type": "string"
      },
      "thresholds": {
         "default": [
            {
               "type": "lower_limit",
               "value": 0.7
            }
         ],
         "description": "The metric thresholds.",
         "items": {
            "$ref": "#/$defs/MetricThreshold"
         },
         "title": "Thresholds",
         "type": "array"
      },
      "tasks": {
         "default": [
            "retrieval_augmented_generation"
         ],
         "description": "The list of supported tasks.",
         "items": {
            "$ref": "#/$defs/TaskType"
         },
         "title": "Tasks",
         "type": "array"
      },
      "group": {
         "$ref": "#/$defs/MetricGroup",
         "default": "retrieval_quality",
         "description": "The metric group.",
         "title": "Group"
      },
      "is_reference_free": {
         "default": true,
         "description": "Decides whether this metric needs a reference for computation",
         "title": "Is Reference Free",
         "type": "boolean"
      },
      "method": {
         "default": "token_precision",
         "description": "The method used to compute the metric. This field is optional and when `llm_judge` is provided, the method would be set to `llm_as_judge`.",
         "enum": [
            "token_precision",
            "sentence_bert_bge",
            "sentence_bert_mini_lm",
            "llm_as_judge",
            "granite_guardian"
         ],
         "title": "Method",
         "type": "string"
      },
      "metric_dependencies": {
         "default": [],
         "description": "Metrics that needs to be evaluated first",
         "items": {
            "$ref": "#/$defs/GenAIMetric"
         },
         "title": "Metric Dependencies",
         "type": "array"
      },
      "llm_judge": {
         "anyOf": [
            {
               "$ref": "#/$defs/LLMJudge"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The LLM judge used to compute the metric.",
         "title": "LLM Judge"
      },
      "compute_per_context": {
         "default": false,
         "description": "The flag to compute the relevance score of each context. The default value is False. Setting the flag to True increases the latency and cost of metric computation.",
         "title": "Compute per context",
         "type": "boolean"
      }
   },
   "$defs": {
      "AzureOpenAICredentials": {
         "properties": {
            "url": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "description": "Azure OpenAI url. This attribute can be read from `AZURE_OPENAI_HOST` environment variable.",
               "title": "Url"
            },
            "api_key": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "description": "API key for Azure OpenAI. This attribute can be read from `AZURE_OPENAI_API_KEY` environment variable.",
               "title": "Api Key"
            },
            "api_version": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "description": "The model API version from Azure OpenAI. This attribute can be read from `AZURE_OPENAI_API_VERSION` environment variable.",
               "title": "Api Version"
            }
         },
         "required": [
            "url",
            "api_key",
            "api_version"
         ],
         "title": "AzureOpenAICredentials",
         "type": "object"
      },
      "AzureOpenAIFoundationModel": {
         "description": "The Azure OpenAI foundation model details\n\nExamples:\n    1. Create Azure OpenAI foundation model by passing the credentials during object creation.\n        .. code-block:: python\n\n            azure_openai_foundation_model = AzureOpenAIFoundationModel(\n                model_id=\"gpt-4o-mini\",\n                provider=AzureOpenAIModelProvider(\n                    credentials=AzureOpenAICredentials(\n                        api_key=azure_api_key,\n                        url=azure_host_url,\n                        api_version=azure_api_model_version,\n                    )\n                )\n            )\n\n2. Create Azure OpenAI foundation model by setting the credentials in environment variables:\n    * ``AZURE_OPENAI_API_KEY`` is used to set the api key for OpenAI.\n    * ``AZURE_OPENAI_HOST`` is used to set the url for Azure OpenAI.\n    * ``AZURE_OPENAI_API_VERSION`` is uses to set the the api version for Azure OpenAI.\n\n        .. code-block:: python\n\n            openai_foundation_model = AzureOpenAIFoundationModel(\n                model_id=\"gpt-4o-mini\",\n            )",
         "properties": {
            "model_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The name of the foundation model.",
               "title": "Model Name"
            },
            "provider": {
               "$ref": "#/$defs/AzureOpenAIModelProvider",
               "description": "Azure OpenAI provider"
            },
            "model_id": {
               "description": "Model deployment name from Azure OpenAI",
               "title": "Model Id",
               "type": "string"
            }
         },
         "required": [
            "model_id"
         ],
         "title": "AzureOpenAIFoundationModel",
         "type": "object"
      },
      "AzureOpenAIModelProvider": {
         "properties": {
            "type": {
               "$ref": "#/$defs/ModelProviderType",
               "default": "azure_openai",
               "description": "The type of model provider."
            },
            "credentials": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/AzureOpenAICredentials"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "Azure OpenAI credentials."
            }
         },
         "title": "AzureOpenAIModelProvider",
         "type": "object"
      },
      "GenAIMetric": {
         "description": "Defines the Generative AI metric interface",
         "properties": {
            "name": {
               "description": "The name of the metric",
               "title": "Metric Name",
               "type": "string"
            },
            "thresholds": {
               "default": [],
               "description": "The list of thresholds",
               "items": {
                  "$ref": "#/$defs/MetricThreshold"
               },
               "title": "Thresholds",
               "type": "array"
            },
            "tasks": {
               "description": "The task types this metric is associated with.",
               "items": {
                  "$ref": "#/$defs/TaskType"
               },
               "title": "Tasks",
               "type": "array"
            },
            "group": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/MetricGroup"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The metric group this metric belongs to."
            },
            "is_reference_free": {
               "default": true,
               "description": "Decides whether this metric needs a reference for computation",
               "title": "Is Reference Free",
               "type": "boolean"
            },
            "method": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The method used to compute the metric.",
               "title": "Method"
            },
            "metric_dependencies": {
               "default": [],
               "description": "Metrics that needs to be evaluated first",
               "items": {
                  "$ref": "#/$defs/GenAIMetric"
               },
               "title": "Metric Dependencies",
               "type": "array"
            }
         },
         "required": [
            "name",
            "tasks"
         ],
         "title": "GenAIMetric",
         "type": "object"
      },
      "LLMJudge": {
         "description": "Defines the LLMJudge.\n\nThe LLMJudge class contains the details of the llm judge model to be used for computing the metric.\n\nExamples:\n    1. Create LLMJudge using watsonx.ai foundation model:\n        .. code-block:: python\n\n            wx_ai_foundation_model = WxAIFoundationModel(\n                model_id=\"google/flan-ul2\",\n                project_id=PROJECT_ID,\n                provider=WxAIModelProvider(\n                    credentials=WxAICredentials(api_key=wx_apikey)\n                )\n            )\n            llm_judge = LLMJudge(model=wx_ai_foundation_model)",
         "properties": {
            "model": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/WxAIFoundationModel"
                  },
                  {
                     "$ref": "#/$defs/OpenAIFoundationModel"
                  },
                  {
                     "$ref": "#/$defs/AzureOpenAIFoundationModel"
                  },
                  {
                     "$ref": "#/$defs/RITSFoundationModel"
                  }
               ],
               "description": "The foundation model to be used as judge",
               "title": "Model"
            }
         },
         "required": [
            "model"
         ],
         "title": "LLMJudge",
         "type": "object"
      },
      "MetricGroup": {
         "enum": [
            "retrieval_quality",
            "answer_quality",
            "content_safety",
            "performance",
            "usage",
            "tool_call_quality",
            "readability"
         ],
         "title": "MetricGroup",
         "type": "string"
      },
      "MetricThreshold": {
         "description": "The class that defines the threshold for a metric.",
         "properties": {
            "type": {
               "description": "Threshold type. One of 'lower_limit', 'upper_limit'",
               "enum": [
                  "lower_limit",
                  "upper_limit"
               ],
               "title": "Type",
               "type": "string"
            },
            "value": {
               "default": 0,
               "description": "The value of metric threshold",
               "title": "Threshold value",
               "type": "number"
            }
         },
         "required": [
            "type"
         ],
         "title": "MetricThreshold",
         "type": "object"
      },
      "ModelProviderType": {
         "description": "Supported model provider types for Generative AI",
         "enum": [
            "ibm_watsonx.ai",
            "azure_openai",
            "rits",
            "openai",
            "custom"
         ],
         "title": "ModelProviderType",
         "type": "string"
      },
      "OpenAICredentials": {
         "description": "Defines the OpenAICredentials class to specify the OpenAI server details.\n\nExamples:\n    1. Create OpenAICredentials with default parameters. By default Dallas region is used.\n        .. code-block:: python\n\n            openai_credentials = OpenAICredentials(api_key=api_key,\n                                                   url=openai_url)\n\n    2. Create OpenAICredentials by reading from environment variables.\n        .. code-block:: python\n\n            os.environ[\"OPENAI_API_KEY\"] = \"...\"\n            os.environ[\"OPENAI_URL\"] = \"...\"\n            openai_credentials = OpenAICredentials.create_from_env()",
         "properties": {
            "url": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "title": "Url"
            },
            "api_key": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "title": "Api Key"
            }
         },
         "required": [
            "url",
            "api_key"
         ],
         "title": "OpenAICredentials",
         "type": "object"
      },
      "OpenAIFoundationModel": {
         "description": "The OpenAI foundation model details\n\nExamples:\n    1. Create OpenAI foundation model by passing the credentials during object creation. Note that the url is optional and will be set to the default value for OpenAI. To change the default value, the url should be passed to ``OpenAICredentials`` object.\n        .. code-block:: python\n\n            openai_foundation_model = OpenAIFoundationModel(\n                model_id=\"gpt-4o-mini\",\n                provider=OpenAIModelProvider(\n                    credentials=OpenAICredentials(\n                        api_key=api_key,\n                        url=openai_url,\n                    )\n                )\n            )\n\n    2. Create OpenAI foundation model by setting the credentials in environment variables:\n        * ``OPENAI_API_KEY`` is used to set the api key for OpenAI.\n        * ``OPENAI_URL`` is used to set the url for OpenAI\n\n        .. code-block:: python\n\n            openai_foundation_model = OpenAIFoundationModel(\n                model_id=\"gpt-4o-mini\",\n            )",
         "properties": {
            "model_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The name of the foundation model.",
               "title": "Model Name"
            },
            "provider": {
               "$ref": "#/$defs/OpenAIModelProvider",
               "description": "OpenAI provider"
            },
            "model_id": {
               "description": "Model name from OpenAI",
               "title": "Model Id",
               "type": "string"
            }
         },
         "required": [
            "model_id"
         ],
         "title": "OpenAIFoundationModel",
         "type": "object"
      },
      "OpenAIModelProvider": {
         "properties": {
            "type": {
               "$ref": "#/$defs/ModelProviderType",
               "default": "openai",
               "description": "The type of model provider."
            },
            "credentials": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/OpenAICredentials"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "OpenAI credentials. This can also be set by using `OPENAI_API_KEY` environment variable."
            }
         },
         "title": "OpenAIModelProvider",
         "type": "object"
      },
      "RITSCredentials": {
         "properties": {
            "hostname": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": "https://inference-3scale-apicast-production.apps.rits.fmaas.res.ibm.com",
               "description": "The rits hostname",
               "title": "Hostname"
            },
            "api_key": {
               "title": "Api Key",
               "type": "string"
            }
         },
         "required": [
            "api_key"
         ],
         "title": "RITSCredentials",
         "type": "object"
      },
      "RITSFoundationModel": {
         "properties": {
            "model_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The name of the foundation model.",
               "title": "Model Name"
            },
            "provider": {
               "$ref": "#/$defs/RITSModelProvider",
               "description": "The provider of the model."
            }
         },
         "title": "RITSFoundationModel",
         "type": "object"
      },
      "RITSModelProvider": {
         "properties": {
            "type": {
               "$ref": "#/$defs/ModelProviderType",
               "default": "rits",
               "description": "The type of model provider."
            },
            "credentials": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/RITSCredentials"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "RITS credentials."
            }
         },
         "title": "RITSModelProvider",
         "type": "object"
      },
      "TaskType": {
         "description": "Supported task types for generative AI models",
         "enum": [
            "question_answering",
            "classification",
            "summarization",
            "generation",
            "extraction",
            "retrieval_augmented_generation"
         ],
         "title": "TaskType",
         "type": "string"
      },
      "WxAICredentials": {
         "description": "Defines the WxAICredentials class to specify the watsonx.ai server details.\n\nExamples:\n    1. Create WxAICredentials with default parameters. By default Dallas region is used.\n        .. code-block:: python\n\n            wxai_credentials = WxAICredentials(api_key=\"...\")\n\n    2. Create WxAICredentials by specifying region url.\n        .. code-block:: python\n\n            wxai_credentials = WxAICredentials(api_key=\"...\",\n                                               url=\"https://au-syd.ml.cloud.ibm.com\")\n\n    3. Create WxAICredentials by reading from environment variables.\n        .. code-block:: python\n\n            os.environ[\"WATSONX_APIKEY\"] = \"...\"\n            # [Optional] Specify watsonx region specific url. Default is https://us-south.ml.cloud.ibm.com .\n            os.environ[\"WATSONX_URL\"] = \"https://eu-gb.ml.cloud.ibm.com\"\n            wxai_credentials = WxAICredentials.create_from_env()\n\n    4. Create WxAICredentials for on-prem.\n        .. code-block:: python\n\n            wxai_credentials = WxAICredentials(url=\"https://<hostname>\",\n                                               username=\"...\"\n                                               api_key=\"...\",\n                                               version=\"5.2\")\n\n    5. Create WxAICredentials by reading from environment variables for on-prem.\n        .. code-block:: python\n\n            os.environ[\"WATSONX_URL\"] = \"https://<hostname>\"\n            os.environ[\"WATSONX_VERSION\"] = \"5.2\"\n            os.environ[\"WATSONX_USERNAME\"] = \"...\"\n            os.environ[\"WATSONX_APIKEY\"] = \"...\"\n            # Only one of api_key or password is needed\n            #os.environ[\"WATSONX_PASSWORD\"] = \"...\"\n            wxai_credentials = WxAICredentials.create_from_env()",
         "properties": {
            "url": {
               "default": "https://us-south.ml.cloud.ibm.com",
               "description": "The url for watsonx ai service",
               "examples": [
                  "https://us-south.ml.cloud.ibm.com",
                  "https://eu-de.ml.cloud.ibm.com",
                  "https://eu-gb.ml.cloud.ibm.com",
                  "https://jp-tok.ml.cloud.ibm.com",
                  "https://au-syd.ml.cloud.ibm.com"
               ],
               "title": "watsonx.ai url",
               "type": "string"
            },
            "api_key": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The user api key. Required for using watsonx as a service and one of api_key or password is required for using watsonx on-prem software.",
               "strip_whitespace": true,
               "title": "Api Key"
            },
            "version": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The watsonx on-prem software version. Required for using watsonx on-prem software.",
               "title": "Version"
            },
            "username": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The user name. Required for using watsonx on-prem software.",
               "title": "User name"
            },
            "password": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The user password. One of api_key or password is required for using watsonx on-prem software.",
               "title": "Password"
            },
            "instance_id": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": "openshift",
               "description": "The watsonx.ai instance id. Default value is openshift.",
               "title": "Instance id"
            }
         },
         "title": "WxAICredentials",
         "type": "object"
      },
      "WxAIFoundationModel": {
         "description": "The IBM watsonx.ai foundation model details\n\nTo initialize the foundation model, you can either pass in the credentials directly or set the environment.\nYou can follow these examples to create the provider.\n\nExamples:\n    1. Create foundation model by specifying the credentials during object creation:\n        .. code-block:: python\n\n            # Specify the credentials during object creation\n            wx_ai_foundation_model = WxAIFoundationModel(\n                model_id=\"google/flan-ul2\",\n                project_id=<PROJECT_ID>,\n                provider=WxAIModelProvider(\n                    credentials=WxAICredentials(\n                        url=wx_url, # This is optional field, by default US-Dallas region is selected\n                        api_key=wx_apikey,\n                    )\n                )\n            )\n\n    2. Create foundation model by setting the credentials environment variables:\n        * The api key can be set using one of the environment variables ``WXAI_API_KEY``, ``WATSONX_APIKEY``, or ``WXG_API_KEY``. These will be read in the order of precedence.\n        * The url is optional and will be set to US-Dallas region by default. It can be set using one of the environment variables ``WXAI_URL``, ``WATSONX_URL``, or ``WXG_URL``. These will be read in the order of precedence.\n\n        .. code-block:: python\n\n            wx_ai_foundation_model = WxAIFoundationModel(\n                model_id=\"google/flan-ul2\",\n                project_id=<PROJECT_ID>,\n            )\n\n    3. Create foundation model by specifying watsonx.governance software credentials during object creation:\n        .. code-block:: python\n\n            wx_ai_foundation_model = WxAIFoundationModel(\n                model_id=\"google/flan-ul2\",\n                project_id=project_id,\n                provider=WxAIModelProvider(\n                    credentials=WxAICredentials(\n                        url=wx_url,\n                        api_key=wx_apikey,\n                        username=wx_username,\n                        version=wx_version,\n                    )\n                )\n            )\n\n    4. Create foundation model by setting watsonx.governance software credentials environment variables:\n        * The api key can be set using one of the environment variables ``WXAI_API_KEY``, ``WATSONX_APIKEY``, or ``WXG_API_KEY``. These will be read in the order of precedence.\n        * The url can be set using one of these environment variable ``WXAI_URL``, ``WATSONX_URL``, or ``WXG_URL``. These will be read in the order of precedence.\n        * The username can be set using one of these environment variable ``WXAI_USERNAME``, ``WATSONX_USERNAME``, or ``WXG_USERNAME``. These will be read in the order of precedence.\n        * The version of watsonx.governance software can be set using one of these environment variable ``WXAI_VERSION``, ``WATSONX_VERSION``, or ``WXG_VERSION``. These will be read in the order of precedence.\n\n        .. code-block:: python\n\n            wx_ai_foundation_model = WxAIFoundationModel(\n                model_id=\"google/flan-ul2\",\n                project_id=project_id,\n            )",
         "properties": {
            "model_name": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The name of the foundation model.",
               "title": "Model Name"
            },
            "provider": {
               "$ref": "#/$defs/WxAIModelProvider",
               "description": "The provider of the model."
            },
            "model_id": {
               "description": "The unique identifier for the watsonx.ai model.",
               "title": "Model Id",
               "type": "string"
            },
            "project_id": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The project ID associated with the model.",
               "title": "Project Id"
            },
            "space_id": {
               "anyOf": [
                  {
                     "type": "string"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The space ID associated with the model.",
               "title": "Space Id"
            }
         },
         "required": [
            "model_id"
         ],
         "title": "WxAIFoundationModel",
         "type": "object"
      },
      "WxAIModelProvider": {
         "description": "This class represents a model provider configuration for IBM watsonx.ai. It includes the provider type and\ncredentials required to authenticate and interact with the watsonx.ai platform. If credentials are not explicitly\nprovided, it attempts to load them from environment variables.\n\nExamples:\n    1. Create provider using credentials object:\n        .. code-block:: python\n\n            credentials = WxAICredentials(\n                url=\"https://us-south.ml.cloud.ibm.com\",\n                api_key=\"your-api-key\"\n            )\n            provider = WxAIModelProvider(credentials=credentials)\n\n    2. Create provider using environment variables:\n        .. code-block:: python\n\n            import os\n\n            os.environ['WATSONX_URL'] = \"https://us-south.ml.cloud.ibm.com\"\n            os.environ['WATSONX_APIKEY'] = \"your-api-key\"\n\n            provider = WxAIModelProvider()",
         "properties": {
            "type": {
               "$ref": "#/$defs/ModelProviderType",
               "default": "ibm_watsonx.ai",
               "description": "The type of model provider."
            },
            "credentials": {
               "anyOf": [
                  {
                     "$ref": "#/$defs/WxAICredentials"
                  },
                  {
                     "type": "null"
                  }
               ],
               "default": null,
               "description": "The credentials used to authenticate with watsonx.ai. If not provided, they will be loaded from environment variables."
            }
         },
         "title": "WxAIModelProvider",
         "type": "object"
      }
   }
}

Fields:
Validators:
field compute_per_context: Annotated[bool, FieldInfo(annotation=NoneType, required=False, default=False, title='Compute per context', description='The flag to compute the relevance score of each context. The default value is False. Setting the flag to True increases the latency and cost of metric computation.')] = False

The flag to compute the relevance score of each context. The default value is False. Setting the flag to True increases the latency and cost of metric computation.

Validated by:
field group: ', frozen=True)] = MetricGroup.RETRIEVAL_QUALITY

The metric group.

Validated by:
field llm_judge: Annotated[LLMJudge | None, FieldInfo(annotation=NoneType, required=False, default=None, title='LLM Judge', description='The LLM judge used to compute the metric.')] = None

The LLM judge used to compute the metric.

Validated by:
field method: Annotated[Literal['token_precision', 'sentence_bert_bge', 'sentence_bert_mini_lm', 'llm_as_judge', 'granite_guardian'], FieldInfo(annotation=NoneType, required=False, default='token_precision', title='Method', description='The method used to compute the metric. This field is optional and when `llm_judge` is provided, the method would be set to `llm_as_judge`.')] = 'token_precision'

The method used to compute the metric. This field is optional and when llm_judge is provided, the method would be set to llm_as_judge.

Validated by:
field name: Annotated[Literal['context_relevance'], FieldInfo(annotation=NoneType, required=False, default='context_relevance', title='Name', description='The context relevance metric name.', frozen=True)] = 'context_relevance'

The context relevance metric name.

Validated by:
field tasks: ')] = [TaskType.RAG]

The list of supported tasks.

Validated by:
field thresholds: Annotated[list[MetricThreshold], FieldInfo(annotation=NoneType, required=False, default=[MetricThreshold(type='lower_limit', value=0.7)], title='Thresholds', description='The metric thresholds.')] = [MetricThreshold(type='lower_limit', value=0.7)]

The metric thresholds.

Validated by:
evaluate(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs) → AggregateMetricResult
get_combined_context_score(contexts_result)
get_combined_context_scores(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, **kwargs)

Method to compute context relevance on the complete context. Returns the metric result along with a list of the scores.

get_context_scores(contexts_result, combined_context_scores)
get_per_context_scores(data: DataFrame | dict, configuration: GenAIConfiguration | AgenticAIConfiguration, context_fields: list, combined_context_scores: list | None, **kwargs)
model_post_init(context: Any, /) → None

We need to both initialize private attributes and call the user-defined model_post_init method.

validator set_llm_judge_default_method  »  all fields
pydantic model ibm_watsonx_gov.metrics.context_relevance.context_relevance_metric.ContextRelevanceResult

Bases: RecordMetricResult

Show JSON schema
{
   "title": "ContextRelevanceResult",
   "type": "object",
   "properties": {
      "name": {
         "default": "context_relevance",
         "title": "Name",
         "type": "string"
      },
      "method": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The method used to compute this metric result.",
         "examples": [
            "token_recall"
         ],
         "title": "Method"
      },
      "provider": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The provider used to compute this metric result.",
         "title": "Provider"
      },
      "value": {
         "anyOf": [
            {
               "type": "number"
            },
            {
               "type": "string"
            },
            {
               "type": "boolean"
            },
            {
               "type": "null"
            }
         ],
         "description": "The metric value.",
         "title": "Value"
      },
      "errors": {
         "anyOf": [
            {
               "items": {
                  "$ref": "#/$defs/Error"
               },
               "type": "array"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The list of error messages",
         "title": "Errors"
      },
      "additional_info": {
         "additionalProperties": {
            "items": {
               "type": "number"
            },
            "type": "array"
         },
         "default": {
            "contexts_values": []
         },
         "propertyNames": {
            "const": "contexts_values"
         },
         "title": "Additional Info",
         "type": "object"
      },
      "group": {
         "$ref": "#/$defs/MetricGroup",
         "default": "retrieval_quality"
      },
      "thresholds": {
         "default": [],
         "description": "The metric thresholds",
         "items": {
            "$ref": "#/$defs/MetricThreshold"
         },
         "title": "Thresholds",
         "type": "array"
      },
      "record_id": {
         "description": "The record identifier.",
         "examples": [
            "record1"
         ],
         "title": "Record Id",
         "type": "string"
      },
      "record_timestamp": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "description": "The record timestamp.",
         "examples": [
            "2025-01-01T00:00:00.000000Z"
         ],
         "title": "Record Timestamp"
      }
   },
   "$defs": {
      "Error": {
         "properties": {
            "code": {
               "description": "The error code",
               "title": "Code",
               "type": "string"
            },
            "message_en": {
               "description": "The error message in English.",
               "title": "Message En",
               "type": "string"
            },
            "parameters": {
               "default": [],
               "description": "The list of parameters to construct the message in a different locale.",
               "items": {},
               "title": "Parameters",
               "type": "array"
            }
         },
         "required": [
            "code",
            "message_en"
         ],
         "title": "Error",
         "type": "object"
      },
      "MetricGroup": {
         "enum": [
            "retrieval_quality",
            "answer_quality",
            "content_safety",
            "performance",
            "usage",
            "tool_call_quality",
            "readability"
         ],
         "title": "MetricGroup",
         "type": "string"
      },
      "MetricThreshold": {
         "description": "The class that defines the threshold for a metric.",
         "properties": {
            "type": {
               "description": "Threshold type. One of 'lower_limit', 'upper_limit'",
               "enum": [
                  "lower_limit",
                  "upper_limit"
               ],
               "title": "Type",
               "type": "string"
            },
            "value": {
               "default": 0,
               "description": "The value of metric threshold",
               "title": "Threshold value",
               "type": "number"
            }
         },
         "required": [
            "type"
         ],
         "title": "MetricThreshold",
         "type": "object"
      }
   },
   "required": [
      "value",
      "record_id"
   ]
}

Config:

  • arbitrary_types_allowed: bool = True

  • use_enum_values: bool = True

Fields:
field additional_info: dict[Literal['contexts_values'], list[float]] = {'contexts_values': []}
field group: MetricGroup = MetricGroup.RETRIEVAL_QUALITY
field name: str = 'context_relevance'