Evaluation¶

Evaluation is the foundation of RAG optimization in ai4rag. Every parameter configuration is judged by how well it performs on your benchmark dataset, using standardized metrics that measure different aspects of RAG quality.

Why Evaluation Matters¶

RAG systems can fail in subtle ways:

• Generate answers that sound plausible but contradict the retrieved documents (hallucination)
• Retrieve irrelevant documents that don't help answer the question
• Produce incorrect answers even when the right information is available

ai4rag uses multiple evaluator types to detect these failures and guide optimization toward configurations that produce accurate, grounded, and relevant responses.

Available Metrics¶

ai4rag evaluates four complementary aspects of RAG performance using two evaluator types — Unitxt (reference-based) and LLM-as-a-Judge:

Faithfulness¶

What it measures: Whether the generated answer is grounded in the retrieved context.

Why it matters: This metric detects hallucination. A high faithfulness score means the model is not inventing information beyond what was retrieved from your knowledge base.

Score range: 0.0 to 1.0 (higher is better)

Example failure (low faithfulness):

Question: "What is the capital of France?"
Retrieved context: "France is a country in Western Europe."
Answer: "The capital of France is Paris."
Faithfulness: Low (Paris is not mentioned in the retrieved context)

Example success (high faithfulness):

Question: "Where is France located?"
Retrieved context: "France is a country in Western Europe."
Answer: "France is located in Western Europe."
Faithfulness: High (answer is fully grounded in the context)

Answer Correctness¶

What it measures: How correct the generated answer is compared to the ground truth answers in your benchmark data.

Why it matters: This is the ultimate test of whether your RAG system produces accurate responses. Even if the answer is grounded in context, it might still be incomplete or wrong.

Score range: 0.0 to 1.0 (higher is better)

Example:

Question: "What vector databases does ai4rag support?"
Ground truth: ["ChromaDB and Milvus via OGX", "Milvus and ChromaDB"]
Answer: "ai4rag supports ChromaDB and Milvus."
Answer Correctness: High (matches ground truth)

Context Correctness¶

What it measures: How relevant the retrieved documents are to answering the question.

Why it matters: Good retrieval is essential for good answers. This metric evaluates whether your chunking, embedding, and retrieval strategy is finding the right information.

Score range: 0.0 to 1.0 (higher is better)

How it works: Compares the document IDs of retrieved chunks against the correct_answer_document_ids in your benchmark data.

Example:

Question: "How do I configure hybrid search?"
Correct document IDs: ["hybrid_search_guide.md", "vector_stores.md"]
Retrieved document IDs: ["hybrid_search_guide.md", "installation.md"]
Context Correctness: Medium (1 of 2 correct documents retrieved)

Answer Relevance (LLM Judge)¶

What it measures: Whether the generated response directly and helpfully addresses the user's question, as judged by an LLM.

Why it matters: This metric provides an independent quality signal that does not require ground-truth answers. It detects off-topic, unhelpful, or incoherent responses that reference-based metrics might miss.

Score range: 0.0 to 1.0 (higher is better)

How it works: An LLM judge scores the response on a 1–5 rubric using structured JSON output. The raw score is normalized to [0.0, 1.0]. Confidence intervals are computed via bootstrapping.

Overall Score¶

What it measures: The mean of all other evaluated metrics.

Why it matters: A single aggregate score for optimization that balances all quality dimensions. This is the default optimization metric in ai4rag.

Score range: 0.0 to 1.0 (higher is better)

How Evaluation Works¶

Multi-Evaluator Architecture¶

ai4rag supports multiple evaluator types working together. Each evaluator handles the metrics matching its type:

• UnitxtEvaluator — wraps the unitxt library for reference-based RAG metrics (faithfulness, answer_correctness, context_correctness)
• LLMaJEvaluator — uses an LLM as a judge for answer_relevance
• Custom metrics — computed from the results of other evaluators (e.g. overall_score is the mean of all other metrics)

For each RAG configuration being tested:

1. Generate answers for all benchmark questions using the current configuration
2. Collect evaluation data (question, answer, contexts, ground truths)
3. Route metrics to evaluators — each evaluator receives only its own metrics
4. Merge results into a single EvaluationMetricsResult
5. Compute custom metrics (e.g. overall_score) on top of the merged results
6. Return scores with confidence intervals

EvaluationData Structure¶

Each question's data is packaged as an EvaluationData instance:

from ai4rag.evaluator.base_evaluator import EvaluationData

evaluation_data = EvaluationData(
    question="What is ai4rag?",
    answer="ai4rag is a RAG optimization engine...",
    contexts=[
        "ai4rag optimizes RAG templates using hyperparameter optimization.",
        "The engine is provider-agnostic and works with any LLM."
    ],
    context_ids=["readme.md", "overview.md"],
    ground_truths=[
        "ai4rag is a RAG optimization engine",
        "ai4rag optimizes RAG configurations"
    ],
    ground_truths_context_ids=["readme.md", "architecture.md"],
    question_id="q0"
)

Result Structure¶

Evaluation results are returned as an EvaluationMetricsResult TypedDict with two sections: aggregate metrics and per-question breakdowns.

Aggregate Metrics¶

For each metric, you get:

• name: Metric identifier
• evaluator: Which evaluator produced it ("unitxt", "judge", or "custom")
• scores.mean: Average score across all questions
• scores.ci_low: Lower bound of 95% confidence interval
• scores.ci_high: Upper bound of 95% confidence interval

Example:

{
    "metrics": [
        {
            "name": "faithfulness",
            "evaluator": "unitxt",
            "description": "Measures whether the generated answer is grounded in the retrieved context.",
            "scores": {"mean": 0.72, "ci_low": 0.61, "ci_high": 0.83},
        },
        {
            "name": "answer_correctness",
            "evaluator": "unitxt",
            "description": "Measures how accurately the generated answer matches the ground-truth.",
            "scores": {"mean": 0.68, "ci_low": 0.55, "ci_high": 0.81},
        },
        {
            "name": "answer_relevance",
            "evaluator": "judge",
            "description": "LLM judge score for how directly the response addresses the question.",
            "scores": {"mean": 0.85, "ci_low": 0.78, "ci_high": 0.92},
            "model_id": "ollama/llama3.2:3b",
        },
        {
            "name": "overall_score",
            "evaluator": "custom",
            "description": "Aggregate score computed as the mean of all other evaluated metrics.",
            "scores": {"mean": 0.75, "ci_low": 0.65, "ci_high": 0.85},
        },
    ],
    "question_scores": [
        # Per-question breakdown (see below)
    ],
}

Per-Question Scores¶

Detailed breakdown showing how each question performed:

{
    "question_scores": [
        {
            "question_id": "q0",
            "metrics": [
                {"name": "faithfulness", "evaluator": "unitxt", "value": 0.71},
                {"name": "answer_correctness", "evaluator": "unitxt", "value": 0.65},
                {"name": "answer_relevance", "evaluator": "judge", "value": 0.90},
                {"name": "overall_score", "evaluator": "custom", "value": 0.75},
            ],
        },
        {
            "question_id": "q1",
            "metrics": [
                {"name": "faithfulness", "evaluator": "unitxt", "value": 0.73},
                {"name": "answer_correctness", "evaluator": "unitxt", "value": 0.70},
                {"name": "answer_relevance", "evaluator": "judge", "value": 0.80},
                {"name": "overall_score", "evaluator": "custom", "value": 0.74},
            ],
        },
    ]
}

This granular data helps you identify:

• Which questions are consistently difficult across all configurations
• Which configurations excel at specific question types
• Outliers that might indicate benchmark data quality issues

Choosing the Optimization Metric¶

ai4rag optimizes for a single objective metric. By default, this is overall_score (the mean of all other metrics), but you can change it when creating your experiment.

The optimization_metric parameter accepts either a RAGMetric instance from the Metrics registry or a metric name string:

Default: Overall Score¶

from ai4rag.core.experiment.experiment import AI4RAGExperiment

experiment = AI4RAGExperiment(
    # ... other parameters
    # optimization_metric defaults to Metrics.OVERALL_SCORE
)

Why overall_score is the default: It balances all quality dimensions — grounding, accuracy, retrieval precision, and response relevance — into a single aggregate score, preventing optimization from over-fitting to one aspect at the expense of others.

Optimizing for a Specific Metric¶

You can target any metric from the Metrics registry:

from ai4rag.core.experiment.experiment import AI4RAGExperiment
from ai4rag.evaluator.metric import Metrics

# Using a RAGMetric instance
experiment = AI4RAGExperiment(
    # ... other parameters
    optimization_metric=Metrics.FAITHFULNESS,
)

# Or using a metric name string
experiment = AI4RAGExperiment(
    # ... other parameters
    optimization_metric="answer_correctness",
)

Trade-offs¶

Benchmark Data Quality¶

The quality of your evaluation depends entirely on the quality of your benchmark data.

Benchmark Data Format¶

Your benchmark_data.json must follow this schema:

[
  {
    "question": "What is ai4rag?",
    "correct_answers": [
      "ai4rag is a RAG optimization engine",
      "ai4rag optimizes RAG templates using hyperparameter optimization"
    ],
    "correct_answer_document_ids": ["readme.md", "overview.md"]
  },
  {
    "question": "Which vector databases are supported?",
    "correct_answers": [
      "ChromaDB and Milvus via OGX"
    ],
    "correct_answer_document_ids": ["vector_stores.md", "quick_start.md"]
  }
]

Best Practices for Benchmark Data¶

1. Diverse Question Types

Include different question patterns:

[
  {
    "question": "What is X?",  // Factual
    "correct_answers": ["X is a RAG optimization engine"]
  },
  {
    "question": "How do I configure Y?",  // Procedural
    "correct_answers": ["To configure Y, set the parameter..."]
  },
  {
    "question": "When should I use Z?",  // Conceptual
    "correct_answers": ["Use Z when you need..."]
  }
]

2. Multiple Correct Answers

Provide alternative phrasings for the same correct answer:

{
  "question": "What vector databases does ai4rag support?",
  "correct_answers": [
    "ChromaDB and Milvus",
    "Milvus and ChromaDB",
    "ChromaDB (in-memory) and Milvus via OGX"
  ]
}

This makes evaluation more robust to phrasing variations.

3. Accurate Document IDs

Ensure correct_answer_document_ids match the document_id metadata in your knowledge base:

# When loading documents
from langchain_core.documents import Document

documents = [
    Document(
        page_content="...",
        metadata={"document_id": "readme.md"}  # Must match benchmark data
    )
]

4. Representative Coverage

Your benchmark should cover:

• Common questions users will ask
• Edge cases (ambiguous questions, multi-step reasoning)
• Questions that require different amounts of context
• Questions answerable from single vs. multiple documents

5. Ground Truth Verification

Manually verify that:

• All correct answers are actually correct
• All document IDs actually contain the information needed
• Questions are unambiguous and answerable from your knowledge base

Configuring Evaluators¶

By default, AI4RAGExperiment uses only the UnitxtEvaluator. To enable LLM-as-a-Judge evaluation alongside Unitxt, pass both evaluators:

from ai4rag.evaluator.unitxt_evaluator import UnitxtEvaluator
from ai4rag.evaluator.llmaj_evaluator import LLMaJEvaluator
from ai4rag.rag.foundation_models.ogx import OGXFoundationModel

judge_model = OGXFoundationModel(model_id="ollama/llama3.2:3b", client=client)

experiment = AI4RAGExperiment(
    # ... other parameters
    evaluators=[UnitxtEvaluator(), LLMaJEvaluator(model=judge_model)],
)

When both evaluators are configured, the experiment automatically evaluates answer_relevance via the LLM judge alongside the Unitxt reference-based metrics. The default metrics list adjusts to include answer_relevance when a judge evaluator is present.

You can also explicitly control which metrics to evaluate:

from ai4rag.evaluator.metric import Metrics

experiment = AI4RAGExperiment(
    # ... other parameters
    evaluators=[UnitxtEvaluator(), LLMaJEvaluator(model=judge_model)],
    metrics=[Metrics.FAITHFULNESS, Metrics.JUDGE_ANSWER_RELEVANCE, Metrics.OVERALL_SCORE],
)

Code Example¶

Here's a complete example showing how evaluation is used in the experiment loop:

import os
from pathlib import Path
from dotenv import load_dotenv
from ogx_client import OgxClient

from ai4rag.core.experiment.experiment import AI4RAGExperiment
from ai4rag.search_space.src.parameter import Parameter
from ai4rag.search_space.src.search_space import AI4RAGSearchSpace
from ai4rag.rag.foundation_models.ogx import OGXFoundationModel
from ai4rag.rag.embedding.ogx import OGXEmbeddingModel
from ai4rag.core.hpo.gam_opt import GAMOptSettings
from ai4rag.evaluator.metric import Metrics
from ai4rag.evaluator.unitxt_evaluator import UnitxtEvaluator
from ai4rag.evaluator.llmaj_evaluator import LLMaJEvaluator
from ai4rag.utils.event_handler import LocalEventHandler

from dev_utils.file_store import FileStore
from dev_utils.utils import read_benchmark_from_json

# Setup
load_dotenv()
client = OgxClient(base_url=os.getenv("BASE_URL"), api_key=os.getenv("APIKEY"))

# Load data
documents = FileStore(Path("./knowledge_base")).load_as_documents()
benchmark_data = read_benchmark_from_json(Path("./benchmark_data.json"))

# Define search space
search_space = AI4RAGSearchSpace(
    params=[
        Parameter(
            name="foundation_model",
            param_type="C",
            values=[OGXFoundationModel(model_id="ollama/llama3.2:3b", client=client)],
        ),
        Parameter(
            name="embedding_model",
            param_type="C",
            values=[
                OGXEmbeddingModel(
                    model_id="ollama/nomic-embed-text:latest",
                    client=client,
                    params={"embedding_dimension": 768, "context_length": 8192},
                )
            ],
        ),
        Parameter(name="chunk_size", param_type="C", values=[512, 1024]),
        Parameter(name="number_of_chunks", param_type="C", values=[3, 5, 7]),
    ]
)

# Configure evaluators — Unitxt for reference-based metrics, LLMaJ for judge-based
judge_model = OGXFoundationModel(model_id="ollama/llama3.2:3b", client=client)

# Run optimization (optimizes for overall_score by default)
experiment = AI4RAGExperiment(
    client=client,
    documents=documents,
    benchmark_data=benchmark_data,
    search_space=search_space,
    vector_store_type="ogx",
    ogx_vector_io_provider_id="milvus",
    optimizer_settings=GAMOptSettings(max_evals=8, n_random_nodes=3),
    evaluators=[UnitxtEvaluator(), LLMaJEvaluator(model=judge_model)],
    optimization_metric=Metrics.OVERALL_SCORE,
    event_handler=LocalEventHandler(output_path="./results"),
)

experiment.search()

# Access results
best = experiment.results.get_best_evaluations(k=1)[0]
for m in best.scores["metrics"]:
    print(f"{m['name']}: {m['scores']['mean']:.2f}")

Troubleshooting¶

All Scores Are Low¶

Possible causes:

1. Poor benchmark quality: Questions don't match knowledge base
2. Model mismatch: Foundation model isn't suitable for the task
3. Insufficient context: number_of_chunks is too low
4. Bad retrieval: Chunking or embedding strategy isn't working

Actions:

• Manually test a few benchmark questions against your knowledge base
• Verify that document IDs in benchmark data match your actual documents
• Try increasing number_of_chunks in your search space
• Inspect retrieved contexts in the evaluation results JSON files

Faithfulness Is High but Answer Correctness Is Low¶

Cause: The model is generating grounded answers, but they're not matching the ground truth.

Actions:

• Review your ground truth answers - are they too specific?
• Provide multiple acceptable phrasings in correct_answers
• Check if the retrieved context actually contains the information needed
• Consider optimizing for ANSWER_CORRECTNESS instead

Context Correctness Is High but Other Metrics Are Low¶

Cause: Retrieval is finding the right documents, but generation is failing.

Actions:

• Try a different foundation model
• Increase max_new_tokens if answers are being cut off
• Check prompt templates (system message, user message)
• Verify that include_chunk_metadata isn't confusing the model

Evaluation Fails with UnitxtEvaluator Error¶

Cause: Missing required fields in evaluation data.

Actions:

• Ensure all benchmark questions have non-empty correct_answers
• Verify correct_answer_document_ids are provided
• Check that generated answers aren't empty (model timeout issue)

Related Topics¶

• Optimizers: How optimization uses evaluation scores
• Search Space: Parameters that affect evaluation results
• Event Handlers: Accessing detailed evaluation results
• Quick Start: Creating benchmark data

Summary¶

Evaluation in ai4rag:

• Four metrics: Faithfulness (grounding), Answer Correctness (accuracy), Context Correctness (retrieval quality), Answer Relevance (LLM judge)
• Multi-evaluator architecture: Unitxt for reference-based metrics, LLM-as-a-Judge for response quality
• Overall score: Cross-metric mean used as the default optimization target
• Single objective: Optimizes for one metric, but computes all configured metrics
• Benchmark-driven: Quality depends on your benchmark data
• Confidence intervals: Statistical rigor built-in
• Per-question breakdown: Detailed diagnostics for analysis

High-quality evaluation starts with high-quality benchmark data. Invest time in creating diverse, accurate, and representative questions for the best optimization results.