User Guide Overview¶
Welcome to the ai4rag user guide. This section provides comprehensive documentation for using ai4rag to optimize your RAG configurations.
What You'll Learn¶
This user guide covers:
- Search Space: How to define and constrain parameter search spaces
- Optimizers: Understanding HPO algorithms and their configuration
- Evaluation: Metrics, scoring, and result interpretation
- Event Handlers: Tracking and monitoring experiments
Core Concepts¶
RAG Template vs RAG Pattern¶
- RAG Template: A RAG implementation with parameter placeholders
- RAG Pattern: A RAG implementation with optimal parameter values (the output)
ai4rag transforms templates into patterns through optimization.
Published patterns include template_id, identifying the chosen AgenticRAG or SequentialRAG template through ai4rag.rag.template.registry. With LocalEventHandler, each published pattern has pattern.json and evaluation_results.json; agentic patterns also include conversations.json with retrieval and tool activity. See Event Handlers for the output schema.
Each evaluated pattern also retains its conversations in EvaluationResult.conversations, accessible through experiment.results. The field is None when inference emits no conversations. Evaluation JSON records can link to the full conversations through an optional conversation_id.
Search Space¶
The search space defines all possible parameter combinations:
from ai4rag.search_space.src.search_space import AI4RAGSearchSpace
from ai4rag.search_space.src.parameter import Parameter
search_space = AI4RAGSearchSpace(
params=[
# Configured foundation-model and embedding-model instances:
Parameter(name="foundation_model", values=[model]),
Parameter(name="embedding_model", values=[embedding]),
Parameter(name="template_id", values=["agentic_rag", "sequential_rag"]),
Parameter(name="chunk_size", param_type="C", values=[200, 400, 800]),
Parameter(name="retrieval_method", param_type="C", values=["simple", "window"]),
]
)
The categorical template_id dimension lets vector-backend experiments compare agentic and sequential execution. Neo4j requires vector_store_type="neo4j", template_id="agentic_graph_rag", and graph search; its defaults select these automatically. See Template Selection.
Hyperparameter Optimization¶
The HPO engine explores the search space intelligently:
- Sample configurations from the search space
- Evaluate each configuration using benchmark data
- Use results to guide future sampling
- Return the best-performing configuration
Evaluation Metrics¶
ai4rag uses multiple evaluator types for comprehensive quality assessment:
- Faithfulness (Unitxt): Answer grounded in retrieved context
- Answer Correctness (Unitxt): Answer matches ground truth
- Context Correctness (Unitxt): Retrieved documents are relevant
- Answer Relevance (LLM Judge): Response directly addresses the question
- Overall Score (Custom): Mean of all other metrics (default optimization target)
Typical Workflow¶
graph LR
A[Prepare Knowledge Base] --> B[Create Benchmark Data]
B --> C[Define Search Space]
C --> D[Configure Optimizer]
D --> E[Run Experiment]
E --> F[Analyze Results]
F --> G{Satisfied?}
G -->|No| C
G -->|Yes| H[Deploy RAG Pattern] - Prepare: Gather documents and create benchmark questions
- Define: Specify search space and constraints
- Optimize: Run experiment with chosen HPO algorithm
- Analyze: Review results and metrics
- Iterate: Refine search space if needed
- Deploy: Use the best RAG pattern in production
Next Steps¶
Start with the most relevant topic for your needs:
- New to ai4rag? Begin with Search Space
- Optimizing performance? Check Optimizers
- Understanding results? Read Evaluation
- Production deployment? See Event Handlers