ai4rag¶

RAG Templates Optimization Engine¶

ai4rag is an optimization engine for RAG (Retrieval-Augmented Generation) patterns that is LLM and Vector Database provider-agnostic. It accepts benchmark data, search space definition, optimizer configuration then returns a leaderboard with benchmarked RAG Template instances (called RAG Patterns).

Key Features¶

• Provider-agnostic: Works with any LLM and vector database: for more information please see Provider-agnostic section in User Guide
• Hyperparameter Optimization: Uses advanced HPO algorithms (GAM-based optimizer) to find optimal RAG configurations
• Comprehensive Evaluation: Built-in metrics for faithfulness, answer correctness, and context correctness
• Flexible Search Space: Define and constrain any RAG parameter (models, chunk sizes, retrieval methods, etc.)
• Event-Driven Architecture: Track experiment progress with custom event handlers
• Production Ready: Designed for real-world RAG optimization workflows

Quick Example¶

from ai4rag.core.experiment.experiment import AI4RAGExperiment
from ai4rag.core.hpo.gam_opt import GAMOptSettings
from ai4rag.rag.vector_store import MilvusLiteConfig
from ai4rag.search_space.src.search_space import AI4RAGSearchSpace
from ai4rag.utils.event_handler import LocalEventHandler
from pathlib import Path

# Define search space
search_space = AI4RAGSearchSpace(params=[...])

# Configure optimizer
optimizer_settings = GAMOptSettings(max_evals=10, n_random_nodes=4)

# Run experiment
experiment = AI4RAGExperiment(
    documents=documents,
    benchmark_data=benchmark_data,
    search_space=search_space,
    vector_store_config=MilvusLiteConfig(db_path="./ai4rag.db"),  # local Milvus Lite, zero setup
    optimizer_settings=optimizer_settings,
    event_handler=LocalEventHandler(
        output_path=Path(__file__).parent / "ai4rag_results"
    )
)

best_pattern = experiment.search()

How It Works¶

graph TB
    A[Documents]
    B[Benchmark Data]
    C[Search Space Definition]

    D[Experiment Engine]
    E[HPO Optimizer]

    subgraph X[RAG Pattern]
        G[Chunking]
        H[Embedding]
        I[Vector Store]
        J[Retrieval]
        K[Generation]
    end

    M[Evaluation & Metrics Computation]

    N[Best RAG Pattern]
    O[Results Artifacts]
    P[Events Callbacks]

    A --> D
    B --> D
    C --> D
    E <--> D
    D --> X
    G --> H
    H --> I
    I --> J
    J --> K
    X --> M
    M --> E
    D --> N
    D --> O
    D --> P

1. Documents and benchmark_data.json are prepared following desired schema
2. Search Space defines possible parameter combinations (models, chunk sizes, retrieval methods, etc.)
3. Optimizer (optimization engine) explores configurations using an objective function with given RAG Template
4. Evaluation of each configuration using selected metrics based on the Evaluator (default unitxt)
5. Results are returned containing the optimal RAG Pattern with best performance

What's Included¶

Core Components¶

• Experiment Engine: Orchestrates the full optimization lifecycle
• Hyperparameter Optimizer: GAM-based optimization algorithm
• Search Space: Flexible parameter definition with validation rules
• Evaluator: metrics calculation (faithfulness, answer_correctness, context_correctness)

RAG Components¶

• Foundation Model: any OpenAI-compatible chat endpoint via OpenAIFoundationModel — or bring your own via the BaseFoundationModel interface
• Embedding Model: any OpenAI-compatible embedding endpoint via OpenAIEmbeddingModel — or bring your own via the BaseEmbeddingModel interface
• Vector Store: remote Milvus, embedded Milvus Lite, or PostgreSQL/pgvector via direct clients — or bring your own via the BaseVectorStore interface
• Chunking: document splitting into smaller chunks
• Retrieval: simple and window-based retrieval strategies
• Templates: complete RAG implementations defined as a RAGTemplate

Requirements¶

To run an optimization you need a foundation model (for text generation) and an embedding model (for document embeddings). ai4rag reaches both through the stock openai SDK, so any OpenAI-compatible endpoint works — a hosted service, a self-managed server (vLLM, TGI, Ollama, …), or an OpenShift AI Models-as-a-Service (MaaS) deployment, the integration ai4rag ships helpers for out of the box.

The vector store is independent of the model provider: connect directly to a remote Milvus server via MilvusConfig, embedded Milvus Lite via MilvusLiteConfig, or PostgreSQL/pgvector via PGVectorConfig.

Getting Started¶

Community and Support¶

• GitHub Repository: IBM/ai4rag
• Issue Tracker: Report bugs or request features
• Contributing: See our contribution guidelines

License¶

ai4rag is released under the Apache License 2.0. See LICENSE for details.

Copyright © 2025-2026 IBM Corp.