Skip to content

Architecture Overview

ai4rag is designed as a modular RAG optimization engine with clear separation of concerns. This document provides a high-level overview of the system architecture.


Design Principles

Provider Agnostic

ai4rag is LLM and Vector Database provider agnostic. It integrates with various backends through:

  • OpenAI-compatible model endpoints: Foundation and embedding models are reached through the stock openai SDK, so any OpenAI-compatible endpoint works (OpenShift MaaS is the built-in integration)
  • Direct Vector Store Clients: Milvus (remote server, via MilvusConfig), Milvus Lite (embedded, local file, via MilvusLiteConfig), PGVector, and Neo4j (graph-only, via Neo4jConfig) are integrated directly
  • Pluggable Components: Foundation models, embeddings, and vector stores are all defined by abstract base classes — implement one to plug in your own provider

Template-Based Approach

The core concept is RAG Templates vs RAG Patterns:

  • RAG Template: A RAG implementation with uninitialized parameters
  • RAG Pattern: A RAG implementation with optimal parameter values (output of optimization)

Optimization-Driven

Hyperparameter optimization is the central workflow:

  1. Define a search space of possible configurations
  2. Use HPO algorithms to explore configurations
  3. Evaluate each configuration using metrics
  4. Return the best-performing configuration

High-Level Architecture

graph TB
    A[Documents]
    B[Benchmark Data]
    C[Search Space Definition]

    D[Experiment Engine]
    E[HPO Optimizer]

    subgraph X[RAG Pattern]
        G[Chunking]
        H[Embedding]
        I[Vector Store]
        J[Retrieval]
        K[Generation]
    end

    M[Evaluation & Metrics Computation]

    N[Best RAG Pattern]
    O[Results Artifacts]
    P[Events Callbacks]

    A --> D
    B --> D
    C --> D
    E <--> D
    D --> X
    G --> H
    H --> I
    I --> J
    J --> K
    X --> M
    M --> E
    D --> N
    D --> O
    D --> P

Component Layers

1. Core Layer

Experiment Engine (ai4rag/core/experiment/)

  • Orchestrates the entire optimization workflow
  • Manages experiment lifecycle (setup, execution, teardown)
  • Coordinates between HPO, RAG components, and evaluation
  • Produces results and artifacts

HPO Optimizers (ai4rag/core/hpo/)

  • Implements optimization algorithms
  • Explores search space intelligently
  • Suggests next configurations to evaluate
  • Tracks optimization progress

2. Search Space Layer

Search Space (ai4rag/search_space/)

  • Defines parameter ranges and types
  • Validates configurations against rules
  • Provides parameter sampling
  • Enforces constraints (e.g., chunk_size > chunk_overlap)
  • Selects templates through categorical template_id and validates template/search-mode compatibility with the backend

3. RAG Components Layer

Chunking (ai4rag/rag/chunking/)

  • Splits DoclingDocument objects into AI4RAGChunk instances
  • Two chunker implementations: DoclingChunker (structure-aware, via docling HybridChunker) and LangChainChunker (token-based, via LangChain RecursiveCharacterTextSplitter)
  • Configurable chunk size and overlap

Embedding (ai4rag/rag/embedding/)

  • Generates text embeddings
  • Integrates with any OpenAI-compatible embedding endpoint via OpenAIEmbeddingModel (e.g. OpenShift MaaS)
  • Handles batching and error recovery

Vector Stores (ai4rag/rag/vector_store/)

  • Stores and retrieves document embeddings
  • Supports Milvus (remote server, MilvusConfig), Milvus Lite (embedded, local file, MilvusLiteConfig), PostgreSQL/pgvector, and Neo4j (graph-only, Neo4jConfig) via direct clients
  • Provides similarity search capabilities (or graph-expanded retrieval for Neo4j)

Retrieval (ai4rag/rag/retrieval/)

  • Retrieves relevant chunks for queries
  • Supports simple and window-based retrieval
  • Configurable top-k parameter
  • LLMDocumentCompressor extracts query-relevant, verbatim excerpts with source metadata for AgenticRAG's optional context-overflow retry

Foundation Models (ai4rag/rag/foundation_models/)

  • Generates answers using LLMs
  • Integrates with any OpenAI-compatible chat endpoint via OpenAIFoundationModel (e.g. OpenShift MaaS)
  • Formats prompts with retrieved context

Templates (ai4rag/rag/template/)

  • Complete RAG implementations
  • Combines all RAG components
  • Provides end-to-end RAG pipeline
  • Two implementations: SequentialRAG (single fixed retrieve-then-generate pass) and AgenticRAG (a LangChain agent that can rewrite its query and retrieve again before answering — the default used by AI4RAGExperiment and ModelsPreSelector)
  • ai4rag.rag.template.registry maps persisted template IDs to allowed classes and import paths
  • AgenticRAG.generate() returns conversation records with initial retrieval, agent messages, and tool activity alongside its answer and reference documents
  • Context compression is enabled by default for AgenticRAG; a recognized context overflow triggers one retry, with retained context and compression statistics recorded in the result
  • Budget middleware removes tools for a final answer once the retrieval or agent model-call budget is exhausted
  • Tool-selection middleware enforces one tool action per response before history storage and execution; see AgenticRAG for provider compatibility behavior
  • ai4rag.rag.template.agent_trace.AgentTrace keeps invocation-scoped counts and bounded events, emitting a content-free failure record for unrecovered errors after initial retrieval

4. Evaluation Layer

Evaluator (ai4rag/evaluator/)

  • Multi-evaluator architecture with dispatch by metric type
  • UnitxtEvaluator — wraps the unitxt library for reference-based metrics (faithfulness, answer_correctness, context_correctness)
  • LLMaJEvaluator — LLM-as-a-Judge for answer_relevance scoring with structured output
  • RagasEvaluator — optional RAGAS metrics (faithfulness, answer_relevancy, context_precision, context_recall) using the configured foundation and embedding models
  • ragas_adapters delegates RAGAS calls to those models; ragas_diagnostics classifies failures so confirmed context overflows bypass retries of the identical request
  • custom_metrics — derived metrics such as overall_score (mean of all other metrics)
  • RAGMetric dataclass and Metrics registry for typed metric definitions
  • Automatic judge model selection via calibration when multiple generation models are available
  • Returns structured EvaluationMetricsResult with confidence intervals and per-question breakdowns

5. Pipeline Components Layer

Data Components (ai4rag/utils/data/)

  • Reusable functions for pipeline data stages: document discovery, text extraction, and test data loading
  • Extracted from Kubeflow Pipeline components for standalone use
  • Provider-agnostic — accepts injected S3 clients and OpenAI-compatible model clients
  • prepare_extraction() returns a PreparedExtraction configuration based on discovered file types, downloading pinned OCR/Whisper models for connected runs or validating local bundles for disconnected runs
  • LocalHuggingFaceAsrPipeline extends Docling's ASR pipeline for local Transformers-format Whisper models, preserving segment timestamps without runtime model downloads

RAG Optimization Orchestration (pipelines-components repo)

  • Search space preparation with model pre-selection
  • Full RAG optimization orchestration
  • Language detection for multilingual benchmark support
  • Built on top of ai4rag.search_space and ai4rag.core; not part of this repo

Assets Generation (pipelines-components repo)

  • Notebook, leaderboard, and starter-kit generation is owned by kfp_components.components.training.autorag.shared.assets_generator
  • The former ai4rag.assets_generator package, its bundled templates, and the OpenShell gateway setup script were removed in 0.26.0 without an import shim
  • See Assets generator migration for the required caller and runtime updates

6. Utilities Layer

Event Handlers (ai4rag/utils/event_handler/)

  • Provides hooks for experiment events
  • Allows custom tracking and monitoring
  • Useful for production integrations
  • Publishes optional agent trajectories separately from scores; LocalEventHandler writes them to each pattern's conversations.json

Validators (ai4rag/utils/)

  • Validates input data formats
  • Checks configuration consistency
  • Provides error messages

Data Flow

Indexing Phase

Documents
    ↓
Chunking (chunk_size, chunk_overlap)
    ↓
Embedding Model (embedding_model)
    ↓
Vector Store (Milvus / Milvus Lite / PGVector via vector_store_config)

Query Phase (Per Configuration)

Question (from benchmark)
    ↓
Embedding Model
    ↓
Vector Store Search (number_of_chunks, retrieval_method, window_size)
    ↓
Retrieved Chunks
    ↓
Foundation Model (with context + question)
    ↓
Generated Answer

Evaluation Phase

Generated Answer
Ground Truth Answer
Retrieved Documents
Ground Truth Documents
    ↓
Multi-Evaluator Dispatch
  ├─ UnitxtEvaluator → faithfulness, answer_correctness, context_correctness
  ├─ LLMaJEvaluator  → answer_relevance
  └─ RagasEvaluator → faithfulness, answer_relevancy, context_precision, context_recall (optional)
    ↓
Custom Metrics (overall_score = mean of all)
    ↓
Optimization Score
    ↓
HPO Optimizer (feedback for next configuration)

Extension Points

ai4rag is designed for extensibility:

  1. Custom Foundation Models: Implement BaseFoundationModel
  2. Custom Embedding Models: Implement BaseEmbeddingModel
  3. Custom Vector Stores: Implement BaseVectorStore
  4. Custom HPO Algorithms: Extend base optimizer classes
  5. Custom Event Handlers: Implement BaseEventHandler
  6. Custom Metrics: Integrate additional evaluators

Technology Stack

  • Python: 3.12 & 3.13
  • OpenAI SDK: Foundation-model & embedding integration over any OpenAI-compatible endpoint (e.g. OpenShift MaaS)
  • Docling Core: Document representation and structure-aware chunking
  • LangChain: Token-based text splitting
  • Unitxt: Evaluation metrics
  • Pandas: Results management
  • Pydantic: Data validation

Next Steps