Skip to content

Quick Start

This guide walks you through running your first RAG optimization experiment with ai4rag. For the sake of quick-start OpenShift AI Models-as-a-Service (MaaS) will be used, but this can be run with any independently deployed OpenAI-compatible models as long as they are introduced to the experiment with the proper wrapper.


Data loading

To run the experiment you need to provide documents as DoclingDocument instances (from the docling-core library). Where those documents come from is up to you — ai4rag does not require object storage. A local folder is enough; S3 is only involved if you use the pipeline components that read from a bucket.

Whatever the source, one rule decides whether your experiment scores correctly:

A document's key is its name

Each document's DoclingDocument.name is its key — the identifier that follows it through chunking, indexing and evaluation, and the value your benchmark data must reference in correct_answer_document_keys. Pick keys that are unique across the corpus — a path relative to your documents folder locally (manuals/xr-200/setup.pdf), or the full object key when reading from a bucket — rather than a bare file name, so that two files sharing a basename in different folders stay distinct.



Prerequisites

Before starting, ensure you have:

  • Installed ai4rag (Installation Guide)
  • Access to an OpenShift MaaS deployment (or any OpenAI-compatible models) that can be used for the experiment
  • Environment variables set (MAAS_BASE_URL, MAAS_API_KEY) to communicate with MaaS

Step-by-Step Guide with MaaS

1. Prepare the MaaS Client

OpenShift MaaS serves every model from a single OpenAI-compatible endpoint — MAAS_BASE_URL. One client discovers the available models and is reused, unchanged, to serve every model wrapper.

The dev_utils helpers wrap this setup — create_dev_maas_client() reads MAAS_BASE_URL / MAAS_API_KEY and builds that client, and build_maas_model() checks that a model id is available and binds it to the correct wrapper on the same client:

from dotenv import load_dotenv, find_dotenv
from dev_utils.utils import create_dev_maas_client

load_dotenv(find_dotenv())

# Single client, used both to discover models and to serve every wrapper.
client = create_dev_maas_client()  # reads MAAS_BASE_URL / MAAS_API_KEY

Public API

dev_utils is only available when cloning the repository. For the equivalent setup using the public API (the single OpenAI client built with create_maas_client), see Provider-Agnostic Design.


2. Prepare Knowledge Base Documents

Convert a local folder of documents with Docling, naming each one by its path relative to that folder. No object storage is involved:

from pathlib import Path
from docling.document_converter import DocumentConverter

documents_root = Path("path/to/your/documents")
converter = DocumentConverter()

documents = []
for file_path in sorted(p for p in documents_root.rglob("*") if p.is_file()):
    document = converter.convert(file_path).document
    # The key: unique across the corpus, and what benchmark data references.
    document.name = str(file_path.relative_to(documents_root))
    documents.append(document)

print(f"Loaded {len(documents)} documents: {[d.name for d in documents]}")

Always set name yourself

Docling derives a name from the file stem, dropping both the folder and the extension — notes.txt becomes notes. Two files named setup.pdf in different folders would end up sharing a key, and the second would displace the first during indexing. Assigning document.name explicitly, as above, avoids this.

Conversion is the slow part, so save the results and reload them on later runs:

from ai4rag.utils.docling_io import load_docling_documents

extracted_dir = Path("./extracted_text")
for document in documents:
    output_path = extracted_dir / f"{document.name}.json"
    output_path.parent.mkdir(parents=True, exist_ok=True)
    document.save_as_json(output_path)

# On subsequent runs, skip conversion entirely -- names are preserved in the JSON.
documents = load_docling_documents(extracted_dir)

Reading from object storage instead

If your corpus already lives in a bucket, discover_documents() and extract_text() do the same job, naming each document by its full S3 object key — so a benchmark written against a bucket must spell keys out in full, prefix included (datasets/rag/docs/manuals/xr-200/setup.pdf). Use prepare_extraction() from ai4rag.utils.data.text_extraction between discovery and extraction to select OCR/audio models for connected or disconnected runs. See Pipeline Components.

dev_utils.file_store.FileStore

The repository also ships a FileStore helper used by the examples and tests. It is development-only — it is excluded from the installed package and it names documents by bare file name, so it is not suitable for a corpus with nested folders.

PDF parsing uses PDFium, and PDF model inference defaults to CPU to avoid native PyTorch MPS crashes during concurrent conversion on macOS. Pass device="auto" or a specific accelerator to FileStore to enable GPU inference. PDF caches use .pdfium.json; older .json PDF caches are bypassed and regenerated automatically.


3. Prepare Benchmark Data

Create a benchmark_data.json file with questions and ground truth answers. Each correct_answer_document_keys entry must be the name of a document you loaded in step 2:

[
  {
    "question": "How do I set up the XR-200?",
    "correct_answers": [
      "Connect the power supply, then run the setup wizard."
    ],
    "correct_answer_document_keys": ["manuals/xr-200/setup.pdf"]
  },
  {
    "question": "Which vector databases are supported?",
    "correct_answers": [
      "Milvus, PostgreSQL/pgvector, and Neo4j are supported. Neo4j uses graph retrieval."
    ],
    "correct_answer_document_keys": ["overview.md", "reference/stores.md"]
  }
]

Load the benchmark data:

import pandas as pd

benchmark_data = pd.read_json("path/to/benchmark_data.json")

Benchmark Quality

High-quality benchmark data is crucial for meaningful optimization. Ensure questions are based on your knowledge base and answers are accurate.


4. Define Search Space

Specify which parameters to optimize and their possible values:

from ai4rag.search_space.src.parameter import Parameter
from ai4rag.search_space.src.search_space import AI4RAGSearchSpace
from dev_utils.utils import build_maas_model

search_space = AI4RAGSearchSpace(
    params=[
        # Foundation model for generation
        Parameter(
            name="foundation_model",
            param_type="C",
            values=[
                build_maas_model(client, model_id="qwen3-8b-fp8-dynamic", model_type="llm")
            ],
        ),
        # Embedding model
        Parameter(
            name="embedding_model",
            param_type="C",
            values=[
                build_maas_model(
                    client,
                    model_id="bge-m3",
                    model_type="embedding",
                    embedding_params={
                        "embedding_dimension": 1024,
                        "context_length": 8192
                    },
                )
            ],
        ),
        # Chunking parameters
        Parameter(
            name="chunk_size",
            param_type="C",
            values=[200, 400, 800, 1000],
        ),
        Parameter(
            name="chunk_overlap",
            param_type="C",
            values=[0, 50, 100, 200],
        ),
        # Retrieval parameters
        Parameter(
            name="retrieval_method",
            param_type="C",
            values=["simple", "window"],
        ),
        Parameter(
            name="number_of_chunks",
            param_type="C",
            values=[3, 5, 7, 10],
        ),
    ]
)

5. Configure Optimizer

Set up the hyperparameter optimization algorithm:

from ai4rag.core.hpo.gam_opt import GAMOptSettings

optimizer_settings = GAMOptSettings(
    max_evals=10,      # Total number of configurations to evaluate
    n_random_nodes=4   # Number of random explorations before using GAM
)

Optimization Strategy

  • Random phase (n_random_nodes): Explores the search space randomly to avoid falling into local minimum (greater value = better solutions space exploration)
  • GAM phase: Uses a model to suggest promising configurations

6. Run the Experiment

Choosing a Vector Store

The vector store is selected by passing a vector_store_config to AI4RAGExperiment:

  • MilvusLiteConfig(db_path="./ai4rag.db") (or MilvusLiteConfig()) — zero-config, embedded Milvus Lite backed by a local file. Supports hybrid search (dense + BM25); intended for local development, tests, and small-scale workloads, not production.
  • MilvusConfig.from_env() or MilvusConfig(uri="http(s)://host:19530") — a running Milvus server (or Zilliz Cloud). uri must be an http(s):// URL — anything else (a bare host, a local file path, an empty string) raises ValueError rather than silently falling back to a local database. Supports hybrid search (dense + BM25).
  • PGVectorConfig.from_env() or PGVectorConfig(host=...) — a running PostgreSQL instance with pgvector. Supports hybrid search (dense + full-text).
  • Neo4jConfig.from_env() or Neo4jConfig(uri=...) — a running Neo4j instance with APOC. Graph-only retrieval (search_mode="graph"); see Neo4j Indexing and Graph Search.

All four classes live in ai4rag.rag.vector_store and can be built explicitly or from environment variables via .from_env().

RAG template

Each candidate selects its template through the categorical template_id search parameter. Vector backends (Milvus, Milvus Lite, pgvector) accept agentic_rag (default) or sequential_rag; Neo4j requires agentic_graph_rag and graph search. Both agentic IDs use AgenticRAG; sequential uses SequentialRAG. The selected ID is retained in results and streamed patterns.

Create and run the optimization experiment:

from ai4rag.core.experiment.experiment import AI4RAGExperiment
from ai4rag.rag.vector_store import MilvusConfig
from ai4rag.utils.event_handler import LocalEventHandler

experiment = AI4RAGExperiment(
    documents=documents,
    benchmark_data=benchmark_data,
    search_space=search_space,
    vector_store_config=MilvusConfig.from_env(),  # See "Choosing a Vector Store" above
    optimizer_settings=optimizer_settings,
    event_handler=LocalEventHandler(output_path="<path_to_store_results>"),  # Tracks progress
)

# Run optimization
experiment.search()

best_pattern = experiment.results.get_best_evaluations(k=1)[0]

print(f"Best pattern: {best_pattern.pattern_name}, score: {best_pattern.final_score}")

7. Review Results

After completion, check the output_path directory for:

  • <pattern_name>/pattern.json: Published pattern settings, selected template_id, and aggregate scores
  • <pattern_name>/evaluation_results.json: Per-question answers, retrieved contexts, scores, and optional conversation_id links to full conversations
  • <pattern_name>/conversations.json: Agent messages, retrievals, and tool activity when the template emits trajectories

Published pattern names use spreadsheet-style alphabetic suffixes: PatternA, PatternB, …, PatternZ, PatternAA. Update integrations that assume numeric names such as Pattern1 when upgrading to 0.26.0.

Only the best successful warm-start candidate is published as PatternA; other warm-start candidates do not get output directories. See Event Handlers for the artifact schema.

Access conversations directly from an evaluated pattern:

best_pattern = experiment.results.get_best_evaluations(k=1)[0]
for conversation in best_pattern.conversations or []:
    print(conversation["conversation_id"], conversation["question"])

conversations is None when inference emits no conversation records. Each result's to_dict() also includes this field.


Complete Example

Here's the full code in one place:

from pathlib import Path

import pandas as pd
from docling.document_converter import DocumentConverter
from dotenv import load_dotenv

from ai4rag.core.experiment.experiment import AI4RAGExperiment
from ai4rag.search_space.src.parameter import Parameter
from ai4rag.search_space.src.search_space import AI4RAGSearchSpace
from ai4rag.rag.vector_store import MilvusConfig
from ai4rag.core.hpo.gam_opt import GAMOptSettings
from ai4rag.utils.event_handler import LocalEventHandler

from dev_utils.utils import build_maas_model, create_dev_maas_client

# 1. Setup client
load_dotenv()
client = create_dev_maas_client()  # reads MAAS_BASE_URL / MAAS_API_KEY

# 2. Load documents from a local folder, keyed by their relative path
documents_root = Path("./knowledge_base")
converter = DocumentConverter()
documents = []
for file_path in sorted(p for p in documents_root.rglob("*") if p.is_file()):
    document = converter.convert(file_path).document
    document.name = str(file_path.relative_to(documents_root))
    documents.append(document)

# 3. Load benchmark data -- its keys must match the document names above
benchmark_data = pd.read_json("./benchmark_data.json")

# 4. Define search space
search_space = AI4RAGSearchSpace(
    params=[
        Parameter(
            name="foundation_model",
            param_type="C",
            values=[build_maas_model(client, model_id="qwen3-8b-fp8-dynamic", model_type="llm")],
        ),
        Parameter(
            name="embedding_model",
            param_type="C",
            values=[
                build_maas_model(
                    client,
                    model_id="bge-m3",
                    model_type="embedding",
                    embedding_params={"embedding_dimension": 1024, "context_length": 8192},
                )
            ],
        ),
        Parameter(name="chunk_size", param_type="C", values=[200, 400, 800]),
        Parameter(name="chunk_overlap", param_type="C", values=[0, 50, 100]),
        Parameter(name="retrieval_method", param_type="C", values=["simple", "window"]),
        Parameter(name="number_of_chunks", param_type="C", values=[3, 5, 7]),
    ]
)

# 5. Configure optimizer
optimizer_settings = GAMOptSettings(max_evals=10, n_random_nodes=4)

# 6. Run experiment
experiment = AI4RAGExperiment(
    documents=documents,
    benchmark_data=benchmark_data,
    search_space=search_space,
    vector_store_config=MilvusConfig.from_env(),
    optimizer_settings=optimizer_settings,
    event_handler=LocalEventHandler(output_path="./results"),
)

best_pattern = experiment.search()
print(f"Optimization complete! Best pattern: {best_pattern}")

Next Steps


Assets generator migration

Starting with 0.26.0, ai4rag.assets_generator is removed without a compatibility shim. Notebook, leaderboard, and starter-kit generation now lives in pipelines-components. Install its matching wheel and use the compatible runtime image, then replace the old package import with:

from kfp_components.components.training.autorag.shared import assets_generator

Update submodule imports to use this package path as well. Recompile pipelines that embed the old import path, or keep their compatible ai4rag version and runtime image until they can migrate. OpenShell gateway setup is also maintained in pipelines-components; scripts/setup-openshell-gateway.sh is no longer shipped by ai4rag. The core experiment, search-space, and data-extraction APIs remain in ai4rag.

Template migration and pre-selection

The former rag_template experiment argument has been removed. Use Parameter(name="template_id", values=("sequential_rag",)) in the search space, or pass template_ids=["sequential_rag"] in MaaS constraints. Omitting template_ids or passing None selects the backend default; an empty list is invalid. AgenticRAG and its agentic_rag_template import path are retained. Replace SimpleRAG / simple_rag_template with SequentialRAG / sequential_rag_template, and the persisted simple_rag ID with sequential_rag. The template_id_for_class() helper has been removed; resolve an explicit ID with get_template_spec(template_id) from ai4rag.rag.template instead. Use agentic_rag for vector backends and agentic_graph_rag for Neo4j; both use AgenticRAG. Set the search space's vector_store_type to match the experiment's backend. When reusing known observations, include the actual template_id associated with the historical score.

Automatic model pre-selection always uses temporary Milvus Lite and AgenticRAG, including sequential-only, graph, and multiple-template searches. This vector proxy can prune models that would perform better with sequential or graph execution. It ranks by OVERALL_SCORE, independently of the experiment’s optimization metric. Existing model-count thresholds and search(skip_mps=True) still apply. Graph RAG extension enablement belongs to the calling product; the library adds no feature flag.