Skip to content

Installation

Requirements

  • Python: 3.12 or 3.13 (strictly required)
  • Operating System: macOS or Linux
  • A model provider: a foundation model and an embedding model reachable over any OpenAI-compatible endpoint (a hosted API, a self-managed vLLM/TGI/Ollama server, or an OpenShift MaaS deployment), accessed through the openai SDK — or your own BaseFoundationModel / BaseEmbeddingModel implementation
  • A vector store: Milvus Lite (embedded, local-file, no setup required), a running Milvus server/PostgreSQL (pgvector) instance for server-backed retrieval (both support hybrid search), or a Neo4j instance (with APOC) for graph-only retrieval

External models and vector store integration

ai4rag is designed to be provider-agnostic. It means you can use any model from any source as long as it satisfies BaseFoundationModel interface. The same rule applies to embedding model. Vector stores are selected via a typed vector_store_config (MilvusConfig for a remote server, MilvusLiteConfig for embedded local storage, PGVectorConfig, or Neo4jConfig for graph-only retrieval) passed directly to the experiment. A custom vector store can also be plugged in by delivering your own BaseVectorStore implementation.


Basic Installation

Install ai4rag using pip:

pip install "git+https://github.com/IBM/ai4rag.git@main"

This installs the core package with all required dependencies. Using "@main" will download and install latest version of ai4rag. If you want to use specific version, please use e.g. "@v0.1.1"

The core package and text-extraction extra require docling-slim~=2.145.0 (>=2.145.0,<2.146.0), including upstream security fixes. Refresh dependency environments, runtime images, and offline package mirrors when upgrading.

Vector store clients — pymilvus (with the milvus-lite extra), pgvector, asyncpg, neo4j, and neo4j-graphrag — are core dependencies and install automatically. There is no separate vector-store extra to install. No extra step is needed to use a remote Milvus server, embedded Milvus Lite, PostgreSQL/pgvector, or Neo4j as a vector store.

langchain and langchain-openai are also core dependencies, powering the AgenticRAG template that AI4RAGExperiment uses by default (see RAG Templates).

nest-asyncio~=1.6.0 is installed as a core dependency for notebook event-loop compatibility. Call nest_asyncio.apply() before graph extraction when using an environment with an active event loop.

certifi is installed as a core dependency without version bounds to supply public CA roots alongside deployment-mounted MaaS certificates. Include it in offline package mirrors when upgrading.

OCR and audio ingestion

Text extraction from born-digital documents (PDF, DOCX, Markdown, HTML, …) works out of the box. Scanned PDFs/images (via RapidOCR) and audio transcription (via a local Hugging Face Whisper model) additionally require the text-extraction extra, which pulls in torch, docling-ibm-models, rapidocr~=3.9.2, onnxruntime~=1.25.0, huggingface-hub>=0.30,<2, and the audio dependencies:

pip install "ai4rag[text-extraction] @ git+https://github.com/IBM/ai4rag.git@main"

RapidOCR and ONNX Runtime are declared directly by this extra. After discovery, call prepare_extraction() to select the required models. By default it downloads pinned PP-OCRv4 files for PDF/image inputs when no local artifacts path is configured. With disconnected=True, it requires the local Docling/RapidOCR bundle through DOCLING_ARTIFACTS_PATH, including the recognition dictionary. Text-only corpora need neither OCR nor Whisper models.

Audio extraction requires the system ffmpeg executable on PATH, with decoders for the input audio formats. Connected prepare_extraction() calls download a pinned Transformers-format Whisper model when neither HF_MODEL_DIR nor an explicit model path is set. Disconnected clusters must provide a complete local model through HF_MODEL_DIR (or DoclingExtractionConfig.asr_model_path). extract_text() itself does not download ASR weights, and openai-whisper checkpoints are not compatible. See Pipeline Components for usage.


Development Installation

This project uses uv for dependency and environment management. Install it first if you haven't already:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then clone and set up the project:

# Clone the repository
git clone https://github.com/IBM/ai4rag.git
cd ai4rag

# Install all development dependencies (creates .venv automatically)
uv sync --extra dev

The dev optional dependencies include:

  • Testing tools (pytest, pytest-cov, pytest-mock)
  • Code quality tools (black, pylint, isort)
  • Documentation tools (mkdocs, mkdocs-material)
  • Development utilities (beautifulsoup4, dotenv, ipykernel)

To install only a specific subset of dependencies, use the corresponding extra name (see pyproject.toml):

uv sync --extra test        # testing tools only
uv sync --extra code_check  # linting/formatting tools only
uv sync --extra docs        # documentation tools only

Optional Extras

Extra Dependencies Purpose
text-extraction docling-slim~=2.145.0 conversion/audio extras, huggingface-hub>=0.30,<2, rapidocr~=3.9.2, onnxruntime~=1.25.0 OCR and audio ingestion
test pytest, pytest-cov, pytest-mock Testing tools
code_check black, pylint, isort Code quality tools
docs mkdocs, mkdocs-material Documentation tools
dev All of the above Full development environment

Model Provider Setup (OpenShift MaaS)

ai4rag reaches foundation and embedding models through the stock openai SDK (installed automatically as a core dependency), so it works with any OpenAI-compatible endpoint — a hosted API, a self-managed server (vLLM, TGI, Ollama, …), or an OpenShift AI Models-as-a-Service (MaaS) deployment. The steps below use MaaS, the provider ai4rag ships helpers for; to use a different endpoint, point the same openai client at its URL (or supply your own BaseFoundationModel / BaseEmbeddingModel implementation). The vector store is configured independently, via direct clients (remote Milvus, embedded Milvus Lite, PGVector, or Neo4j) — see Vector Store Setup below.

1. Get Access to a MaaS Deployment

Obtain access to an OpenShift AI MaaS deployment that exposes:

  • At least one foundation model (e.g., qwen3-8b-fp8-dynamic)
  • At least one embedding model (e.g., bge-m3)

MaaS serves all models from a single OpenAI-compatible endpoint — MAAS_BASE_URL. No extra package is required — the openai SDK ships with ai4rag as a core dependency.

2. Note Your Credentials

Record the MaaS base URL and API key for use in ai4rag:

  • MAAS_BASE_URL — the OpenAI-compatible endpoint URL (e.g. https://<host>); a trailing /v1 is appended automatically if not already present
  • MAAS_API_KEY — a single API key for the single client that serves listing, chat, and embeddings

Vector Store Setup

ai4rag connects to the vector store directly — no MaaS deployment is required for this part. Pick a provider and pass its config to AI4RAGExperiment as vector_store_config:

Provider Config Hybrid search (dense + keyword) Setup
Milvus Lite (embedded, local file) MilvusLiteConfig(db_path="./ai4rag.db") dense + BM25 None — zero-server, backed by a local file
Milvus (server) MilvusConfig(uri="http(s)://host:19530") dense + BM25 Requires a reachable Milvus (or Zilliz Cloud) instance
PGVector PGVectorConfig dense + tsvector full-text Requires a reachable PostgreSQL instance with the pgvector extension
Neo4j (graph-only) Neo4jConfig graph search only Requires a reachable Neo4j instance with APOC enabled
from ai4rag.rag.vector_store import MilvusConfig, MilvusLiteConfig, Neo4jConfig, PGVectorConfig

# Zero-config, embedded Milvus Lite backed by a local file (great for local experimentation)
vector_store_config = MilvusLiteConfig(db_path="./ai4rag.db")

# Or build configs from environment variables
vector_store_config = MilvusConfig.from_env()       # reads MILVUS_URI (must be http(s)://), MILVUS_TOKEN, MILVUS_CA_CERT
vector_store_config = MilvusLiteConfig.from_env()   # reads MILVUS_LITE_DB_PATH (optional; defaults to "./ai4rag_milvus_lite.db")
vector_store_config = PGVectorConfig.from_env()     # reads PGVECTOR_HOST, PGVECTOR_PORT, PGVECTOR_DB, PGVECTOR_USER, PGVECTOR_PASSWORD
vector_store_config = Neo4jConfig.from_env()        # also reads optional NEO4J_CA_CERT (inline PEM text)

Each config class exposes the environment variables it reads via its env_vars attribute, and can be constructed explicitly instead of from the environment, e.g. MilvusConfig(uri="https://localhost:19530") for a remote server or MilvusLiteConfig(db_path="./ai4rag.db") for embedded local storage.

Verified TLS for Neo4j

Use neo4j+s:// or bolt+s:// with a publicly trusted certificate. For a private or self-signed CA, supply its PEM certificate text through Neo4jConfig.ca_cert or NEO4J_CA_CERT; a file path is not accepted as certificate content. ai4rag enables encrypted, verified transport with either a base or +s URI when a custom CA is set. neo4j+ssc:// and bolt+ssc:// are rejected.

Default CA discovery for Milvus

HTTPS Milvus connections without ca_cert use the CAs loaded by Python's default SSL context. Include a mounted bundle in that trust configuration, for example via SSL_CERT_FILE, or supply inline PEM through MILVUS_CA_CERT. An empty loaded trust store raises a configuration error. See Vector Store Backends.

Why MilvusConfig and MilvusLiteConfig are separate

MilvusConfig (remote Milvus server / Zilliz Cloud) validates that uri is an http(s):// URL and raises ValueError for anything else — a bare host, a local file path, or an empty string. This is a deliberate safety check: a mistyped or unreachable MILVUS_URI now fails loudly at construction time instead of silently being interpreted as a local file path and creating a throwaway Milvus Lite database. To opt into the embedded local engine explicitly, use MilvusLiteConfig(db_path=...) instead, which in turn rejects http(s):// values.

Milvus Lite is not a production store

MilvusLiteConfig is intended for local development, tests, and small-scale workloads (prototyping, up to roughly 1M vectors) — not production serving. For production or large corpora, use a remote Milvus server or Zilliz Cloud via MilvusConfig, or pgvector. Milvus Lite also computes BM25 statistics segment-locally rather than corpus-wide, so hybrid-search ranking fidelity — and any benchmark/HPO scores measured on it — may not transfer exactly to a production server; and it serializes writes, so only one process should open a given .db file at a time.


MaaS Environment Configuration

Store your MaaS credentials securely in a .env file:

# .env
MAAS_BASE_URL="<maas_endpoint_url>"
MAAS_API_KEY="<maas_api_key>"

Private CA certificates for MaaS

Pass a PEM bundle path as ca_bundle= to create_maas_client(), or set MAAS_CA_BUNDLE. When both are unset and SSL_CERT_FILE is also unset, the factory adds a mounted ca.crt under SSL_CERT_DIR to certifi's public trust. See Private CA certificates on OpenShift AI for precedence and platform configuration.

Security

Never commit your .env file to version control. Add it to .gitignore.

Load environment variables in your code:

import os
from dotenv import load_dotenv, find_dotenv

load_dotenv(find_dotenv())

base_url = os.getenv("MAAS_BASE_URL")
api_key = os.getenv("MAAS_API_KEY")

Verify Installation

Check that ai4rag is installed correctly:

import ai4rag
print(ai4rag.__version__)

Test MaaS connectivity:

import os
from ai4rag.utils.clients.maas_client import create_maas_client

# Single client — serves listing, chat, and embeddings for every model.
client = create_maas_client(
    base_url=os.getenv("MAAS_BASE_URL"),
    api_key=os.getenv("MAAS_API_KEY"),
)

# List available models
models = client.models.list().data
print(f"Available models: {[m.id for m in models]}")

Next Steps