Skip to content

Changelog

All notable changes to ai4rag will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.


0.26.0

Changed

  • BREAKING CHANGE: Experiment — generated pattern names use alphabetic suffixes (PatternA, PatternB, …, PatternZ, PatternAA) instead of numeric suffixes; warm-start names retain -warm-start, and the best warm-start pattern is published as PatternA
  • Dependencies — upgraded the core and text-extraction extra's docling-slim requirement from ~=2.121.0 to ~=2.145.0 for upstream security fixes; removed version bounds on certifi
  • Dependencies — refreshed the lockfile with certifi 2026.10.23, Pillow 13.0.0, and Transformers 5.22.1 across platforms for security fixes, alongside filelock 4.2.6, Tornado 6.6, and pypdf 6.21.4

Fixed

  • Experiment — logs the original name and score of the best warm-start pattern before publishing it under its final name
  • GAM optimizer — excludes invalid and non-finite scores from warm-start success counts, coverage, surrogate training, and final selection; numerical surrogate-fit failures skip the affected iteration instead of aborting optimization
  • Vector store — restored RRF and weighted hybrid search with Milvus Lite by disabling unsupported sparse-query term dropping; remote Milvus keeps its existing setting
  • Agentic RAG — requests one tool call per model response and, when a provider returns multiple calls, retains only the first eligible offered call before storing history or executing tools; batches without an eligible call fail clearly, and budget finalization remains enforced

Removed

  • BREAKING CHANGE: Assets generator — removed ai4rag.assets_generator and its bundled notebook, leaderboard, and starter-kit templates without a compatibility shim; generation now lives in kfp_components.components.training.autorag.shared.assets_generator in pipelines-components
  • OpenShell gateway — removed scripts/setup-openshell-gateway.sh from ai4rag; gateway setup is maintained with the relocated assets generator in pipelines-components

0.25.4

Changed

  • Assets generator — generated starter kits run the RHOAI OpenShell CLI through Podman and build a Python 3.12 runtime on the Red Hat sandbox base; CLI and sandbox image selection follows the installed RHOAI CSV, with registry fallbacks
  • OpenShell gateway — setup provisions the downstream Red Hat OpenShell 3.6 gateway with mTLS authentication and RHOAI image references; an explicit --reset option supports migration from the incompatible upstream 0.0.99 gateway database
  • Dependencies — added an explicit certifi>=2026.7.22,<2027 dependency for public CA roots used alongside mounted MaaS certificates

Fixed

  • Text extraction — local Whisper ASR uses native timestamp-aware long-form generation instead of fixed 30-second pipeline windows; multilingual models explicitly transcribe the source language while retaining automatic detection unless the model config specifies a language
  • MaaS TLS — add mounted SSL_CERT_DIR/ca.crt bundles to public trust when ca_bundle, MAAS_CA_BUNDLE, and SSL_CERT_FILE are unset; defaults remain unchanged when the mount is absent
  • MaaS TLS — preserve hashed certificate-directory trust when adding plain bundles and retain configured CA trust in synchronous and asynchronous agent chat clients
  • Assets generator — OpenShell starter kits route Milvus gRPC through a local CONNECT proxy, retain native database tunneling, and use the cluster API endpoint with a CA bundle checked against that endpoint for service-account authentication
  • Assets generator — OpenShell deployment respects the gateway namespace, handles Kyverno pull-secret conflicts with runtime-image mirroring and sandbox retries, and improves build completion checks and agent startup recovery
  • Assets generator — make refresh-certs registers the cluster gateway and refreshes mTLS credentials; sandbox creation checks gateway connectivity before replacing an existing sandbox and reports recovery instructions when unavailable

0.25.3

Changed

  • Evaluator — increased the default RAGAS judge completion limit from 1,024 to 2,048 tokens to reduce truncated faithfulness JSON and missing scores for longer answers
  • Chunker — reduced LangChainChunker's default chunk size and the maximum chunk_sizes value accepted by MaaS search-space preparation from 2,048 to 1,024 tokens; default chunk-size choices are now 512 and 1,024
  • Assets generator — moved one-time OpenShell gateway setup to the repository's scripts/setup-openshell-gateway.sh; generated starter kits require an existing gateway registered as openshift

Fixed

  • Evaluator — confirmed RAGAS context-overflow requests skip retries with the identical prompt, while other failures retain existing retry handling; incomplete evaluations log question IDs and missing metric names
  • Evaluator — RAGAS metric selection now checks the evaluator as well as the metric name, avoiding collisions with metrics from other evaluators
  • Experiment — result ranking places patterns with no final score after scored patterns instead of failing when comparing scores with None

Removed

  • Assets generator — removed the setup-gateway and teardown-gateway Makefile targets from generated starter kits

0.25.2

Fixed

  • Text extraction — corrected guidance for unsupported version-1 M4A chnl channel-layout boxes to recommend conversion to standard AAC-LC M4A, MP3, or WAV, removing the FFmpeg upgrade recommendation

0.25.1

Changed

  • Dependencies — restored the base and text-extraction extra's docling-slim~=2.121.0 requirement for ai4rag and Kubeflow Pipelines compatibility, reverting the >=2.132.0,<3.0.0 requirement introduced in 0.25.0
  • Dependencies — refreshed the lockfile with Docling Core 2.96.0, Docling IBM Models 3.15.0, Docling Parse 7.21.0, and macOS-specific Transformers 5.8.1 and Tokenizers 0.22.2; updated pypdf from 6.20.0 to 6.21.1 and GitPython from 3.2.0 to 3.2.1
  • Development workflow — replaced @MichalSteczko with @dryszka among the default pull-request review owners

Fixed

  • Text extraction — unsupported version-1 M4A chnl channel-layout boxes now produce an actionable error recommending conversion to standard AAC-LC M4A, MP3, or WAV, or a runtime upgrade to FFmpeg 8.1.2 or newer

0.25.0

Added

  • Text extraction — new prepare_extraction() helper returns a PreparedExtraction configuration based on discovered file types, downloads pinned PP-OCRv4 and Whisper Tiny models for connected runs when local paths are unset, and validates local bundles for disconnected runs; text-only corpora need neither bundle
  • Agentic RAG — failed agent invocations emit a RAG_FAILURE_TRACE JSON record with a diagnostic ID, model/tool call counts, retrieval counts, and bounded recent events across context-compression attempts, excluding prompts, document content, tool arguments, and exception messages
  • Retrieval — LLMDocumentCompressor.compress() accepts an optional keyword-only config for invocation callbacks, including tracing compression model calls

Changed

  • Indexing notebooks — MaaS and knowledge-graph indexing default to connected mode and support disconnected Run All execution through RHOAI_NOTEBOOK_CLUSTER_MODE or the overriding NOTEBOOK_CLUSTER_MODE, with model preparation after document discovery
  • Indexing notebooks — the pattern language selects English or Chinese OCR automatically, and extraction uses one worker to reduce memory pressure on small workbenches
  • Dependencies — raised the base and text-extraction extra's docling-slim requirement to >=2.132.0,<3.0.0, addressing PYSEC-2026-4208/4209/4210/4211/4213/4214/4216, CVE-2026-105744, and related upstream fixes for resource exhaustion, file access, plugin imports, and network trust bypasses
  • Dependencies — added huggingface-hub>=0.30,<2 to the text-extraction extra and refreshed the lockfile, including docling-slim 2.140.0, docling-ibm-models 4.0.3 across platforms, Transformers 5.20.1, and PyTorch 2.13.0
  • Assets generator — starter-kit documentation recommends the Red Hat OpenShell sandbox base image for production deployments

Fixed

  • Agentic RAG — exhausted retrieval or model-call budgets force a final answer without tools, preventing recursion loops while preserving response-language instructions; providers returning tool calls after exhaustion raise a clear error
  • Text extraction — local Whisper ASR decodes audio from seekable file paths with FFmpeg before transcription, fixing M4A files with trailing metadata that failed when piped through Transformers; missing FFmpeg, decoding failures, and empty audio now produce actionable errors
  • Assets generator — Neo4j starter-kit connections tunnel native TCP through OpenShell's HTTP CONNECT proxy, extending the existing pgvector support

0.24.0

Added

  • Search space — categorical template_id and MaaS template_ids constraints let vector-backend experiments compare agentic_rag and sequential_rag; Neo4j defaults to agentic_graph_rag
  • Agentic RAG — context-overflow responses reporting "maximum context length" trigger one retry with query-relevant document excerpts; enable_context_compression defaults to True and can disable the fallback
  • Retrieval — new LLMDocumentCompressor extracts verified verbatim excerpts while preserving source metadata, omits chunks with no relevant evidence, and limits each compression pass to 4,096 estimated tokens
  • Agentic RAG — conversation records include context_compression_fallback and compression statistics for initial context and subsequent retrievals during the retry
  • Assets generator — generated inference notebooks and starter kits accept settings.agentic.enable_context_compression, defaulting to enabled when absent

Changed

  • BREAKING CHANGE: Experiment — template selection moves from the rag_template constructor argument to each candidate's template_id; results, evaluation caching, streamed patterns, notebooks, and starter kits retain the selected ID, while matching indexing settings still reuse collections
  • BREAKING CHANGE: RAG template — SimpleRAG and its simple_rag_template module are renamed to SequentialRAG and sequential_rag_template; persisted simple_rag IDs must become sequential_rag
  • Search space — construction validates template and search-mode compatibility: vector backends accept agentic_rag or sequential_rag with vector/hybrid search, while Neo4j requires agentic_graph_rag with graph search
  • Models pre-selector — model pre-selection always uses AgenticRAG with temporary Milvus Lite, including sequential-only and graph searches; experiment pre-selection continues ranking by OVERALL_SCORE, with existing thresholds and skip_mps support

Fixed

  • Assets generator — generated starter kits instantiate the selected template and retain effective Neo4j graph-retrieval settings through configuration and deployment
  • Assets generator — OpenShell deployment preserves database CA certificates across shell boundaries, including PEM or base64-encoded pgvector CA input
  • Assets generator — pgvector connections in OpenShell sandboxes tunnel native TCP through the sandbox's HTTP CONNECT proxy

Removed

  • RAG template — removed template_id_for_class(); use get_template_spec(template_id) to resolve a selected template

0.23.0

Added

  • Experiment — EvaluationResult.conversations retains agent conversation records with each evaluated pattern, making them accessible through experiment results and to_dict()
  • Experiment — evaluation JSON records include conversation_id links to full agent conversations when conversation and evaluation counts match; mismatched counts produce a warning and omit the links
  • Vector store — Neo4j kg_extraction_config.max_completion_tokens accepts a positive integer to control entity-extraction output independently of answer generation, defaulting to 1024
  • Assets generator — generated starter kits support Neo4j OpenShell deployment, loading database credentials, validating required settings, and configuring egress from the Neo4j URI (default port 7687)

Changed

  • Dependencies — the text-extraction extra now includes rapidocr~=3.9.2 and onnxruntime~=1.25.0, providing OCR dependencies through the standard installation
  • Dependencies — refreshed uv.lock with Transformers 5.8.1 → 5.10.1 and platform-specific Docling IBM Models versions (3.13.2 on macOS, 3.15.0 elsewhere)
  • Notebooks — the MaaS indexing notebook installs OCR dependencies through ai4rag[text-extraction], replacing separate RapidOCR and ONNX Runtime installation commands
  • Development utilities — FileStore defaults PDF model inference to CPU and accepts a keyword-only device override; PDF caches use .pdfium.json so legacy cached extractions are regenerated

Fixed

  • Experiment — pattern publication, including the selected warm-start pattern, reads conversations from its EvaluationResult, keeping conversation artifacts attached to the result being published
  • Text extraction — production PDF extraction and the local FileStore helper use Docling's PDFium backend to recover text that the native parser emitted as NUL characters
  • Vector store — Neo4j entity extraction reduces default chunk concurrency from eight to two and uses a dedicated completion-token limit; collection reuse tracks that extraction limit rather than the answer-generation limit
  • Assets generator — starter-kit deployment forwards MILVUS_CA_CERT, PGVECTOR_CA_CERT, and NEO4J_CA_CERT to the sandbox and agent; Milvus deployment uses MILVUS_CA_CERT instead of the obsolete MILVUS_SERVER_CERT key

0.22.0

Added

  • Text extraction — added a local Hugging Face Transformers Whisper backend and DoclingExtractionConfig.asr_model_path (falling back to HF_MODEL_DIR) for offline audio transcription, preserving segment timestamps in Docling documents
  • Vector store — Neo4jConfig.ca_cert (NEO4J_CA_CERT) accepts inline PEM CA certificates for encrypted, verified connections to private or self-signed Neo4j endpoints
  • RAG template — added RAGTemplateSpec, get_template_spec(), and template_id_for_class() to resolve supported persisted template IDs (simple_rag, agentic_rag, and agentic_graph_rag)
  • Agentic RAG — inference responses now include a conversation record with the question, initial retrieval, agent messages, and query-rewrite/retriever tool inputs and outputs; LocalEventHandler saves conversations.json for each published agentic pattern, including the selected warm-start pattern, and KFPEventHandler includes the records in its pattern output

Changed

  • Experiment — published patterns now include template_id so generated inference notebooks can reproduce the selected RAG template
  • BREAKING CHANGE: RAG template — pattern publication now requires a registered built-in template class; custom BaseRAGTemplate subclasses need registry support before they can publish patterns
  • BREAKING CHANGE: Experiment — Neo4j pattern payloads replace settings.knowledge_graph with settings.graph_extraction, containing the extraction policy rather than a separate extraction-model block; notebook generation reads this policy or indexing.pipeline_spec.parameters.graph_extraction_config
  • BREAKING CHANGE: Text extraction — audio extraction now requires a complete local Transformers-format Whisper model through asr_model_path or HF_MODEL_DIR; the previous Docling/openai-whisper backend and runtime ASR model downloads are no longer used
  • BREAKING CHANGE: Vector store — Neo4jConfig rejects neo4j+ssc:// and bolt+ssc:// URIs that bypass certificate verification; use verified +s transport or supply a trusted private CA
  • Text extraction — OCR no longer unconditionally requires DOCLING_ARTIFACTS_PATH: explicit OCR model paths and bundled wheel models work without it, and connected environments can use Docling's model downloader when offline artifacts are unset; configured offline artifacts are still validated
  • Vector store — Neo4j knowledge-graph extraction suppresses verbose GraphRAG logs containing document text, retains warnings and errors, reports extraction start/completion, and hides passwords and CA text from Neo4jConfig representations
  • Notebooks — the MaaS indexing workflow selects OCR and audio artifacts after document discovery, requires only the models needed by the selected corpus, and installs tested RapidOCR/ONNX Runtime versions
  • Notebooks — knowledge-graph indexing now includes S3 discovery, text extraction, offline artifact checks, and notebook event-loop setup; graph inference includes guided setup and optional benchmark evaluation examples
  • Dependencies — added nest-asyncio~=1.6.0 for notebook event-loop compatibility and refreshed uv.lock, including neo4j-graphrag 1.19.0 → 1.22.0, docling-core 2.96.0 → 2.101.1, and langchain-openai 1.6.7 → 1.7.0

Fixed

  • Notebooks — standard and graph inference notebooks instantiate the pattern's selected template instead of always using SimpleRAG; legacy patterns without template_id retain SimpleRAG with a warning, and unsupported IDs raise a clear error
  • Notebook generation — normalized knowledge-graph notebook cell sources so generated notebooks contain complete executable cells
  • Agentic RAG — corrected the invocation-state lock annotation to avoid Pydantic validation warnings on Python 3.12
  • Vector store — HTTPS Milvus connections without an explicit ca_cert now use the CAs loaded by Python's default SSL context, including configured CA bundles; an empty loaded trust store raises a clear configuration error
  • Vector store — updated the Neo4j text splitter for the newer GraphRAG chunk-iterator interface while preserving each canonical chunk's text and index

0.21.0

Added

  • Agentic RAG — new AgenticRAG RAG template (ai4rag.rag.template.AgenticRAG), a LangChain ReAct-style agent that can rewrite the search query and retrieve additional context across up to max_retrieval_steps tool calls before answering, instead of a single fixed retrieval pass
  • Experiment — AI4RAGExperiment and ModelsPreSelector accept a rag_template constructor argument (a BaseRAGTemplate subclass) selecting which template evaluates each candidate pattern
  • Assets generator — generate_starter_kit() (ai4rag.assets_generator.starter_kit) packages a published pattern's pattern.json into a deployable starter_kit.zip: a self-contained agentic-RAG microservice (LangChain agent, OpenShift/OpenShell container image, Makefile, auth wrapper, pyproject.toml/uv.lock) pre-filled with that pattern's generation, embedding, retrieval, and vector-store settings
  • Network — new ai4rag.utils.network module (ensure_safe_url, is_url_scheme_safe, is_local_or_cluster_host) centralizing the scheme/host trust policy enforced across MaaS, S3, Milvus, Neo4j, and pgvector
  • MaaS client — create_maas_client() gains a ca_bundle parameter (or MAAS_CA_BUNDLE env var) to verify a MaaS endpoint against a private or self-signed CA
  • Vector store — PGVectorConfig gains ca_cert (PGVECTOR_CA_CERT env var) to verify a self-signed PostgreSQL server certificate
  • Dependencies — langchain and langchain-openai are new core dependencies, powering AgenticRAG

Changed

  • BREAKING CHANGE: Vector store — MilvusConfig.server_cert is renamed to ca_cert (MILVUS_SERVER_CERT env var renamed to MILVUS_CA_CERT), for consistency with the new PGVectorConfig.ca_cert
  • BREAKING CHANGE: Security — MaaS, S3, and PGVectorStore client factories no longer retry with unverified TLS (verify=False) after a certificate-verification failure; they now raise, directing the caller to set MAAS_CA_BUNDLE/AWS_CA_BUNDLE/ca_cert. MilvusConfig.uri and Neo4jConfig.uri now reject a plaintext scheme (http://, neo4j://, bolt://) against any host that isn't local (localhost/127.0.0.1/::1) or in-cluster (*.cluster.local); PGVectorStore requests fully verified TLS for any non-local host. discover_documents() and load_test_data() no longer probe-and-fall-back to an unverified S3 client on an SSL error
  • Experiment — the default rag_template for AI4RAGExperiment and ModelsPreSelector is now AgenticRAG (previously always SimpleRAG); pass rag_template=SimpleRAG to keep single-pass retrieval behavior
  • RAG template — enriched-message construction (_build_enriched_user_message, _render_enriched_user_message, _build_rag_messages) moved from SimpleRAG up to BaseRAGTemplate, now shared with AgenticRAG
  • Dependencies — bumped transitive tornado dependency from 6.5.8 to 6.5.9; refreshed uv.lock

Fixed

  • Vector store — HTTPS Milvus connections without explicit ca_cert now use the CA certificates loaded by Python's default SSL context, allowing configured mounted CA bundles to be used without a hardcoded gRPC certificate path; an empty default trust store raises a clear configuration error
  • Evaluator — UnitxtEvaluator.evaluate() no longer crashes inside unitxt's TokenOverlap metric when a question's retrieval returns zero chunks; an empty contexts list is substituted with a single blank reference before scoring, so the question scores zero instead of aborting the whole evaluation run

0.20.0

Added

  • Vector store — new Neo4j backend (Neo4jConfig, Neo4jGraphStore, selected via vector_store_type="neo4j") for graph-based RAG. add_documents() stores canonical chunks and, when a foundation model is supplied, runs SimpleKGPipeline to extract entities and relationships per chunk (kg_extraction_config={"mode": "constrained"} by default, or "free" with max_entities_per_chunk/max_relationships_per_chunk caps), linking them back via FROM_CHUNK. The new search_mode="graph" fuses direct vector retrieval with graph-expanded retrieval (entity neighbors and relationship-hop traversal) via reciprocal-rank fusion, tunable through Neo4jGraphRetrievalConfig (route_k, include_entity_neighbors, entity_neighbor_limit, entity_pivot_limit, entity_relationship_hops, relationship_neighbor_limit). neo4j and neo4j-graphrag are new core dependencies
  • Experiment — AI4RAGExperiment accepts kg_extraction_config and graph_retrieval_config (mapping or Neo4jGraphRetrievalConfig) constructor arguments, applied whenever the resolved search_mode is "graph"; the published pattern payload now carries a knowledge_graph block (extraction model, model params, extraction config) alongside the existing indexing/retrieval settings
  • Search space — AI4RAGSearchSpace, get_default_ai4rag_search_space_parameters(), and prepare_search_space_with_maas() take a new vector_store_type parameter (default "milvus"); when "neo4j", the default search space fixes chunk_size/chunk_overlap (1024/0,64) and search_mode="graph", omitting the hybrid ranker parameters that Neo4j does not support
  • Notebooks — two new templates, mass_creating_knowledge_graph_template.ipynb and mass_inference_knowledge_graph_template.ipynb, for running Neo4j GraphRAG indexing and inference at scale
  • Docs — new architecture page "Neo4j Indexing and Graph Search" documenting the indexing and dual-route retrieval flow, plus prerequisites (APOC, required NEO4J_* env vars)

Changed

  • Vector store — validate_search_params() now takes supported_modes and store_class keyword arguments instead of hardcoding ("vector", "hybrid"), so each backend validates against its own supported modes and raises a backend-named error for an unsupported one; Milvus and pgvector pass supported_modes=("vector", "hybrid") explicitly, so they are unaffected by the new "graph" mode
  • Notebook generation — create_placeholder_mapping() emits FM_PARAMS, GRAPH_RETRIEVAL_CONFIG, and KG_EXTRACTION_CONFIG placeholders consumed by the new knowledge-graph notebook templates

Fixed

  • Text extraction — DoclingExtractionConfig.device (default "cpu") is now honored when building Docling's PDF, image, DOCX, and PPTX pipeline options; previously AcceleratorOptions was hardcoded to device="cpu", so a GPU-configured extraction run silently executed on CPU. extract_text() now also fails fast with a RuntimeError when do_ocr=True and device starts with "cuda" but ONNX Runtime has no available CUDAExecutionProvider, instead of discovering the missing GPU support deep inside a worker process

0.19.0

Added

  • Chunker — DoclingChunker now preserves Docling ASR segment timing on audio/hybrid chunks, emitting audio_start_seconds and audio_end_seconds in chunk metadata as the min/max timing across every source segment a chunk spans. LangChainChunker logs a warning when it receives audio documents, since converting them to markdown discards the Docling timing provenance needed to populate these fields — use DoclingChunker to retain them
  • Text extraction — extract_text() gains ssl_cert_path (an explicit PEM CA bundle used to verify the S3 endpoint, taking precedence over AWS_CA_BUNDLE) and s3_client (inject a pre-configured client instead of having extract_text build one from credentials or the environment)
  • S3 client — create_s3_client()'s verify parameter now also accepts a CA bundle path (bool | str, previously bool only); its default (True) now resolves AWS_CA_BUNDLE when set, where boto3 previously ignored that variable whenever verify was passed as the literal True
  • Notebooks — the MaaS indexing template documents how to trust a private/self-signed CA on OpenShift AI via the DSCInitialization (DSCI) trust bundle, and the S3 setup cell now uses the mounted workbench CA bundle (or AWS_CA_BUNDLE) to verify the S3 endpoint

Changed

  • Text extraction — the S3 client used by an extraction run is now created once and shared across all download threads, instead of a fresh client being constructed per download
  • Notebooks — the generated MaaS indexing template pins a tested opencv-python-headless==4.14.0.94 reinstall step for disconnected clusters, and checks discover_documents()'s signature before calling it, so a notebook generated against this release still runs against an ai4rag install predating the prefixes parameter (erroring clearly if more than one input location was configured)
  • BREAKING CHANGE: Text extraction / Notebooks — DOCLING_ARTIFACTS_PATH is now mandatory wherever Docling needs model artifacts: extract_text() raises FileNotFoundError whenever OCR is requested (do_ocr=True) without DOCLING_ARTIFACTS_PATH set — this check runs before custom ocr_*_model_paths are even considered, so setting those does not exempt you from it — and the generated MaaS indexing notebook now raises at startup if it's unset. Both previously fell back to Docling's runtime model downloader / an "online mode" message. Bake Docling artifacts into the workbench image and set the environment variable before upgrading

Fixed

  • RAG optimization component — the warm-start phase now evaluates only the configured optimization metric for each candidate pattern instead of recomputing every configured metric; the remaining metrics are computed and merged in only for the best warm-start candidate, once it is selected and published as a pattern. Avoids redundant evaluator calls (including LLM-judge metrics) for warm-start candidates that are discarded, without changing the scores of the pattern that's ultimately published

Removed

  • BREAKING CHANGE: Text extraction — removed the automatic insecure retry that caught SSLError during S3 downloads and silently re-attempted with certificate verification disabled (verify=False). Configure a trusted CA via AWS_CA_BUNDLE or extract_text(ssl_cert_path=...) instead

0.18.0

Added

  • RAG optimization component — GAM-based optimization gains a warm-start phase: GAMOptSettings.warm_start_strategy ("random" by default, or "greedy"/"balanced" via fields_to_balance) controls how the initial random evaluations are chosen before GAM iterations begin, auto-adjusting the warm-start size (with a logged notice) when the configured n_random_nodes is too small to guarantee full coverage of categorical values. A new GAMOptSettings.max_iterations setting caps how many evaluated patterns are retained and published, independent of the max_evals evaluation budget
  • Document discovery — benchmark keys are now validated against the discovered corpus. Each test_data_doc_names entry must identify exactly one document: an exact object-key match always wins, and a bare file name is accepted only when it resolves to a single document. Anything else raises the new BenchmarkKeyError (a ValueError subclass, exported from ai4rag.utils.data) listing the offending keys, the locations searched, and the expected key format. Previously a correct_answer_document_keys entry that matched no ingested object — a prefix-relative key, say — simply retrieved nothing, leaving the question ungrounded and the scores quietly wrong. Pass validate_test_data_keys=False to downgrade the failure to a warning. A benchmark document that is discovered but dropped by the sampling budget is reported separately, also as a warning

Changed

  • BREAKING CHANGE: Document discovery — discover_documents() now ingests several bucket locations at once: the prefix: str parameter is replaced by prefixes: str | list[str] | None, and DiscoveryResult.prefix by DiscoveryResult.prefixes: tuple[str, ...] (to_dict() and documents_descriptor.json emit "prefixes" accordingly). A bare string is still accepted and coerced to a one-element list, and omitting the argument still lists the whole bucket, so single-location callers only need to rename the keyword. Every prefix is listed and merged into one corpus deduplicated by object key — overlapping selections such as docs/ and docs/manuals/ are safe — and the sampling_max_size_gb budget applies to that union rather than to each location separately
  • Document discovery — listings are now paginated. list_objects_v2 returns at most 1000 keys per call, so a location holding more than that was silently truncated; with several locations sharing one budget the truncation would have starved the later prefixes entirely
  • BREAKING CHANGE: Asset generation — create_placeholder_mapping() and generate_notebook_from_template() take input_data_keys: list[str] in place of input_data_key: str, and the indexing notebook template's INPUT_DATA_KEY placeholder becomes INPUT_DATA_KEYS, rendered as a Python list literal. The generated indexing.ipynb therefore rediscovers every location the pipeline ingested, not just the first
  • BREAKING CHANGE: Event handler — the pattern payload's settings block field vector_store_binding is renamed to store_binding (backend-agnostic naming). Code reading on_pattern_creation() payloads must switch to the new field name
  • RAG optimization component — only the best successful warm-start candidate is published as a pattern (as Pattern1), feeding into the subsequent GAM iterations; the other warm-start evaluations remain internal to the optimizer and are not streamed individually. GAM training now fits factor terms for categorical parameters and spline terms for numeric ones (previously every parameter was label-encoded and modeled with a spline), improving prediction quality once warm-start coverage is enabled

Fixed

  • Notebooks — fixed the MaaS indexing notebook template: corrected the table of contents/appendix links, removed a duplicate "Download HuggingFace Models" step (renumbering the subsequent steps), and cleaned up incorrect configuration descriptions

0.17.0

Added

  • Vector store — Milvus Lite, the embedded, zero-server mode of the Milvus backend, is now the recommended local/zero-config replacement for the removed Chroma store: use the new MilvusLiteConfig(db_path="./ai4rag.db") (or MilvusLiteConfig() for the default path). Unlike Chroma, Milvus Lite supports hybrid (dense + BM25) search, though it computes BM25 statistics segment-locally rather than corpus-wide, so hybrid-search ranking fidelity may not transfer exactly to a production Milvus server. pymilvus[milvus-lite] is now a core dependency, so no extra installation step is needed
  • Vector store — new temporary_milvus_lite_store() helper (ai4rag.rag.vector_store.local_store) providing a disposable, file-backed Milvus Lite store for internal throwaway indexes (model pre-selection, judge calibration), replacing the previous ephemeral in-memory Chroma store used for the same purpose
  • Notebooks — the MaaS indexing notebook template gains a "Prerequisites for Disconnected Clusters" section (validation cells, configuration examples, download instructions) for pre-downloading Docling and HuggingFace artifacts before running on an air-gapped cluster; the README documents the pre-download/transfer workflow (DOCLING_ARTIFACTS_PATH, HF_HOME, HF_HUB_OFFLINE)

Changed

  • Vector store — the Milvus config was split into MilvusConfig (remote server / Zilliz Cloud only, which now validates that uri is an http(s):// URL and raises ValueError otherwise) and a new MilvusLiteConfig (embedded, local only, configured via db_path, which conversely rejects http(s):// values). Previously, a single MilvusConfig selected between a remote server and embedded Milvus Lite based on whether uri looked like a URL or a local file path; a mistyped or unreachable MILVUS_URI could therefore be silently interpreted as a local path and create an unintended throwaway local database. That silent fallback is no longer possible — a misconfigured server URI now fails loudly instead. ai4rag.rag.vector_store now also exports MilvusLiteConfig, and the vector_store_type search-space parameter accepts "milvus_lite" in addition to "milvus" and "pgvector". Breaking: get_vector_store() now requires MilvusLiteConfig (not MilvusConfig) when config.provider == "milvus_lite"
  • Search space — hybrid search parameters (search_mode, ranker_strategy, ranker_k, ranker_alpha) are now unconditionally part of the default search space, since every remaining vector store backend supports hybrid search. Breaking: AI4RAGSearchSpace, prepare_search_space_with_maas(), and get_default_ai4rag_search_space_parameters() no longer accept a vector_store_type parameter
  • Event handler — VectorStoreSettings payload fields provider_id / vector_store_id replaced by provider_type / collection_name. Breaking: code reading on_pattern_creation() payloads must switch to the new field names
  • Dependencies — refreshed uv.lock (transitive dependency updates)

Fixed

  • CI — the publish-pypi maintainer check now reads the actor's role_name from the GitHub API instead of the removed permission field

Removed

  • BREAKING CHANGE: Vector store — removed the ChromaDB vector store backend (ChromaConfig, ChromaVectorStore), the chromadb dependency, and the now-unused ai4rag.utils.compat.ensure_sqlite3() compatibility shim, due to known security vulnerabilities in the chromadb package. ai4rag.rag.vector_store no longer exports ChromaConfig; the "chroma" value for vector_store_type is no longer accepted (only "milvus", "milvus_lite", and "pgvector" are supported)

0.16.0

Added

  • Document discovery / extraction — .msg (Outlook email) files are now a supported extension for discovery and text extraction
  • RAG template — SimpleRAG.chat(), a chat-completions-style entry point that RAG-enriches the last user turn of a conversation while passing prior history through untouched

Changed

  • Benchmark data — BenchmarkData now requires correct_answer_document_keys instead of correct_answer_document_ids; a document key is a document's DoclingDocument.name, the identifier carried through chunking, indexing, and evaluation. Breaking: correct_answer_document_ids is rejected with a message explaining what a key is; evaluation results report document_key instead of document_id
  • Document discovery / extraction — documents extracted from object storage are now named by their full object key (prefix included) instead of the bare file name, so two documents with the same basename under different prefixes no longer collide on the output path and silently overwrite one another
  • Assets generator — relocated from ai4rag.utils.assets_generator to ai4rag.assets_generator, alongside other top-level packages instead of under the general-purpose utils namespace. Breaking: ai4rag.utils.assets_generator no longer exists; import from ai4rag.assets_generator instead
  • RAG template — BaseRAGTemplate and SimpleRAG now only compose a retriever and a foundation model for retrieval-and-generation; index building is an upstream concern owned by ai4rag.rag.vector_store. Breaking: chunker, embedding_model, and vector_store constructor arguments and build_index have been removed; build indexes via ai4rag.rag.vector_store directly before constructing a template
  • Dependencies — bumped docling-slim from ~=2.107.0 to ~=2.121.0 (base and text-extraction extras)

Fixed

  • Notebooks — the indexing notebook template now declares the text-extraction extras it requires
  • Search space preparation — MaaS foundation/embedding model validation failures now log the underlying root cause (status code, error code, and message extracted from the openai client error, with traceback) instead of discarding it and logging only the model id

0.15.0

Changed

  • Vector store (pgvector) — PGVectorStore now uses asyncpg instead of psycopg to talk to Postgres, so installing ai4rag no longer requires a libpq binding in any form (bundled, compiled, or system-provided); the public sync API (search/add_documents/clean_collection/close) is unchanged, backed internally by a dedicated per-instance background event loop. Breaking: database errors now surface as asyncpg exceptions (e.g. asyncpg.exceptions.PostgresConnectionError) instead of psycopg ones
  • Vector store (pgvector) — hybrid search's dense and keyword queries now run concurrently instead of sequentially
  • Vector store (pgvector) — retry-on-dropped-connection behavior, previously insert-only, now also covers search, keyword search, and clean_collection
  • Vector store (pgvector) — hot-path queries (search, keyword search, insert) now enforce a 90s command timeout on both pool acquisition and query execution, so a connection silently killed by an idle middlebox fails fast instead of hanging on the OS's TCP retransmission timer; idle pooled connections are now recycled after 60s (was 300s)
  • Dependencies — replaced psycopg[binary,pool] with asyncpg as the pgvector backend client

Fixed

  • Vector store (pgvector) — fixed a latent ordering bug where register_vector() could look up the vector type's OID before CREATE EXTENSION IF NOT EXISTS vector had run, on a fresh database
  • Vector store (pgvector) — fixed a race between close() and in-flight queries that could raise a bare AttributeError; close() now drains in-flight calls before tearing down the event loop, and a store closed mid-call raises a clear RuntimeError instead

Removed

  • Vector store (pgvector) — psycopg/psycopg_pool are no longer dependencies of ai4rag

0.14.0

Added

  • Leaderboard — build_leaderboard_html() gains an optimization_metric_evaluator parameter so the pinned leaderboard column resolves to the evaluator that drove optimization instead of always falling back to unitxt/custom when metric names collide across evaluators (e.g. faithfulness)

Changed

  • Core — split ai4rag.components into ai4rag.utils (pure, dependency-free business-logic helpers) and pipelines-components (KFP-only orchestration): ai4rag.components.data → ai4rag.utils.data, ai4rag.components.assets_generator → ai4rag.utils.assets_generator (NotebookCell and create_placeholder_mapping are no longer re-exported from the package __init__; import them from their submodules), ai4rag.components.utils.{s3,maas_client} grouped into a new ai4rag.utils.clients package, and ai4rag.components.utils.docling_io moved to ai4rag.utils.docling_io

Fixed

  • Experiment — AI4RAGExperiment now only includes the unitxt default metric bundle (answer_correctness, faithfulness, context_correctness, overall_score) when a unitxt evaluator is actually configured, instead of always including it regardless of active evaluators

Removed

  • RAG optimization component — ai4rag.components.optimization (run_rag_optimization, OptimizationResult) removed entirely; RAG-optimization orchestration is now a KFP-only concern living in pipelines-components, built on ai4rag.core and ai4rag.search_space
  • Components — the ai4rag.components package no longer exists; import from ai4rag.utils instead

0.13.0

Added

  • Text extraction — optional RapidOCR via Docling, configured through a single DoclingExtractionConfig passed to extract_text (do_ocr, ocr_lang, custom ONNX model paths); OCR remains off by default
  • Document discovery / extraction — JPEG, PNG, and TIFF image extensions supported for OCR-capable ingestion
  • Text extraction — audio ingestion (.wav, .mp3, .m4a, .aac, .ogg, .flac) transcribed via Docling's ASR pipeline (Whisper), with automatic language detection

Changed

  • Dependencies — split docling's heavy conversion stack (torch, docling-ibm-models, rapidocr, whisper) out of the base install into a new text-extraction extra; core RAG code (chunking, experiment, evaluator, templates) only needs docling-slim[feat-chunking]. The test extra now depends on ai4rag[text-extraction] so unit tests keep exercising the full conversion stack
  • Model pre-selection — default foundation/embedding model counts used by ModelsPreSelector extracted into a new PreSelectorConstants (ai4rag.utils.constants)

Fixed

  • Text extraction / OCR — fail fast with bake instructions when DOCLING_ARTIFACTS_PATH is set but RapidOCR ONNX models are missing; current PyPI rapidocr wheels no longer ship model files (bake via tmp/Containerfile.autorag-dev)
  • Vector store (pgvector) — embedding models whose dimension exceeds pgvector's 2000-dimension HNSW/IVFFlat index cap are no longer rejected at construction; PGVectorStore now skips building the HNSW index above the limit and logs a warning, falling back to an exact sequential scan (storage, keyword search, and hybrid fusion are unaffected)
  • Vector store (pgvector) — the connection pool and backing table are now created lazily on the first add_documents()/search call, guarded by double-checked locking, instead of during __init__; avoids idle pooled connections accumulating while the (potentially slow) embedding step runs before the first insert
  • Model access — create_maas_client() now normalizes base_url by stripping any trailing slash and appending /v1 if missing, so both host-only and fully-qualified endpoint URLs work (previously the URL was used verbatim, requiring the exact /v1-suffixed endpoint)

Removed

  • Vector store — ChromaVectorStore, MilvusVectorStore, and PGVectorStore are no longer re-exported from ai4rag.rag.vector_store; import each directly from its backend module instead (ai4rag.rag.vector_store.chroma, .milvus, .pgvector). This keeps each backend's client library (chromadb, pymilvus, psycopg) — and its own import-time requirements, notably chromadb's minimum sqlite3 version — isolated to callers that actually use that backend. BaseVectorStore, ChromaConfig/MilvusConfig/PGVectorConfig, get_vector_store, get_vector_store_config, and get_vector_store_env_vars are unaffected and remain importable from ai4rag.rag.vector_store
  • Internal utilities — removed the unused ai4rag.utils.validators module

0.12.0

Added

  • Vector store — direct backend clients for Chroma, Milvus, and PostgreSQL/pgvector (ChromaVectorStore, MilvusVectorStore, PGVectorStore), each selected via a typed, frozen config dataclass (ChromaConfig, MilvusConfig, PGVectorConfig) passed as a single vector_store_config
  • Vector store — reranker module implementing RRF and weighted fusion for hybrid search
  • Vector store — BaseVectorStore now supports close() and the context-manager protocol; each optimization trial scopes its store in a with block, so connections and pools are no longer leaked per trial
  • Vector store (pgvector) — connection pooling via psycopg_pool.ConnectionPool with a configurable PGVectorConfig.pool_max_size (default 10); AI4RAGExperiment sizes the pool from inference_max_threads so it tracks real query concurrency
  • Evaluator — optional RagasEvaluator (with RAGAS adapter classes AI4RAGRagasLLM / AI4RAGRagasEmbeddings) enabling RAGAS-based metrics; ragas is now a regular dependency
  • RAG optimization component — llm_judge_mode selector (base / ragas / all / none) on run_rag_optimization() to choose which LLM-as-a-Judge evaluators run
  • Evaluator — build_aggregate_metric() shared helper on BaseEvaluator for constructing aggregate metric payloads
  • Search space preparation — build_search_space_report() and serialize_model() in ai4rag.search_space.prepare, co-locating the model↔spec round-trip (serialize_model() is the write mirror of the model restore path)
  • Model access — create_maas_client() and shared model discovery/restore helpers get_foundation_models() / get_embedding_models() in ai4rag.search_space.prepare.models, accepting either bare model ids (discovery) or serialized report specs (restore)
  • Assets generator — get_vector_store_config() / get_vector_store_env_vars() factories that build a backend config from a provider discriminator and expose each backend's required environment variables for documentation
  • Dependencies — added openai as the model-access SDK (replacing ogx-client), plus chromadb, pymilvus, pgvector, and psycopg[binary,pool] for the direct vector-store clients

Changed

  • Model provider — replaced the OGX integration with any OpenAI-compatible endpoint; the shipped integration targets OpenShift AI Models-as-a-Service (MaaS), which serves listing, chat, and embeddings from a single endpoint
  • Vector store — get_vector_store() and AI4RAGExperiment now take a single vector_store_config and dispatch on config.provider, replacing the vector_store_type string plus the OGX vector_io provider id
  • Vector store — collection-name resolution centralized in BaseVectorStore, enforcing a mandatory ai4rag prefix as the cross-backend isolation guard
  • Vector store — hybrid-search reranking parameter renamed impact_factor → k
  • Search space — default vector_store_type changed from ogx to milvus; the default Chroma search space no longer includes the window retrieval method
  • Search space preparation — renamed prepare_search_space_with_ogx to prepare_search_space_with_maas, now accepting an openai.OpenAI client. Because MaaS models.list() carries no metadata (model type, embedding dimension, context length), the payload must declare foundation and embedding model IDs explicitly; embedding dimension and context length are auto-detected at construction time
  • Model ids — model ids are used verbatim, exactly as models.list() reports them (including any / characters); there is no more model-specific URL derivation or id stripping
  • Client factory — replaced create_ogx_client with create_maas_client, a single client that serves listing, chat, and embeddings for every model at the one MaaS endpoint
  • Notebook templates — renamed the generated ogx_{indexing,inference} templates to maas_{indexing,inference}, each building a single OpenAI client from MAAS_BASE_URL / MAAS_API_KEY and reusing it for every model; the inference notebook now also rebuilds the pattern's detected generation language and passes it to OpenAIFoundationModel, so answers keep the benchmark's language
  • Experiment / evaluator — metrics and optimization_metric now require RAGMetric instances selected from Metrics and reject bare metric-name strings, which are ambiguous now that a name (e.g. faithfulness) is shared across the unitxt and RAGAS evaluators
  • Model helpers — model-instantiation helpers moved to ai4rag.search_space.prepare.models, removing the components↔search_space coupling
  • Search space report — model pre-selection decoupled from report building into an explicit ModelsPreSelector step; SearchSpaceReport slimmed to the search-space dict and no longer carries selected_models or a per-model base_url, and pattern.json no longer carries base_url
  • Leaderboard — aggregate scores are keyed by a collision-free key (unitxt and custom metrics keep their bare name; other evaluators are prefixed, e.g. ragas_faithfulness), so colliding metric names each get their own column instead of overwriting one another

Fixed

  • Vector store (Milvus) — forced consistency_level="Strong" on vector/hybrid search so a query immediately following an add_documents() upsert can no longer race Milvus's default bounded-staleness read and return zero hits against a collection that does contain matching data
  • Vector store (pgvector) — corrected inner_product scoring: the <#> operator returns the negative inner product, so the score is now derived by negation (cosine/l2/l1 keep 1/dist), fixing an inverted ranking
  • Vector store (pgvector) — guarded lazy index creation with double-checked locking (plus a UniqueViolation fallback) so concurrent search threads no longer race on CREATE INDEX
  • Experiment — an optimization metric that is produced but unscored (None mean) is now recorded as a failed — not fatal — iteration; a genuinely absent metric still raises a RAGExperimentError with an evaluator-qualified message
  • Components — added db_secret_name to the indexing pipeline params
  • Core — ensure_ascii=False when JSON-dumping documents that may reach the end user, preserving non-ASCII characters
  • Benchmark data — reject BenchmarkData records with zero correct answers, preventing a downstream unitxt TokenOverlap crash on max() of an empty iterable
  • Experiment — benchmark JSON is now read with an explicit UTF-8 encoding

Removed

  • OGX — removed all OGX support: the ogx-client dependency, OGXFoundationModel, OGXEmbeddingModel, OGXVectorStore, OGXModelParameters, OGXEmbeddingParams, create_ogx_client, the ogx_utils module, the ogx_inference_base_url helper, and the OGX_CLIENT_BASE_URL / OGX_CLIENT_API_KEY environment variables (replaced by MAAS_BASE_URL / MAAS_API_KEY)
  • Assets generator — removed the OGX-only pattern_builder and prompt_filters modules and the build_pattern_json export; indexing-spec enrichment is now inlined
  • Search space preparation — prepare_search_space_report() and the search_space_preparation module removed from ai4rag.components.optimization; build a search space with prepare_search_space_with_maas(), then call build_search_space_report() from ai4rag.search_space.prepare
  • Experiment — EvaluationResult no longer carries a rag_pattern field; a trial's vector store is closed once the trial finishes, so read pattern_name / scores from EvaluationResult instead of calling .generate() on a previously returned pattern
  • Dependencies — removed langchain-chroma; Chroma is now used directly via chromadb
  • Samples — removed the outdated samples/run_ai4rag.ipynb notebook

0.11.0

Added

  • Text extraction — added support for 9 additional document formats (.odt, .odp, .adoc, .tex, .epub, .eml, .qmd, .rmd, .xhtml) in document discovery and text extraction, alongside existing PDF, DOCX, PPTX, Markdown, HTML, and plain-text support

Changed

  • Data component — SUPPORTED_EXTENSIONS extracted into a shared ai4rag.components.data.constants module, removing duplication between document discovery and text extraction
  • Dependencies — replaced the docling meta-package with docling-slim[standard,feat-chunking,format-opendocument], and dropped the standalone docling-core dependency, now pulled in transitively via the feat-chunking extra

Fixed

  • Notebooks — updated the ogx_inference_template.ipynb test-data-loading example to call ai4rag.components.data.test_data_loader.load_test_data(), replacing a stale reference to the removed kfp_components pipeline API

0.10.4

Fixed

  • Experiment — fixed incorrect dictionary key "method" used to check the chunking method when determining whether to include metadata; now uses the canonical AI4RAGParamNames.CHUNKING_METHOD constant, ensuring include_metadata is correctly set for hybrid chunking during experiment execution

0.10.3

Fixed

  • RAG optimization component — GAMOptimizer initial random phase now uses stratified sampling to guarantee that every unique value of each string-valued categorical parameter (e.g. search_mode, chunking_method) is evaluated at least once before GAM training begins, preventing biased exploration when the search space is skewed toward a dominant category; warm-start observations are accounted for so stratification does not waste early slots on already-covered values; a warning is emitted when n_random_nodes is too small for full categorical coverage

0.10.2

Changed

  • Dependencies — updated ogx-client dependency from ~=1.1.0 to ~=1.2.0
  • Data component — ChunkingConstraints.METHODS changed from mutable list to immutable tuple for correctness

Fixed

  • Evaluator — hardened LLM-as-a-Judge JSON response parsing with lightweight repair for common malformed outputs (single-quoted JSON, markdown-fenced blocks, JSON embedded in surrounding prose); added explicit output format instructions to the judge prompt; separated LLM call failures from JSON parse failures with distinct warning messages and raw response logging
  • Evaluator — calculate_overall_score() now propagates None directly for unevaluated metrics instead of converting to float("nan")
  • Experiment — streamed pattern now includes the include_metadata chunking field, ensuring metadata-aware chunking configurations are fully captured in pattern output

Removed

  • Data component — removed index_documents() function and documents_indexing module from ai4rag.components.data; the component was unused

0.10.1

Added

  • OGX client — timeout fallback for embedding and chat requests: on APITimeoutError, retries once with a 20-minute timeout and disabled client-level retries to accommodate slow CPU-deployed models

Changed

  • Chunking — AI4RAGChunk now carries a deterministic chunk_id field (SHA-256 of document ID, sequence number, and text), replacing ad-hoc hash-based ID generation in vector stores
  • Chunking — hybrid chunking method now automatically includes document metadata during experiment execution
  • Vector store — ChromaVectorStore and OGXVectorStore deduplication and chunk identification now use the deterministic AI4RAGChunk.chunk_id instead of independent hash computations

Fixed

  • Prompt templates — partially reverted default RAG prompt templates for all model families (Granite, Llama, Mistral, OpenAI, default) to use model-native prompting patterns, removing shared instruction boilerplate
  • Prompt filters — decoupled HPO_CITATION_FRAGMENTS from internal _RAG_CITATION_INSTRUCTION constant, using inline string literals for portability

0.10.0

Added

  • Evaluator — LLMaJEvaluator for LLM-as-a-Judge evaluation, scoring answer_relevance on a 1–5 rubric with structured JSON output and bootstrap confidence intervals; scores are normalized to [0.0, 1.0]
  • Evaluator — automatic judge model selection via select_judge_model() — when multiple generation models are available, a calibration round scores each candidate on a benchmark subset and picks the one with the highest spread-and-stability score
  • Evaluator — RAGMetric frozen dataclass and Metrics registry replacing raw metric-name strings throughout the evaluator and experiment APIs
  • Evaluator — custom_metrics module with calculate_overall_score() — computes a cross-metric mean as a built-in custom metric (overall_score)
  • Experiment — multi-evaluator dispatch: AI4RAGExperiment now accepts an evaluators list and routes each metric to the evaluator matching its EVALUATOR_TYPE
  • Experiment — metrics parameter on AI4RAGExperiment for explicit control over which metrics are evaluated; defaults are derived from configured evaluators when omitted
  • RAG optimization component — indexing_pipeline_params parameter on run_rag_optimization() for enriching pattern.json with indexing pipeline settings
  • OGX client utilities — ogx_inference_base_url() helper for building /v1-suffixed inference endpoint URLs

Changed

  • Evaluator — BaseEvaluator.evaluate_metrics() signature now accepts Sequence[RAGMetric] and returns a structured EvaluationMetricsResult TypedDict (was list[str] → dict)
  • Evaluator — UnitxtEvaluator updated to work with the new RAGMetric-based metric dispatch and return EvaluationMetricsResult
  • Event handler — BaseEventHandler.on_pattern_creation() payload and evaluation results now fully typed via PatternPayload and EvaluationRecord TypedDicts with nested structured types (AggregateMetricPayload, VectorStoreSettings, ChunkingSettings, RetrievalSettings, GenerationSettings)
  • Experiment — optimization_metric parameter accepts RAGMetric | str (was str only); default changed from faithfulness to overall_score
  • Experiment — evaluation results internally use structured EvaluationMetricsResult throughout the scoring, streaming, and caching pipeline
  • RAG optimization component — default optimization metric changed from faithfulness to overall_score; supported metrics now include overall_score
  • RAG optimization component — judge model selection and LLM-as-a-Judge evaluation are now automatically enabled during run_rag_optimization()
  • RAG optimization component — artefact generation extracted into _generate_output_artifacts() for clearer separation of concerns
  • Dependencies — refreshed uv.lock and sorted dependency declarations in pyproject.toml

0.9.3

Added

  • Search space preparation — chunk_overlaps parameter on prepare_search_space_report() and prepare_search_space_with_ogx() for constraining the chunk-overlap dimension of the search space (e.g. [0, 128]), with Pydantic range validation against ChunkingConstraints bounds

Changed

  • RAG optimization component — renamed max_threads keyword argument to inference_max_threads on run_rag_optimization() to align with prepare_search_space_report() naming
  • Search space report — verbose representation in prepare_search_space_report() now derives values from valid (rule-filtered) combinations instead of raw parameter lists, ensuring the report reflects only reachable configurations
  • Search space constraint validation — removed automatic deduplication of chunking_methods and chunk_sizes from AI4RAGConstraints validators — duplicates are no longer silently collapsed
  • Dependencies — removed the langchain meta-package; only the needed sub-packages (langchain-chroma, langchain-text-splitters) are retained

0.9.2

Added

  • chunk_sizes parameter on prepare_search_space_report() for constraining the chunk-size dimension of the search space (e.g. [256, 512])

Changed

  • Chunking constraint validation (chunking_methods, chunk_sizes) now happens via Pydantic before any I/O, providing clearer error messages with exact field paths
  • Minimum allowed chunk size lowered from 512 to 128, enabling finer-grained chunking strategies
  • Duplicate values in chunking_methods and chunk_sizes are now automatically deduplicated

0.9.1

Fixed

  • LLM-based language detection now uses JSON-schema structured output (response_format) instead of fragile regex extraction, ensuring reliable ISO 639-1 code parsing from model responses

0.9.0

Added

  • Multilingual support — Language dataclass and language parameter on BaseFoundationModel for language-aware prompt template generation
  • ai4rag.search_space.prepare.language_detection module for LLM-based benchmark language detection with ISO 639-1 mapping
  • CharApproxTokenizer — lightweight, model-agnostic tokenizer approximating token count via character ratio, replacing the tiktoken dependency
  • ai4rag.components.assets_generator.prompt_filters module for filtering OGX runtime injection duplicates from HPO prompt templates during Responses API export
  • Progressive chunk truncation in OGXEmbeddingModel — oversized chunks are truncated before embedding instead of failing
  • max_threads parameter on run_rag_optimization() for controlling concurrent benchmark evaluation threads

Changed

  • Breaking: BaseFoundationModel constructor accepts a language parameter; user_message_text and context_template_text are now validated properties instead of RAGPromptTemplateString descriptors
  • Breaking: BaseFoundationModel.chat() signature now accepts **kwargs
  • Breaking: Search space report format changed from YAML to JSON — search_space_preparation and rag_templates_optimization no longer use pyyaml
  • Breaking: OGXEmbeddingModel._embed_text() renamed to _call_embedding_api()
  • Replaced tiktoken dependency with character-based token approximation across all chunkers
  • Upgraded docling dependency to 2.107.0 and adapted to new API
  • Prompt template system refactored — RAGPromptTemplateString descriptor replaced with setter-based validation via validate_prompt_templates_placeholders()
  • Responses API payload aligned with previous chat/completion format
  • Component functions (text_extraction, search_space_preparation, rag_templates_optimization) made more customizable with additional parameters
  • Removed mike dependency and documentation versioning from CI/CD

Fixed

  • Chunks exceeding embedding model context length now truncated with progressive margins instead of causing API failures
  • random_state parameter properly wired through BaseOptimizer to GAMOptimizer and RandomOptimizer for deterministic optimization runs
  • Removed unnecessary docling install cell from indexing notebook template

Removed

  • tiktoken dependency — replaced by CharApproxTokenizer
  • RAGPromptTemplateString descriptor class from ai4rag.rag.foundation_models.utils
  • YAML serialization support for model instances in search space reports

0.8.1

Changed

  • Downgraded docling-core dependency from ~=2.84.0 to ~=2.83.0 to resolve compatibility issues

0.8.0

Added

  • ai4rag.components package — pipeline step business logic consolidated from pipelines-components, usable standalone or within KFP wrappers
    • components.data: discover_documents(), extract_text(), index_documents(), load_test_data()
    • components.optimization: prepare_search_space_report(), run_rag_optimization(), detect_benchmark_language()
    • Shared utilities: create_s3_client(), create_ogx_client(), load_docling_documents()
  • ai4rag.components.assets_generator — notebook, leaderboard, and pattern artefact generation
    • Notebook / NotebookCell classes with importlib.resources template loading
    • build_leaderboard_html() for styled HTML leaderboard generation
    • build_pattern_json() for RAG pattern definition building
    • generate_notebook_from_template() for notebook rendering from templates
    • Bundled notebook and script templates as package data
  • ai4rag.utils.compat.ensure_sqlite3() — centralized pysqlite3 patch for RHEL 9 / older sqlite
  • boto3 and multiprocess added as core dependencies
  • docling promoted from dev-only to core dependency
  • Pipeline Components user guide and API reference documentation

Changed

  • Breaking: Event handler PatternPayload schema restructured — pattern_name → name, execution_time → duration_seconds, vector_store → vector_store_binding, datasource_type → provider_id, collection_name → vector_store_id
  • Breaking: Removed schema_version and producer fields from PatternPayload
  • Breaking: Removed distance_metric from EmbeddingSettings and indexing params
  • VectorStoreSettings and EmbeddingSettings TypedDicts now use total=False for optional fields
  • ogx-client dependency updated from ~=1.0.0 to ~=1.1.0
  • docling-core dependency updated from ~=2.74.1 to ~=2.84.0
  • Hybrid search payload now conditionally includes ranker_k (only for rrf strategy) and ranker_alpha (only for weighted strategy)
  • Vector store provider type is now resolved dynamically from the OGX server when available
  • max_combinations from search space now included in pattern creation payload

Removed

  • ai4rag.search_space.src.models module (FoundationModels and EmbeddingModels enum classes) — model IDs are now plain strings throughout the codebase
  • EmbeddingModels.get_distance_metric() utility — distance metric is no longer tracked

0.7.0

Added

  • DoclingChunker — structure-aware, token-aware chunker wrapping docling's HybridChunker, operating directly on DoclingDocument objects and preserving document hierarchy (headings, tables, figures) during chunking
  • AI4RAGChunk — framework-agnostic chunk dataclass replacing langchain Document as the pipeline's canonical chunk representation
  • "hybrid" chunking method in the default search space, enabling DoclingChunker alongside the existing "recursive" method
  • Search space validation rule _rule_chunk_overlap_for_chunking_method enforcing chunker-specific overlap constraints (hybrid requires overlap = 0; recursive requires overlap > 0)
  • Minimum context length validation for embedding models — models with context_length below 700 tokens are now rejected during initialization with a descriptive error

Changed

  • Breaking: BaseChunker.split_documents() now accepts Sequence[DoclingDocument] and returns list[AI4RAGChunk] (was Sequence[Document] → list[Document])
  • Breaking: BaseVectorStore.add_documents() now accepts Sequence[AI4RAGChunk] (was Sequence[Document])
  • Breaking: BaseVectorStore.search() now returns list[AI4RAGChunk] (was list[dict])
  • LangChainChunker updated to accept DoclingDocument input (converts to markdown internally) and return AI4RAGChunk output
  • ChromaVectorStore and OGXVectorStore updated to work with AI4RAGChunk, with internal conversions handled transparently
  • OGXEmbeddingModel embedding batch size reduced from 2048 to 1024
  • Added docling-core as a project dependency

0.6.3

Fixed

  • Duplicate chunk IDs in OGXVectorStore.add_documents() now detected and skipped with a warning, preventing insertion failures when documents produce identical chunk hashes

0.6.2

Added

  • VectorStoreInitializationError exception for clearer diagnostics when vector store creation or retrieval fails

Fixed

  • Vector store initialization errors are now caught and wrapped with contextual information (embedding model ID, vector store provider) instead of propagating raw exceptions
  • Simplified experiment error summary — removed redundant log-file reminder suffix from error messages

0.6.1

Changed

  • Upgraded ogx-client dependency from ~=0.8.0 to ~=1.0.0
  • Updated documentation to require OGX Server >= 1.0.0

0.6.0

Added

  • uv package manager support as an alternative to pip for dependency management and development workflows
  • AGENTS.md file with AI agent guidelines for contributing to the project

Changed

  • Rebranded all Llama Stack integrations to OGX: LSEmbeddingModel → OGXEmbeddingModel, LSFoundationModel → OGXFoundationModel, LSVectorStore → OGXVectorStore, prepare_search_space_with_llama_stack → prepare_search_space_with_ogx (and all related classes, modules, and configuration keys)
  • Replaced llama-stack-client dependency with ogx-client
  • Improved logging during model selection and validation — clearer messages when models are filtered or skipped
  • Updated CI/CD workflows to use uv for dependency installation and test execution
  • Updated all documentation to reflect the Llama Stack → OGX rebranding

Removed

  • OpenAI model wrappers (OpenAIEmbeddingModel, OpenAIFoundationModel)
  • dev_utils/run_experiment_with_openai_models.py example script
  • Llama Stack example notebooks from dev_utils/llama_stack_examples/

0.5.5

Changed

  • Improved error messages when models registered in Llama Stack do not respond — errors now distinguish between "not registered" and "registered but not responding" models, with actionable guidance
  • Improved pre-selector logging to show total model counts before selection and selected counts after
  • Added logging of selected foundation and embedding models during search space preparation
  • Removed pydantic-based payload validation overhead in prepare_search_space_with_llama_stack — replaced with direct dataclass instantiation
  • Removed validation_error_decoder module and pydantic dependency from search space preparation

0.5.4

Fixed

  • Fixed chunk ID collisions in LSVectorStore — chunks from the same document no longer share the same chunk_id; IDs are now derived from chunk content via hashing
  • Fixed chunk_metadata in LSVectorStore to only contain document_id, with full document metadata preserved in a separate metadata field

0.5.3

Changed

  • Bumped llama-stack-client dependency from ~=0.6.0 to ~=0.7.1
  • Updated documentation and installation instructions to require Llama Stack >= 0.7.0

0.5.2

Changed

  • Vector store type now supports any Llama Stack provider via the ls_<provider_id> pattern (e.g., ls_milvus, ls_qdrant), instead of only the hardcoded ls_milvus
  • vector_store_type parameter on AI4RAGExperiment changed from Literal["chroma", "ls_milvus"] to str for flexibility

Fixed

  • Fixed provider_id extraction in get_vector_store — previously hardcoded to "milvus", now correctly derived from the ls_<provider_id> vector store type

0.5.1

Added

  • Batch processing for Llama Stack embeddings (2048 chunk limit) and vector store document insertion, preventing failures with large document sets

Changed

  • Hybrid search re-enabled by default in the ls_milvus default search space (was disabled in 0.5.0 due to upstream instability)
  • Default chunk sizes narrowed from (512, 1024, 2048, 4096) to (1024, 2048) and overlaps from (128, 256, 512) to (128, 256) for faster optimization
  • Chroma vector store batch size simplified to a fixed default of 2048 instead of querying client internals

0.5.0

Added

  • KFPEventHandler: new event handler for Kubeflow Pipelines (KFP) integration, enabling experiment progress tracking inside KFP pipeline components
  • known_observations parameter on GAMOptimiser and AI4RAGExperiment, allowing the optimizer to be pre-seeded with prior evaluation results so redundant evaluations are skipped
  • __hash__ method on BaseFoundationModel based on model_id
  • New functional test suite under tests/functional/ with end-to-end experiment coverage using mocked models

Changed

  • GAMOptSettings: removed lower-bound constraints on n_random_nodes and max_evals; both now accept 0, which is required for KFP pipeline component usage
  • Bumped llama-stack-client dependency from ~=0.5.0 to ~=0.6.0
  • Hybrid search disabled by default in the default search space due to upstream Llama Stack instability
  • BaseEventHandler payload TypedDicts enriched with full structured types (MetricCI, PatternScores, VectorStoreSettings, ChunkingSettings, etc.)
  • Tests reorganized into tests/unit/ and tests/functional/ subdirectories
  • Documentation and development workflow guides updated

Fixed

  • Fixed crash when the metadata field is absent from the models.list() response returned by Llama Stack

0.4.2

Added

  • Search space validation rule _rule_ranker_k_for_rrf_only ensuring ranker_k is only used with rrf ranker strategy
  • Vector store validation that ranker_k is only valid when ranker_strategy='rrf'

Changed

  • Removed numpy dependency from UnitxtEvaluator; replaced with pandas-native DataFrame.mask() and pd.isna()
  • Default search space: added 4096 to default chunk sizes
  • Default search space: simplified hybrid search defaults — removed normalized ranker strategy, reduced ranker_k values to (0, 60) and ranker_alpha values to (1, 0.5)

0.4.1

Changed

  • Updated hybrid search reranker API to match Llama Stack 0.5.x: ranker → reranker_type/reranker_params, k → impact_factor (for RRF strategy)
  • ranker_k parameter is now only passed for rrf ranker strategy (previously passed for all strategies)
  • Bumped llama-stack-client dependency from ~=0.4.2 to ~=0.5.0
  • Updated documentation and installation instructions to require Llama Stack >= 0.5.0

0.4.0

Added

  • Hybrid search support for ls_milvus vector store: new search_mode ("vector" or "hybrid"), ranker_strategy ("rrf", "weighted", "normalized"), ranker_k, and ranker_alpha parameters
  • Search space validation rules for hybrid search consistency (_rule_search_mode_ranker_consistency, _rule_ranker_alpha_for_weighted_only)
  • AI4RAGSearchSpace now accepts vector_store_type parameter to tailor default parameters and validation rules per vector store
  • Default search space for chroma now includes window retrieval method and window sizes (0, 1, 3, 5)
  • Embedding params are now serialized and included in indexing params passed to the vector store
  • __hash__ method added to BaseEmbeddingModel based on model_id
  • New documentation page for hybrid search (docs/user-guide/hybrid-search.md)

Changed

  • LlamaStackRAG renamed to SimpleRAG and moved from llama_stack_rag_template.py to simple_rag_template.py to reflect its provider-agnostic nature
  • Retriever now accepts and forwards search_mode, ranker_strategy, ranker_k, and ranker_alpha to the vector store
  • LSVectorStore.search() now accepts hybrid search parameters and validates their consistency
  • Event stream payload restructured: pattern_name, scores, execution_time, final_score, schema_version, and producer are now top-level fields; settings.retrieval includes search_mode and ranker details for hybrid mode
  • get_default_ai4rag_search_space_parameters() now accepts vector_store_type to control which parameters are included in the default search space

Fixed

  • Fixed incorrect logger call in LocalEventHandler.on_pattern_creation (missing format argument)
  • Added encoding="utf-8" to file open calls in LocalEventHandler

0.3.0

Added

  • Auto-detection of embedding model embedding_dimension and context_length when not explicitly provided
  • Model availability validation against the Llama Stack server during search space preparation
  • Search space validation rule ensuring chunk_size respects embedding model context length
  • New prepare_search_space_with_llama_stack utility for streamlined search space setup

Changed

  • Foundation model chat() API now accepts structured message list instead of separate system/user message strings
  • Default search space expanded with additional chunk_size (512) and chunk_overlap (128) values
  • Chunk size validation rule now requires chunk_size > 2 * chunk_overlap
  • LSEmbeddingParams refactored from TypedDict to @dataclass

Fixed

  • Embedding model backwards compatibility in vector store for both legacy dict and new dataclass params

0.2.1

Changed

  • Default optimizer is now GAMOptimizer
  • Default retrieval methods no longer contain window method, as this is not supported for ls_milvus at the moment
  • Parameter no longer requires to specify param_type. C type is used as default

Fixed

  • Bug in GAMOptimizer that unabled its usage (failing during deepcopy)

0.2.0

Added

  • Support for LocalEventHandler
  • Support for external models introduced via OpenAI client
  • CI/CD tooling
  • Added RAG pattern object streaming with and added it to results, so that pattern can be reused post experiment

Fixed

  • Documentation and README.md update
  • Updated samples
  • Updated docstrings

Changed

  • Loose required parameters for AI4RAGExperiment
  • Change "Optimiser" to "Optimizer" in all references

0.1.0

Added

  • Initial working implementation of ai4rag that can be used with llama-stack for RAG Template optimization

Version Numbering

ai4rag follows Semantic Versioning:

  • Major.Minor.Patch (e.g., 1.2.3)
  • Major: Breaking changes
  • Minor: New features, backward compatible
  • Patch: Bug fixes, backward compatible

Release Process

Releases are created by maintainers by tagging a commit on main.

See Development Workflow for detailed release procedures.


Stay Updated