Changelog¶
All notable changes to ai4rag will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
0.26.0¶
Changed¶
- BREAKING CHANGE: Experiment — generated pattern names use alphabetic suffixes (
PatternA,PatternB, …,PatternZ,PatternAA) instead of numeric suffixes; warm-start names retain-warm-start, and the best warm-start pattern is published asPatternA - Dependencies — upgraded the core and
text-extractionextra'sdocling-slimrequirement from~=2.121.0to~=2.145.0for upstream security fixes; removed version bounds oncertifi - Dependencies — refreshed the lockfile with certifi 2026.10.23, Pillow 13.0.0, and Transformers 5.22.1 across platforms for security fixes, alongside filelock 4.2.6, Tornado 6.6, and pypdf 6.21.4
Fixed¶
- Experiment — logs the original name and score of the best warm-start pattern before publishing it under its final name
- GAM optimizer — excludes invalid and non-finite scores from warm-start success counts, coverage, surrogate training, and final selection; numerical surrogate-fit failures skip the affected iteration instead of aborting optimization
- Vector store — restored RRF and weighted hybrid search with Milvus Lite by disabling unsupported sparse-query term dropping; remote Milvus keeps its existing setting
- Agentic RAG — requests one tool call per model response and, when a provider returns multiple calls, retains only the first eligible offered call before storing history or executing tools; batches without an eligible call fail clearly, and budget finalization remains enforced
Removed¶
- BREAKING CHANGE: Assets generator — removed
ai4rag.assets_generatorand its bundled notebook, leaderboard, and starter-kit templates without a compatibility shim; generation now lives inkfp_components.components.training.autorag.shared.assets_generatorin pipelines-components - OpenShell gateway — removed
scripts/setup-openshell-gateway.shfrom ai4rag; gateway setup is maintained with the relocated assets generator in pipelines-components
0.25.4¶
Changed¶
- Assets generator — generated starter kits run the RHOAI OpenShell CLI through Podman and build a Python 3.12 runtime on the Red Hat sandbox base; CLI and sandbox image selection follows the installed RHOAI CSV, with registry fallbacks
- OpenShell gateway — setup provisions the downstream Red Hat OpenShell 3.6 gateway with mTLS authentication and RHOAI image references; an explicit
--resetoption supports migration from the incompatible upstream 0.0.99 gateway database - Dependencies — added an explicit
certifi>=2026.7.22,<2027dependency for public CA roots used alongside mounted MaaS certificates
Fixed¶
- Text extraction — local Whisper ASR uses native timestamp-aware long-form generation instead of fixed 30-second pipeline windows; multilingual models explicitly transcribe the source language while retaining automatic detection unless the model config specifies a language
- MaaS TLS — add mounted
SSL_CERT_DIR/ca.crtbundles to public trust whenca_bundle,MAAS_CA_BUNDLE, andSSL_CERT_FILEare unset; defaults remain unchanged when the mount is absent - MaaS TLS — preserve hashed certificate-directory trust when adding plain bundles and retain configured CA trust in synchronous and asynchronous agent chat clients
- Assets generator — OpenShell starter kits route Milvus gRPC through a local CONNECT proxy, retain native database tunneling, and use the cluster API endpoint with a CA bundle checked against that endpoint for service-account authentication
- Assets generator — OpenShell deployment respects the gateway namespace, handles Kyverno pull-secret conflicts with runtime-image mirroring and sandbox retries, and improves build completion checks and agent startup recovery
- Assets generator —
make refresh-certsregisters the cluster gateway and refreshes mTLS credentials; sandbox creation checks gateway connectivity before replacing an existing sandbox and reports recovery instructions when unavailable
0.25.3¶
Changed¶
- Evaluator — increased the default RAGAS judge completion limit from 1,024 to 2,048 tokens to reduce truncated faithfulness JSON and missing scores for longer answers
- Chunker — reduced
LangChainChunker's default chunk size and the maximumchunk_sizesvalue accepted by MaaS search-space preparation from 2,048 to 1,024 tokens; default chunk-size choices are now 512 and 1,024 - Assets generator — moved one-time OpenShell gateway setup to the repository's
scripts/setup-openshell-gateway.sh; generated starter kits require an existing gateway registered asopenshift
Fixed¶
- Evaluator — confirmed RAGAS context-overflow requests skip retries with the identical prompt, while other failures retain existing retry handling; incomplete evaluations log question IDs and missing metric names
- Evaluator — RAGAS metric selection now checks the evaluator as well as the metric name, avoiding collisions with metrics from other evaluators
- Experiment — result ranking places patterns with no final score after scored patterns instead of failing when comparing scores with
None
Removed¶
- Assets generator — removed the
setup-gatewayandteardown-gatewayMakefile targets from generated starter kits
0.25.2¶
Fixed¶
- Text extraction — corrected guidance for unsupported version-1 M4A
chnlchannel-layout boxes to recommend conversion to standard AAC-LC M4A, MP3, or WAV, removing the FFmpeg upgrade recommendation
0.25.1¶
Changed¶
- Dependencies — restored the base and
text-extractionextra'sdocling-slim~=2.121.0requirement for ai4rag and Kubeflow Pipelines compatibility, reverting the>=2.132.0,<3.0.0requirement introduced in 0.25.0 - Dependencies — refreshed the lockfile with Docling Core 2.96.0, Docling IBM Models 3.15.0, Docling Parse 7.21.0, and macOS-specific Transformers 5.8.1 and Tokenizers 0.22.2; updated pypdf from 6.20.0 to 6.21.1 and GitPython from 3.2.0 to 3.2.1
- Development workflow — replaced
@MichalSteczkowith@dryszkaamong the default pull-request review owners
Fixed¶
- Text extraction — unsupported version-1 M4A
chnlchannel-layout boxes now produce an actionable error recommending conversion to standard AAC-LC M4A, MP3, or WAV, or a runtime upgrade to FFmpeg 8.1.2 or newer
0.25.0¶
Added¶
- Text extraction — new
prepare_extraction()helper returns aPreparedExtractionconfiguration based on discovered file types, downloads pinned PP-OCRv4 and Whisper Tiny models for connected runs when local paths are unset, and validates local bundles for disconnected runs; text-only corpora need neither bundle - Agentic RAG — failed agent invocations emit a
RAG_FAILURE_TRACEJSON record with a diagnostic ID, model/tool call counts, retrieval counts, and bounded recent events across context-compression attempts, excluding prompts, document content, tool arguments, and exception messages - Retrieval —
LLMDocumentCompressor.compress()accepts an optional keyword-onlyconfigfor invocation callbacks, including tracing compression model calls
Changed¶
- Indexing notebooks — MaaS and knowledge-graph indexing default to connected mode and support disconnected Run All execution through
RHOAI_NOTEBOOK_CLUSTER_MODEor the overridingNOTEBOOK_CLUSTER_MODE, with model preparation after document discovery - Indexing notebooks — the pattern language selects English or Chinese OCR automatically, and extraction uses one worker to reduce memory pressure on small workbenches
- Dependencies — raised the base and
text-extractionextra'sdocling-slimrequirement to>=2.132.0,<3.0.0, addressing PYSEC-2026-4208/4209/4210/4211/4213/4214/4216, CVE-2026-105744, and related upstream fixes for resource exhaustion, file access, plugin imports, and network trust bypasses - Dependencies — added
huggingface-hub>=0.30,<2to thetext-extractionextra and refreshed the lockfile, includingdocling-slim2.140.0,docling-ibm-models4.0.3 across platforms, Transformers 5.20.1, and PyTorch 2.13.0 - Assets generator — starter-kit documentation recommends the Red Hat OpenShell sandbox base image for production deployments
Fixed¶
- Agentic RAG — exhausted retrieval or model-call budgets force a final answer without tools, preventing recursion loops while preserving response-language instructions; providers returning tool calls after exhaustion raise a clear error
- Text extraction — local Whisper ASR decodes audio from seekable file paths with FFmpeg before transcription, fixing M4A files with trailing metadata that failed when piped through Transformers; missing FFmpeg, decoding failures, and empty audio now produce actionable errors
- Assets generator — Neo4j starter-kit connections tunnel native TCP through OpenShell's HTTP CONNECT proxy, extending the existing pgvector support
0.24.0¶
Added¶
- Search space — categorical
template_idand MaaStemplate_idsconstraints let vector-backend experiments compareagentic_ragandsequential_rag; Neo4j defaults toagentic_graph_rag - Agentic RAG — context-overflow responses reporting "maximum context length" trigger one retry with query-relevant document excerpts;
enable_context_compressiondefaults toTrueand can disable the fallback - Retrieval — new
LLMDocumentCompressorextracts verified verbatim excerpts while preserving source metadata, omits chunks with no relevant evidence, and limits each compression pass to 4,096 estimated tokens - Agentic RAG — conversation records include
context_compression_fallbackandcompressionstatistics for initial context and subsequent retrievals during the retry - Assets generator — generated inference notebooks and starter kits accept
settings.agentic.enable_context_compression, defaulting to enabled when absent
Changed¶
- BREAKING CHANGE: Experiment — template selection moves from the
rag_templateconstructor argument to each candidate'stemplate_id; results, evaluation caching, streamed patterns, notebooks, and starter kits retain the selected ID, while matching indexing settings still reuse collections - BREAKING CHANGE: RAG template —
SimpleRAGand itssimple_rag_templatemodule are renamed toSequentialRAGandsequential_rag_template; persistedsimple_ragIDs must becomesequential_rag - Search space — construction validates template and search-mode compatibility: vector backends accept
agentic_ragorsequential_ragwith vector/hybrid search, while Neo4j requiresagentic_graph_ragwith graph search - Models pre-selector — model pre-selection always uses
AgenticRAGwith temporary Milvus Lite, including sequential-only and graph searches; experiment pre-selection continues ranking byOVERALL_SCORE, with existing thresholds andskip_mpssupport
Fixed¶
- Assets generator — generated starter kits instantiate the selected template and retain effective Neo4j graph-retrieval settings through configuration and deployment
- Assets generator — OpenShell deployment preserves database CA certificates across shell boundaries, including PEM or base64-encoded pgvector CA input
- Assets generator — pgvector connections in OpenShell sandboxes tunnel native TCP through the sandbox's HTTP CONNECT proxy
Removed¶
- RAG template — removed
template_id_for_class(); useget_template_spec(template_id)to resolve a selected template
0.23.0¶
Added¶
- Experiment —
EvaluationResult.conversationsretains agent conversation records with each evaluated pattern, making them accessible through experiment results andto_dict() - Experiment — evaluation JSON records include
conversation_idlinks to full agent conversations when conversation and evaluation counts match; mismatched counts produce a warning and omit the links - Vector store — Neo4j
kg_extraction_config.max_completion_tokensaccepts a positive integer to control entity-extraction output independently of answer generation, defaulting to 1024 - Assets generator — generated starter kits support Neo4j OpenShell deployment, loading database credentials, validating required settings, and configuring egress from the Neo4j URI (default port 7687)
Changed¶
- Dependencies — the
text-extractionextra now includesrapidocr~=3.9.2andonnxruntime~=1.25.0, providing OCR dependencies through the standard installation - Dependencies — refreshed
uv.lockwith Transformers 5.8.1 → 5.10.1 and platform-specific Docling IBM Models versions (3.13.2 on macOS, 3.15.0 elsewhere) - Notebooks — the MaaS indexing notebook installs OCR dependencies through
ai4rag[text-extraction], replacing separate RapidOCR and ONNX Runtime installation commands - Development utilities —
FileStoredefaults PDF model inference to CPU and accepts a keyword-onlydeviceoverride; PDF caches use.pdfium.jsonso legacy cached extractions are regenerated
Fixed¶
- Experiment — pattern publication, including the selected warm-start pattern, reads conversations from its
EvaluationResult, keeping conversation artifacts attached to the result being published - Text extraction — production PDF extraction and the local
FileStorehelper use Docling's PDFium backend to recover text that the native parser emitted as NUL characters - Vector store — Neo4j entity extraction reduces default chunk concurrency from eight to two and uses a dedicated completion-token limit; collection reuse tracks that extraction limit rather than the answer-generation limit
- Assets generator — starter-kit deployment forwards
MILVUS_CA_CERT,PGVECTOR_CA_CERT, andNEO4J_CA_CERTto the sandbox and agent; Milvus deployment usesMILVUS_CA_CERTinstead of the obsoleteMILVUS_SERVER_CERTkey
0.22.0¶
Added¶
- Text extraction — added a local Hugging Face Transformers Whisper backend and
DoclingExtractionConfig.asr_model_path(falling back toHF_MODEL_DIR) for offline audio transcription, preserving segment timestamps in Docling documents - Vector store —
Neo4jConfig.ca_cert(NEO4J_CA_CERT) accepts inline PEM CA certificates for encrypted, verified connections to private or self-signed Neo4j endpoints - RAG template — added
RAGTemplateSpec,get_template_spec(), andtemplate_id_for_class()to resolve supported persisted template IDs (simple_rag,agentic_rag, andagentic_graph_rag) - Agentic RAG — inference responses now include a
conversationrecord with the question, initial retrieval, agent messages, and query-rewrite/retriever tool inputs and outputs;LocalEventHandlersavesconversations.jsonfor each published agentic pattern, including the selected warm-start pattern, andKFPEventHandlerincludes the records in its pattern output
Changed¶
- Experiment — published patterns now include
template_idso generated inference notebooks can reproduce the selected RAG template - BREAKING CHANGE: RAG template — pattern publication now requires a registered built-in template class; custom
BaseRAGTemplatesubclasses need registry support before they can publish patterns - BREAKING CHANGE: Experiment — Neo4j pattern payloads replace
settings.knowledge_graphwithsettings.graph_extraction, containing the extraction policy rather than a separate extraction-model block; notebook generation reads this policy orindexing.pipeline_spec.parameters.graph_extraction_config - BREAKING CHANGE: Text extraction — audio extraction now requires a complete local Transformers-format Whisper model through
asr_model_pathorHF_MODEL_DIR; the previous Docling/openai-whisper backend and runtime ASR model downloads are no longer used - BREAKING CHANGE: Vector store —
Neo4jConfigrejectsneo4j+ssc://andbolt+ssc://URIs that bypass certificate verification; use verified+stransport or supply a trusted private CA - Text extraction — OCR no longer unconditionally requires
DOCLING_ARTIFACTS_PATH: explicit OCR model paths and bundled wheel models work without it, and connected environments can use Docling's model downloader when offline artifacts are unset; configured offline artifacts are still validated - Vector store — Neo4j knowledge-graph extraction suppresses verbose GraphRAG logs containing document text, retains warnings and errors, reports extraction start/completion, and hides passwords and CA text from
Neo4jConfigrepresentations - Notebooks — the MaaS indexing workflow selects OCR and audio artifacts after document discovery, requires only the models needed by the selected corpus, and installs tested RapidOCR/ONNX Runtime versions
- Notebooks — knowledge-graph indexing now includes S3 discovery, text extraction, offline artifact checks, and notebook event-loop setup; graph inference includes guided setup and optional benchmark evaluation examples
- Dependencies — added
nest-asyncio~=1.6.0for notebook event-loop compatibility and refresheduv.lock, includingneo4j-graphrag1.19.0 → 1.22.0,docling-core2.96.0 → 2.101.1, andlangchain-openai1.6.7 → 1.7.0
Fixed¶
- Notebooks — standard and graph inference notebooks instantiate the pattern's selected template instead of always using
SimpleRAG; legacy patterns withouttemplate_idretainSimpleRAGwith a warning, and unsupported IDs raise a clear error - Notebook generation — normalized knowledge-graph notebook cell sources so generated notebooks contain complete executable cells
- Agentic RAG — corrected the invocation-state lock annotation to avoid Pydantic validation warnings on Python 3.12
- Vector store — HTTPS Milvus connections without an explicit
ca_certnow use the CAs loaded by Python's default SSL context, including configured CA bundles; an empty loaded trust store raises a clear configuration error - Vector store — updated the Neo4j text splitter for the newer GraphRAG chunk-iterator interface while preserving each canonical chunk's text and index
0.21.0¶
Added¶
- Agentic RAG — new
AgenticRAGRAG template (ai4rag.rag.template.AgenticRAG), a LangChain ReAct-style agent that can rewrite the search query and retrieve additional context across up tomax_retrieval_stepstool calls before answering, instead of a single fixed retrieval pass - Experiment —
AI4RAGExperimentandModelsPreSelectoraccept arag_templateconstructor argument (aBaseRAGTemplatesubclass) selecting which template evaluates each candidate pattern - Assets generator —
generate_starter_kit()(ai4rag.assets_generator.starter_kit) packages a published pattern'spattern.jsoninto a deployablestarter_kit.zip: a self-contained agentic-RAG microservice (LangChain agent, OpenShift/OpenShell container image, Makefile, auth wrapper,pyproject.toml/uv.lock) pre-filled with that pattern's generation, embedding, retrieval, and vector-store settings - Network — new
ai4rag.utils.networkmodule (ensure_safe_url,is_url_scheme_safe,is_local_or_cluster_host) centralizing the scheme/host trust policy enforced across MaaS, S3, Milvus, Neo4j, and pgvector - MaaS client —
create_maas_client()gains aca_bundleparameter (orMAAS_CA_BUNDLEenv var) to verify a MaaS endpoint against a private or self-signed CA - Vector store —
PGVectorConfiggainsca_cert(PGVECTOR_CA_CERTenv var) to verify a self-signed PostgreSQL server certificate - Dependencies —
langchainandlangchain-openaiare new core dependencies, poweringAgenticRAG
Changed¶
- BREAKING CHANGE: Vector store —
MilvusConfig.server_certis renamed toca_cert(MILVUS_SERVER_CERTenv var renamed toMILVUS_CA_CERT), for consistency with the newPGVectorConfig.ca_cert - BREAKING CHANGE: Security — MaaS, S3, and PGVectorStore client factories no longer retry with unverified TLS (
verify=False) after a certificate-verification failure; they now raise, directing the caller to setMAAS_CA_BUNDLE/AWS_CA_BUNDLE/ca_cert.MilvusConfig.uriandNeo4jConfig.urinow reject a plaintext scheme (http://,neo4j://,bolt://) against any host that isn't local (localhost/127.0.0.1/::1) or in-cluster (*.cluster.local);PGVectorStorerequests fully verified TLS for any non-local host.discover_documents()andload_test_data()no longer probe-and-fall-back to an unverified S3 client on an SSL error - Experiment — the default
rag_templateforAI4RAGExperimentandModelsPreSelectoris nowAgenticRAG(previously alwaysSimpleRAG); passrag_template=SimpleRAGto keep single-pass retrieval behavior - RAG template — enriched-message construction (
_build_enriched_user_message,_render_enriched_user_message,_build_rag_messages) moved fromSimpleRAGup toBaseRAGTemplate, now shared withAgenticRAG - Dependencies — bumped transitive
tornadodependency from 6.5.8 to 6.5.9; refresheduv.lock
Fixed¶
- Vector store — HTTPS Milvus connections without explicit
ca_certnow use the CA certificates loaded by Python's default SSL context, allowing configured mounted CA bundles to be used without a hardcoded gRPC certificate path; an empty default trust store raises a clear configuration error - Evaluator —
UnitxtEvaluator.evaluate()no longer crashes inside unitxt'sTokenOverlapmetric when a question's retrieval returns zero chunks; an emptycontextslist is substituted with a single blank reference before scoring, so the question scores zero instead of aborting the whole evaluation run
0.20.0¶
Added¶
- Vector store — new Neo4j backend (
Neo4jConfig,Neo4jGraphStore, selected viavector_store_type="neo4j") for graph-based RAG.add_documents()stores canonical chunks and, when a foundation model is supplied, runsSimpleKGPipelineto extract entities and relationships per chunk (kg_extraction_config={"mode": "constrained"}by default, or"free"withmax_entities_per_chunk/max_relationships_per_chunkcaps), linking them back viaFROM_CHUNK. The newsearch_mode="graph"fuses direct vector retrieval with graph-expanded retrieval (entity neighbors and relationship-hop traversal) via reciprocal-rank fusion, tunable throughNeo4jGraphRetrievalConfig(route_k,include_entity_neighbors,entity_neighbor_limit,entity_pivot_limit,entity_relationship_hops,relationship_neighbor_limit).neo4jandneo4j-graphragare new core dependencies - Experiment —
AI4RAGExperimentacceptskg_extraction_configandgraph_retrieval_config(mapping orNeo4jGraphRetrievalConfig) constructor arguments, applied whenever the resolvedsearch_modeis"graph"; the published pattern payload now carries aknowledge_graphblock (extraction model, model params, extraction config) alongside the existing indexing/retrieval settings - Search space —
AI4RAGSearchSpace,get_default_ai4rag_search_space_parameters(), andprepare_search_space_with_maas()take a newvector_store_typeparameter (default"milvus"); when"neo4j", the default search space fixeschunk_size/chunk_overlap(1024/0,64) andsearch_mode="graph", omitting the hybrid ranker parameters that Neo4j does not support - Notebooks — two new templates,
mass_creating_knowledge_graph_template.ipynbandmass_inference_knowledge_graph_template.ipynb, for running Neo4j GraphRAG indexing and inference at scale - Docs — new architecture page "Neo4j Indexing and Graph Search" documenting the indexing and dual-route retrieval flow, plus prerequisites (APOC, required
NEO4J_*env vars)
Changed¶
- Vector store —
validate_search_params()now takessupported_modesandstore_classkeyword arguments instead of hardcoding("vector", "hybrid"), so each backend validates against its own supported modes and raises a backend-named error for an unsupported one; Milvus and pgvector passsupported_modes=("vector", "hybrid")explicitly, so they are unaffected by the new"graph"mode - Notebook generation —
create_placeholder_mapping()emitsFM_PARAMS,GRAPH_RETRIEVAL_CONFIG, andKG_EXTRACTION_CONFIGplaceholders consumed by the new knowledge-graph notebook templates
Fixed¶
- Text extraction —
DoclingExtractionConfig.device(default"cpu") is now honored when building Docling's PDF, image, DOCX, and PPTX pipeline options; previouslyAcceleratorOptionswas hardcoded todevice="cpu", so a GPU-configured extraction run silently executed on CPU.extract_text()now also fails fast with aRuntimeErrorwhendo_ocr=Trueanddevicestarts with"cuda"but ONNX Runtime has no availableCUDAExecutionProvider, instead of discovering the missing GPU support deep inside a worker process
0.19.0¶
Added¶
- Chunker —
DoclingChunkernow preserves Docling ASR segment timing on audio/hybrid chunks, emittingaudio_start_secondsandaudio_end_secondsin chunk metadata as the min/max timing across every source segment a chunk spans.LangChainChunkerlogs a warning when it receives audio documents, since converting them to markdown discards the Docling timing provenance needed to populate these fields — useDoclingChunkerto retain them - Text extraction —
extract_text()gainsssl_cert_path(an explicit PEM CA bundle used to verify the S3 endpoint, taking precedence overAWS_CA_BUNDLE) ands3_client(inject a pre-configured client instead of havingextract_textbuild one from credentials or the environment) - S3 client —
create_s3_client()'sverifyparameter now also accepts a CA bundle path (bool | str, previouslyboolonly); its default (True) now resolvesAWS_CA_BUNDLEwhen set, where boto3 previously ignored that variable wheneververifywas passed as the literalTrue - Notebooks — the MaaS indexing template documents how to trust a private/self-signed CA on OpenShift AI via the
DSCInitialization(DSCI) trust bundle, and the S3 setup cell now uses the mounted workbench CA bundle (orAWS_CA_BUNDLE) to verify the S3 endpoint
Changed¶
- Text extraction — the S3 client used by an extraction run is now created once and shared across all download threads, instead of a fresh client being constructed per download
- Notebooks — the generated MaaS indexing template pins a tested
opencv-python-headless==4.14.0.94reinstall step for disconnected clusters, and checksdiscover_documents()'s signature before calling it, so a notebook generated against this release still runs against an ai4rag install predating theprefixesparameter (erroring clearly if more than one input location was configured) - BREAKING CHANGE: Text extraction / Notebooks —
DOCLING_ARTIFACTS_PATHis now mandatory wherever Docling needs model artifacts:extract_text()raisesFileNotFoundErrorwhenever OCR is requested (do_ocr=True) withoutDOCLING_ARTIFACTS_PATHset — this check runs before customocr_*_model_paths are even considered, so setting those does not exempt you from it — and the generated MaaS indexing notebook now raises at startup if it's unset. Both previously fell back to Docling's runtime model downloader / an "online mode" message. Bake Docling artifacts into the workbench image and set the environment variable before upgrading
Fixed¶
- RAG optimization component — the warm-start phase now evaluates only the configured optimization metric for each candidate pattern instead of recomputing every configured metric; the remaining metrics are computed and merged in only for the best warm-start candidate, once it is selected and published as a pattern. Avoids redundant evaluator calls (including LLM-judge metrics) for warm-start candidates that are discarded, without changing the scores of the pattern that's ultimately published
Removed¶
- BREAKING CHANGE: Text extraction — removed the automatic insecure retry that caught
SSLErrorduring S3 downloads and silently re-attempted with certificate verification disabled (verify=False). Configure a trusted CA viaAWS_CA_BUNDLEorextract_text(ssl_cert_path=...)instead
0.18.0¶
Added¶
- RAG optimization component — GAM-based optimization gains a warm-start phase:
GAMOptSettings.warm_start_strategy("random"by default, or"greedy"/"balanced"viafields_to_balance) controls how the initial random evaluations are chosen before GAM iterations begin, auto-adjusting the warm-start size (with a logged notice) when the configuredn_random_nodesis too small to guarantee full coverage of categorical values. A newGAMOptSettings.max_iterationssetting caps how many evaluated patterns are retained and published, independent of themax_evalsevaluation budget - Document discovery — benchmark keys are now validated against the discovered corpus. Each
test_data_doc_namesentry must identify exactly one document: an exact object-key match always wins, and a bare file name is accepted only when it resolves to a single document. Anything else raises the newBenchmarkKeyError(aValueErrorsubclass, exported fromai4rag.utils.data) listing the offending keys, the locations searched, and the expected key format. Previously acorrect_answer_document_keysentry that matched no ingested object — a prefix-relative key, say — simply retrieved nothing, leaving the question ungrounded and the scores quietly wrong. Passvalidate_test_data_keys=Falseto downgrade the failure to a warning. A benchmark document that is discovered but dropped by the sampling budget is reported separately, also as a warning
Changed¶
- BREAKING CHANGE: Document discovery —
discover_documents()now ingests several bucket locations at once: theprefix: strparameter is replaced byprefixes: str | list[str] | None, andDiscoveryResult.prefixbyDiscoveryResult.prefixes: tuple[str, ...](to_dict()anddocuments_descriptor.jsonemit"prefixes"accordingly). A bare string is still accepted and coerced to a one-element list, and omitting the argument still lists the whole bucket, so single-location callers only need to rename the keyword. Every prefix is listed and merged into one corpus deduplicated by object key — overlapping selections such asdocs/anddocs/manuals/are safe — and thesampling_max_size_gbbudget applies to that union rather than to each location separately - Document discovery — listings are now paginated.
list_objects_v2returns at most 1000 keys per call, so a location holding more than that was silently truncated; with several locations sharing one budget the truncation would have starved the later prefixes entirely - BREAKING CHANGE: Asset generation —
create_placeholder_mapping()andgenerate_notebook_from_template()takeinput_data_keys: list[str]in place ofinput_data_key: str, and the indexing notebook template'sINPUT_DATA_KEYplaceholder becomesINPUT_DATA_KEYS, rendered as a Python list literal. The generatedindexing.ipynbtherefore rediscovers every location the pipeline ingested, not just the first - BREAKING CHANGE: Event handler — the pattern payload's settings block field
vector_store_bindingis renamed tostore_binding(backend-agnostic naming). Code readingon_pattern_creation()payloads must switch to the new field name - RAG optimization component — only the best successful warm-start candidate is published as a pattern (as
Pattern1), feeding into the subsequent GAM iterations; the other warm-start evaluations remain internal to the optimizer and are not streamed individually. GAM training now fits factor terms for categorical parameters and spline terms for numeric ones (previously every parameter was label-encoded and modeled with a spline), improving prediction quality once warm-start coverage is enabled
Fixed¶
- Notebooks — fixed the MaaS indexing notebook template: corrected the table of contents/appendix links, removed a duplicate "Download HuggingFace Models" step (renumbering the subsequent steps), and cleaned up incorrect configuration descriptions
0.17.0¶
Added¶
- Vector store — Milvus Lite, the embedded, zero-server mode of the Milvus backend, is now the recommended local/zero-config replacement for the removed Chroma store: use the new
MilvusLiteConfig(db_path="./ai4rag.db")(orMilvusLiteConfig()for the default path). Unlike Chroma, Milvus Lite supports hybrid (dense + BM25) search, though it computes BM25 statistics segment-locally rather than corpus-wide, so hybrid-search ranking fidelity may not transfer exactly to a production Milvus server.pymilvus[milvus-lite]is now a core dependency, so no extra installation step is needed - Vector store — new
temporary_milvus_lite_store()helper (ai4rag.rag.vector_store.local_store) providing a disposable, file-backed Milvus Lite store for internal throwaway indexes (model pre-selection, judge calibration), replacing the previous ephemeral in-memory Chroma store used for the same purpose - Notebooks — the MaaS indexing notebook template gains a "Prerequisites for Disconnected Clusters" section (validation cells, configuration examples, download instructions) for pre-downloading Docling and HuggingFace artifacts before running on an air-gapped cluster; the README documents the pre-download/transfer workflow (
DOCLING_ARTIFACTS_PATH,HF_HOME,HF_HUB_OFFLINE)
Changed¶
- Vector store — the Milvus config was split into
MilvusConfig(remote server / Zilliz Cloud only, which now validates thaturiis anhttp(s)://URL and raisesValueErrorotherwise) and a newMilvusLiteConfig(embedded, local only, configured viadb_path, which conversely rejectshttp(s)://values). Previously, a singleMilvusConfigselected between a remote server and embedded Milvus Lite based on whetherurilooked like a URL or a local file path; a mistyped or unreachableMILVUS_URIcould therefore be silently interpreted as a local path and create an unintended throwaway local database. That silent fallback is no longer possible — a misconfigured server URI now fails loudly instead.ai4rag.rag.vector_storenow also exportsMilvusLiteConfig, and thevector_store_typesearch-space parameter accepts"milvus_lite"in addition to"milvus"and"pgvector". Breaking:get_vector_store()now requiresMilvusLiteConfig(notMilvusConfig) whenconfig.provider == "milvus_lite" - Search space — hybrid search parameters (
search_mode,ranker_strategy,ranker_k,ranker_alpha) are now unconditionally part of the default search space, since every remaining vector store backend supports hybrid search. Breaking:AI4RAGSearchSpace,prepare_search_space_with_maas(), andget_default_ai4rag_search_space_parameters()no longer accept avector_store_typeparameter - Event handler —
VectorStoreSettingspayload fieldsprovider_id/vector_store_idreplaced byprovider_type/collection_name. Breaking: code readingon_pattern_creation()payloads must switch to the new field names - Dependencies — refreshed
uv.lock(transitive dependency updates)
Fixed¶
- CI — the
publish-pypimaintainer check now reads the actor'srole_namefrom the GitHub API instead of the removedpermissionfield
Removed¶
- BREAKING CHANGE: Vector store — removed the ChromaDB vector store backend (
ChromaConfig,ChromaVectorStore), thechromadbdependency, and the now-unusedai4rag.utils.compat.ensure_sqlite3()compatibility shim, due to known security vulnerabilities in thechromadbpackage.ai4rag.rag.vector_storeno longer exportsChromaConfig; the"chroma"value forvector_store_typeis no longer accepted (only"milvus","milvus_lite", and"pgvector"are supported)
0.16.0¶
Added¶
- Document discovery / extraction —
.msg(Outlook email) files are now a supported extension for discovery and text extraction - RAG template —
SimpleRAG.chat(), a chat-completions-style entry point that RAG-enriches the last user turn of a conversation while passing prior history through untouched
Changed¶
- Benchmark data —
BenchmarkDatanow requirescorrect_answer_document_keysinstead ofcorrect_answer_document_ids; a document key is a document'sDoclingDocument.name, the identifier carried through chunking, indexing, and evaluation. Breaking:correct_answer_document_idsis rejected with a message explaining what a key is; evaluation results reportdocument_keyinstead ofdocument_id - Document discovery / extraction — documents extracted from object storage are now named by their full object key (prefix included) instead of the bare file name, so two documents with the same basename under different prefixes no longer collide on the output path and silently overwrite one another
- Assets generator — relocated from
ai4rag.utils.assets_generatortoai4rag.assets_generator, alongside other top-level packages instead of under the general-purpose utils namespace. Breaking:ai4rag.utils.assets_generatorno longer exists; import fromai4rag.assets_generatorinstead - RAG template —
BaseRAGTemplateandSimpleRAGnow only compose a retriever and a foundation model for retrieval-and-generation; index building is an upstream concern owned byai4rag.rag.vector_store. Breaking:chunker,embedding_model, andvector_storeconstructor arguments andbuild_indexhave been removed; build indexes viaai4rag.rag.vector_storedirectly before constructing a template - Dependencies — bumped
docling-slimfrom~=2.107.0to~=2.121.0(base andtext-extractionextras)
Fixed¶
- Notebooks — the indexing notebook template now declares the
text-extractionextras it requires - Search space preparation — MaaS foundation/embedding model validation failures now log the underlying root cause (status code, error code, and message extracted from the
openaiclient error, with traceback) instead of discarding it and logging only the model id
0.15.0¶
Changed¶
- Vector store (pgvector) —
PGVectorStorenow usesasyncpginstead ofpsycopgto talk to Postgres, so installing ai4rag no longer requires a libpq binding in any form (bundled, compiled, or system-provided); the public sync API (search/add_documents/clean_collection/close) is unchanged, backed internally by a dedicated per-instance background event loop. Breaking: database errors now surface asasyncpgexceptions (e.g.asyncpg.exceptions.PostgresConnectionError) instead ofpsycopgones - Vector store (pgvector) — hybrid search's dense and keyword queries now run concurrently instead of sequentially
- Vector store (pgvector) — retry-on-dropped-connection behavior, previously insert-only, now also covers search, keyword search, and
clean_collection - Vector store (pgvector) — hot-path queries (search, keyword search, insert) now enforce a 90s command timeout on both pool acquisition and query execution, so a connection silently killed by an idle middlebox fails fast instead of hanging on the OS's TCP retransmission timer; idle pooled connections are now recycled after 60s (was 300s)
- Dependencies — replaced
psycopg[binary,pool]withasyncpgas the pgvector backend client
Fixed¶
- Vector store (pgvector) — fixed a latent ordering bug where
register_vector()could look up thevectortype's OID beforeCREATE EXTENSION IF NOT EXISTS vectorhad run, on a fresh database - Vector store (pgvector) — fixed a race between
close()and in-flight queries that could raise a bareAttributeError;close()now drains in-flight calls before tearing down the event loop, and a store closed mid-call raises a clearRuntimeErrorinstead
Removed¶
- Vector store (pgvector) —
psycopg/psycopg_poolare no longer dependencies of ai4rag
0.14.0¶
Added¶
- Leaderboard —
build_leaderboard_html()gains anoptimization_metric_evaluatorparameter so the pinned leaderboard column resolves to the evaluator that drove optimization instead of always falling back to unitxt/custom when metric names collide across evaluators (e.g.faithfulness)
Changed¶
- Core — split
ai4rag.componentsintoai4rag.utils(pure, dependency-free business-logic helpers) andpipelines-components(KFP-only orchestration):ai4rag.components.data→ai4rag.utils.data,ai4rag.components.assets_generator→ai4rag.utils.assets_generator(NotebookCellandcreate_placeholder_mappingare no longer re-exported from the package__init__; import them from their submodules),ai4rag.components.utils.{s3,maas_client}grouped into a newai4rag.utils.clientspackage, andai4rag.components.utils.docling_iomoved toai4rag.utils.docling_io
Fixed¶
- Experiment —
AI4RAGExperimentnow only includes the unitxt default metric bundle (answer_correctness,faithfulness,context_correctness,overall_score) when a unitxt evaluator is actually configured, instead of always including it regardless of active evaluators
Removed¶
- RAG optimization component —
ai4rag.components.optimization(run_rag_optimization,OptimizationResult) removed entirely; RAG-optimization orchestration is now a KFP-only concern living inpipelines-components, built onai4rag.coreandai4rag.search_space - Components — the
ai4rag.componentspackage no longer exists; import fromai4rag.utilsinstead
0.13.0¶
Added¶
- Text extraction — optional RapidOCR via Docling, configured through a single
DoclingExtractionConfigpassed toextract_text(do_ocr,ocr_lang, custom ONNX model paths); OCR remains off by default - Document discovery / extraction — JPEG, PNG, and TIFF image extensions supported for OCR-capable ingestion
- Text extraction — audio ingestion (
.wav,.mp3,.m4a,.aac,.ogg,.flac) transcribed via Docling's ASR pipeline (Whisper), with automatic language detection
Changed¶
- Dependencies — split docling's heavy conversion stack (
torch,docling-ibm-models,rapidocr,whisper) out of the base install into a newtext-extractionextra; core RAG code (chunking, experiment, evaluator, templates) only needsdocling-slim[feat-chunking]. Thetestextra now depends onai4rag[text-extraction]so unit tests keep exercising the full conversion stack - Model pre-selection — default foundation/embedding model counts used by
ModelsPreSelectorextracted into a newPreSelectorConstants(ai4rag.utils.constants)
Fixed¶
- Text extraction / OCR — fail fast with bake instructions when
DOCLING_ARTIFACTS_PATHis set but RapidOCR ONNX models are missing; current PyPIrapidocrwheels no longer ship model files (bake viatmp/Containerfile.autorag-dev) - Vector store (pgvector) — embedding models whose dimension exceeds pgvector's 2000-dimension HNSW/IVFFlat index cap are no longer rejected at construction;
PGVectorStorenow skips building the HNSW index above the limit and logs a warning, falling back to an exact sequential scan (storage, keyword search, and hybrid fusion are unaffected) - Vector store (pgvector) — the connection pool and backing table are now created lazily on the first
add_documents()/search call, guarded by double-checked locking, instead of during__init__; avoids idle pooled connections accumulating while the (potentially slow) embedding step runs before the first insert - Model access —
create_maas_client()now normalizesbase_urlby stripping any trailing slash and appending/v1if missing, so both host-only and fully-qualified endpoint URLs work (previously the URL was used verbatim, requiring the exact/v1-suffixed endpoint)
Removed¶
- Vector store —
ChromaVectorStore,MilvusVectorStore, andPGVectorStoreare no longer re-exported fromai4rag.rag.vector_store; import each directly from its backend module instead (ai4rag.rag.vector_store.chroma,.milvus,.pgvector). This keeps each backend's client library (chromadb,pymilvus,psycopg) — and its own import-time requirements, notably chromadb's minimum sqlite3 version — isolated to callers that actually use that backend.BaseVectorStore,ChromaConfig/MilvusConfig/PGVectorConfig,get_vector_store,get_vector_store_config, andget_vector_store_env_varsare unaffected and remain importable fromai4rag.rag.vector_store - Internal utilities — removed the unused
ai4rag.utils.validatorsmodule
0.12.0¶
Added¶
- Vector store — direct backend clients for Chroma, Milvus, and PostgreSQL/pgvector (
ChromaVectorStore,MilvusVectorStore,PGVectorStore), each selected via a typed, frozen config dataclass (ChromaConfig,MilvusConfig,PGVectorConfig) passed as a singlevector_store_config - Vector store —
rerankermodule implementing RRF and weighted fusion for hybrid search - Vector store —
BaseVectorStorenow supportsclose()and the context-manager protocol; each optimization trial scopes its store in awithblock, so connections and pools are no longer leaked per trial - Vector store (pgvector) — connection pooling via
psycopg_pool.ConnectionPoolwith a configurablePGVectorConfig.pool_max_size(default 10);AI4RAGExperimentsizes the pool frominference_max_threadsso it tracks real query concurrency - Evaluator — optional
RagasEvaluator(with RAGAS adapter classesAI4RAGRagasLLM/AI4RAGRagasEmbeddings) enabling RAGAS-based metrics;ragasis now a regular dependency - RAG optimization component —
llm_judge_modeselector (base/ragas/all/none) onrun_rag_optimization()to choose which LLM-as-a-Judge evaluators run - Evaluator —
build_aggregate_metric()shared helper onBaseEvaluatorfor constructing aggregate metric payloads - Search space preparation —
build_search_space_report()andserialize_model()inai4rag.search_space.prepare, co-locating the model↔spec round-trip (serialize_model()is the write mirror of the model restore path) - Model access —
create_maas_client()and shared model discovery/restore helpersget_foundation_models()/get_embedding_models()inai4rag.search_space.prepare.models, accepting either bare model ids (discovery) or serialized report specs (restore) - Assets generator —
get_vector_store_config()/get_vector_store_env_vars()factories that build a backend config from a provider discriminator and expose each backend's required environment variables for documentation - Dependencies — added
openaias the model-access SDK (replacingogx-client), pluschromadb,pymilvus,pgvector, andpsycopg[binary,pool]for the direct vector-store clients
Changed¶
- Model provider — replaced the OGX integration with any OpenAI-compatible endpoint; the shipped integration targets OpenShift AI Models-as-a-Service (MaaS), which serves listing, chat, and embeddings from a single endpoint
- Vector store —
get_vector_store()andAI4RAGExperimentnow take a singlevector_store_configand dispatch onconfig.provider, replacing thevector_store_typestring plus the OGXvector_ioprovider id - Vector store — collection-name resolution centralized in
BaseVectorStore, enforcing a mandatoryai4ragprefix as the cross-backend isolation guard - Vector store — hybrid-search reranking parameter renamed
impact_factor→k - Search space — default
vector_store_typechanged fromogxtomilvus; the default Chroma search space no longer includes thewindowretrieval method - Search space preparation — renamed
prepare_search_space_with_ogxtoprepare_search_space_with_maas, now accepting anopenai.OpenAIclient. Because MaaSmodels.list()carries no metadata (model type, embedding dimension, context length), the payload must declare foundation and embedding model IDs explicitly; embedding dimension and context length are auto-detected at construction time - Model ids — model ids are used verbatim, exactly as
models.list()reports them (including any/characters); there is no more model-specific URL derivation or id stripping - Client factory — replaced
create_ogx_clientwithcreate_maas_client, a single client that serves listing, chat, and embeddings for every model at the one MaaS endpoint - Notebook templates — renamed the generated
ogx_{indexing,inference}templates tomaas_{indexing,inference}, each building a singleOpenAIclient fromMAAS_BASE_URL/MAAS_API_KEYand reusing it for every model; the inference notebook now also rebuilds the pattern's detected generation language and passes it toOpenAIFoundationModel, so answers keep the benchmark's language - Experiment / evaluator —
metricsandoptimization_metricnow requireRAGMetricinstances selected fromMetricsand reject bare metric-name strings, which are ambiguous now that a name (e.g.faithfulness) is shared across the unitxt and RAGAS evaluators - Model helpers — model-instantiation helpers moved to
ai4rag.search_space.prepare.models, removing the components↔search_space coupling - Search space report — model pre-selection decoupled from report building into an explicit
ModelsPreSelectorstep;SearchSpaceReportslimmed to the search-space dict and no longer carriesselected_modelsor a per-modelbase_url, andpattern.jsonno longer carriesbase_url - Leaderboard — aggregate scores are keyed by a collision-free key (unitxt and custom metrics keep their bare name; other evaluators are prefixed, e.g.
ragas_faithfulness), so colliding metric names each get their own column instead of overwriting one another
Fixed¶
- Vector store (Milvus) — forced
consistency_level="Strong"on vector/hybrid search so a query immediately following anadd_documents()upsert can no longer race Milvus's default bounded-staleness read and return zero hits against a collection that does contain matching data - Vector store (pgvector) — corrected
inner_productscoring: the<#>operator returns the negative inner product, so the score is now derived by negation (cosine/l2/l1 keep1/dist), fixing an inverted ranking - Vector store (pgvector) — guarded lazy index creation with double-checked locking (plus a
UniqueViolationfallback) so concurrent search threads no longer race onCREATE INDEX - Experiment — an optimization metric that is produced but unscored (
Nonemean) is now recorded as a failed — not fatal — iteration; a genuinely absent metric still raises aRAGExperimentErrorwith an evaluator-qualified message - Components — added
db_secret_nameto the indexing pipeline params - Core —
ensure_ascii=Falsewhen JSON-dumping documents that may reach the end user, preserving non-ASCII characters - Benchmark data — reject
BenchmarkDatarecords with zero correct answers, preventing a downstream unitxtTokenOverlapcrash onmax()of an empty iterable - Experiment — benchmark JSON is now read with an explicit UTF-8 encoding
Removed¶
- OGX — removed all OGX support: the
ogx-clientdependency,OGXFoundationModel,OGXEmbeddingModel,OGXVectorStore,OGXModelParameters,OGXEmbeddingParams,create_ogx_client, theogx_utilsmodule, theogx_inference_base_urlhelper, and theOGX_CLIENT_BASE_URL/OGX_CLIENT_API_KEYenvironment variables (replaced byMAAS_BASE_URL/MAAS_API_KEY) - Assets generator — removed the OGX-only
pattern_builderandprompt_filtersmodules and thebuild_pattern_jsonexport; indexing-spec enrichment is now inlined - Search space preparation —
prepare_search_space_report()and thesearch_space_preparationmodule removed fromai4rag.components.optimization; build a search space withprepare_search_space_with_maas(), then callbuild_search_space_report()fromai4rag.search_space.prepare - Experiment —
EvaluationResultno longer carries arag_patternfield; a trial's vector store is closed once the trial finishes, so readpattern_name/scoresfromEvaluationResultinstead of calling.generate()on a previously returned pattern - Dependencies — removed
langchain-chroma; Chroma is now used directly viachromadb - Samples — removed the outdated
samples/run_ai4rag.ipynbnotebook
0.11.0¶
Added¶
- Text extraction — added support for 9 additional document formats (
.odt,.odp,.adoc,.tex,.epub,.eml,.qmd,.rmd,.xhtml) in document discovery and text extraction, alongside existing PDF, DOCX, PPTX, Markdown, HTML, and plain-text support
Changed¶
- Data component —
SUPPORTED_EXTENSIONSextracted into a sharedai4rag.components.data.constantsmodule, removing duplication between document discovery and text extraction - Dependencies — replaced the
doclingmeta-package withdocling-slim[standard,feat-chunking,format-opendocument], and dropped the standalonedocling-coredependency, now pulled in transitively via thefeat-chunkingextra
Fixed¶
- Notebooks — updated the
ogx_inference_template.ipynbtest-data-loading example to callai4rag.components.data.test_data_loader.load_test_data(), replacing a stale reference to the removedkfp_componentspipeline API
0.10.4¶
Fixed¶
- Experiment — fixed incorrect dictionary key
"method"used to check the chunking method when determining whether to include metadata; now uses the canonicalAI4RAGParamNames.CHUNKING_METHODconstant, ensuringinclude_metadatais correctly set for hybrid chunking during experiment execution
0.10.3¶
Fixed¶
- RAG optimization component —
GAMOptimizerinitial random phase now uses stratified sampling to guarantee that every unique value of each string-valued categorical parameter (e.g.search_mode,chunking_method) is evaluated at least once before GAM training begins, preventing biased exploration when the search space is skewed toward a dominant category; warm-start observations are accounted for so stratification does not waste early slots on already-covered values; a warning is emitted whenn_random_nodesis too small for full categorical coverage
0.10.2¶
Changed¶
- Dependencies — updated
ogx-clientdependency from~=1.1.0to~=1.2.0 - Data component —
ChunkingConstraints.METHODSchanged from mutable list to immutable tuple for correctness
Fixed¶
- Evaluator — hardened LLM-as-a-Judge JSON response parsing with lightweight repair for common malformed outputs (single-quoted JSON, markdown-fenced blocks, JSON embedded in surrounding prose); added explicit output format instructions to the judge prompt; separated LLM call failures from JSON parse failures with distinct warning messages and raw response logging
- Evaluator —
calculate_overall_score()now propagatesNonedirectly for unevaluated metrics instead of converting tofloat("nan") - Experiment — streamed pattern now includes the
include_metadatachunking field, ensuring metadata-aware chunking configurations are fully captured in pattern output
Removed¶
- Data component — removed
index_documents()function anddocuments_indexingmodule fromai4rag.components.data; the component was unused
0.10.1¶
Added¶
- OGX client — timeout fallback for embedding and chat requests: on
APITimeoutError, retries once with a 20-minute timeout and disabled client-level retries to accommodate slow CPU-deployed models
Changed¶
- Chunking —
AI4RAGChunknow carries a deterministicchunk_idfield (SHA-256 of document ID, sequence number, and text), replacing ad-hoc hash-based ID generation in vector stores - Chunking — hybrid chunking method now automatically includes document metadata during experiment execution
- Vector store —
ChromaVectorStoreandOGXVectorStorededuplication and chunk identification now use the deterministicAI4RAGChunk.chunk_idinstead of independent hash computations
Fixed¶
- Prompt templates — partially reverted default RAG prompt templates for all model families (Granite, Llama, Mistral, OpenAI, default) to use model-native prompting patterns, removing shared instruction boilerplate
- Prompt filters — decoupled
HPO_CITATION_FRAGMENTSfrom internal_RAG_CITATION_INSTRUCTIONconstant, using inline string literals for portability
0.10.0¶
Added¶
- Evaluator —
LLMaJEvaluatorfor LLM-as-a-Judge evaluation, scoringanswer_relevanceon a 1–5 rubric with structured JSON output and bootstrap confidence intervals; scores are normalized to [0.0, 1.0] - Evaluator — automatic judge model selection via
select_judge_model()— when multiple generation models are available, a calibration round scores each candidate on a benchmark subset and picks the one with the highest spread-and-stability score - Evaluator —
RAGMetricfrozen dataclass andMetricsregistry replacing raw metric-name strings throughout the evaluator and experiment APIs - Evaluator —
custom_metricsmodule withcalculate_overall_score()— computes a cross-metric mean as a built-in custom metric (overall_score) - Experiment — multi-evaluator dispatch:
AI4RAGExperimentnow accepts anevaluatorslist and routes each metric to the evaluator matching itsEVALUATOR_TYPE - Experiment —
metricsparameter onAI4RAGExperimentfor explicit control over which metrics are evaluated; defaults are derived from configured evaluators when omitted - RAG optimization component —
indexing_pipeline_paramsparameter onrun_rag_optimization()for enrichingpattern.jsonwith indexing pipeline settings - OGX client utilities —
ogx_inference_base_url()helper for building/v1-suffixed inference endpoint URLs
Changed¶
- Evaluator —
BaseEvaluator.evaluate_metrics()signature now acceptsSequence[RAGMetric]and returns a structuredEvaluationMetricsResultTypedDict (waslist[str]→dict) - Evaluator —
UnitxtEvaluatorupdated to work with the newRAGMetric-based metric dispatch and returnEvaluationMetricsResult - Event handler —
BaseEventHandler.on_pattern_creation()payload and evaluation results now fully typed viaPatternPayloadandEvaluationRecordTypedDicts with nested structured types (AggregateMetricPayload,VectorStoreSettings,ChunkingSettings,RetrievalSettings,GenerationSettings) - Experiment —
optimization_metricparameter acceptsRAGMetric | str(wasstronly); default changed fromfaithfulnesstooverall_score - Experiment — evaluation results internally use structured
EvaluationMetricsResultthroughout the scoring, streaming, and caching pipeline - RAG optimization component — default optimization metric changed from
faithfulnesstooverall_score; supported metrics now includeoverall_score - RAG optimization component — judge model selection and LLM-as-a-Judge evaluation are now automatically enabled during
run_rag_optimization() - RAG optimization component — artefact generation extracted into
_generate_output_artifacts()for clearer separation of concerns - Dependencies — refreshed
uv.lockand sorted dependency declarations inpyproject.toml
0.9.3¶
Added¶
- Search space preparation —
chunk_overlapsparameter onprepare_search_space_report()andprepare_search_space_with_ogx()for constraining the chunk-overlap dimension of the search space (e.g.[0, 128]), with Pydantic range validation againstChunkingConstraintsbounds
Changed¶
- RAG optimization component — renamed
max_threadskeyword argument toinference_max_threadsonrun_rag_optimization()to align withprepare_search_space_report()naming - Search space report — verbose representation in
prepare_search_space_report()now derives values from valid (rule-filtered) combinations instead of raw parameter lists, ensuring the report reflects only reachable configurations - Search space constraint validation — removed automatic deduplication of
chunking_methodsandchunk_sizesfromAI4RAGConstraintsvalidators — duplicates are no longer silently collapsed - Dependencies — removed the
langchainmeta-package; only the needed sub-packages (langchain-chroma,langchain-text-splitters) are retained
0.9.2¶
Added¶
chunk_sizesparameter onprepare_search_space_report()for constraining the chunk-size dimension of the search space (e.g.[256, 512])
Changed¶
- Chunking constraint validation (
chunking_methods,chunk_sizes) now happens via Pydantic before any I/O, providing clearer error messages with exact field paths - Minimum allowed chunk size lowered from 512 to 128, enabling finer-grained chunking strategies
- Duplicate values in
chunking_methodsandchunk_sizesare now automatically deduplicated
0.9.1¶
Fixed¶
- LLM-based language detection now uses JSON-schema structured output (
response_format) instead of fragile regex extraction, ensuring reliable ISO 639-1 code parsing from model responses
0.9.0¶
Added¶
- Multilingual support —
Languagedataclass andlanguageparameter onBaseFoundationModelfor language-aware prompt template generation ai4rag.search_space.prepare.language_detectionmodule for LLM-based benchmark language detection with ISO 639-1 mappingCharApproxTokenizer— lightweight, model-agnostic tokenizer approximating token count via character ratio, replacing thetiktokendependencyai4rag.components.assets_generator.prompt_filtersmodule for filtering OGX runtime injection duplicates from HPO prompt templates during Responses API export- Progressive chunk truncation in
OGXEmbeddingModel— oversized chunks are truncated before embedding instead of failing max_threadsparameter onrun_rag_optimization()for controlling concurrent benchmark evaluation threads
Changed¶
- Breaking:
BaseFoundationModelconstructor accepts alanguageparameter;user_message_textandcontext_template_textare now validated properties instead ofRAGPromptTemplateStringdescriptors - Breaking:
BaseFoundationModel.chat()signature now accepts**kwargs - Breaking: Search space report format changed from YAML to JSON —
search_space_preparationandrag_templates_optimizationno longer usepyyaml - Breaking:
OGXEmbeddingModel._embed_text()renamed to_call_embedding_api() - Replaced
tiktokendependency with character-based token approximation across all chunkers - Upgraded
doclingdependency to2.107.0and adapted to new API - Prompt template system refactored —
RAGPromptTemplateStringdescriptor replaced with setter-based validation viavalidate_prompt_templates_placeholders() - Responses API payload aligned with previous chat/completion format
- Component functions (
text_extraction,search_space_preparation,rag_templates_optimization) made more customizable with additional parameters - Removed
mikedependency and documentation versioning from CI/CD
Fixed¶
- Chunks exceeding embedding model context length now truncated with progressive margins instead of causing API failures
random_stateparameter properly wired throughBaseOptimizertoGAMOptimizerandRandomOptimizerfor deterministic optimization runs- Removed unnecessary
doclinginstall cell from indexing notebook template
Removed¶
tiktokendependency — replaced byCharApproxTokenizerRAGPromptTemplateStringdescriptor class fromai4rag.rag.foundation_models.utils- YAML serialization support for model instances in search space reports
0.8.1¶
Changed¶
- Downgraded
docling-coredependency from~=2.84.0to~=2.83.0to resolve compatibility issues
0.8.0¶
Added¶
ai4rag.componentspackage — pipeline step business logic consolidated frompipelines-components, usable standalone or within KFP wrapperscomponents.data:discover_documents(),extract_text(),index_documents(),load_test_data()components.optimization:prepare_search_space_report(),run_rag_optimization(),detect_benchmark_language()- Shared utilities:
create_s3_client(),create_ogx_client(),load_docling_documents()
ai4rag.components.assets_generator— notebook, leaderboard, and pattern artefact generationNotebook/NotebookCellclasses withimportlib.resourcestemplate loadingbuild_leaderboard_html()for styled HTML leaderboard generationbuild_pattern_json()for RAG pattern definition buildinggenerate_notebook_from_template()for notebook rendering from templates- Bundled notebook and script templates as package data
ai4rag.utils.compat.ensure_sqlite3()— centralized pysqlite3 patch for RHEL 9 / older sqliteboto3andmultiprocessadded as core dependenciesdoclingpromoted from dev-only to core dependency- Pipeline Components user guide and API reference documentation
Changed¶
- Breaking: Event handler
PatternPayloadschema restructured —pattern_name→name,execution_time→duration_seconds,vector_store→vector_store_binding,datasource_type→provider_id,collection_name→vector_store_id - Breaking: Removed
schema_versionandproducerfields fromPatternPayload - Breaking: Removed
distance_metricfromEmbeddingSettingsand indexing params VectorStoreSettingsandEmbeddingSettingsTypedDicts now usetotal=Falsefor optional fieldsogx-clientdependency updated from~=1.0.0to~=1.1.0docling-coredependency updated from~=2.74.1to~=2.84.0- Hybrid search payload now conditionally includes
ranker_k(only forrrfstrategy) andranker_alpha(only forweightedstrategy) - Vector store provider type is now resolved dynamically from the OGX server when available
max_combinationsfrom search space now included in pattern creation payload
Removed¶
ai4rag.search_space.src.modelsmodule (FoundationModelsandEmbeddingModelsenum classes) — model IDs are now plain strings throughout the codebaseEmbeddingModels.get_distance_metric()utility — distance metric is no longer tracked
0.7.0¶
Added¶
DoclingChunker— structure-aware, token-aware chunker wrapping docling'sHybridChunker, operating directly onDoclingDocumentobjects and preserving document hierarchy (headings, tables, figures) during chunkingAI4RAGChunk— framework-agnostic chunk dataclass replacing langchainDocumentas the pipeline's canonical chunk representation"hybrid"chunking method in the default search space, enablingDoclingChunkeralongside the existing"recursive"method- Search space validation rule
_rule_chunk_overlap_for_chunking_methodenforcing chunker-specific overlap constraints (hybridrequires overlap = 0;recursiverequires overlap > 0) - Minimum context length validation for embedding models — models with
context_lengthbelow 700 tokens are now rejected during initialization with a descriptive error
Changed¶
- Breaking:
BaseChunker.split_documents()now acceptsSequence[DoclingDocument]and returnslist[AI4RAGChunk](wasSequence[Document]→list[Document]) - Breaking:
BaseVectorStore.add_documents()now acceptsSequence[AI4RAGChunk](wasSequence[Document]) - Breaking:
BaseVectorStore.search()now returnslist[AI4RAGChunk](waslist[dict]) LangChainChunkerupdated to acceptDoclingDocumentinput (converts to markdown internally) and returnAI4RAGChunkoutputChromaVectorStoreandOGXVectorStoreupdated to work withAI4RAGChunk, with internal conversions handled transparentlyOGXEmbeddingModelembedding batch size reduced from 2048 to 1024- Added
docling-coreas a project dependency
0.6.3¶
Fixed¶
- Duplicate chunk IDs in
OGXVectorStore.add_documents()now detected and skipped with a warning, preventing insertion failures when documents produce identical chunk hashes
0.6.2¶
Added¶
VectorStoreInitializationErrorexception for clearer diagnostics when vector store creation or retrieval fails
Fixed¶
- Vector store initialization errors are now caught and wrapped with contextual information (embedding model ID, vector store provider) instead of propagating raw exceptions
- Simplified experiment error summary — removed redundant log-file reminder suffix from error messages
0.6.1¶
Changed¶
- Upgraded
ogx-clientdependency from~=0.8.0to~=1.0.0 - Updated documentation to require OGX Server >= 1.0.0
0.6.0¶
Added¶
uvpackage manager support as an alternative topipfor dependency management and development workflowsAGENTS.mdfile with AI agent guidelines for contributing to the project
Changed¶
- Rebranded all Llama Stack integrations to OGX:
LSEmbeddingModel→OGXEmbeddingModel,LSFoundationModel→OGXFoundationModel,LSVectorStore→OGXVectorStore,prepare_search_space_with_llama_stack→prepare_search_space_with_ogx(and all related classes, modules, and configuration keys) - Replaced
llama-stack-clientdependency withogx-client - Improved logging during model selection and validation — clearer messages when models are filtered or skipped
- Updated CI/CD workflows to use
uvfor dependency installation and test execution - Updated all documentation to reflect the Llama Stack → OGX rebranding
Removed¶
- OpenAI model wrappers (
OpenAIEmbeddingModel,OpenAIFoundationModel) dev_utils/run_experiment_with_openai_models.pyexample script- Llama Stack example notebooks from
dev_utils/llama_stack_examples/
0.5.5¶
Changed¶
- Improved error messages when models registered in Llama Stack do not respond — errors now distinguish between "not registered" and "registered but not responding" models, with actionable guidance
- Improved pre-selector logging to show total model counts before selection and selected counts after
- Added logging of selected foundation and embedding models during search space preparation
- Removed
pydantic-based payload validation overhead inprepare_search_space_with_llama_stack— replaced with direct dataclass instantiation - Removed
validation_error_decodermodule andpydanticdependency from search space preparation
0.5.4¶
Fixed¶
- Fixed chunk ID collisions in
LSVectorStore— chunks from the same document no longer share the samechunk_id; IDs are now derived from chunk content via hashing - Fixed
chunk_metadatainLSVectorStoreto only containdocument_id, with full document metadata preserved in a separatemetadatafield
0.5.3¶
Changed¶
- Bumped
llama-stack-clientdependency from~=0.6.0to~=0.7.1 - Updated documentation and installation instructions to require Llama Stack >= 0.7.0
0.5.2¶
Changed¶
- Vector store type now supports any Llama Stack provider via the
ls_<provider_id>pattern (e.g.,ls_milvus,ls_qdrant), instead of only the hardcodedls_milvus vector_store_typeparameter onAI4RAGExperimentchanged fromLiteral["chroma", "ls_milvus"]tostrfor flexibility
Fixed¶
- Fixed
provider_idextraction inget_vector_store— previously hardcoded to"milvus", now correctly derived from thels_<provider_id>vector store type
0.5.1¶
Added¶
- Batch processing for Llama Stack embeddings (2048 chunk limit) and vector store document insertion, preventing failures with large document sets
Changed¶
- Hybrid search re-enabled by default in the
ls_milvusdefault search space (was disabled in 0.5.0 due to upstream instability) - Default chunk sizes narrowed from
(512, 1024, 2048, 4096)to(1024, 2048)and overlaps from(128, 256, 512)to(128, 256)for faster optimization - Chroma vector store batch size simplified to a fixed default of 2048 instead of querying client internals
0.5.0¶
Added¶
KFPEventHandler: new event handler for Kubeflow Pipelines (KFP) integration, enabling experiment progress tracking inside KFP pipeline componentsknown_observationsparameter onGAMOptimiserandAI4RAGExperiment, allowing the optimizer to be pre-seeded with prior evaluation results so redundant evaluations are skipped__hash__method onBaseFoundationModelbased onmodel_id- New functional test suite under
tests/functional/with end-to-end experiment coverage using mocked models
Changed¶
GAMOptSettings: removed lower-bound constraints onn_random_nodesandmax_evals; both now accept0, which is required for KFP pipeline component usage- Bumped
llama-stack-clientdependency from~=0.5.0to~=0.6.0 - Hybrid search disabled by default in the default search space due to upstream Llama Stack instability
BaseEventHandlerpayload TypedDicts enriched with full structured types (MetricCI,PatternScores,VectorStoreSettings,ChunkingSettings, etc.)- Tests reorganized into
tests/unit/andtests/functional/subdirectories - Documentation and development workflow guides updated
Fixed¶
- Fixed crash when the
metadatafield is absent from themodels.list()response returned by Llama Stack
0.4.2¶
Added¶
- Search space validation rule
_rule_ranker_k_for_rrf_onlyensuringranker_kis only used withrrfranker strategy - Vector store validation that
ranker_kis only valid whenranker_strategy='rrf'
Changed¶
- Removed
numpydependency fromUnitxtEvaluator; replaced with pandas-nativeDataFrame.mask()andpd.isna() - Default search space: added
4096to default chunk sizes - Default search space: simplified hybrid search defaults — removed
normalizedranker strategy, reducedranker_kvalues to(0, 60)andranker_alphavalues to(1, 0.5)
0.4.1¶
Changed¶
- Updated hybrid search reranker API to match Llama Stack 0.5.x:
ranker→reranker_type/reranker_params,k→impact_factor(for RRF strategy) ranker_kparameter is now only passed forrrfranker strategy (previously passed for all strategies)- Bumped
llama-stack-clientdependency from~=0.4.2to~=0.5.0 - Updated documentation and installation instructions to require Llama Stack >= 0.5.0
0.4.0¶
Added¶
- Hybrid search support for
ls_milvusvector store: newsearch_mode("vector" or "hybrid"),ranker_strategy("rrf", "weighted", "normalized"),ranker_k, andranker_alphaparameters - Search space validation rules for hybrid search consistency (
_rule_search_mode_ranker_consistency,_rule_ranker_alpha_for_weighted_only) AI4RAGSearchSpacenow acceptsvector_store_typeparameter to tailor default parameters and validation rules per vector store- Default search space for
chromanow includeswindowretrieval method and window sizes (0, 1, 3, 5) - Embedding params are now serialized and included in indexing params passed to the vector store
__hash__method added toBaseEmbeddingModelbased onmodel_id- New documentation page for hybrid search (
docs/user-guide/hybrid-search.md)
Changed¶
LlamaStackRAGrenamed toSimpleRAGand moved fromllama_stack_rag_template.pytosimple_rag_template.pyto reflect its provider-agnostic natureRetrievernow accepts and forwardssearch_mode,ranker_strategy,ranker_k, andranker_alphato the vector storeLSVectorStore.search()now accepts hybrid search parameters and validates their consistency- Event stream payload restructured:
pattern_name,scores,execution_time,final_score,schema_version, andproducerare now top-level fields;settings.retrievalincludessearch_modeand ranker details for hybrid mode get_default_ai4rag_search_space_parameters()now acceptsvector_store_typeto control which parameters are included in the default search space
Fixed¶
- Fixed incorrect logger call in
LocalEventHandler.on_pattern_creation(missing format argument) - Added
encoding="utf-8"to file open calls inLocalEventHandler
0.3.0¶
Added¶
- Auto-detection of embedding model
embedding_dimensionandcontext_lengthwhen not explicitly provided - Model availability validation against the Llama Stack server during search space preparation
- Search space validation rule ensuring
chunk_sizerespects embedding model context length - New
prepare_search_space_with_llama_stackutility for streamlined search space setup
Changed¶
- Foundation model
chat()API now accepts structured message list instead of separate system/user message strings - Default search space expanded with additional
chunk_size(512) andchunk_overlap(128) values - Chunk size validation rule now requires
chunk_size > 2 * chunk_overlap LSEmbeddingParamsrefactored fromTypedDictto@dataclass
Fixed¶
- Embedding model backwards compatibility in vector store for both legacy dict and new dataclass params
0.2.1¶
Changed¶
- Default optimizer is now
GAMOptimizer - Default retrieval methods no longer contain
windowmethod, as this is not supported forls_milvusat the moment Parameterno longer requires to specifyparam_type.Ctype is used as default
Fixed¶
- Bug in
GAMOptimizerthat unabled its usage (failing during deepcopy)
0.2.0¶
Added¶
- Support for
LocalEventHandler - Support for external models introduced via
OpenAIclient - CI/CD tooling
- Added RAG pattern object streaming with and added it to results, so that pattern can be reused post experiment
Fixed¶
- Documentation and
README.mdupdate - Updated samples
- Updated docstrings
Changed¶
- Loose required parameters for
AI4RAGExperiment - Change "Optimiser" to "Optimizer" in all references
0.1.0¶
Added¶
- Initial working implementation of
ai4ragthat can be used withllama-stackfor RAG Template optimization
Version Numbering¶
ai4rag follows Semantic Versioning:
- Major.Minor.Patch (e.g., 1.2.3)
- Major: Breaking changes
- Minor: New features, backward compatible
- Patch: Bug fixes, backward compatible
Release Process¶
Releases are created by maintainers by tagging a commit on main.
See Development Workflow for detailed release procedures.
Stay Updated¶
- Watch the GitHub repository for releases
- Subscribe to release notifications
- Check the releases page for version history