Data Components¶
Data processing functions for the AutoRAG pipeline.
Discovery¶
documents_discovery ¶
Classes¶
DiscoveryResult dataclass ¶
DiscoveryResult(
bucket: str, prefixes: tuple[str, ...], documents: list[DocumentDescriptor], total_size_bytes: int, count: int
)
Outcome of a document discovery run.
Attributes:
-
bucket(str) –S3 bucket name.
-
prefixes(tuple[str, ...]) –S3 key prefixes used during listing. A single empty string means the whole bucket was listed.
-
documents(list[DocumentDescriptor]) –Discovered (and optionally sampled) documents, deduplicated by object key across all prefixes.
-
total_size_bytes(int) –Combined size of all discovered documents.
-
count(int) –Number of discovered documents.
Methods:¶
__post_init__ ¶
to_dict ¶
Serialise the result to a JSON-compatible dictionary.
Source code in ai4rag/utils/data/documents_discovery.py
save ¶
Write documents_descriptor.json into the given directory.
Parameters:
-
path(str | Path) –Directory where the descriptor file will be created. The directory is created if it does not exist.
-
filename(str, default:DOCUMENTS_DESCRIPTOR_FILENAME) –Name of the file to be used within the output directory.
Source code in ai4rag/utils/data/documents_discovery.py
DocumentDescriptor dataclass ¶
Metadata for a single document discovered in an S3 bucket.
Attributes:
-
key(str) –Full S3 object key. It both fetches the object and identifies the document downstream: it becomes the
DoclingDocumentname and is what benchmark data must reference. -
size_bytes(int) –Object size in bytes.
BenchmarkKeyError ¶
Bases: ValueError
Benchmark data references documents that are not part of the discovered corpus.
Functions:¶
discover_documents ¶
discover_documents(
bucket_name: str,
prefixes: str | list[str] | None = None,
test_data_doc_names: list[str] | None = None,
sampling_enabled: bool = True,
sampling_max_size_gb: float = SAMPLING_MAX_SIZE_GB,
supported_extensions: set[str] | None = None,
validate_test_data_keys: bool = True,
s3_client: Any | None = None,
) -> DiscoveryResult
Discover documents across one or more bucket locations and optionally sample them.
Lists objects under every entry of prefixes, merges the results into a single corpus deduplicated by object key, filters by file extension, and applies size-based sampling when enabled. The sampling budget is shared by the whole union, not applied per prefix. Documents referenced by test_data_doc_names are prioritized during sampling so that benchmark-relevant files are always included when the budget permits.
Parameters:
-
bucket_name(str) –S3-compatible bucket name.
-
prefixes(str | list[str] | None, default:None) –Object-key prefixes to narrow the listing. A bare string is treated as a one-element list.
None, an empty list, or a list containing an empty string lists the whole bucket. Overlapping prefixes are safe: objects matched by more than one are kept once. -
test_data_doc_names(list[str] | None, default:None) –Keys of documents referenced by the benchmark test data. Each is matched against the full object key, or -- when the name resolves to exactly one document -- against a bare file name. Matched documents are sorted first so that sampling picks them before other files.
-
sampling_enabled(bool, default:True) –When
True, only documents up to sampling_max_size_gb total are returned. -
sampling_max_size_gb(float, default:1.0) –Maximum cumulative size (in gigabytes) when sampling is enabled.
-
supported_extensions(set[str] | None, default:None) –File extensions to accept. Defaults to :data:
~ai4rag.utils.data.constants.SUPPORTED_EXTENSIONS. -
validate_test_data_keys(bool, default:True) –When
True, every entry of test_data_doc_names must identify exactly one discovered document, otherwise :class:BenchmarkKeyErroris raised. Set toFalseto downgrade the failure to a warning. -
s3_client(Any | None, default:None) –Pre-configured
boto3S3 client. WhenNone, one is created via :func:ai4rag.utils.clients.s3.create_s3_client.
Returns:
-
DiscoveryResult–Discovery outcome with document metadata.
Raises:
-
RuntimeError–If no supported documents are found under any of the prefixes.
-
BenchmarkKeyError–If a benchmark key matches no discovered document, or matches more than one, while validate_test_data_keys is enabled.
-
ValueError–If sampling produces an empty selection.
Source code in ai4rag/utils/data/documents_discovery.py
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 | |
Text Extraction¶
PDF extraction uses Docling's PyPdfiumDocumentBackend to preserve text that the native parser can emit as NUL characters. OCR and table processing remain controlled by DoclingExtractionConfig.
prepare_extraction() selects OCR and audio models from discovered document types before extraction. It returns a PreparedExtraction with config, artifacts_path, needs_ocr, and needs_asr. Connected preparation can download pinned models; disconnected preparation validates local bundles. See Text Extraction for an example.
text_extraction ¶
Classes¶
PreparedExtraction dataclass ¶
PreparedExtraction(config: DoclingExtractionConfig, artifacts_path: str | None, needs_ocr: bool, needs_asr: bool)
Validated model settings for one discovered corpus.
Attributes:
-
config(DoclingExtractionConfig) –Configuration to pass to :func:
extract_text. -
artifacts_path(str | None) –Local Docling artifacts directory, when one is configured.
-
needs_ocr(bool) –Whether the selected documents include a PDF or image.
-
needs_asr(bool) –Whether the selected documents include audio.
DoclingExtractionConfig dataclass ¶
DoclingExtractionConfig(
do_table_structure: bool = False,
do_ocr: bool = False,
ocr_lang: tuple[str, ...] = DEFAULT_OCR_LANG,
ocr_det_model_path: str | None = None,
ocr_cls_model_path: str | None = None,
ocr_rec_model_path: str | None = None,
ocr_rec_keys_path: str | None = None,
asr_model_path: str | None = None,
device: str = "cpu",
)
Docling converter settings shared with extraction worker processes.
An instance of this config is the single knob callers (e.g. pipelines-components) use to control conversion behaviour. It is constructed once and passed to :func:extract_text, which forwards it unchanged to every worker process so each worker builds an identically configured DocumentConverter. Being frozen makes it hashable and safe to ship across the spawn process boundary.
Attributes:
-
do_table_structure(bool) –What: run Docling's TableFormer to reconstruct rows/columns from the detected PDF layout. Why: table structure parsing is comparatively expensive, so it stays off by default and is opted into only when the corpus contains tables worth reconstructing.
-
do_ocr(bool) –What: run RapidOCR on pages Docling flags as needing OCR (scanned PDFs, images). Why: born-digital documents already carry a text layer, so OCR is off by default to avoid the runtime cost; enable it for scanned or image inputs.
-
ocr_lang(tuple[str, ...]) –What: the RapidOCR language selection (e.g.
("english",)or("english", "chinese")). Why: there is no auto-detection -- this value picks which bundled model set is loaded. Latin-script languages map to the English models; only Chinese switches to the Chinese models (see :func:_rapidocr_artifacts_rel_paths). Ignored whendo_ocrisFalse. Explicit OCR model paths take precedence over bundled wheel models, followed by validatedDOCLING_ARTIFACTS_PATHartifacts. Without configured artifacts, connected environments can use Docling's normal model downloader. -
ocr_det_model_path(str | None) –What/Why: optional path to a custom RapidOCR text-detection ONNX model, for disconnected clusters or specialised model sets that differ from the bundled defaults.
-
ocr_cls_model_path(str | None) –What/Why: optional path to a custom RapidOCR angle-classification ONNX model (same rationale as
ocr_det_model_path). -
ocr_rec_model_path(str | None) –What/Why: optional path to a custom RapidOCR text-recognition ONNX model (same rationale as
ocr_det_model_path). -
ocr_rec_keys_path(str | None) –What/Why: optional path to the character-keys dictionary matching the custom recognition model; required when the recognition model uses a non-default character set.
-
asr_model_path(str | None) –Directory containing the approved local Hugging Face Transformers Whisper model. When
None, the audio pipeline resolvesHF_MODEL_DIRin its worker process. Audio extraction fails closed when neither location provides a complete local model. ASR runs on CPU independently ofdeviceand never downloads model weights. -
device(str) –Device for Docling's PDF, image, DOCX, and PPTX pipelines. Defaults to
"cpu"so existing extraction runs keep their device selection. CUDA OCR requires a GPU-enabled ONNX Runtime.
ExtractionResult dataclass ¶
Outcome of a text extraction run.
Attributes:
-
processed_count(int) –Number of documents successfully extracted.
-
total_documents(int) –Total number of input documents.
-
error_count(int) –Number of documents that failed during download or extraction.
Functions:¶
prepare_extraction ¶
prepare_extraction(
documents: list[dict],
language: str | None = None,
*,
disconnected: bool = False,
docling_artifacts_path: str | None = None,
asr_model_path: str | None = None
) -> PreparedExtraction
Select and validate extraction models for discovered documents.
Connected runs download the pinned PP-OCRv4 files when no local artifacts directory is configured. Disconnected runs require those files locally. Audio uses a local Whisper model when configured. Connected runs download the pinned Whisper snapshot when no local model is configured. Text-only runs require neither model bundle.
Parameters:
-
documents(list[dict]) –Selected document descriptors containing a
key. -
language(str | None, default:None) –Document language or pattern language code. Chinese selects the Chinese OCR bundle; every other value selects English.
-
disconnected(bool, default:False) –Whether downloads are unavailable.
-
docling_artifacts_path(str | None, default:None) –Local Docling artifacts directory. Defaults to
DOCLING_ARTIFACTS_PATH. -
asr_model_path(str | None, default:None) –Local Whisper directory. Defaults to
HF_MODEL_DIRwhen set.
Returns:
-
PreparedExtraction–The validated configuration and local artifacts path for extraction.
Source code in ai4rag/utils/data/text_extraction.py
extract_text ¶
extract_text(
documents: list[dict],
bucket: str,
output_dir: str | Path,
s3_endpoint: str | None = None,
s3_access_key: str | None = None,
s3_secret_key: str | None = None,
s3_region: str | None = None,
ssl_cert_path: str | None = None,
error_tolerance: float | None = None,
max_extraction_workers: int | None = None,
docling_artifacts_path: str | None = None,
docling_config: DoclingExtractionConfig | None = None,
s3_client: Any | None = None,
) -> ExtractionResult
Download documents from S3 and extract text using Docling.
Each input document is downloaded from S3, converted to a :class:DoclingDocument via the Docling library, and persisted as a JSON file in output_dir. Conversion runs in a separate process pool (multiprocess library, "spawn" context) while downloads happen concurrently in a thread pool.
Parameters:
-
documents(list[dict]) –List of document descriptor dicts, each with at least a
"key"and"size_bytes"entry (as produced by :func:~ai4rag.utils.data.documents_discovery.discover_documents). The"key"entry also names the extracted document. -
bucket(str) –S3-compatible bucket name.
-
output_dir(str | Path) –Local directory where DoclingDocument JSON files are written.
-
s3_endpoint(str | None, default:None) –S3-compatible endpoint URL. Falls back to
AWS_S3_ENDPOINT. -
s3_access_key(str | None, default:None) –AWS access key. Falls back to
AWS_ACCESS_KEY_ID. -
s3_secret_key(str | None, default:None) –AWS secret key. Falls back to
AWS_SECRET_ACCESS_KEY. -
s3_region(str | None, default:None) –AWS region. Falls back to
AWS_DEFAULT_REGION. -
ssl_cert_path(str | None, default:None) –Optional path to a PEM CA bundle used to verify the S3 endpoint. When provided, it takes precedence over
AWS_CA_BUNDLE. The file must be readable; certificate validation remains enabled. -
error_tolerance(float | None, default:None) –Fraction of documents (0.0--1.0) allowed to fail.
Nonemeans zero tolerance. -
max_extraction_workers(int | None, default:None) –Number of parallel worker processes. Defaults to
min(max(1, cpu_count // 2), 8). -
docling_artifacts_path(str | None, default:None) –Path to pre-downloaded Docling model artifacts for offline use. Falls back to
DOCLING_ARTIFACTS_PATHenvironment variable. -
docling_config(DoclingExtractionConfig | None, default:None) –Ready :class:
DoclingExtractionConfigcontrolling table-structure, OCR, and device selection. Callers (e.g.pipelines-components) construct it once and pass it here; it is forwarded unchanged to every worker process.None(default) usesDoclingExtractionConfig()-- table structure and OCR both disabled. See :class:DoclingExtractionConfigfor the per-fieldwhat/whyand for howocr_langmaps to bundled OCR models. -
s3_client(Any | None, default:None) –Pre-configured S3 client used for every download. When omitted, one client is created from the S3 credentials supplied to this function or from the environment. The client is shared only by download threads; it is never sent to Docling worker processes.
Returns:
-
ExtractionResult–Summary of the extraction run.
Raises:
-
RuntimeError–If CUDA OCR lacks an ONNX Runtime GPU provider, or the error count exceeds the allowed tolerance.
Source code in ai4rag/utils/data/text_extraction.py
317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 | |
Local Whisper ASR¶
Audio extraction uses a local Transformers-format Whisper model configured through DoclingExtractionConfig.asr_model_path or HF_MODEL_DIR. See Audio Transcription for the required artifacts and migration from openai-whisper checkpoints.
local_hf_asr_pipeline ¶
Offline Hugging Face Whisper backend for Docling audio extraction.
Classes¶
LocalHuggingFaceAsrPipelineOptions ¶
Bases: AsrPipelineOptions
Options for :class:LocalHuggingFaceAsrPipeline.
Attributes:
-
asr_model_path(str | None) –Directory containing an approved Hugging Face Transformers Whisper model.
Noneis permitted while constructing a converter so non-audio runs do not require audio model artifacts; audio conversion then fails before any remote lookup is attempted.
LocalHuggingFaceAsrPipeline ¶
Bases: AsrPipeline
Docling ASR pipeline that loads only a local Transformers model.
The native Docling ASR initializer creates an openai-whisper model, whose checkpoint format differs from the approved Transformers modelcar. This pipeline intentionally initializes :class:BasePipeline directly and lazily loads the local model on the first audio document.
Source code in ai4rag/utils/data/local_hf_asr_pipeline.py
Test Data Loading¶
test_data_loader ¶
Classes¶
TestDataResult dataclass ¶
Outcome of loading (and optionally sampling) benchmark test data.
Attributes:
-
data(list[dict]) –Benchmark records, each containing question, correct_answers, and correct_answer_document_keys.
-
record_count(int) –Number of records in
data. -
sampled(bool) –Trueif the data was randomly sampled down.
TestDataLoaderError ¶
Bases: Exception
Raised when test data cannot be loaded or validated.
Functions:¶
load_test_data ¶
load_test_data(
bucket_name: str, key: str, benchmark_sample_size: int = BENCHMARK_SAMPLE_SIZE, s3_client: Any | None = None
) -> TestDataResult
Download benchmark test data from S3 and optionally sample it.
Parameters:
-
bucket_name(str) –S3-compatible bucket containing the test data file.
-
key(str) –Full S3 object key to the JSON test data file.
-
benchmark_sample_size(int, default:25) –Maximum number of records to keep. When the dataset exceeds this limit a reproducible random sample is drawn (seed 42). Set to
0to disable sampling and keep all records. -
s3_client(Any | None, default:None) –Pre-configured
boto3S3 client. WhenNone, one is created via :func:ai4rag.utils.clients.s3.create_s3_client.
Returns:
-
TestDataResult–Loaded (and optionally sampled) benchmark data.
Raises:
-
FileNotFoundError–If the object does not exist in S3.
-
TestDataLoaderError–If the file is not valid JSON or the records have an unexpected structure.