Skip to content

Data Components

Data processing functions for the AutoRAG pipeline.

Discovery

documents_discovery

Classes

DiscoveryResult dataclass

DiscoveryResult(
    bucket: str, prefixes: tuple[str, ...], documents: list[DocumentDescriptor], total_size_bytes: int, count: int
)

Outcome of a document discovery run.

Attributes:

  • bucket (str) –

    S3 bucket name.

  • prefixes (tuple[str, ...]) –

    S3 key prefixes used during listing. A single empty string means the whole bucket was listed.

  • documents (list[DocumentDescriptor]) –

    Discovered (and optionally sampled) documents, deduplicated by object key across all prefixes.

  • total_size_bytes (int) –

    Combined size of all discovered documents.

  • count (int) –

    Number of discovered documents.

Methods:
__post_init__
__post_init__() -> None

Freeze the prefix collection even when callers provide a list.

Source code in ai4rag/utils/data/documents_discovery.py
def __post_init__(self) -> None:
    """Freeze the prefix collection even when callers provide a list."""
    object.__setattr__(self, "prefixes", tuple(self.prefixes))
to_dict
to_dict() -> dict

Serialise the result to a JSON-compatible dictionary.

Source code in ai4rag/utils/data/documents_discovery.py
def to_dict(self) -> dict:
    """Serialise the result to a JSON-compatible dictionary."""
    return {
        "bucket": self.bucket,
        "prefixes": list(self.prefixes),
        "documents": [{"key": d.key, "size_bytes": d.size_bytes} for d in self.documents],
        "total_size_bytes": self.total_size_bytes,
        "count": self.count,
    }
save
save(path: str | Path, filename: str = DOCUMENTS_DESCRIPTOR_FILENAME) -> None

Write documents_descriptor.json into the given directory.

Parameters:

  • path (str | Path) –

    Directory where the descriptor file will be created. The directory is created if it does not exist.

  • filename (str, default: DOCUMENTS_DESCRIPTOR_FILENAME ) –

    Name of the file to be used within the output directory.

Source code in ai4rag/utils/data/documents_discovery.py
def save(self, path: str | Path, filename: str = DOCUMENTS_DESCRIPTOR_FILENAME) -> None:
    """Write ``documents_descriptor.json`` into the given directory.

    Parameters
    ----------
    path : str | Path
        Directory where the descriptor file will be created. The
        directory is created if it does not exist.
    filename : str
        Name of the file to be used within the output directory.
    """
    out_dir = Path(path)
    out_dir.mkdir(parents=True, exist_ok=True)
    descriptor_path = out_dir / filename
    with open(descriptor_path, "w", encoding="utf-8") as fh:
        json.dump(self.to_dict(), fh, indent=2, ensure_ascii=False)
    _logger.info("Documents descriptor written to %s", descriptor_path)

DocumentDescriptor dataclass

DocumentDescriptor(key: str, size_bytes: int)

Metadata for a single document discovered in an S3 bucket.

Attributes:

  • key (str) –

    Full S3 object key. It both fetches the object and identifies the document downstream: it becomes the DoclingDocument name and is what benchmark data must reference.

  • size_bytes (int) –

    Object size in bytes.

BenchmarkKeyError

Bases: ValueError

Benchmark data references documents that are not part of the discovered corpus.

Functions:

discover_documents

discover_documents(
    bucket_name: str,
    prefixes: str | list[str] | None = None,
    test_data_doc_names: list[str] | None = None,
    sampling_enabled: bool = True,
    sampling_max_size_gb: float = SAMPLING_MAX_SIZE_GB,
    supported_extensions: set[str] | None = None,
    validate_test_data_keys: bool = True,
    s3_client: Any | None = None,
) -> DiscoveryResult

Discover documents across one or more bucket locations and optionally sample them.

Lists objects under every entry of prefixes, merges the results into a single corpus deduplicated by object key, filters by file extension, and applies size-based sampling when enabled. The sampling budget is shared by the whole union, not applied per prefix. Documents referenced by test_data_doc_names are prioritized during sampling so that benchmark-relevant files are always included when the budget permits.

Parameters:

  • bucket_name (str) –

    S3-compatible bucket name.

  • prefixes (str | list[str] | None, default: None ) –

    Object-key prefixes to narrow the listing. A bare string is treated as a one-element list. None, an empty list, or a list containing an empty string lists the whole bucket. Overlapping prefixes are safe: objects matched by more than one are kept once.

  • test_data_doc_names (list[str] | None, default: None ) –

    Keys of documents referenced by the benchmark test data. Each is matched against the full object key, or -- when the name resolves to exactly one document -- against a bare file name. Matched documents are sorted first so that sampling picks them before other files.

  • sampling_enabled (bool, default: True ) –

    When True, only documents up to sampling_max_size_gb total are returned.

  • sampling_max_size_gb (float, default: 1.0 ) –

    Maximum cumulative size (in gigabytes) when sampling is enabled.

  • supported_extensions (set[str] | None, default: None ) –

    File extensions to accept. Defaults to :data:~ai4rag.utils.data.constants.SUPPORTED_EXTENSIONS.

  • validate_test_data_keys (bool, default: True ) –

    When True, every entry of test_data_doc_names must identify exactly one discovered document, otherwise :class:BenchmarkKeyError is raised. Set to False to downgrade the failure to a warning.

  • s3_client (Any | None, default: None ) –

    Pre-configured boto3 S3 client. When None, one is created via :func:ai4rag.utils.clients.s3.create_s3_client.

Returns:

Raises:

  • RuntimeError –

    If no supported documents are found under any of the prefixes.

  • BenchmarkKeyError –

    If a benchmark key matches no discovered document, or matches more than one, while validate_test_data_keys is enabled.

  • ValueError –

    If sampling produces an empty selection.

Source code in ai4rag/utils/data/documents_discovery.py
def discover_documents(
    bucket_name: str,
    prefixes: str | list[str] | None = None,
    test_data_doc_names: list[str] | None = None,
    sampling_enabled: bool = True,
    sampling_max_size_gb: float = SAMPLING_MAX_SIZE_GB,
    supported_extensions: set[str] | None = None,
    validate_test_data_keys: bool = True,
    s3_client: Any | None = None,
) -> DiscoveryResult:
    """Discover documents across one or more bucket locations and optionally sample them.

    Lists objects under every entry of *prefixes*, merges the results into a
    single corpus deduplicated by object key, filters by file extension, and
    applies size-based sampling when enabled.  The sampling budget is shared by
    the whole union, not applied per prefix.  Documents referenced by
    ``test_data_doc_names`` are prioritized during sampling so that
    benchmark-relevant files are always included when the budget permits.

    Parameters
    ----------
    bucket_name : str
        S3-compatible bucket name.
    prefixes : str | list[str] | None, default=None
        Object-key prefixes to narrow the listing.  A bare string is treated as
        a one-element list.  ``None``, an empty list, or a list containing an
        empty string lists the whole bucket.  Overlapping prefixes are safe:
        objects matched by more than one are kept once.
    test_data_doc_names : list[str] | None, default=None
        Keys of documents referenced by the benchmark test data.  Each is
        matched against the full object key, or -- when the name resolves to
        exactly one document -- against a bare file name.  Matched documents
        are sorted first so that sampling picks them before other files.
    sampling_enabled : bool, default=True
        When ``True``, only documents up to *sampling_max_size_gb* total
        are returned.
    sampling_max_size_gb : float, default=1.0
        Maximum cumulative size (in gigabytes) when sampling is enabled.
    supported_extensions : set[str] | None, default=None
        File extensions to accept.  Defaults to
        :data:`~ai4rag.utils.data.constants.SUPPORTED_EXTENSIONS`.
    validate_test_data_keys : bool, default=True
        When ``True``, every entry of *test_data_doc_names* must identify
        exactly one discovered document, otherwise :class:`BenchmarkKeyError`
        is raised.  Set to ``False`` to downgrade the failure to a warning.
    s3_client : Any | None, default=None
        Pre-configured ``boto3`` S3 client.  When ``None``, one is created
        via :func:`ai4rag.utils.clients.s3.create_s3_client`.

    Returns
    -------
    DiscoveryResult
        Discovery outcome with document metadata.

    Raises
    ------
    RuntimeError
        If no supported documents are found under any of the prefixes.
    BenchmarkKeyError
        If a benchmark key matches no discovered document, or matches more
        than one, while *validate_test_data_keys* is enabled.
    ValueError
        If sampling produces an empty selection.
    """
    if supported_extensions is None:
        supported_extensions = set(SUPPORTED_EXTENSIONS)

    resolved_prefixes = _normalize_prefixes(prefixes)
    ext_tuple = tuple(supported_extensions)
    max_size_bytes = float(sampling_max_size_gb) * 1024**3 if sampling_enabled else float(inf)

    if s3_client is None:
        s3_client = create_s3_client()
    contents = _list_objects_union(s3_client, bucket_name, resolved_prefixes)
    supported_files = [c for c in contents if c["Key"].endswith(ext_tuple)]

    if not supported_files:
        raise RuntimeError(f"No supported documents found in {_location(bucket_name, resolved_prefixes)}.")

    test_keys: set[str] = set()
    if test_data_doc_names:
        test_keys = _resolve_test_data_keys(
            test_data_doc_names,
            supported_files,
            bucket_name=bucket_name,
            prefixes=resolved_prefixes,
            strict=validate_test_data_keys,
        )
        supported_files.sort(key=lambda c: c["Key"] not in test_keys)

    total_size = 0
    selected: list[DocumentDescriptor] = []
    for file_info in supported_files:
        size = file_info["Size"]
        if total_size + size > max_size_bytes:
            continue
        selected.append(DocumentDescriptor(key=file_info["Key"], size_bytes=size))
        total_size += size

    if not selected:
        raise ValueError(
            "No documents to process. Check that the bucket/prefixes are correct and contain supported files."
        )

    dropped = sorted(test_keys - {d.key for d in selected})
    if dropped:
        _logger.warning(
            "%d benchmark-referenced document(s) could not fit within the %.2f GB sampling budget and were skipped: %s",
            len(dropped),
            sampling_max_size_gb,
            ", ".join(dropped),
        )

    result = DiscoveryResult(
        bucket=bucket_name,
        prefixes=tuple(resolved_prefixes),
        documents=selected,
        total_size_bytes=total_size,
        count=len(selected),
    )
    _logger.info(
        "Discovered %d document(s) across %d location(s), total size %d bytes",
        result.count,
        len(resolved_prefixes),
        result.total_size_bytes,
    )
    return result

Text Extraction

PDF extraction uses Docling's PyPdfiumDocumentBackend to preserve text that the native parser can emit as NUL characters. OCR and table processing remain controlled by DoclingExtractionConfig.

prepare_extraction() selects OCR and audio models from discovered document types before extraction. It returns a PreparedExtraction with config, artifacts_path, needs_ocr, and needs_asr. Connected preparation can download pinned models; disconnected preparation validates local bundles. See Text Extraction for an example.

text_extraction

Classes

PreparedExtraction dataclass

PreparedExtraction(config: DoclingExtractionConfig, artifacts_path: str | None, needs_ocr: bool, needs_asr: bool)

Validated model settings for one discovered corpus.

Attributes:

  • config (DoclingExtractionConfig) –

    Configuration to pass to :func:extract_text.

  • artifacts_path (str | None) –

    Local Docling artifacts directory, when one is configured.

  • needs_ocr (bool) –

    Whether the selected documents include a PDF or image.

  • needs_asr (bool) –

    Whether the selected documents include audio.

DoclingExtractionConfig dataclass

DoclingExtractionConfig(
    do_table_structure: bool = False,
    do_ocr: bool = False,
    ocr_lang: tuple[str, ...] = DEFAULT_OCR_LANG,
    ocr_det_model_path: str | None = None,
    ocr_cls_model_path: str | None = None,
    ocr_rec_model_path: str | None = None,
    ocr_rec_keys_path: str | None = None,
    asr_model_path: str | None = None,
    device: str = "cpu",
)

Docling converter settings shared with extraction worker processes.

An instance of this config is the single knob callers (e.g. pipelines-components) use to control conversion behaviour. It is constructed once and passed to :func:extract_text, which forwards it unchanged to every worker process so each worker builds an identically configured DocumentConverter. Being frozen makes it hashable and safe to ship across the spawn process boundary.

Attributes:

  • do_table_structure (bool) –

    What: run Docling's TableFormer to reconstruct rows/columns from the detected PDF layout. Why: table structure parsing is comparatively expensive, so it stays off by default and is opted into only when the corpus contains tables worth reconstructing.

  • do_ocr (bool) –

    What: run RapidOCR on pages Docling flags as needing OCR (scanned PDFs, images). Why: born-digital documents already carry a text layer, so OCR is off by default to avoid the runtime cost; enable it for scanned or image inputs.

  • ocr_lang (tuple[str, ...]) –

    What: the RapidOCR language selection (e.g. ("english",) or ("english", "chinese")). Why: there is no auto-detection -- this value picks which bundled model set is loaded. Latin-script languages map to the English models; only Chinese switches to the Chinese models (see :func:_rapidocr_artifacts_rel_paths). Ignored when do_ocr is False. Explicit OCR model paths take precedence over bundled wheel models, followed by validated DOCLING_ARTIFACTS_PATH artifacts. Without configured artifacts, connected environments can use Docling's normal model downloader.

  • ocr_det_model_path (str | None) –

    What/Why: optional path to a custom RapidOCR text-detection ONNX model, for disconnected clusters or specialised model sets that differ from the bundled defaults.

  • ocr_cls_model_path (str | None) –

    What/Why: optional path to a custom RapidOCR angle-classification ONNX model (same rationale as ocr_det_model_path).

  • ocr_rec_model_path (str | None) –

    What/Why: optional path to a custom RapidOCR text-recognition ONNX model (same rationale as ocr_det_model_path).

  • ocr_rec_keys_path (str | None) –

    What/Why: optional path to the character-keys dictionary matching the custom recognition model; required when the recognition model uses a non-default character set.

  • asr_model_path (str | None) –

    Directory containing the approved local Hugging Face Transformers Whisper model. When None, the audio pipeline resolves HF_MODEL_DIR in its worker process. Audio extraction fails closed when neither location provides a complete local model. ASR runs on CPU independently of device and never downloads model weights.

  • device (str) –

    Device for Docling's PDF, image, DOCX, and PPTX pipelines. Defaults to "cpu" so existing extraction runs keep their device selection. CUDA OCR requires a GPU-enabled ONNX Runtime.

ExtractionResult dataclass

ExtractionResult(processed_count: int, total_documents: int, error_count: int)

Outcome of a text extraction run.

Attributes:

  • processed_count (int) –

    Number of documents successfully extracted.

  • total_documents (int) –

    Total number of input documents.

  • error_count (int) –

    Number of documents that failed during download or extraction.

Functions:

prepare_extraction

prepare_extraction(
    documents: list[dict],
    language: str | None = None,
    *,
    disconnected: bool = False,
    docling_artifacts_path: str | None = None,
    asr_model_path: str | None = None
) -> PreparedExtraction

Select and validate extraction models for discovered documents.

Connected runs download the pinned PP-OCRv4 files when no local artifacts directory is configured. Disconnected runs require those files locally. Audio uses a local Whisper model when configured. Connected runs download the pinned Whisper snapshot when no local model is configured. Text-only runs require neither model bundle.

Parameters:

  • documents (list[dict]) –

    Selected document descriptors containing a key.

  • language (str | None, default: None ) –

    Document language or pattern language code. Chinese selects the Chinese OCR bundle; every other value selects English.

  • disconnected (bool, default: False ) –

    Whether downloads are unavailable.

  • docling_artifacts_path (str | None, default: None ) –

    Local Docling artifacts directory. Defaults to DOCLING_ARTIFACTS_PATH.

  • asr_model_path (str | None, default: None ) –

    Local Whisper directory. Defaults to HF_MODEL_DIR when set.

Returns:

  • PreparedExtraction –

    The validated configuration and local artifacts path for extraction.

Source code in ai4rag/utils/data/text_extraction.py
def prepare_extraction(
    documents: list[dict],
    language: str | None = None,
    *,
    disconnected: bool = False,
    docling_artifacts_path: str | None = None,
    asr_model_path: str | None = None,
) -> PreparedExtraction:
    """Select and validate extraction models for discovered documents.

    Connected runs download the pinned PP-OCRv4 files when no local artifacts
    directory is configured. Disconnected runs require those files locally.
    Audio uses a local Whisper model when configured. Connected runs download
    the pinned Whisper snapshot when no local model is configured.
    Text-only runs require neither model bundle.

    Parameters
    ----------
    documents
        Selected document descriptors containing a ``key``.
    language
        Document language or pattern language code. Chinese selects the Chinese
        OCR bundle; every other value selects English.
    disconnected
        Whether downloads are unavailable.
    docling_artifacts_path
        Local Docling artifacts directory. Defaults to ``DOCLING_ARTIFACTS_PATH``.
    asr_model_path
        Local Whisper directory. Defaults to ``HF_MODEL_DIR`` when set.

    Returns
    -------
    PreparedExtraction
        The validated configuration and local artifacts path for extraction.
    """
    extensions = {Path(document["key"]).suffix.lower() for document in documents}
    needs_ocr = bool(extensions & OCR_EXTENSIONS)
    needs_asr = bool(extensions & AUDIO_EXTENSIONS)
    bundle = "chinese" if (language or "").strip().lower() in _CHINESE_OCR_LANGS else "english"
    model_paths, local_path = (
        _prepare_ocr_model_paths(bundle, disconnected, docling_artifacts_path) if needs_ocr else ({}, None)
    )
    local_asr_path = _prepare_asr_model_path(asr_model_path, disconnected) if needs_asr else None
    config = DoclingExtractionConfig(do_ocr=needs_ocr, ocr_lang=bundle, asr_model_path=local_asr_path, **model_paths)
    return PreparedExtraction(
        config=config,
        artifacts_path=local_path,
        needs_ocr=needs_ocr,
        needs_asr=needs_asr,
    )

extract_text

extract_text(
    documents: list[dict],
    bucket: str,
    output_dir: str | Path,
    s3_endpoint: str | None = None,
    s3_access_key: str | None = None,
    s3_secret_key: str | None = None,
    s3_region: str | None = None,
    ssl_cert_path: str | None = None,
    error_tolerance: float | None = None,
    max_extraction_workers: int | None = None,
    docling_artifacts_path: str | None = None,
    docling_config: DoclingExtractionConfig | None = None,
    s3_client: Any | None = None,
) -> ExtractionResult

Download documents from S3 and extract text using Docling.

Each input document is downloaded from S3, converted to a :class:DoclingDocument via the Docling library, and persisted as a JSON file in output_dir. Conversion runs in a separate process pool (multiprocess library, "spawn" context) while downloads happen concurrently in a thread pool.

Parameters:

  • documents (list[dict]) –

    List of document descriptor dicts, each with at least a "key" and "size_bytes" entry (as produced by :func:~ai4rag.utils.data.documents_discovery.discover_documents). The "key" entry also names the extracted document.

  • bucket (str) –

    S3-compatible bucket name.

  • output_dir (str | Path) –

    Local directory where DoclingDocument JSON files are written.

  • s3_endpoint (str | None, default: None ) –

    S3-compatible endpoint URL. Falls back to AWS_S3_ENDPOINT.

  • s3_access_key (str | None, default: None ) –

    AWS access key. Falls back to AWS_ACCESS_KEY_ID.

  • s3_secret_key (str | None, default: None ) –

    AWS secret key. Falls back to AWS_SECRET_ACCESS_KEY.

  • s3_region (str | None, default: None ) –

    AWS region. Falls back to AWS_DEFAULT_REGION.

  • ssl_cert_path (str | None, default: None ) –

    Optional path to a PEM CA bundle used to verify the S3 endpoint. When provided, it takes precedence over AWS_CA_BUNDLE. The file must be readable; certificate validation remains enabled.

  • error_tolerance (float | None, default: None ) –

    Fraction of documents (0.0--1.0) allowed to fail. None means zero tolerance.

  • max_extraction_workers (int | None, default: None ) –

    Number of parallel worker processes. Defaults to min(max(1, cpu_count // 2), 8).

  • docling_artifacts_path (str | None, default: None ) –

    Path to pre-downloaded Docling model artifacts for offline use. Falls back to DOCLING_ARTIFACTS_PATH environment variable.

  • docling_config (DoclingExtractionConfig | None, default: None ) –

    Ready :class:DoclingExtractionConfig controlling table-structure, OCR, and device selection. Callers (e.g. pipelines-components) construct it once and pass it here; it is forwarded unchanged to every worker process. None (default) uses DoclingExtractionConfig() -- table structure and OCR both disabled. See :class:DoclingExtractionConfig for the per-field what/why and for how ocr_lang maps to bundled OCR models.

  • s3_client (Any | None, default: None ) –

    Pre-configured S3 client used for every download. When omitted, one client is created from the S3 credentials supplied to this function or from the environment. The client is shared only by download threads; it is never sent to Docling worker processes.

Returns:

Raises:

  • RuntimeError –

    If CUDA OCR lacks an ONNX Runtime GPU provider, or the error count exceeds the allowed tolerance.

Source code in ai4rag/utils/data/text_extraction.py
def extract_text(  # pylint: disable=too-many-locals,too-many-arguments,too-many-positional-arguments,too-many-statements
    documents: list[dict],
    bucket: str,
    output_dir: str | Path,
    s3_endpoint: str | None = None,
    s3_access_key: str | None = None,
    s3_secret_key: str | None = None,
    s3_region: str | None = None,
    ssl_cert_path: str | None = None,
    error_tolerance: float | None = None,
    max_extraction_workers: int | None = None,
    docling_artifacts_path: str | None = None,
    docling_config: DoclingExtractionConfig | None = None,
    s3_client: Any | None = None,
) -> ExtractionResult:
    """Download documents from S3 and extract text using Docling.

    Each input document is downloaded from S3, converted to a
    :class:`DoclingDocument` via the Docling library, and persisted as a
    JSON file in *output_dir*.  Conversion runs in a separate process pool
    (``multiprocess`` library, ``"spawn"`` context) while downloads happen
    concurrently in a thread pool.

    Parameters
    ----------
    documents
        List of document descriptor dicts, each with at least a ``"key"``
        and ``"size_bytes"`` entry (as produced by
        :func:`~ai4rag.utils.data.documents_discovery.discover_documents`).
        The ``"key"`` entry also names the extracted document.
    bucket
        S3-compatible bucket name.
    output_dir
        Local directory where DoclingDocument JSON files are written.
    s3_endpoint
        S3-compatible endpoint URL.  Falls back to ``AWS_S3_ENDPOINT``.
    s3_access_key
        AWS access key.  Falls back to ``AWS_ACCESS_KEY_ID``.
    s3_secret_key
        AWS secret key.  Falls back to ``AWS_SECRET_ACCESS_KEY``.
    s3_region
        AWS region.  Falls back to ``AWS_DEFAULT_REGION``.
    ssl_cert_path
        Optional path to a PEM CA bundle used to verify the S3 endpoint. When
        provided, it takes precedence over ``AWS_CA_BUNDLE``. The file must be
        readable; certificate validation remains enabled.
    error_tolerance
        Fraction of documents (0.0--1.0) allowed to fail.  ``None`` means
        zero tolerance.
    max_extraction_workers
        Number of parallel worker processes.  Defaults to
        ``min(max(1, cpu_count // 2), 8)``.
    docling_artifacts_path
        Path to pre-downloaded Docling model artifacts for offline use.
        Falls back to ``DOCLING_ARTIFACTS_PATH`` environment variable.
    docling_config
        Ready :class:`DoclingExtractionConfig` controlling table-structure,
        OCR, and device selection. Callers (e.g. ``pipelines-components``) construct
        it once and pass it here; it is forwarded unchanged to every worker
        process.  ``None`` (default) uses ``DoclingExtractionConfig()`` --
        table structure and OCR both disabled.  See
        :class:`DoclingExtractionConfig` for the per-field ``what``/``why``
        and for how ``ocr_lang`` maps to bundled OCR models.
    s3_client
        Pre-configured S3 client used for every download. When omitted, one
        client is created from the S3 credentials supplied to this function or
        from the environment. The client is shared only by download threads;
        it is never sent to Docling worker processes.

    Returns
    -------
    ExtractionResult
        Summary of the extraction run.

    Raises
    ------
    RuntimeError
        If CUDA OCR lacks an ONNX Runtime GPU provider, or the error count
        exceeds the allowed tolerance.
    """
    import tempfile

    import multiprocess as multiprocessing

    out_dir = Path(output_dir)
    out_dir.mkdir(parents=True, exist_ok=True)

    if not documents:
        _logger.info("No documents to process.")
        return ExtractionResult(processed_count=0, total_documents=0, error_count=0)

    pipeline_config = docling_config or DoclingExtractionConfig()
    _require_cuda_ocr_provider(pipeline_config)

    if s3_client is None:
        s3_creds = _resolve_s3_credentials(s3_endpoint, s3_access_key, s3_secret_key, s3_region)
        s3_client = _make_s3_client(s3_creds, ssl_cert_path=ssl_cert_path)
    artifacts_path = _resolve_artifacts_path(docling_artifacts_path)

    has_custom_models = bool(
        pipeline_config.ocr_det_model_path
        or pipeline_config.ocr_cls_model_path
        or pipeline_config.ocr_rec_model_path
        or pipeline_config.ocr_rec_keys_path
    )
    _logger.info("Docling table structure parsing: %s", pipeline_config.do_table_structure)
    _logger.info(
        "Docling OCR (RapidOCR): enabled=%s lang=%s custom_models=%s",
        pipeline_config.do_ocr,
        pipeline_config.ocr_lang if pipeline_config.do_ocr else (),
        has_custom_models,
    )

    documents = sorted(documents, key=lambda d: d.get("size_bytes", 0), reverse=True)

    effective_workers = _effective_worker_count(max_extraction_workers)
    _logger.info(
        "Starting text extraction for %d documents. extraction_workers=%d, download_threads=%d.",
        len(documents),
        effective_workers,
        DOWNLOAD_MAX_THREADS,
    )

    if artifacts_path is not None:
        os.environ.setdefault("HF_HUB_OFFLINE", "1")

    mp_context = multiprocessing.get_context("spawn")  # pylint: disable=no-member
    with (
        tempfile.TemporaryDirectory() as download_dir,
        mp_context.Pool(
            processes=effective_workers,
            initializer=_text_extraction_pool_initializer,
            initargs=(pipeline_config,),
        ) as process_pool,
    ):
        download_start = time.perf_counter()
        extraction_tasks, download_errors = _download_and_submit(
            docs=documents,
            bucket=bucket,
            download_path=Path(download_dir),
            process_pool=process_pool,
            out_dir=out_dir,
            s3_client=s3_client,
        )
        _logger.info(
            "Downloads finished in %.1fs; %d file(s) queued for extraction, %d download error(s).",
            time.perf_counter() - download_start,
            len(extraction_tasks),
            len(download_errors),
        )
        _raise_if_threshold_exceeded(download_errors, len(documents), error_tolerance)

        extraction_errors: list[dict] = []
        processed_count = 0
        pending = list(extraction_tasks)
        completed = 0

        while pending:
            still_pending = []
            for file_path, task in pending:
                if task.ready():
                    completed += 1
                    try:
                        success, tb = task.get()
                    except Exception:
                        tb = traceback.format_exc()
                        _logger.error("Worker crashed for %s:\n%s", file_path, tb)
                        success = False
                    Path(file_path).unlink(missing_ok=True)
                    if success:
                        processed_count += 1
                    else:
                        extraction_errors.append({"file": file_path, "traceback": tb})
                    _logger.info("Extraction progress %d/%d", completed, len(extraction_tasks))
                else:
                    still_pending.append((file_path, task))
            pending = still_pending
            if pending:
                time.sleep(0.01)

    all_errors = download_errors + extraction_errors
    total_errors = len(all_errors)
    _logger.info(
        "Text extraction completed. Total processed: %d/%d, Errors: %d",
        processed_count,
        len(documents),
        total_errors,
    )
    _raise_if_threshold_exceeded(
        error_details=all_errors,
        total_docs=len(documents),
        tolerance=error_tolerance,
    )

    return ExtractionResult(
        processed_count=processed_count,
        total_documents=len(documents),
        error_count=total_errors,
    )

Local Whisper ASR

Audio extraction uses a local Transformers-format Whisper model configured through DoclingExtractionConfig.asr_model_path or HF_MODEL_DIR. See Audio Transcription for the required artifacts and migration from openai-whisper checkpoints.

local_hf_asr_pipeline

Offline Hugging Face Whisper backend for Docling audio extraction.

Classes

LocalHuggingFaceAsrPipelineOptions

Bases: AsrPipelineOptions

Options for :class:LocalHuggingFaceAsrPipeline.

Attributes:

  • asr_model_path (str | None) –

    Directory containing an approved Hugging Face Transformers Whisper model. None is permitted while constructing a converter so non-audio runs do not require audio model artifacts; audio conversion then fails before any remote lookup is attempted.

LocalHuggingFaceAsrPipeline

LocalHuggingFaceAsrPipeline(pipeline_options: LocalHuggingFaceAsrPipelineOptions)

Bases: AsrPipeline

Docling ASR pipeline that loads only a local Transformers model.

The native Docling ASR initializer creates an openai-whisper model, whose checkpoint format differs from the approved Transformers modelcar. This pipeline intentionally initializes :class:BasePipeline directly and lazily loads the local model on the first audio document.

Source code in ai4rag/utils/data/local_hf_asr_pipeline.py
def __init__(  # pylint: disable=super-init-not-called,non-parent-init-called
    self, pipeline_options: LocalHuggingFaceAsrPipelineOptions
):
    BasePipeline.__init__(self, pipeline_options)
    self.keep_backend = True
    self.pipeline_options = pipeline_options
    self._transcriber = None

Test Data Loading

test_data_loader

Classes

TestDataResult dataclass

TestDataResult(data: list[dict], record_count: int, sampled: bool)

Outcome of loading (and optionally sampling) benchmark test data.

Attributes:

  • data (list[dict]) –

    Benchmark records, each containing question, correct_answers, and correct_answer_document_keys.

  • record_count (int) –

    Number of records in data.

  • sampled (bool) –

    True if the data was randomly sampled down.

TestDataLoaderError

Bases: Exception

Raised when test data cannot be loaded or validated.

Functions:

load_test_data

load_test_data(
    bucket_name: str, key: str, benchmark_sample_size: int = BENCHMARK_SAMPLE_SIZE, s3_client: Any | None = None
) -> TestDataResult

Download benchmark test data from S3 and optionally sample it.

Parameters:

  • bucket_name (str) –

    S3-compatible bucket containing the test data file.

  • key (str) –

    Full S3 object key to the JSON test data file.

  • benchmark_sample_size (int, default: 25 ) –

    Maximum number of records to keep. When the dataset exceeds this limit a reproducible random sample is drawn (seed 42). Set to 0 to disable sampling and keep all records.

  • s3_client (Any | None, default: None ) –

    Pre-configured boto3 S3 client. When None, one is created via :func:ai4rag.utils.clients.s3.create_s3_client.

Returns:

Raises:

  • FileNotFoundError –

    If the object does not exist in S3.

  • TestDataLoaderError –

    If the file is not valid JSON or the records have an unexpected structure.

Source code in ai4rag/utils/data/test_data_loader.py
def load_test_data(
    bucket_name: str,
    key: str,
    benchmark_sample_size: int = BENCHMARK_SAMPLE_SIZE,
    s3_client: Any | None = None,
) -> TestDataResult:
    """Download benchmark test data from S3 and optionally sample it.

    Parameters
    ----------
    bucket_name : str
        S3-compatible bucket containing the test data file.
    key : str
        Full S3 object key to the JSON test data file.
    benchmark_sample_size : int, default=25
        Maximum number of records to keep.  When the dataset exceeds this
        limit a reproducible random sample is drawn (seed 42).  Set to
        ``0`` to disable sampling and keep all records.
    s3_client : Any | None, default=None
        Pre-configured ``boto3`` S3 client.  When ``None``, one is created
        via :func:`ai4rag.utils.clients.s3.create_s3_client`.

    Returns
    -------
    TestDataResult
        Loaded (and optionally sampled) benchmark data.

    Raises
    ------
    FileNotFoundError
        If the object does not exist in S3.
    TestDataLoaderError
        If the file is not valid JSON or the records have an unexpected
        structure.
    """
    if not bucket_name:
        raise TypeError("bucket_name must be a non-empty string")

    if s3_client is None:
        s3_client = create_s3_client()

    raw_data = _download_object(s3_client, bucket_name, key)
    benchmark_data = _parse_and_validate(raw_data)

    sampled = False
    if 0 < benchmark_sample_size < len(benchmark_data):
        original_count = len(benchmark_data)
        rng = random.Random(42)
        benchmark_data = rng.sample(benchmark_data, benchmark_sample_size)
        sampled = True
        _logger.info("Sampled %d records from %d total.", benchmark_sample_size, original_count)
    else:
        _logger.info("No sampling applied; record count: %d.", len(benchmark_data))

    return TestDataResult(data=benchmark_data, record_count=len(benchmark_data), sampled=sampled)