docling-pipelines

Global Configuration Reference

This document provides a comprehensive reference for all global configuration parameters that can be set in docpipe flow definitions. Global configuration parameters control flow-level behavior and are specified in the global_config section of your flow JSON.

Overview

Global configuration parameters are set at the flow level and apply to all operators in the pipeline unless overridden at the operator level. These parameters control:

Execution behavior: How the flow processes data
Batching strategy: Whether and how to split data into batches
Incremental processing: Tracking and processing only changed data
Orchestration: Prefect-based workflow execution settings
Storage: Where intermediate and final data is stored

Configuration Structure

Global configuration is specified in the global_config section of your flow definition:

{
  "flow_name": "My Pipeline",
  "description": "Example pipeline with global configuration",
  "global_config": {
    "force_ingest": false,
    "micro_batch_size": 50,
    "prefect": {
      "batch_execution": {
        "strategy": "thread-pool"
      }
    }
  },
  "flow": [
    {
      "type": "ingest_source",
      "name": "ingest",
      "config": {
        "provider": "filesystem",
        "connection_params": {"paths": ["./documents"]}
      }
    },
    {
      "type": "extract_operator",
      "name": "extract",
      "config": {
        "text_extraction": {"provider": "docling_library"}
      },
      "depends_on": ["ingest"]
    }
  ]
}

Execution Control

Parameters that control how the flow executes and processes data.

`disable_validation`

Type: boolean
Default: false
Description: Disables flow validation before execution. Not recommended for production use.

Valid Values:

true: Skip flow validation (faster startup, risky)
false: Validate flow before execution (recommended)

Example:

{
  "global_config": {
    "disable_validation": true
  }
}

Warning: Disabling validation skips the main flow validation pass and can lead to runtime errors that validation would normally catch.

What Gets Skipped When disable_validation: true:

DAG structure validation: Checks for empty DAG, unnamed operators, duplicate operator names
First operator validation: Verification that the first operator is an Ingest category operator
ACL operator placement validation: Checks that ACL operators are correctly positioned in the flow
Disjoint operators validation: Detection of disconnected or isolated operators in the flow
Cycle detection: Checks for circular dependencies in the DAG
Operator availability validation: Verification that all operator types are registered and available
Node-level validation: Individual operator configuration and parameter validation
Last operator validation: Warning if the last operator is not a VectorDB operator

`skip_custom_op_validation`

Type: boolean
Default: false
Description: Skips validation for custom operators while still validating built-in operators.

Valid Values:

true: Skip custom operator validation
false: Validate all operators including custom ones

Example:

{
  "global_config": {
    "skip_custom_op_validation": true
  }
}

`output_folder`

Type: string
Description: Directory for storing final output files. Can be either a relative path (relative to workspace directory) or an absolute path. If not specified, the system generates a unique path based on job execution IDs.
Valid Values: Valid directory path (relative or absolute)

Example:

{
  "global_config": {
    "data_local_config": {
      "output_folder": "./data/output"
    }
  }
}

`data_storage_type`

Type: string
Description: Controls intermediate execution data storage behavior during flow execution.
Valid Values:

"memory": Use in-memory intermediate data handling
"local": Use local filesystem-backed intermediate data handling

Example:

{
  "global_config": {
    "data_storage_type": "local"
  }
}

Note: Memory storage is fastest but limited by available RAM. Use "local" for large datasets.

`memmap_threshold`

Type: integer
Default: 100
Description: Threshold in MB after which persistent storage is used for chunks and embeddings. Valid Values: Must be greater than 1.

Example:

{
  "global_config": {
    "memmap_threshold": 100
  }
}

Incremental Processing

Configuration for tracking and processing only changed documents.

`force_ingest`

Type: boolean
Default: false
Description: Forces re-ingestion of all documents, even if they were previously processed. Useful for reprocessing data after operator configuration changes.

Valid Values:

true: Re-ingest all documents regardless of previous processing
false: Skip documents that were already processed (incremental mode)

Example:

{
  "global_config": {
    "force_ingest": true
  }
}

Use Cases:

Reprocessing entire dataset after fixing extraction logic
Reprocess entire dataset when the target DB (ex: collection, table, etc.) is changed.
Reprocess entire data set after modifying some changes to operator, examples: PII/HAP configuration is changed.
Testing flow changes on full dataset
Recovering from corrupted incremental metadata

`retain_deleted_docs`

Type: boolean
Default: true
Description: Controls whether documents deleted from the source should be retained in the output or removed.

Valid Values:

true: Keep documents in output even if deleted from source
false: Remove documents from output when deleted from source

Example:

{
  "global_config": {
    "retain_deleted_docs": false
  }
}

Use Cases:

Maintaining historical records (set to true)
Keeping output synchronized with source (set to false)

Important Limitations:

This feature only works when the source directory/path still exists
If you delete the entire source directory (e.g., ./sample_documents), the system will report an error: Path does not exist
To handle deleted directories, you must either:
- Keep the source directory structure intact (even if empty)
- Set force_ingest: true to reset incremental metadata

Centralized Incremental Metadata Configuration

Docling Pipelines uses a centralized configuration system for incremental metadata storage. Instead of configuring incremental metadata in each flow JSON file, you configure it once in a repository-level docling-pipelines-config.yaml file.

Configuration Structure :

The incremental metadata configuration is defined in docling-pipelines-config.yaml:

Option 1: Inherit from global_storage

global_storage:
   type: "filesystem"  # Options: filesystem, postgresql
   config:
      base_dir: "./data"
      lock_timeout: 30.0
   # PostgreSQL configuration (only needed if type is "postgres")
   postgres:
      host: "localhost"
      port: 5432
      database: "docpipe"
      user: "docpipe_user"
      password: "${POSTGRES_PASSWORD}" # pragma: allowlist secret
      schema: "incremental_metadata"

incremental_metadata: {} # inherit from global_storage

Option 2: Override with service-specific configuration

global_storage:
  type: "filesystem"
  config:
    base_dir: "./data"
    lock_timeout: 30.0

# incremental_metadata overrides global_storage
incremental_metadata:
  storage:
    type: "filesystem"
    config:
      base_dir: "./incremental_data"  # Different path
      lock_timeout: 60.0

Supported Storage types :

Docling Pipelines supports two storage types for incremental metadata:

Filesystem (default)
- Efficient columnar storage using Apache Parquet format
- Best for: Development, small to medium-scale deployments
- Configuration: Required config.base_dir and config.lock_timeout
- Features: Thread-safe operations with file locking, atomic writes
PostgreSQL
- Relational database storage
- Best for: Production deployments, multi-user environments, high concurrency
- Configuration: Requires database connection parameters

Environment Variables :

You can override configuration using environment variables:

DOCPIPE_INCREMENTAL_BASE_DIR: Override the base directory for incremental metadata
DOCPIPE_INCREMENTAL_STORAGE_BACKEND: Override the storage backend (filesystem, postgresql)
DOCPIPE_CONFIG_PATH: Specify a custom path to docling-pipelines-config.yaml

Example:

export DOCPIPE_INCREMENTAL_BASE_DIR="/data/incremental"
export DOCPIPE_INCREMENTAL_STORAGE_BACKEND="filesystem"
docling-pipelines --flow-file my_flow.json

Configuration Precedence :

Configuration is resolved in the following order (highest to lowest priority):

Environment variables (DOCPIPE_INCREMENTAL_BASE_DIR, DOCPIPE_INCREMENTAL_STORAGE_BACKEND)
Service-specific configuration (e.g., incremental_metadata.storage in docling-pipelines-config.yaml)
Global storage configuration (e.g., global_storage in docling-pipelines-config.yaml)
System defaults (Filesystem backend with ./data)

Flow JSON Configuration :

In your flow JSON files, you no longer need to specify incremental metadata configuration. The system automatically uses the centralized configuration:

{
  "flow_name": "My Pipeline",
  "description": "Pipeline with centralized incremental metadata",
  "global_config": {
    "force_ingest": false,
    "retain_deleted_docs": true
  },
  "flow": [
    {
      "type": "ingest_source",
      "name": "ingest",
      "config": {
        "provider": "filesystem",
        "connection_params": {"paths": ["./documents"]}
      }
    }
  ]
}

Migration from Flow-Level Configuration :

If you have existing flows with flow-level incremental metadata configuration, follow these steps:

Create docling-pipelines-config.yaml in your repository root:

 incremental_metadata:
   storage:
     type: "filesystem"
     config:
       base_dir: "./incremental_data"
       lock_timeout: 60.0

Remove incremental_metadata from flow JSON:
- Delete the incremental_metadata section from global_config
- The system will automatically use the centralized configuration

Verify configuration:

docling-pipelines --flow-file your_flow.json --validate

Use Cases :

Processing only new or modified documents: Incremental metadata tracks document hashes and modification times
Resuming interrupted pipeline runs: Metadata persists across runs, allowing pipelines to resume where they left off
Efficient updates to large document collections: Only process changed documents, not the entire collection
Multi-flow coordination: Share incremental metadata across multiple flows using the same base directory

Related Documentation :

Incremental Metadata Configuration

Orchestration configuration

Prefect Configuration

Configuration for prefect-based workflow orchestration and batch execution strategies.

`prefect`

Type: object
Description: Prefect orchestration settings including batch execution strategy and work pool configuration.

Structure :

{
  "prefect": {
    "batch_execution": {
      "strategy": "thread-pool",
      "work_pool_name": "my-pool",
      "deployment_name": "my-deployment",
      "batch_storage": {
        "type": "local",
        "path": "./batch_data"
      }
    }
  }
}

Batch Execution Strategy

The batch_execution section controls how batches are executed.

`strategy`

Type: string
Default: "thread-pool"
Description: Execution strategy for batch processing.

Valid Values:

"thread-pool": Local execution using ThreadPoolTaskRunner (default, simplest)
"work-pool-process": Distributed execution using Prefect process work pools
"work-pool-docker": Distributed execution using Docker containers

Example - Thread Pool (Local):

{
  "prefect": {
    "batch_execution": {
      "strategy": "thread-pool"
    }
  }
}

Example - Docker Work Pool:

{
  "prefect": {
    "batch_execution": {
      "strategy": "work-pool-docker",
      "work_pool_name": "docpipe-docker-pool",
      "deployment_name": "batch-processor",
      "deployment_path": "/opt/docpipe",
      "image": "docpipe:latest",
      "batch_storage": {
        "type": "local",
        "path": "/data/batches"
      }
    }
  }
}

Work Pool Configuration

Required when using work pool strategies (work-pool-*).

`work_pool_name`

Type: string
Required: Yes (for work pool strategies)
Description: Name of the Prefect work pool to use for batch execution.

Example:

{
  "work_pool_name": "docpipe-production-pool"
}

`deployment_name`

Type: string
Required: Yes (for work pool strategies)
Description: Name for the Prefect deployment that will execute batches.

Example:

{
  "deployment_name": "document-processing-v1"
}

`deployment_path`

Type: string
Default: Current working directory
Description: Runtime path where flow code is available in the worker environment.

Example:

{
  "deployment_path": "/opt/docpipe"
}

`env`

Type: object (dictionary of string key-value pairs)
Default: {}
Description: Environment variables injected into the worker job process or container. Used to pass configuration, credentials, or runtime settings to workers.

Example:

{
  "env": {
    "OLLAMA_HOST": "http://ollama-service:11434",
    "LOG_LEVEL": "INFO",
    "PREFECT_MODE": "cloud",
    "PREFECT_URL": "https://api.prefect.cloud",
    "DOCPIPE_STORAGE_BACKEND": "postgresql",
    "PYTHONPATH": "/app/src",
    "LOCAL_FLOWS_DIR": "/app/flows",
    "DOCPIPE_DATA_PATH": "/data/docpipe"
  }
}

Note: System-required environment variables (PREFECT_API_URL, PYTHONPATH, etc.) are automatically set by the adapter. User-provided env values supplement or override these defaults.

`image`

Type: string
Required: Yes (for containerized work pools)
Description: Container image for Docker work pools.

Example:

{
  "image": "myregistry/docpipe:v1.2.3"
}

`image_pull_policy`

Type: string
Default: "Never"
Description: Policy for pulling container images.

Valid Values:

"Always": Always pull the image
"IfNotPresent": Pull only if not present locally
"Never": Never pull, use local image only

Example:

{
  "image_pull_policy": "Never"
}

Docker - Specific Configuration

`networks`

Type: array of strings
Default: []
Description: Docker networks for spawned containers.

Example:

{
  "networks": ["docpipe-network", "monitoring-network"]
}

Batch Storage Configuration

The batch_storage section controls where batch data is stored during distributed execution.

`type`

Type: string
Default: "inline"
Description: Storage type for batch data.

Valid Values:

"inline": Serialize batch data directly in deployment parameters (small batches only)
"local": Store batch data on local filesystem

`path`

Type: string
Required: Yes (when type is "local")
Description: Filesystem path for storing batch data when using local storage type.

Example - Local Storage:

{
  "batch_storage": {
    "type": "local",
    "path": "/data/batches"
  }
}

Related Documentation: Prefect Distributed Execution Guide

Micro-Batching Configuration

Parameters for controlling micro-batching behavior. Micro-batching must be explicitly enabled and splits large datasets into smaller batches for parallel processing.

`micro_batch_size`

Type: integer
Default: 100
Description: Note that the batch_size is not strictly enforced. The files are adjusted in the batches to make the batch size uniform across the batches, but limiting the number of documents in a batch to the given batch_size.
Valid Values: Positive integer

Example:

{
  "global_config": {
    "micro_batch_size": 50
  }
}

Tuning Guidelines:

Small batches (10-50): Better for large documents or memory-constrained environments
Medium batches (50-200): Good balance for most use cases
Large batches (200-1000): Suitable for small documents with high throughput requirements

`max_concurrent_batches`

Type: integer
Default: 10
Description: Maximum number of batches that can execute concurrently. Controls parallelism and resource usage.
Valid Values: Positive integer

Example:

{
  "global_config": {
    "max_concurrent_batches": 5
  }
}

Tuning Guidelines:

Low concurrency (1-5): Reduces memory usage, suitable for resource-constrained environments
Medium concurrency (5-15): Good balance for most systems
High concurrency (15-50): Maximizes throughput on high-resource systems

Note: Higher concurrency requires more memory and CPU resources.

Operator Overrides

You can override configuration for specific operators using their name or ID.

`<operator_name>` or `<operator_id>`

Type: object
Description: Override configuration for a specific operator by its name or ID.

Example:

{
  "global_config": {
    "doc_column": "content",
    "extract_operator": {
      "doc_column": "document",
      "max_workers": 8
    }
  }
}

In this example, all operators use doc_column: "content" except the extract_operator which uses doc_column: "document".

Complete Example

Here’s a comprehensive example showing the separation between flow JSON and docling-pipelines-config.yaml:

Flow JSON

{
  "flow_name": "Production Document Processing Pipeline",
  "description": "Process documents with micro-batching and Docker execution",
  "global_config": {
    "force_ingest": false,
    "retain_deleted_docs": true,
    "micro_batch_size": 100,
    "max_concurrent_batches": 10,
    "data_local_config": {
        "output_folder": "./output"
    },
    "data_storage_type": "local",
    "prefect": {
      "batch_execution": {
        "strategy": "work-pool-docker",
        "work_pool_name": "docpipe-docker-pool",
        "deployment_name": "doc-processor-v1",
        "deployment_path": "/opt/docpipe",
        "image": "myregistry/docpipe:1.0.0",
        "env": {
          "OLLAMA_HOST": "http://ollama-service:11434",
          "LOG_LEVEL": "INFO"
        },
        "batch_storage": {
          "type": "local",
          "path": "/data/batches"
        }
      }
    }
  },
  "flow": [
    {
      "type": "ingest_source",
      "name": "ingest_source_filesystem",
      "config": {
        "provider": "filesystem",
        "connection_params": {"paths": ["./sample_documents"]},
        "include_filter": "pdf,txt,docx"
      }
    },
    {
      "type": "extract_operator",
      "name": "extract_with_docling",
      "config": {
        "doc_column": "content"
      },
      "depends_on": ["ingest_source_filesystem"]
    }
  ]
}

Centralized Configuration (`docling-pipelines-config.yaml`)

# Repository-level configuration for incremental metadata
incremental_metadata:
   storage:
      type: "filesystem"
      config:
         base_dir: "./incremental_data"
         lock_timeout: 60.0
#  PostgreSQL configuration (only needed if type is "postgres")
#  postgres:
#    host: "localhost"
#    port: 5432
#    database: "docpipe"
#    user: "docpipe_user"
#    password: "${POSTGRES_PASSWORD}" # pragma: allowlist secret
#    schema: "incremental_metadata"

Execution

# The flow automatically uses the centralized incremental metadata configuration
docling-pipelines --flow-file production_pipeline.json

Key Points:

Incremental metadata configuration is in docling-pipelines-config.yaml
Flow JSON focuses on pipeline structure and execution parameters
Centralized configuration is shared across all flows in the repository
Environment variables can override configuration for specific runs

This site is open source. Improve this page.

docling-pipelines

Global Configuration Reference

Overview

Configuration Structure

Execution Control

disable_validation

skip_custom_op_validation

output_folder

data_storage_type

memmap_threshold

Incremental Processing

force_ingest

retain_deleted_docs

Centralized Incremental Metadata Configuration

Configuration Structure :

Supported Storage types :

Environment Variables :

Configuration Precedence :

Flow JSON Configuration :

Migration from Flow-Level Configuration :

Use Cases :

Orchestration configuration

Prefect Configuration

prefect

Batch Execution Strategy

strategy

Work Pool Configuration

work_pool_name

deployment_name

deployment_path

env

image

image_pull_policy

Docker - Specific Configuration

networks

Batch Storage Configuration

type

path

Micro-Batching Configuration

micro_batch_size

max_concurrent_batches

Operator Overrides

<operator_name> or <operator_id>

Complete Example

Flow JSON

Centralized Configuration (docling-pipelines-config.yaml)

Execution

`disable_validation`

`skip_custom_op_validation`

`output_folder`

`data_storage_type`

`memmap_threshold`

`force_ingest`

`retain_deleted_docs`

`prefect`

`strategy`

`work_pool_name`

`deployment_name`

`deployment_path`

`env`

`image`

`image_pull_policy`

`networks`

`type`

`path`

`micro_batch_size`

`max_concurrent_batches`

`<operator_name>` or `<operator_id>`

Centralized Configuration (`docling-pipelines-config.yaml`)