docling-pipelines

Docling Pipelines Documentation

Welcome to the Docling Pipelines documentation! This guide will help you navigate our comprehensive documentation and find exactly what you need.

📚 Documentation Book

The Docling-pipelines Book is a 12-chapter guide covering architecture, installation, flow authoring, document processing, vector storage, LLM integration, deployment, and more. It is authored in AsciiDoc and built with Antora into a static website (hosted on GitHub Pages) and a PDF.

🚀 Getting Started

New to Docling Pipelines? Start here:

• Quick Start Guide - Get your first pipeline running in 5 minutes
• Complete Setup Guide - Detailed installation and configuration
• Troubleshooting Guide - Solutions to common issues

📖 Guides

Task-oriented guides to help you accomplish specific goals:

Core Guides

• Flow Authoring Format - Learn the simplified flow authoring format
• Flow Configuration Guide - Complete flow configuration reference
• Python API Guide - Use Docling Pipelines programmatically

Developer Guides

• Create Connector Guide - Build custom data source connectors
• Custom Operators Guide - Create your own operators
• External Operator Integration - Integrate external operators
• Testing Standards - Coverage requirements, test organisation, fixtures, and naming conventions

Advanced Topics

• Advanced Configuration - Production deployment and optimization
• Security Best Practices - Credential management, authentication, ACL, and production hardening
• Unified LLM Architecture - LLM integration patterns
• Document Libraries - Managing document collections
• Document Class Utils - Document schema utilities
• Vault Integration - HashiCorp Vault secret resolution via vault:// URIs
• UI User Guide - Using the React frontend

Best Practices

• Logging Best Practices - Security-first logging, sensitive data handling, and NFR Point 12 compliance

⚙️ Operator Configurations

Configuration examples and patterns for all operators:

Core Pipeline Operators

• IngestSource - Ingest documents from local filesystem or external sources (S3, SharePoint, etc.)
• Extract - Extract text and entities from documents
• Chunker - Split documents into chunks
• Embeddings - Generate vector embeddings
• VectorDB - Store vectors in OpenSearch or Milvus
• StorageOutput - Write documents to a file destination (filesystem, etc.)

All Operator Configurations

Browse the complete list of operator configuration examples in the operators/ directory.

📚 Reference

Quick lookup documentation for parameters and APIs:

• Global Configuration Reference - Flow-level configuration parameters
• Operator Reference - Complete operator parameter specifications
• Document Schemas - Document class schema definitions

🌐 REST API

Documentation for the Docling Pipelines REST API server:

• REST API Server - Server setup, all endpoints, authentication, and security overview
• Document Retrieval API - ACL-based document retrieval endpoints
• OAuth2 Authentication - OAuth2 and OIDC authentication setup

🔌 Integrations

Integration-specific documentation:

• OpenSearch - Vector storage with OpenSearch
  • Quick Start
  • Environment Setup
  • Schema Templates
• Milvus - Vector storage with Milvus
• Prefect - Distributed execution with Prefect
  • Distributed Execution Guide

🛠️ Contributing

Resources for contributors:

• Contributing Guide - How to contribute to Docling Pipelines
• Documentation Style Guide - Formatting, Mermaid, and writing conventions for all contributors

🔬 Internals

Internal documentation for maintainers:

• Metadata Aggregation Strategy - Metadata aggregation in micro-batching
• Document Libraries Architecture - Document library system design
• Unified Asset Architecture - Complete reference for the common asset layer: domain models, ports, adapters, factories, and service wiring

🚀 Deployment

Deployment guides for production environments:

• OpenShift Deployment - Deploy Docling Pipelines on OpenShift

💡 Architecture

Understand how Docling Pipelines works:

• Architecture Overview - System design and operator catalog

📋 Documentation Organization

Our documentation is organized by user journey:

docs/
├── guides/              # How-to guides (task-oriented)
├── operators/           # Operator configuration examples
├── reference/           # Parameter and API lookups
├── api/                 # REST API documentation
├── integrations/        # Integration-specific docs
├── internals/           # Internal maintainer docs
└── deployment/          # Deployment guides

Note: Comprehensive operator documentation (architecture, implementation details) is located in the source code directories at src/docpipe/core/operators/*/README.md.

🔍 Finding What You Need

I want to…

• Get started quickly → Quick Start Guide
• Set up my environment → Complete Setup Guide
• Write my first flow → Flow Authoring Format
• Look up a parameter → Global Config Reference or Operator Reference
• See operator config examples → Operator Configs Directory
• Understand operator internals → Check src/docpipe/core/operators/*/README.md in source code
• Use the Python API → Python API Guide
• Use the REST API → REST API Server
• Set up OAuth2 authentication → OAuth2 Authentication Guide
• Create a custom connector → Create Connector Guide
• Secure my deployment → Security Best Practices
• Deploy to production → Advanced Configuration
• Troubleshoot an issue → Troubleshooting Guide
• Contribute code → Contributing Guide

📞 Need Help?

• Check the Troubleshooting Guide for common issues
• Review the Architecture Overview to understand system design
• Browse operator configuration examples for usage patterns
• Check existing GitHub issues or create a new one

---

Happy data processing! 🚀

This site is open source. Improve this page.