docling-pipelines

Document Libraries User Guide

Overview

Document Libraries provide a way to organize and manage collections of Document Sets in docpipe. A Document Library is a metadata container that groups related Document Sets together, enabling better organization, discovery, and management of your document collections.

Key Concepts

Document Library

A Document Library is a named collection that contains references to multiple Document Sets. It stores:

• Metadata: Name, description, tags, timestamps
• Relationships: References to Document Set IDs (many-to-many)
• Aggregate Metrics: Computed statistics from associated Document Sets

Important: Document Libraries store only metadata and references, not the actual document content or data.

Relationship with Document Sets

• One-to-Many: A Document Library can contain multiple Document Sets
• Many-to-Many: A Document Set can belong to multiple Document Libraries
• Reference-Based: Libraries store Document Set IDs, not copies of data

Storage Architecture

Document Libraries follow a hybrid storage approach aligned with Document Sets pattern:

1. Library Metadata (JSON Storage):
   • Uses KeyValueStorage interface for library metadata
   • Stored as JSON in data column (CAMS-compatible)
   • Schema: key, data (JSON), created_at, updated_at
   • Same pattern as Document Sets and Flows
2. Junction Table (Relational Storage):
   • Uses direct SQL for many-to-many relationships
   • Optimized for bulk operations
   • Table: library_documentset_junction

This hybrid approach balances architectural consistency (JSON for assets) with performance (SQL for relationships).

Prerequisites

Environment Setup

# 1. Navigate to project root directory
cd /path/to/docling-pipelines

# 2. Activate virtual environment
source .venv/bin/activate

# 3. Set PYTHONPATH
export PYTHONPATH="$(pwd)/src:${PYTHONPATH}"

# 4. Ensure dependencies are installed
uv sync

API Server

Document Libraries are accessed via REST API endpoints. Start the API server:

# Recommended — uses the installed console entry point
docling-pipelines-api

# Alternative — invoke uvicorn directly (with PYTHONPATH set)
uvicorn docpipe.api.main:app --reload --host 0.0.0.0 --port 8000

The API will be available at http://localhost:8000 with interactive documentation at http://localhost:8000/api/v1/docs.

Note: The direct uvicorn command must be run from the project root with PYTHONPATH set to include the src directory.

Troubleshooting:

• Error: “address already in use”: Port 8000 is already occupied. Either:
  • Stop the existing server: pkill -f "uvicorn docpipe.api.main:app"
  • Use a different port: uvicorn docpipe.api.main:app --port 8001
• Error: “Could not import module”: Ensure PYTHONPATH is set correctly and you’re in the project root directory

API Endpoints

Base URL

All Document Library endpoints are under: /api/v1/document-libraries

Available Operations

1. Create a Document Library

Endpoint: POST /api/v1/document-libraries

Request Body:

{
  "name": "Financial Documents Q1 2024",
  "description": "Collection of financial documents for Q1 2024 analysis",
  "tags": ["finance", "q1-2024", "reports"]
}

Validation Rules:

• name: Required, 3-128 characters, must start with a letter, alphanumeric with spaces/underscores only (NO hyphens)
• description: Optional, max 2000 characters
• tags: Optional, no limit on count or individual tag length

Response (201 Created):

{
  "library_id": "550e8400-e29b-41d4-a716-446655440000",
  "name": "Financial Documents Q1 2024",
  "description": "Collection of financial documents for Q1 2024 analysis",
  "tags": ["finance", "q1-2024", "reports"],
  "document_set_ids": [],
  "aggregate_metrics": {
    "total_document_sets": 0,
    "total_documents": 0,
    "total_size_bytes": 0
  },
  "created_at": "2024-01-15T10:30:00Z",
  "updated_at": "2024-01-15T10:30:00Z"
}

Example using curl:

curl -X POST "http://localhost:8000/api/v1/document-libraries" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Financial Documents Q1 20241",
    "description": "Collection of financial documents for Q1 2024 analysis",
    "tags": ["finance", "q1-2024", "reports"]
  }'

2. Get a Document Library

Endpoint: GET /api/v1/document-libraries/{library_id}

Response (200 OK):

{
  "library_id": "81cea322-ae1b-4142-8f27-e06ff39e211a",
  "name": "Financial Documents Q1 20241",
  "description": "Collection of financial documents for Q1 2024 analysis",
  "purpose": null,
  "original_size": null,
  "final_size": null,
  "tags": [
    "finance",
    "q1-2024",
    "reports"
  ],
  "created_by": null,
  "href": null
}

Example using curl:

curl -X GET "http://localhost:8000/api/v1/document-libraries/550e8400-e29b-41d4-a716-446655440000"

3. Update a Document Library

Endpoint: PATCH /api/v1/document-libraries/{library_id}

Request Body:

• name (required): Library name (3-128 characters, must start with letter, can only contain letters/digits/spaces/underscores)
• description (optional): Updated description
• purpose (optional): Updated purpose
• original_size (optional): Updated original size
• final_size (optional): Updated final size
• tags (optional): Updated tags list

Note: Name field is currently required even for PATCH operations. Name cannot contain hyphens or special characters.

{
  "name": "Financial Documents Q1 2024 Updated",
  "description": "Updated collection description",
  "tags": ["finance", "q1_2024", "reports", "audited"]
}

Response (200 OK): Returns updated library object

Example using curl:

curl -X PATCH "http://localhost:8000/api/v1/document-libraries/550e8400-e29b-41d4-a716-446655440000" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Financial Documents Q1 2024 Updated",
    "description": "Updated collection description",
    "tags": ["finance", "q1_2024", "reports", "audited"]
  }'

4. Delete a Document Library

Endpoint: DELETE /api/v1/document-libraries/{library_id}

Response (204 No Content)

Note: Deleting a library does not delete the associated Document Sets, only the library metadata and relationships.

Example using curl:

curl -X DELETE "http://localhost:8000/api/v1/document-libraries/550e8400-e29b-41d4-a716-446655440000"

5. Add Document Sets to Library (Bulk)

Endpoint: PUT /api/v1/document-libraries/{library_id}/document-sets

Query Parameters:

• document_sets_ids (required): Comma-separated list of document set UUIDs

Response (204 No Content): Document sets added successfully

Example using curl:

curl -X PUT "http://localhost:8000/api/v1/document-libraries/550e8400-e29b-41d4-a716-446655440000/document-sets?document_sets_ids=abc123,def456,ghi789"

6. Remove Document Sets from Library (Bulk)

Endpoint: DELETE /api/v1/document-libraries/{library_id}/document-sets

Query Parameters:

• document_sets_ids (required): Comma-separated list of document set UUIDs

Response (204 No Content): Document sets removed successfully

Note: This removes the associations but does NOT delete the document sets themselves.

Example using curl:

curl -X DELETE "http://localhost:8000/api/v1/document-libraries/550e8400-e29b-41d4-a716-446655440000/document-sets?document_sets_ids=abc123,def456"

7. List All Document Libraries

Endpoint: GET /api/v1/document-libraries

Query Parameters:

• offset: Number of records to skip (default: 0)
• limit: Maximum number of records to return (default: 100)

Response (200 OK):

[
  {
    "library_id": "e8b2cf35-4d2a-4997-b1e2-2470cd05fd73",
    "name": "Financial Documents Q1 2024",
    "description": "Collection of financial documents for Q1 2024 analysis",
    "purpose": null,
    "original_size": null,
    "final_size": null,
    "tags": [
      "finance",
      "q1-2024",
      "reports"
    ],
    "created_by": null,
    "href": null
  },
  {
    "library_id": "81cea322-ae1b-4142-8f27-e06ff39e211a",
    "name": "Financial Documents Q1 20241",
    "description": "Collection of financial documents for Q1 2024 analysis",
    "purpose": null,
    "original_size": null,
    "final_size": null,
    "tags": [
      "finance",
      "q1-2024",
      "reports"
    ],
    "created_by": null,
    "href": null
  }
]

Example using curl:

curl -X GET "http://localhost:8000/api/v1/document-libraries?offset=0&limit=10"

8. List Document Sets in a Library

Endpoint: GET /api/v1/document-libraries/{library_id}/document-sets

Path Parameters:

• library_id (required): UUID of the library

Response (200 OK):

{
  "document_sets": [
    {
      "id": "f824b653-45af-45a6-9336-341c6aeb2d8a",
      "name": "Research Documents1",
      "container_id": null,
      "container_type": null,
      "description": "Collection of research papers and reports.",
      "documents": null,
      "tags": [],
      "propagate_source_acls": null,
      "is_derivative_available": null
    },
    {
      "id": "85d7a47f-9d81-4ffe-a112-a93b88cd9c70",
      "name": "Research Documents",
      "container_id": null,
      "container_type": null,
      "description": "Collection of research papers and reports.",
      "documents": null,
      "tags": [],
      "propagate_source_acls": null,
      "is_derivative_available": null
    }
  ]
}

Example using curl:

curl -X GET "http://localhost:8000/api/v1/document-libraries/550e8400-e29b-41d4-a716-446655440000/document-sets"

Python Client Example

import requests
import json

class DocumentLibraryClient:
    def __init__(self, base_url: str = "http://localhost:8000"):
        self.base_url = base_url
        self.api_path = "/api/v1/document-libraries"

    def create_library(self, name: str, description: str = None, tags: list = None):
        """Create a new document library"""
        url = f"{self.base_url}{self.api_path}"
        payload = {"name": name}
        if description:
            payload["description"] = description
        if tags:
            payload["tags"] = tags

        response = requests.post(url, json=payload)
        response.raise_for_status()
        return response.json()

    def get_library(self, library_id: str):
        """Get a document library by ID"""
        url = f"{self.base_url}{self.api_path}/{library_id}"
        response = requests.get(url)
        response.raise_for_status()
        return response.json()

    def update_library(self, library_id: str, name: str = None,
                      description: str = None, tags: list = None):
        """Update a document library"""
        url = f"{self.base_url}{self.api_path}/{library_id}"
        payload = {}
        if name:
            payload["name"] = name
        if description:
            payload["description"] = description
        if tags:
            payload["tags"] = tags

        response = requests.patch(url, json=payload)
        response.raise_for_status()
        return response.json()

    def delete_library(self, library_id: str):
        """Delete a document library"""
        url = f"{self.base_url}{self.api_path}/{library_id}"
        response = requests.delete(url)
        response.raise_for_status()

    def add_document_sets(self, library_id: str, document_set_ids: list[str]):
        """Add document sets to a library (bulk operation)"""
        url = f"{self.base_url}{self.api_path}/{library_id}/document-sets"
        params = {"document_sets_ids": ",".join(document_set_ids)}
        response = requests.put(url, params=params)
        response.raise_for_status()

    def remove_document_sets(self, library_id: str, document_set_ids: list[str]):
        """Remove document sets from a library (bulk operation)"""
        url = f"{self.base_url}{self.api_path}/{library_id}/document-sets"
        params = {"document_sets_ids": ",".join(document_set_ids)}
        response = requests.delete(url, params=params)
        response.raise_for_status()

    def list_document_sets(self, library_id: str):
        """List document sets in a library"""
        url = f"{self.base_url}{self.api_path}/{library_id}/document-sets"
        response = requests.get(url)
        response.raise_for_status()
        return response.json()

# Usage example
if __name__ == "__main__":
    client = DocumentLibraryClient()

    # Create a library
    library = client.create_library(
        name="Invoice Processing Library",
        description="Collection of invoice document sets",
        tags=["invoices", "accounting", "q1-2024"]
    )
    print(f"Created library: {library['library_id']}")

    # Add document sets (bulk operation)
    client.add_document_sets(library['library_id'], ["docset-001", "docset-002"])

    # List document sets in library
    doc_sets = client.list_document_sets(library['library_id'])
    print(f"Library has {len(doc_sets['document_sets'])} document sets")

Error Handling

Common Error Responses

404 Not Found

{
  "detail": "Document library not found: 550e8400-e29b-41d4-a716-446655440000"
}

409 Conflict

{
  "detail": "Document library with name 'Financial Documents Q1 2024' already exists"
}

422 Validation Error

{
  "detail": [
    {
      "loc": ["body", "name"],
      "msg": "ensure this value has at least 3 characters",
      "type": "value_error.any_str.min_length"
    }
  ]
}

Error Handling in Python

import requests

try:
    response = requests.post(
        "http://localhost:8000/api/v1/document-libraries",
        json={"name": "My Library"}
    )
    response.raise_for_status()
    library = response.json()
except requests.exceptions.HTTPError as e:
    if e.response.status_code == 404:
        print("Library not found")
    elif e.response.status_code == 409:
        print("Library already exists")
    elif e.response.status_code == 422:
        print(f"Validation error: {e.response.json()}")
    else:
        print(f"HTTP error: {e}")
except requests.exceptions.RequestException as e:
    print(f"Request failed: {e}")

Best Practices

1. Naming Conventions

• Use descriptive, meaningful names
• Include time periods or versions when relevant
• Names must start with a letter and contain only letters, digits, spaces, and underscores
• Example: “Financial Reports Q1 2024”, “Legal Contracts 2024_v2”

2. Tagging Strategy

• Use consistent tag naming (lowercase, hyphenated)
• Create a tag taxonomy for your organization
• Examples: “finance”, “q1-2024”, “high-priority”, “audited”

3. Organization Patterns

• By Time Period: “Q1 2024 Documents”, “2024 Annual Reports”
• By Department: “HR Documents”, “Legal Contracts”, “Finance Reports”
• By Project: “Project Alpha Documents”, “Customer Onboarding”
• By Status: “Draft Documents”, “Approved Documents”, “Archived”

4. Aggregate Metrics

• Metrics are automatically computed when document sets are added/removed
• Use metrics for monitoring and reporting
• Metrics include: total_document_sets, total_documents, total_size_bytes

5. Performance Considerations

• Use pagination (skip/limit) for large result sets
• Use search/filter endpoints instead of fetching all libraries
• Cache frequently accessed library metadata

Troubleshooting

Issue: Library Not Found

Symptom: 404 error when accessing a library

Solutions:

1. Verify the library_id is correct
2. Check if the library was deleted
3. Use the list endpoint to see all available libraries

Issue: Duplicate Library Name

Symptom: 409 conflict error when creating a library

Solutions:

1. Choose a different name
2. Update the existing library instead
3. Delete the old library if no longer needed

Issue: Validation Errors

Symptom: 422 validation error

Solutions:

1. Check name length (3-128 characters minimum)
2. Ensure name starts with a letter (a-z, A-Z)
3. Verify description length (max 2000 characters)
4. Use only alphanumeric characters, spaces, and underscores in names (no hyphens or special characters)
5. Name pattern: ^[a-zA-Z][a-zA-Z0-9_ ]*$ (example: “My Library Name” or “Test_Library_123”)

Issue: API Server Not Running

Symptom: Connection refused errors

Solutions:

1. Start the API server: uvicorn app.main:app --reload
2. Verify the server is running on the correct port (default: 8000)
3. Check firewall settings

Advanced Topics

Database Schema

The Document Library feature uses a hybrid storage approach:

1. Library Metadata (KeyValueStorage - JSON):

CREATE TABLE document_libraries (
    key VARCHAR PRIMARY KEY,           -- library id
    data JSON NOT NULL,                -- Full library metadata as JSON
    created_at TIMESTAMP NOT NULL,
    updated_at TIMESTAMP NOT NULL
);

JSON Structure in data column:

{
  "library_id": "uuid",
  "name": "Library Name",
  "description": "Description",
  "purpose": "Purpose",
  "tags": ["tag1", "tag2"],
  "created_by": "user@example.com",
  "href": "https://..."
}

2. Junction Table (Relational Storage):

CREATE TABLE library_documentset_junction (
    library_id VARCHAR NOT NULL,
    document_set_id VARCHAR NOT NULL,
    added_at TIMESTAMP NOT NULL,
    PRIMARY KEY (library_id, document_set_id)
);

Architecture Notes:

• Library metadata uses KeyValueStorage interface (same as Document Sets)
• JSON storage enables flexible schema and CAMS compatibility
• Junction table uses relational storage for performance
• Follows Document Sets pattern: JSON for assets, SQL for relationships

Hexagonal Architecture

The Document Library implementation follows hexagonal architecture (ports & adapters):

• Domain Layer: Pure Python business logic (DocumentLibrary model)
• Ports: Repository interface (DocumentLibraryRepositoryPort)
• Adapters: DuckDB implementation (DuckDBDocumentLibraryRepository)
• Application Layer: Service orchestration (DocumentLibraryService)
• API Layer: FastAPI routes and DTOs

This architecture allows for:

• Easy testing with mocked repositories
• Swapping storage backends (e.g., PostgreSQL, MongoDB)
• Clear separation of concerns

Related Documentation

• ARCHITECTURE.md - System architecture overview
• OPERATOR_REFERENCE.md - Operator documentation
• USER_GUIDE_PIPELINE_SETUP.md - Pipeline setup guide

Support

For issues or questions:

1. Check the troubleshooting section above
2. Review the API documentation at http://localhost:8000/docs
3. Consult the ARCHITECTURE.md for technical details
4. Open an issue on the project repository

This site is open source. Improve this page.