Pipelines

Pipelines provide orchestration capabilities for coordinating complex data workflows. Built on Kubeflow Pipelines (KFP), they allow you to define workflows as Python functions and execute them in a managed environment.

This section covers:

• Pipelines
  • Creating a Pipeline
  • Retrieving Pipelines
  • Duplicating a Pipeline
  • Deleting a Pipeline
  • Running a Pipeline
  • Using Parameter Sets with Pipelines
• Pipeline Components
  • Overview
  • Listing Pipeline Components
  • Using the PipelineComponentId Enum
  • Built-in Components
  • Creating Custom Components
  • Deleting Custom Components
• Pipeline Settings
  • DataStage Optimized Pipeline Runner
• Pipeline Examples
  • CEL Expressions
  • User Variables via Set User Variable Component
  • If / Else Conditions
  • Environment Variables in Bash Scripts