Deploying Pipelines

Deploy Python-defined pipelines to a Brokoli server using the CLI.

brokoli deploy

Deploy a pipeline file to a running server:

brokoli deploy my_pipeline.py --server http://localhost:8080

With authentication:

brokoli deploy my_pipeline.py \
  --server http://localhost:8080 \
  --api-key brk_your_key_here

If the pipeline has a pipeline_id, an existing pipeline with that ID is updated. Otherwise, a new pipeline is created.

Deploy all pipelines in a directory

brokoli deploy pipelines/ --server http://localhost:8080

This discovers all .py files containing Pipeline() definitions and deploys them.

brokoli compile

Preview the YAML that would be sent to the server without deploying:

brokoli compile my_pipeline.py

Output:

name: Daily ETL
schedule: 0 6 * * *
enabled: true
nodes:
- id: load_a1b2c3
  type: source_db
  name: Load Orders
  config:
    query: |
      SELECT *
      FROM orders
      WHERE date = CURRENT_DATE - 1
    conn_id: warehouse
- id: save_d4e5f6
  type: sink_db
  name: Write DWH
  config:
    table: fact_orders
    mode: append
    conn_id: warehouse
edges:
- from: load_a1b2c3
  to: save_d4e5f6

For JSON output instead, use --format json:

brokoli compile my_pipeline.py --format json

brokoli validate

Check a pipeline for errors locally without connecting to a server:

brokoli validate my_pipeline.py

Validates:

All edges reference existing nodes
No duplicate node IDs
Required config fields are present (query, table, url, etc.)
Connection IDs exist on the server (if --server provided)

brokoli export

Export the pipeline to a file:

brokoli export my_pipeline.py -o pipeline.yaml

Defaults to YAML. For JSON:

brokoli export my_pipeline.py -o pipeline.json --format json

CI/CD with GitHub Actions

name: Deploy Pipelines
on:
  push:
    branches: [main]
    paths: ['pipelines/**']

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install Brokoli SDK
        run: pip install brokoli

      - name: Validate pipelines
        run: brokoli validate pipelines/

      - name: Preview YAML (optional)
        run: brokoli compile pipelines/

      - name: Deploy pipelines
        env:
          BROKOLI_TOKEN: ${{ secrets.BROKOLI_TOKEN }}
        run: brokoli deploy pipelines/ --server ${{ secrets.BROKOLI_SERVER }}

Git Sync (Enterprise)

Git Sync automatically deploys pipelines when you push to a Git repository. Configure a repository and branch, and Brokoli pulls pipeline definitions on push (via webhook) or on a schedule.

Key points:

Pipelines are matched by pipeline_id (set in the Python SDK)
Pipelines created via Git Sync have source: "git" and cannot be edited in the UI
Pull and push operations are available via the API:

# Pull latest from configured repo
curl -X POST http://localhost:8080/api/git/pull

# Push a UI-created pipeline to the repo
curl -X POST http://localhost:8080/api/git/push/{pipeline_id}

Environment-specific configuration

Use variables to handle environment differences. Define pipeline logic once, and set connection details per environment:

from brokoli import Pipeline, source_db, sink_db

with Pipeline("etl", pipeline_id="daily-etl") as p:
    src = source_db("Load", conn_id="${var.source_db}", query="SELECT * FROM events")
    sink = sink_db("Save", conn_id="${var.target_db}", table="events", mode="append")
    src >> sink

Preview the YAML to verify variable references:

brokoli compile pipeline.py

Then set source_db and target_db variables differently in staging vs production via the Brokoli UI or API.

On this page