Scheduling

Run pipelines on a schedule with cron expressions, timezone support, catch-up runs, and cross-pipeline dependencies.

Cron expressions

Brokoli uses standard 5-field cron expressions:

┌───────── minute (0-59)
│ ┌─────── hour (0-23)
│ │ ┌───── day of month (1-31)
│ │ │ ┌─── month (1-12)
│ │ │ │ ┌─ day of week (0-6, Sun=0)
│ │ │ │ │
* * * * *

Common schedules

Set the schedule in the pipeline config:

UI

In the pipeline editor, open Settings and enter the cron expression in the Schedule field.

API

curl -X PUT http://localhost:8080/api/pipelines/{id} \
  -H "Content-Type: application/json" \
  -d '{"schedule": "0 6 * * *", "enabled": true}'

Python SDK

with Pipeline("daily-etl", schedule="0 6 * * *") as p:
    ...

Timezone support

By default, schedules run in UTC. Set schedule_timezone to use a different timezone:

{
  "schedule": "0 9 * * 1-5",
  "schedule_timezone": "America/New_York"
}

This runs at 9 AM Eastern time on weekdays. Brokoli handles DST transitions automatically.

Valid timezones are IANA timezone names (e.g., Europe/London, Asia/Tokyo, US/Pacific).

Catch-up runs

When Brokoli restarts, the scheduler checks each pipeline's last run against its schedule. If a run was missed during downtime (and it's been less than 24 hours), it triggers a catch-up run automatically.

This means:

• Short maintenance windows don't cause missed runs
• Runs missed more than 24 hours ago are skipped (to prevent a flood of backfill runs)
• Catch-up runs execute before the cron scheduler starts

Pipeline dependencies

A pipeline can depend on other pipelines. The scheduler only runs the dependent pipeline if all its dependencies succeeded in their most recent run.

{
  "depends_on": ["pipeline-id-1", "pipeline-id-2"]
}

If any dependency's last run is not success, the scheduled run is skipped with a log message.

Check dependency status via API:

curl http://localhost:8080/api/pipelines/{id}/deps

SLA deadlines

Set a daily deadline by which a pipeline must complete:

{
  "sla_deadline": "09:00",
  "sla_timezone": "America/New_York"
}

If the pipeline hasn't completed by the deadline, notifications are triggered (requires enterprise notification provider or webhook hooks).

Backfill

Run a pipeline for a range of dates. Each date is passed as a date parameter:

curl -X POST http://localhost:8080/api/pipelines/{id}/backfill \
  -H "Content-Type: application/json" \
  -d '{"start_date": "2024-01-01", "end_date": "2024-01-31"}'

Your pipeline can reference ${param.date} in node configs to use the backfill date.

Scheduler status

View all active schedules and their next run times:

curl http://localhost:8080/api/scheduler/status

[
  {
    "pipeline_id": "abc123",
    "pipeline_name": "Daily ETL",
    "schedule": "0 6 * * *",
    "next_run": "2024-01-16T06:00:00Z",
    "last_run": "2024-01-15T06:00:02Z"
  }
]

Enabling and disabling

Set "enabled": false to pause a pipeline's schedule without deleting it. The scheduler unregisters disabled pipelines immediately.

curl -X PUT http://localhost:8080/api/pipelines/{id} \
  -H "Content-Type: application/json" \
  -d '{"enabled": false}'