Pipelines API

CRUD operations for pipelines including create, list, update, delete, clone, export, and import.

CRUD operations for pipelines.

List pipelines

curl http://localhost:8080/api/pipelines

Response: array of pipeline summaries (no nodes/edges for performance).

[
  {
    "id": "abc-123",
    "name": "Daily ETL",
    "description": "Load and transform user data",
    "schedule": "0 6 * * *",
    "enabled": true,
    "tags": ["production"],
    "node_count": 5,
    "edge_count": 4,
    "created_at": "2024-01-10T12:00:00Z",
    "updated_at": "2024-01-15T08:30:00Z"
  }
]

Pagination

curl "http://localhost:8080/api/pipelines?page=1&per_page=20"

Paginated response:

{
  "items": [...],
  "total": 42,
  "page": 1,
  "per_page": 20,
  "pages": 3
}

Get pipeline

curl http://localhost:8080/api/pipelines/{id}

Returns the full pipeline including nodes, edges, hooks, and all config.

Create pipeline

curl -X POST http://localhost:8080/api/pipelines \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Pipeline",
    "description": "Loads user data",
    "enabled": true,
    "schedule": "0 6 * * *",
    "schedule_timezone": "America/New_York",
    "tags": ["etl", "users"],
    "params": {"limit": "1000"},
    "nodes": [
      {
        "id": "src",
        "type": "source_db",
        "name": "Load Users",
        "config": {"conn_id": "prod-pg", "query": "SELECT * FROM users LIMIT ${param.limit}"},
        "position": {"x": 100, "y": 200}
      },
      {
        "id": "sink",
        "type": "sink_file",
        "name": "Export",
        "config": {"path": "/data/users.csv", "format": "csv"},
        "position": {"x": 400, "y": 200}
      }
    ],
    "edges": [
      {"from": "src", "to": "sink"}
    ]
  }'

Returns 201 Created with the full pipeline including generated id.

Update pipeline

curl -X PUT http://localhost:8080/api/pipelines/{id} \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My Pipeline (Updated)",
    "enabled": false,
    "nodes": [...],
    "edges": [...]
  }'

Delete pipeline

curl -X DELETE http://localhost:8080/api/pipelines/{id}

Returns 204 No Content.

Clone pipeline

curl -X POST http://localhost:8080/api/pipelines/{id}/clone

Creates a copy with "(Copy)" appended to the name.

Export / Import

Export

curl http://localhost:8080/api/pipelines/{id}/export

Returns the pipeline as a standalone JSON file (suitable for import).

Import

curl -X POST http://localhost:8080/api/pipelines/import \
  -H "Content-Type: application/json" \
  -d @pipeline.json

Validate

curl http://localhost:8080/api/pipelines/{id}/validate

Returns validation results:

{
  "valid": true,
  "errors": [],
  "warnings": ["Node 'transform1' has no downstream connections"]
}

Versions

List versions

curl http://localhost:8080/api/pipelines/{id}/versions

Rollback

curl -X POST http://localhost:8080/api/pipelines/{id}/rollback \
  -H "Content-Type: application/json" \
  -d '{"version_id": "version-abc"}'

Bulk operations

curl -X POST http://localhost:8080/api/pipelines/bulk \
  -H "Content-Type: application/json" \
  -d '{"action": "delete", "ids": ["id1", "id2", "id3"]}'

Validate nodes

Validate individual nodes within a pipeline:

curl -X POST http://localhost:8080/api/pipelines/{id}/validate-nodes

Dead Letter Queue

List DLQ entries

curl http://localhost:8080/api/pipelines/{id}/dlq

Query parameters:

include_resolved (bool, default false) -- include already-resolved entries
limit (int, default 50, max 500) -- number of entries to return

Resolve a DLQ entry

curl -X POST http://localhost:8080/api/pipelines/{id}/dlq/{dlqId}/resolve

Returns {"status": "resolved"}.

Webhook trigger

Trigger a pipeline run via webhook token. The pipeline must have a webhook_token configured. Rate-limited to one trigger per pipeline per 10 seconds.

curl -X POST "http://localhost:8080/api/pipelines/{id}/webhook?token=YOUR_TOKEN"

The token can also be passed via the X-Webhook-Token header:

curl -X POST http://localhost:8080/api/pipelines/{id}/webhook \
  -H "X-Webhook-Token: YOUR_TOKEN"

Returns the run ID and status.

Dependency status

Check whether a pipeline's upstream dependencies are satisfied:

curl http://localhost:8080/api/pipelines/{id}/deps

Returns an array of dependency objects with pipeline_id, name, last_status, and satisfied (boolean).

Summary

Get all pipelines with last-run statistics (status, success/fail counts):

curl http://localhost:8080/api/pipelines/summary

Impact analysis

Check downstream dependencies:

curl http://localhost:8080/api/pipelines/{id}/impact

On this page