Runs API

Trigger, monitor, and manage pipeline runs.

Trigger a run

curl -X POST http://localhost:8080/api/pipelines/{pipeline_id}/run \
  -H "Content-Type: application/json" \
  -d '{"params": {"date": "2024-01-15"}}'

Returns 202 Accepted immediately. The pipeline executes asynchronously.

{
  "id": "run-abc-123",
  "pipeline_id": "pipeline-xyz",
  "status": "pending"
}

Track progress via WebSocket events or poll the run endpoint.

Get a run

curl http://localhost:8080/api/runs/{run_id}

{
  "id": "run-abc-123",
  "pipeline_id": "pipeline-xyz",
  "status": "success",
  "started_at": "2024-01-15T06:00:00Z",
  "finished_at": "2024-01-15T06:01:23Z",
  "node_runs": [
    {
      "id": "nr-1",
      "node_id": "source1",
      "status": "success",
      "row_count": 1500,
      "duration_ms": 2340,
      "started_at": "2024-01-15T06:00:00Z"
    },
    {
      "id": "nr-2",
      "node_id": "transform1",
      "status": "success",
      "row_count": 1200,
      "duration_ms": 450,
      "started_at": "2024-01-15T06:00:02Z"
    }
  ]
}

List runs for a pipeline

curl http://localhost:8080/api/pipelines/{pipeline_id}/runs

Returns the 50 most recent runs. Supports pagination:

curl "http://localhost:8080/api/pipelines/{pipeline_id}/runs?page=1&per_page=20"

Run status lifecycle

Status	Description
`pending`	Run created, waiting for execution slot
`running`	Pipeline is executing
`success`	All nodes completed successfully
`failed`	One or more nodes failed
`cancelled`	Manually cancelled by user

Cancel a run

curl -X POST http://localhost:8080/api/runs/{run_id}/cancel

Cancels the running pipeline. Nodes currently executing will finish, but no new nodes will start.

Resume a failed run

curl -X POST http://localhost:8080/api/runs/{run_id}/resume

Re-runs the pipeline from the first failed node. Nodes that already succeeded are skipped.

Note: Only runs with failed status can be resumed.

Get run logs

curl http://localhost:8080/api/runs/{run_id}/logs

[
  {
    "run_id": "run-abc",
    "node_id": "source1",
    "level": "info",
    "message": "Starting node: Fetch Users (source_api)",
    "timestamp": "2024-01-15T06:00:00Z"
  },
  {
    "run_id": "run-abc",
    "node_id": "source1",
    "level": "info",
    "message": "Node completed: 1500 rows in 2.3s (652 rows/sec)",
    "timestamp": "2024-01-15T06:00:02Z"
  }
]

Export logs

Download logs as a file:

curl http://localhost:8080/api/runs/{run_id}/logs/export

Node data preview

View the output data from a specific node:

curl http://localhost:8080/api/runs/{run_id}/nodes/{node_id}/preview

Returns the first 50 rows of the node's output dataset.

Node profile

View data profiling information:

curl http://localhost:8080/api/runs/{run_id}/nodes/{node_id}/profile

Dry run

Execute a pipeline in preview mode (no persistent run, first 10 rows only):

curl -X POST http://localhost:8080/api/pipelines/{pipeline_id}/dry-run

Returns per-node preview data:

{
  "source1": {
    "node_id": "source1",
    "name": "Fetch Users",
    "status": "success",
    "columns": ["id", "name", "email"],
    "rows": [{"id": 1, "name": "Alice", "email": "alice@example.com"}]
  }
}

Backfill

Run a pipeline for a date range:

curl -X POST http://localhost:8080/api/pipelines/{pipeline_id}/backfill \
  -H "Content-Type: application/json" \
  -d '{"start_date": "2024-01-01", "end_date": "2024-01-31"}'

Returns an array of run IDs (one per date).

Calendar view

Get runs grouped by date for a calendar visualization:

curl http://localhost:8080/api/runs/calendar

Query parameters:

days (int, default 90, max 365) -- number of days of history to return

Returns an array of objects with date, success count, failure count, and total runs per day.

Utility endpoints

These endpoints provide cross-cutting platform functionality.

Lineage

Get the lineage graph across all pipelines (connections, shared tables, variable references):

curl http://localhost:8080/api/lineage

Scheduler status

Get the current state of all scheduled pipelines:

curl http://localhost:8080/api/scheduler/status

Returns an array of schedule entries with pipeline ID, cron expression, next run time, and enabled status.

Dashboard

Get aggregated dashboard data in a single request (pipeline summaries, recent runs, 7-day trends, top failing pipelines):

curl http://localhost:8080/api/dashboard

{
  "pipelines": [...],
  "recent_runs": [...],
  "trends": [
    {"date": "2024-01-09", "success": 12, "failed": 1, "total": 13}
  ],
  "top_failing": [
    {"pipeline_id": "abc", "name": "Broken ETL", "fail_count": 3}
  ]
}

Search

Search across pipelines, connections, and variables:

curl "http://localhost:8080/api/search?q=users&limit=30"

Query parameters:

q (string, required) -- search query (case-insensitive substring match)
limit (int, default 30, max 100) -- max results

System info

Get system status including database size, pipeline count, and active run queue:

curl http://localhost:8080/api/system/info

{
  "version": "0.1.0-dev",
  "db_size": 1048576,
  "db_size_mb": "1.00 MB",
  "pipelines": 12,
  "active_runs": 2,
  "max_concurrent_runs": 5
}

System purge

Delete old run data. Requires settings.edit permission.

curl -X POST http://localhost:8080/api/system/purge \
  -H "Content-Type: application/json" \
  -d '{"days": 30}'

Deletes runs older than the specified number of days (default: 30). Returns {"deleted": <count>, "days": 30}.

WebSocket

Connect to the real-time event stream:

wscat -c ws://localhost:8080/api/ws

Receives JSON messages for run status changes, node completions, and log events. Used by the UI for live updates.

On this page