REST APIs

Connect Brokoli to REST APIs for reading and writing data.

Connection config

Field	Value	Example
`type`	`http`
`host`	Base URL	`https://api.example.com`
`port`	Port (optional)	`443`
`login`	Username (for Basic Auth)	`api_user`
`password`	Password (for Basic Auth)	`secret`
`extra`	JSON with headers, auth config	`{"headers": {"X-API-Key": "..."}}`

Create via API

curl -X POST http://localhost:8080/api/connections \
  -H "Content-Type: application/json" \
  -d '{
    "conn_id": "external-api",
    "type": "http",
    "host": "https://api.example.com",
    "login": "",
    "password": "",
    "extra": "{\"headers\": {\"Authorization\": \"Bearer sk-live-xxx\"}}"
  }'

Authentication methods

API key in header

{
  "extra": "{\"headers\": {\"X-API-Key\": \"your-api-key\"}}"
}

Bearer token

{
  "extra": "{\"headers\": {\"Authorization\": \"Bearer your-token\"}}"
}

Basic auth

Set login and password fields. Brokoli sends them as HTTP Basic Authentication.

Reading data (source_api)

Use conn_id to inherit the base URL and auth from the connection:

{
  "type": "source_api",
  "config": {
    "conn_id": "external-api",
    "url": "/v1/users",
    "method": "GET",
    "json_path": "data.users"
  }
}

The full URL is constructed by combining the connection's host with the node's url.

Custom headers per request

Override or add headers at the node level:

{
  "conn_id": "external-api",
  "url": "/v1/users",
  "headers": {"Accept": "application/json", "X-Custom": "value"}
}

Node-level headers merge with connection-level headers.

JSON path extraction

When the API returns nested data, use json_path to extract the relevant array:

{"json_path": "data.users"}

Pagination

For APIs that paginate results:

{
  "conn_id": "external-api",
  "url": "/v1/events",
  "pagination": {
    "type": "offset",
    "limit_param": "limit",
    "offset_param": "offset",
    "page_size": 100
  }
}

Brokoli fetches pages until an empty result is returned.

Writing data (sink_api)

POST the dataset to an API endpoint:

{
  "type": "sink_api",
  "config": {
    "conn_id": "external-api",
    "url": "/v1/import",
    "method": "POST",
    "headers": {"Content-Type": "application/json"},
    "batch_size": 100
  }
}

With batch_size, rows are sent in batches of 100 per request instead of all at once.

Testing the connection

curl -X POST http://localhost:8080/api/connections/external-api/test

The test sends a GET request to the base URL with the configured authentication. A successful response (2xx/3xx) means the connection works.

{"success": true, "message": "Authenticated successfully (HTTP 200)"}

Using variables

Reference variables in the connection's extra field or in node configs:

{
  "url": "/v1/events?since=${param.date}",
  "headers": {"Authorization": "Bearer ${var.api_token}"}
}

On this page