Webhook
Receive JSON events at a per-connection endpoint.
A Webhook connection gives you an ingest URL and a token. Anything that can send an HTTP POST with a JSON body can deliver data to it, and each delivery becomes one row in a Bronze dataset. Webhook is built in and needs no integration package.
Create a webhook
On Home, right-click the canvas, choose New connection, and pick Webhook under Data sources. Give the connection a name.
Beetl creates the connection and links it to a new dataset straight away. The dataset name
defaults to the connection name followed by _events. You can change it in the Data set name
field.
The Webhook ready dialog shows the ingestion endpoint, the authorization token and an example request. Copy the token before you close the dialog, because the app does not show it again.
Send a first request, then choose Finish.
The dataset appears on Home as Pending and becomes Initialized once the first delivery is written.
Treat the token like a password
Anyone who has the token can write to this dataset. A token works only for its own connection, so create one webhook per sending system and keep each token in that system's secret store.
Send data
POST the JSON body to the endpoint from the dialog and pass the token as a bearer token:
curl -X POST "https://app.beetl.io/api/ingest/<connection_id>" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
--data-binary @payload.jsonThe body must be valid JSON of at most 1,000,000 bytes. Any JSON value is accepted, and one request is always one row. If you send an array of 50 events, they arrive together in a single row, so send one event per request when you want a row per event. For CSV, Parquet or larger loads, use a File Set. The endpoint is also described in the REST reference.
Responses
| Status | Code | Meaning |
|---|---|---|
202 | Accepted. The delivery is written to the dataset shortly after. | |
400 | INVALID_WEBHOOK_PAYLOAD | The body is not valid JSON. |
401 | MISSING_TOKEN | No Authorization: Bearer header was sent. |
401 | INVALID_TOKEN | The token does not belong to this connection, or the connection was deleted. |
404 | OUTPUT_NOT_LINKED | The connection is not linked to a dataset. |
413 | The body is larger than 1,000,000 bytes. | |
422 | CONNECTION_PAUSED | The connection is paused. Resume it from its pane on Home. |
422 | OBJECT_STORE_NOT_CONFIGURED | The tenant has no default object store. See Troubleshooting. |
422 | TENANT_SUSPENDED | The tenant is suspended. |
Beetl does not keep refused deliveries. Any response other than 202 means nothing was stored,
including requests sent while the connection is paused. Have your sender keep the event and try
again once the cause is fixed.
What lands in the dataset
Every webhook dataset has the same 8 columns. The JSON body is in payload, base64-encoded.
| Column | Type | Contents |
|---|---|---|
dt | string | Day the delivery was received, as YYYY-MM-DD in UTC |
message_id | string | Unique ID of the delivery |
ingested_at | timestamp (UTC) | When Beetl received the delivery |
connection_id | string | The webhook connection |
tenant_id | string | Your tenant |
content_type | string, nullable | The Content-Type header you sent |
payload_size | int64 | Size of the body in bytes |
payload | string | The JSON body, base64-encoded |
These columns never change, so a new field in your JSON never causes a schema conflict.
Check that data landed
A 202 confirms that Beetl accepted the delivery. To see the rows, click the dataset on Home and
open the Data preview tab, or query it. SQL uses the bare dataset name, whatever its tier:
WITH decoded AS (
SELECT ingested_at, CAST(decode(payload, 'base64') AS VARCHAR) AS body
FROM shop_orders_events
)
SELECT
ingested_at,
json_get_str(body, 'order_id') AS order_id,
TRY_CAST(json_as_text(body, 'total') AS DECIMAL(12, 2)) AS total,
json_get_str(body, 'customer', 'country') AS country
FROM decoded
ORDER BY ingested_at DESC
LIMIT 100If you read the same fields often, move this decode into a pipeline that writes a typed Silver dataset and query that instead. The JSON functions are listed under JSON functions.
Related
Last updated on