Webhook

A Webhook connection gives you a URL and a bearer token. Anything that can make an HTTP POST can send data to it. Beetl stores the raw body untouched and adds envelope columns around it. No connector is needed, and nothing of Beetl's runs inside your network.

Use Webhook when the sending system emits JSON or XML events and you do not control its output format. If you can produce Parquet with a known schema, use HTTP feed instead. That type gives you real columns rather than an encoded blob.

Creating one

Create a connection of type Webhook and give it a name. No further configuration exists. ConnectionDetails::Webhook is an empty variant, which is the honest signal that there is nothing to configure. On creation Beetl generates an ingest token and shows it once, in a dialog carrying the endpoint, the token and a ready-to-run cURL example.

The endpoint

POST /api/ingest/{source_id}

source_id is the data source id behind the connection, not the connection id. The dialog gives you the full URL, so you rarely need to build it yourself.

curl -X POST "https://<your-beetl-host>/api/ingest/<source_id>" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  --data-binary @payload.json

Authentication

Bearer token, and only bearer token. The middleware pulls the token from the Authorization header and validates it against the data source named in the path. A token is bound to one source, so a token leaked from one webhook cannot write to another.

There is no HMAC signature verification, no shared-secret body signing and no IP allowlist. If your sending system offers signed webhooks, Beetl cannot check the signature. There is also no replay protection, so a request replayed with a valid token is accepted and stored again as a new row.

Content type is not authenticated or restricted. The webhook path accepts any content type and records what you sent.

Responses

A 202 means the payload was published to the batching stream, not that it was written to Delta. The write happens asynchronously.

What lands in the dataset

One Bronze dataset per Webhook connection, with a fixed envelope schema. Your payload is base64-encoded into the payload column and the rest describes the request.

Decoding it back to fields is a SQL step. See JSON functions for the full set.

WITH decoded AS (
  SELECT CAST(decode(payload, 'base64') AS VARCHAR) as json_str
  FROM bronze.my_webhook
)
SELECT
  json_get_str(json_str, 'order_id') as order_id,
  json_get_float(json_str, 'total') as total
FROM decoded
LIMIT 100

Requests flow through NATS JetStream and are batched before being written, in WebhookMerge mode, partitioned daily by dt. Batching means a row is visible shortly after the 202, not instantly.

Known limits

• The feed query parameter does nothing here. The handler parses and validates it, then the webhook path ignores it and writes to the single implicit dataset. Named feeds work on HttpFeed, not on Webhook.
• Failed batches vanish. A batch that keeps failing is NACKed and dropped after five delivery attempts, with no user-visible trace. A dead-letter stream is provisioned in Kubernetes but nothing publishes to it.
• No delivery history. There is no ingestion monitoring surface, no run history and no rejected-payload view. A 202 followed by no rows gives you nothing to inspect.
• No signature verification or replay protection. As above.
• No structured columns. Every consumer of a webhook dataset pays the base64-decode cost on every query. If schema is stable, move to HttpFeed.
• Object store first. Until a tenant registers an object store, every request returns 422 OBJECT_STORE_NOT_CONFIGURED.