Connect your data

Data reaches Beetl in one of five ways. Three of them copy data into a dataset that Beetl writes and keeps up to date. The other two read data where it already sits, so nothing is copied.

All five start on Home. Right-click the canvas and choose New connection for an integration, a webhook or a PostgreSQL database, or Register data set for files in your object store. File Sets have their own File sets card on the canvas, and New file processor starts from a processor that reads the files you add to it.

Which one to use

Use an integration when the data lives in a system with an API, such as a SaaS application, and you want Beetl to fetch it on a schedule. Each integration is a Connect package that a TenantAdmin installs in the tenant. If the system you need has no package yet, a developer can write one; see Build integrations.

Use a webhook when your system can push events as they happen. Nothing needs installing, and nothing of Beetl's runs on your side.

Use a File Set for exports, one-off loads and anything that arrives as files. CSV, NDJSON and Parquet files are imported as they are. For PDFs, Word files, images and other documents, a file processor reads each document into one record, and for Excel workbooks with a fixed layout you can map cells to columns instead. See Documents and spreadsheets.

Use a registered dataset when the files are already in your own bucket and you do not want a second copy. Use an external dataset when the data sits in a PostgreSQL database and you want every query to see its current state. Each query then runs against your database, so heavy use adds load there.

How copied data lands

A webhook connection, a sync and a File Set's processor each write to exactly one dataset. The dataset shows as Pending on Home until its first successful write, then as Initialized. A run that fails leaves the dataset as it was, and an incremental sync never skips records because of a failed run.

New datasets are written to the object store marked as Beetl storage. That is Beetl's hosted store unless an admin has chosen your own bucket instead; see Tenants and roles.

The rules behind this, including write modes, partitioning, limits and what deleting a producer does, are on Connection types, connections and syncs.

Reaching your systems

Integrations and external datasets connect out from Beetl, so the system has to be reachable from the public internet. Integrations reach their providers over HTTPS. A system that only exists inside your private network cannot be reached directly. Export from it into a File Set, or have it push to a webhook.

In this section

Related