Connection types, connections and syncs

Data that Beetl copies into a tenant arrives through a small set of objects. A Connection Type defines how to talk to a kind of system. A Connection is your access to one account or endpoint of that system. A Configured Sync reads one stream of records through a connection. A File Set holds files you upload. Whichever of these produces the data writes to exactly one dataset through an Output Link.

Registered and external datasets are read in place and use none of these. See Connect your data for all five ways in.

Connection types

Webhook is the only connection type built in. It gives a connection a URL and a token, and anything that can POST JSON can send data to it. See Webhook.

Every other type is a Connect package, which the app calls an integration: TypeScript written against the open-source @beetlio/connect SDK (Apache-2.0) and uploaded by a TenantAdmin under Settings → Tenant → Custom connection types. See Build integrations.

A package's manifest declares its settings and credential fields, its authentication mode (none, credentials, or OAuth2), and the syncs it offers, each with a record schema, its settings and an optional checkpoint. That contract is fixed once installed. To ship a change, upload the package as a new connection type and move connections and syncs to it. See The contract rule.

Only packages built with a current SDK (0.5 or later) are supported. Beetl rejects an older package with "Unsupported integration version", and it has to be rebuilt and uploaded again. See Rebuilding older packages.

Connections

A connection is one tenant's access to one provider. Create one by right-clicking the canvas on Home and choosing New connection. Its name, settings and credentials can all be changed later. A connection is Enabled, Paused (no syncs start and its webhook refuses deliveries) or Deleted, which is final.

A connection also has an authentication state: not required, pending, configured, or reauthorization required. Syncs only run once it is configured.

Credentials are saved as named, reusable records, so another connection of the same type can pick them from Use saved credentials. Secret values are never shown again once saved.

For OAuth, you save the OAuth app credentials, then choose Connect with OAuth and authorize in the provider's popup. Each connection holds its own authorization. If the provider later refuses to refresh access, the connection asks you to reconnect. See Integrations.

Configured syncs

A configured sync is one runnable stream of records on a connection. The package declares which streams exist; you add the ones you want with + on the connection's card on Home. A connection can carry several syncs, including the same stream twice with different settings.

A sync runs on demand with Run now, or on a schedule (hourly, daily, weekly or monthly in a timezone you choose, or any cron expression). Only one run per sync is active at a time.

You choose how records are written for each sync, under Write mode when you add or edit it. The default is Append, and you can change it later.

Each run commits as one Delta transaction, so a failed run leaves the dataset unchanged. See Write modes for which one suits a sync.

Partitioning is chosen when the sync is created and cannot change later. Options are none, sync time (Beetl adds a dt column for the day, week, month or year of the run, and a Replace run replaces only its own period), or an output field, any top-level text, number or boolean field of the record. With Replace, an output-field partitioned run still replaces the whole dataset.

Limits and retries

A run that hits a limit fails without writing anything. Beetl retries failures on its own side; a run that fails in the integration's code is not retried. Integration code runs in an isolated sandbox that can reach public hosts over HTTPS. See The sandbox.

Checkpoints

A package can save a checkpoint, such as a cursor or timestamp, so the next run reads only what changed. The checkpoint advances only after the write commits, so a failed run never skips records. An Append sync resumes from its last checkpoint. A Replace run always starts from the beginning.

Changing a sync's settings, including its write mode, resets its checkpoint. The next run then reads everything again and replaces the dataset's contents, and later runs continue as before. Changing only the schedule resets nothing.

File Sets

A File Set is a named folder of uploaded files imported into one dataset, in CSV, newline-delimited JSON or Parquet. Create one from the File sets card on Home. It has no connection and no schedule; Beetl re-imports when the files change. See File Sets.

Output links

An output link connects a producer (webhook connection, sync or File Set) to its dataset. Each producer has one link and each dataset one producer, so two syncs never write the same table. A link can only point at a dataset that has never been written, and a producer without a link cannot write: a webhook answers 404 OUTPUT_NOT_LINKED and a sync cannot start.

Runs, logs and deleting

A connection's details pane on Home has a Runs tab (status, duration, rows added by Append runs, and the reason for a failure) and a Logs tab with each run's log lines, kept for 30 days. A started run cannot be cancelled; pausing stops future runs only. See Runs.

Deleting a producer never deletes its data. Deleting a connection, sync or File Set removes its output link and keeps the dataset, and a connection with syncs must have its syncs deleted first. Because links only attach to unwritten datasets, a dataset whose producer was deleted cannot get a new one.

Installing packages and managing saved credentials is admin-only; everything else on this page is open to every user. See Tenants and roles.

Related