Integrations
Connections from an installed Connect package: credentials, OAuth, syncs and schedules.
An integration connects Beetl to one system, such as a CRM or an ERP, through a Connect package installed in your tenant. You create a connection with your account's settings and credentials, then add syncs. Each sync reads one stream of records into its own dataset. For how these objects fit together, see Connection types, connections and syncs.
Before you start
A TenantAdmin installs the package once per tenant. In Settings → Tenant → Custom connection
types, choose Add custom connection type and upload the .tgz file produced by
beetl-connect pack. It is ready to use once its status shows Ready. Packages are written
against the open-source @beetlio/connect SDK; to write one,
see Build integrations.
You also need the access details the provider issues, such as an API key or an OAuth app's client ID and secret.
Create the connection
Choose the integration
On Home, right-click the canvas and choose New connection. Pick the system under Integrations, or type its name into the search box.
Enter the details
Give the connection a name, and optionally a description. Fill in the package's fields under Configuration, such as an account ID or a region.
Add credentials
Enter the credentials the package asks for. If someone in your tenant already saved credentials for this integration, you can pick them from Use saved credentials instead. Secret values are never shown again once saved.
Choose Connect. A package that needs no credentials shows Create connection instead.
Check the result
If the package supports a connection test, Beetl runs it straight away and shows Connection verified, or the reason the test failed. Fix the settings and choose Test again.
Once the connection exists, changes to its name, description and configuration save on their own. You can add syncs in the same dialog, or close it with Finish and add them later from Home.
Connecting with OAuth
For a package that uses OAuth, first add this redirect URL to the OAuth app at the provider:
https://app.beetl.io/web/-/connections/oauth/callbackThen enter the OAuth app's credentials, or pick saved ones, and choose Connect with OAuth. Sign in and approve access in the provider's popup window. If your browser blocks the popup, allow popups for Beetl and try again.
Saved OAuth app credentials can be reused, but each connection is authorized on its own. If the provider later stops accepting the authorization, the connection shows Reauthorization required and its syncs stop running. Open the connection and choose Reconnect with OAuth.
Add a sync
Pick a stream
On Home, choose + on the connection's card. Pick a stream from the list, then choose Configure sync. You can sync the same stream twice with different settings.
Set it up
Fill in the sync's own fields, if the package defines any. To run it automatically, turn on Schedule and choose every hour, day, week or month, with a time and a timezone. For anything else, open Use an advanced cron expression and enter six fields, starting with seconds. Without a schedule, the sync runs only when you start it.
Choose partitioning
Under Partitioning, choose None, Sync time (Beetl adds a dt column for the day,
week, month or year of each run) or Output field. Partitioning cannot change after the sync
is created, and Merge syncs cannot be partitioned. See
Configured syncs.
Create the sync and its dataset
Choose Create sync. Beetl creates a new output dataset with a suggested name, which you can change under Data set name. Names use lowercase letters, digits and underscores, and the name is what you write in SQL.
Run it
Choose Run now to start a first run instead of waiting for the schedule.
Check runs and logs
Click the connection on Home to open its details pane:
- The Details tab lists each sync with its latest run, and links to its runs and logs.
- The Runs tab lists each run with its status, duration and the rows it added, updated and removed. Expand a failed run to read the reason.
- The Logs tab shows a run's log lines.
After the first successful run, the dataset turns from Pending to Initialized and you can query it.
Change, pause or remove
To edit a connection, open its details pane and choose Configure. This reopens the setup, where you can change settings, replace credentials and edit each sync's fields and schedule.
Changing a sync's fields resets its checkpoint, so the next run reads everything again. Changing only its schedule does not. Before you change a Merge sync, read Checkpoints.
Pause on a sync stops it from running until you choose Resume. Pausing the connection stops all of its syncs. A run that has already started still finishes.
Deleting a sync keeps its dataset. A connection can be deleted once it has no syncs.
Related
Last updated on