PostgreSQL

A PostgreSQL connection reads whole tables out of a Postgres database and lands each one as its own Bronze dataset. It is pull-based. Beetl's cloud never dials your database. Instead beetl-connector, running inside your network, receives an extraction job, runs the query locally, and pushes Parquet back to Beetl over the ingest API.

That means a PostgreSQL connection has a hard prerequisite: a connector deployed at a site, showing Online. See edge deployment.

Configuration

ConnectionDetails::PostgreSql carries exactly five fields. These are the fields the New Connection wizard asks for and the only ones stored.

Field	Type	Notes
`host`	string	Hostname or IP reachable from the connector, not from the internet.
`port`	u16	Postgres port.
`database`	string	Database name.
`username`	string	Role used for extraction. Read access is enough.
`password`	secret	Stored as a `Secret` newtype.

There is no TLS mode field, no SSH tunnel field, no schema filter and no connection-options field. The connector builds an ODBC connection string from those five values and connects through the PostgreSQL Unicode driver:

Driver={PostgreSQL Unicode};Server=<host>;Port=<port>;Database=<database>;Uid=<username>;Pwd=<password>;

Credentials are not encrypted at rest

A secrets backend with AES-256-GCM exists in the server, but connection credentials have not been pointed at it yet. They sit in the event store in plaintext, and they travel to the connector inside the job payload. Use a dedicated read-only role, scoped to the tables you intend to extract.

Test and discovery

A new pull-based connection starts in Pending. The connector picks up a connectivity test, and on success reports back the server version plus a schema and table listing. The connection moves to Active. On failure it moves to Failed and shows the driver error.

Discovery runs two catalog queries against the source database. Schemas come from information_schema.schemata. Tables come from information_schema.tables filtered to table_type = 'BASE TABLE'.

That filter is worth reading twice. Views, materialized views and foreign tables are not discovered, so you cannot select them. If the data you want lives behind a view, materialize it into a table on the source side, or extract the base tables and reassemble them in a Beetl pipeline.

Selecting tables

The connection detail page shows the discovered schemas as a collapsible tree with checkboxes. Selections are saved as schema.table strings, one per selected table, and posted to the connection's selections endpoint. An invalid reference is rejected outright.

Selection is all-or-nothing per table. You cannot pick columns, and you cannot attach a row filter or a WHERE clause.

What extraction actually runs

For each selected table the connector issues one query:

SELECT * FROM schema.table

The full result is written to Parquet in memory and posted to the ingest API as one request:

curl -X POST "https://<your-beetl-host>/api/ingest/<source_id>?feed=<schema>_<table>" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/vnd.apache.parquet" \
  --data-binary @extract.parquet

The feed name is the table reference with dots replaced by underscores, so public.orders extracts to the feed public_orders. Feed names are stable across runs, which is what keeps a table landing in the same dataset every time.

What lands in the dataset

One Bronze dataset per selected table, stored as Delta. Columns and types come from the Arrow schema of the extracted Parquet, so the dataset mirrors the source table.

On the second and later runs the incoming schema is validated against the existing dataset. Same column count and names, types allowed to widen, nullability allowed to relax. Anything else is rejected with SchemaMismatch and the run fails rather than silently rewriting the dataset. A column added in Postgres will therefore break the next extraction until the dataset is dealt with.

Running it

Two triggers exist. Extract Now runs an extraction immediately from the connection card or the graph node. A cron schedule runs it repeatedly, and ScheduleConfig has only two shapes: OnDemand, or Cron with an expression.

Known limits

Full table every run. There is no incremental extraction, no watermark column and no CDC. A 40 million row table is read and transferred in full on every schedule tick.
No schedule editing. The UpdateSchedule command exists in the domain but no web route calls it. The schedule is fixed at creation and shown read-only afterwards.
No connection editing. UpdateConnectionDetails exists as a command with nothing mounted on it. A rotated password means deleting the connection and creating a new one.
Base tables only. Views and materialized views are invisible to discovery.
No run history. There is no per-connection runs endpoint and no log surface, so a failed extraction shows as a status and an error message with no trace behind it.
Credentials in plaintext. As above.
The connector is a single point of failure. Deleting a connector auto-pauses every connection using it. Reassign Connector reattaches one, but that control exists only on the connection detail page at /web/connections/{id}, which nothing links to.