Datasets and tiers
The four kinds of dataset, how they are stored, what happens when a schema changes, and the Bronze / Silver / Gold convention.
A dataset is a named table in your tenant. You query it by name, use it as a pipeline source, chart it and mention it in chat. Every dataset has an Arrow schema, a status and a tier.
Four kinds of dataset
| Kind | Where it comes from | Written by | Format |
|---|---|---|---|
| Materialized | A webhook, sync or file processor through its output link, or a pipeline's destination | Beetl | Delta Lake table |
| App-published | The output of an app, named app_ plus the output name | The app, replaced on each run | Delta Lake table |
| Registered | Files already in one of your object stores | Nobody; read in place | Delta, Parquet, CSV or JSON |
| External | A table in a live PostgreSQL database | Nobody; queried in place | PostgreSQL table |
A pipeline writes only to a destination dataset it created itself; if a dataset with that name came from somewhere else, the run fails. Registered and external datasets are read-only everywhere: they can be pipeline sources but never destinations or output-link targets, and they never trigger a pipeline run. An app-published dataset can be a pipeline source too, but do not make it a destination, because the app replaces it on every run. See Connect your data for how to set up the other kinds.
Names and status
Queries use the bare dataset name, with no schema or tier prefix. Beetl lowercases names and turns
any character that is not a letter or digit into an underscore. A name starts with a letter, is at
most 63 characters, and is unique within the tenant. A file processor's dataset is named
automatically, documents_ followed by a unique ID.
| Status | Meaning |
|---|---|
| Pending | A producer is linked but nothing has been written yet. It cannot be queried. |
| Initialized | Data has been written, or the dataset was registered. |
| Deleted | Removed from the app. |
A pipeline's destination is created by its first successful run, and registered and external datasets are Initialized as soon as registration succeeds.
Schema
Where the schema comes from depends on the producer: a webhook has a fixed envelope of 8 columns
with the JSON body in payload, a sync uses the record schema its integration declares, a file processor takes
it from the files you upload (CSV, NDJSON and Parquet), from the fields you define (documents) or
from your mapping (spreadsheets), a pipeline uses its last stage's output, and registered and external
datasets read it at registration. Partition columns are marked in the schema on Home.
Beetl checks every write against the existing schema. A write is compatible when it has the same column names with the same types; column order and nullability do not matter. A new, missing or retyped column makes it incompatible, and what happens then depends on the producer:
| Producer | On an incompatible schema |
|---|---|
| File processor | The run fails and the dataset is kept as it was. A Documents processor fixes its fields once the dataset exists; to use different fields, create a new dataset with a new processor. See Documents and spreadsheets. |
| Configured sync | The record is rejected and the run stops. The checkpoint does not advance. |
| Webhook | Cannot happen, because the envelope columns are fixed. |
| Pipeline | The run fails before writing, and the destination is left as it was. |
Pipelines refer to sources by dataset ID. A file processor never deletes and recreates its dataset, so pipelines that read it keep working. When you move to a new layout, the new processor writes a new dataset and you point the pipelines at it. A mismatch between files inside one File Set also makes processing fail and keeps the dataset.
Delta Lake and storage
Every materialized dataset is a Delta Lake table. Each write commits as one transaction, so a reader sees all of a write or none of it, and a failed run leaves the table as it was. Queries, previews and pipelines always read the latest version; the app cannot read or restore an earlier one. Beetl maintains these tables automatically, and never touches registered or external data.
A new dataset goes to the tenant's default store (marked Managed data in Settings) and stays in the store it was first written to. See Tenants and roles.
Bronze, Silver and Gold
| Tier | Usually holds | How it is set |
|---|---|---|
| Bronze | Raw data as it arrived | Always, for webhook, sync, file processor, registered and app-published datasets |
| Silver | Cleaned, typed and joined data | Chosen on a pipeline's destination |
| Gold | Data shaped for reporting and dashboards | Chosen on a pipeline's destination |
The tier is a convention your team shares about how processed a dataset is. It appears in the
storage path and in filters, and changes nothing else. Beetl enforces no lineage between tiers, and
the tier is not part of SQL: you write FROM orders whatever tier orders has.
Finding, renaming and deleting
Home shows every dataset as a node next to whatever writes it. Its details pane shows status,
storage location, object store, origin and schema, and the Data preview tab shows the first 30
rows. You can also tag datasets there and filter Home by tag. The Data Catalog at /web/data-sets
lists all datasets but is not in the sidebar.
Any user can delete a dataset. To rename a sync's output dataset, open the connection's Syncs dialog and change Data set name under Output data set. Wait for Saved before closing the dialog. For other datasets, contact Beetl to change the name. A rename keeps the dataset's ID, so pipeline source references keep working, but SQL that uses the old name stops resolving.
Deleting is a soft delete: the dataset disappears from the app and its name is freed, while the data stays in storage. It also removes the output link that writes to it. Deleting a registered or external dataset never touches your files or database. Deleting an app keeps the datasets it published, and you should remove a dataset from an app before you delete the dataset. Beetl does not warn you before deleting a dataset that a pipeline reads, and that pipeline's runs then fail.
Related
Last updated on