Pipelines
Sources, SQL stages and a destination, the Working and Deployed versions, and what starts a run.
A pipeline reads one or more source datasets, runs an ordered list of SQL stages over them, and can write the last stage's result to a destination dataset. Pipelines belong to the tenant, so every user sees and can edit the same ones.
To create one, right-click the canvas on Home and choose New pipeline, or use the + on a dataset node to create a pipeline that already reads that dataset. Existing pipelines open from their node. The Pipelines section has the step-by-step guides.
Sources, stages and destination
Each source is a dataset with an alias that you use as its table name in SQL. Any dataset can be a source, including registered and external ones, which can never be destinations.
Stages run in order, and each can read the sources and any earlier stage by name. They use the same DataFusion engine and PostgreSQL dialect as Query. Previewing a stage runs it and everything before it on your unsaved draft, shows up to 50 rows, and writes nothing.
The destination is optional. It names the dataset the last stage writes, its tier (Gold by default), optional partition columns, and a write mode (Append by default in the editor):
| Write mode | What a run does |
|---|---|
| Append | Adds the result rows. |
| Overwrite, full table | Replaces the whole dataset. |
| Overwrite, predicate | Replaces only rows matching a predicate such as dt = '{{ run.dt }}'. |
| Merge | Updates rows that match on merge keys (or a custom predicate) and inserts the rest. |
See Write modes and partitioning.
Working and Deployed
Every pipeline has two versions. Working is the draft you edit and preview, and Deployed is the version that runs; the editor header shows both, for example "Working v4 · Deployed v3".
Save writes your draft. If someone else, the assistant or an MCP client saved the same pipeline in the meantime, your save is rejected instead of overwriting their change.
Deploy promotes the saved Working version and, if it changed and the pipeline is enabled, starts a first run. Beetl checks the pipeline with a dry run first, and asks for any run values that run needs. Run starts the deployed version again.
A pipeline without a destination is valid and useful for exploring data, but it cannot be deployed or run.
Template variables
Stage SQL, the destination name and write predicates can contain expressions in double braces. If a value is missing, the preview or run fails and names it; Beetl never substitutes an empty string.
| Namespace | Value comes from |
|---|---|
vars.* | String variables stored with the pipeline, edited in its Variables panel. |
run.* | Values for one run: entered when you preview, deploy or run, or mapped from a source update. |
fn.* | fn.today(), fn.now() (both UTC) and fn.uuid(). |
SELECT * FROM orders
WHERE dt = '{{ run.dt }}' AND region = '{{ vars.region }}'See Template variables.
What starts a run
A deployed pipeline runs when you press Run, when you deploy a new version, or when a source with
Trigger on update receives new data. A source's run mapping turns fields of its update into
run.* values; for a partitioned source it defaults to the partition columns, and a write that
touches several partitions starts one run per partition. Each triggering source starts runs on its
own, without waiting for the others.
Pipelines have no schedule of their own. To run one on a timetable, trigger it from a source that updates on that timetable.
Runs of one pipeline execute one at a time; different pipelines run in parallel. A disabled pipeline does not run. Run history is in the Runs tab of the pipeline's node on Home. Pipelines do not produce execution logs. See Runs.
The assistant and MCP clients
Chat and MCP clients with a write key use the same tools. build_pipeline creates a pipeline
(defaulting the destination to overwrite), update_pipeline changes only the fields it sends,
run_pipeline previews the stored draft without writing, and deploy_pipeline deploys. When the
pipeline is open in the editor, the assistant edits your unsaved draft instead.
Deploying from chat waits for your confirmation. Over MCP a write key deploys directly, and the
first run then writes the destination. See The agent trust boundary.
Related
Last updated on