> Documentation index: https://docs.beetl.io/llms.txt

# Pipelines

> Sources, SQL stages and a destination, the Working and Deployed versions, and what starts a run.

Source: https://docs.beetl.io/concepts/pipelines/
Last updated: 2026-10-03

A pipeline reads one or more source datasets, runs an ordered list of SQL stages over them, and can
write the last stage's result to a destination dataset. Pipelines belong to the tenant, so every
user sees and can edit the same ones.

To create one, right-click the canvas on Home and choose New pipeline, or use the + on a dataset
node to create a pipeline that already reads that dataset. Existing pipelines open from their node. The
[Pipelines](https://docs.beetl.io/pipelines) section has the step-by-step guides.

## Sources, stages and destination

Each **source** is a dataset with an alias that you use as its table name in SQL. Any dataset can be
a source, including [registered](https://docs.beetl.io/sources/registered-datasets) and
[external](https://docs.beetl.io/sources/external-datasets) ones, which can never be destinations.

**Stages** run in order, and each can read the sources and any earlier stage by name. They use the
same DataFusion engine and PostgreSQL dialect as [Query](https://docs.beetl.io/query). Previewing a stage runs it and
everything before it on your unsaved draft, shows up to 50 rows, and writes nothing.

The **destination** is optional. It names the dataset the last stage writes, its tier (Gold by
default), optional partition columns, and a write mode (Append by default in the editor):

| Write mode            | What a run does                                                                     |
| --------------------- | ----------------------------------------------------------------------------------- |
| Append                | Adds the result rows.                                                               |
| Overwrite, full table | Replaces the whole dataset.                                                         |
| Overwrite, predicate  | Replaces only rows matching a predicate such as `dt = '{{ run.dt }}'`.              |
| Merge                 | Updates rows that match on merge keys (or a custom predicate) and inserts the rest. |

See [Write modes and partitioning](https://docs.beetl.io/pipelines/write-modes-and-partitioning).

## Working and Deployed

Every pipeline has two versions. **Working** is the draft you edit and preview, and **Deployed** is
the version that runs; the editor header shows both, for example "Working v4 · Deployed v3".

**Save** writes your draft. If someone else, the assistant or an MCP client saved the same pipeline
in the meantime, your save is rejected instead of overwriting their change.

**Deploy** promotes the saved Working version and, if it changed and the pipeline is enabled,
starts a first run. Beetl checks the pipeline with a dry run first, and asks for any run values that run needs. **Run** starts the deployed
version again.

A pipeline without a destination is valid and useful for exploring data, but it cannot be deployed
or run.

## Template variables

Stage SQL, the destination name and write predicates can contain expressions in double braces. If
a value is missing, the preview or run fails and names it; Beetl never substitutes an empty string.

| Namespace | Value comes from                                                                             |
| --------- | -------------------------------------------------------------------------------------------- |
| `vars.*`  | String variables stored with the pipeline, edited in its Variables panel.                    |
| `run.*`   | Values for one run: entered when you preview, deploy or run, or mapped from a source update. |
| `fn.*`    | `fn.today()`, `fn.now()` (both UTC) and `fn.uuid()`.                                         |

```sql
SELECT * FROM orders
WHERE dt = '{{ run.dt }}' AND region = '{{ vars.region }}'
```

See [Template variables](https://docs.beetl.io/pipelines/template-variables).

## What starts a run

A deployed pipeline runs when you press Run, when you deploy a new version, or when a source with
**Trigger on update** receives new data. A source's run mapping turns fields of its update into
`run.*` values; for a partitioned source it defaults to the partition columns, and a write that
touches several partitions starts one run per partition. Each triggering source starts runs on its
own, without waiting for the others.

Pipelines have no schedule of their own. To run one on a timetable, trigger it from a source that
updates on that timetable.

Runs of one pipeline execute one at a time; different pipelines run in parallel. A
disabled pipeline does not run. Run history is in the Runs tab of the pipeline's node on Home.
Pipelines do not produce execution logs. See [Runs](https://docs.beetl.io/concepts/runs-and-jobs).

## The assistant and MCP clients

Chat and MCP clients with a `write` key use the same tools. `build_pipeline` creates a pipeline
(defaulting the destination to overwrite), `update_pipeline` changes only the fields it sends,
`run_pipeline` previews the stored draft without writing, and `deploy_pipeline` deploys. When the
pipeline is open in the editor, the assistant edits your unsaved draft instead.

Deploying from chat waits for your confirmation. Over MCP a `write` key deploys directly, and the
first run then writes the destination. See [The agent trust boundary](https://docs.beetl.io/concepts/agent-trust-boundary).

## Related

- [Builder tour](https://docs.beetl.io/pipelines/builder): Sources, stages, preview and the destination in the editor.

- [Deploy and trigger](https://docs.beetl.io/pipelines/deploy-and-trigger): Deploying, running by hand and running on source updates.

- [Datasets and tiers](https://docs.beetl.io/concepts/datasets-and-tiers): The datasets pipelines read and write.