> Documentation index: https://docs.beetl.io/llms.txt

# Pipelines

> Building, deploying and running pipelines, plus recipes for common cleanup steps.

Source: https://docs.beetl.io/pipelines/
Last updated: 2026-10-03

A pipeline reads source datasets, runs SQL stages over them, and writes the result to a destination
dataset. This section shows how to do that in practice. For the model behind it (the Working and
Deployed versions, write modes and what starts a run), read [Pipelines](https://docs.beetl.io/concepts/pipelines) in
Core concepts first.

You build pipelines in the editor, which you open from Home. Right-click the canvas and choose New
pipeline, or use the + on a dataset node to create a pipeline that already reads it.

## In this section

### Guides

| Page                                                                    | What it covers                                                                        |
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| [Builder tour](https://docs.beetl.io/pipelines/builder)                                      | Opening the editor, adding sources and SQL stages, previewing, and saving your draft. |
| [Deploy and trigger](https://docs.beetl.io/pipelines/deploy-and-trigger)                     | Deploying a pipeline, running it by hand, and running it when a source updates.       |
| [Template variables](https://docs.beetl.io/pipelines/template-variables)                     | `vars`, `run`, and the `fn` helpers.                                                  |
| [Write modes and partitioning](https://docs.beetl.io/pipelines/write-modes-and-partitioning) | Append, overwrite and merge, and how output lands in a dataset.                       |

### Recipes

Recipes are worked examples of steps that come up in most pipelines.

| Page                                                            | What it covers                                                     |
| --------------------------------------------------------------- | ------------------------------------------------------------------ |
| [Deduplicate records](https://docs.beetl.io/pipelines/deduplicate-records)           | Keeping one row per entity in a pipeline stage.                    |
| [Match records across systems](https://docs.beetl.io/pipelines/match-across-systems) | Linking the same customer or order between two sources.            |
| [Quarantine bad rows](https://docs.beetl.io/pipelines/quarantine-bad-rows)           | Splitting rows that fail a check into their own dataset.           |
| [Redact a column](https://docs.beetl.io/pipelines/redact-columns)                    | Dropping, masking or hashing personal data in a pipeline's output. |

## Related

- [Pipelines concept](https://docs.beetl.io/concepts/pipelines): Sources, stages, destination, versions and runs on one page.

- [Query](https://docs.beetl.io/query): The same SQL engine, for one-off questions.