Pipelines
Pipelines
Building, deploying and running pipelines, plus recipes for common cleanup steps.
A pipeline reads source datasets, runs SQL stages over them, and writes the result to a destination dataset. This section shows how to do that in practice. For the model behind it (the Working and Deployed versions, write modes and what starts a run), read Pipelines in Core concepts first.
You build pipelines in the editor, which you open from Home. Right-click the canvas and choose New pipeline, or use the + on a dataset node to create a pipeline that already reads it.
In this section
Guides
| Page | What it covers |
|---|---|
| Builder tour | Opening the editor, adding sources and SQL stages, previewing, and saving your draft. |
| Deploy and trigger | Deploying a pipeline, running it by hand, and running it when a source updates. |
| Template variables | vars, run, and the fn helpers. |
| Write modes and partitioning | Append, overwrite and merge, and how output lands in a dataset. |
Recipes
Recipes are worked examples of steps that come up in most pipelines.
| Page | What it covers |
|---|---|
| Deduplicate records | Keeping one row per entity in a pipeline stage. |
| Match records across systems | Linking the same customer or order between two sources. |
| Quarantine bad rows | Splitting rows that fail a check into their own dataset. |
| Redact a column | Dropping, masking or hashing personal data in a pipeline's output. |
Related
Last updated on