Builder tour
Opening the editor, adding sources and SQL stages, previewing, and saving your draft.
The pipeline editor is where you put a pipeline together: sources at the top, SQL stages in the middle and a destination at the bottom. The model behind it is on Pipelines.
Open the editor
| Starting point | On Home |
|---|---|
| An empty pipeline | Right-click the canvas and choose New pipeline. Beetl creates an untitled pipeline and opens it. |
| A dataset | Click the + on the right edge of the dataset's node, enter a pipeline name and choose Create pipeline. The pipeline appears on the canvas with that dataset as a source called source. |
| An existing pipeline | Double-click its node, or right-click it and choose Open editor. |
The editor opens in a dialog over Home. Open full page moves it to its own page.
The header
Click the pipeline name to rename it, and the new name applies right away without Save.
Below the name you see the versions, for example "Working v4 · Deployed v3", or "Not deployed" before the first deploy. Once a pipeline has been deployed, a Working and Deployed switch appears. The Deployed view is read-only and holds the Run button.
Add sources
In Sources, open the + Add source menu and pick a dataset. Its alias is the name you use as a table in SQL and starts as the dataset name in lowercase with underscores. Click the alias to rename it.
Each source also has a Trigger on update switch and a run mapping, which decide when the deployed pipeline runs. Deploy and trigger covers both.
Write SQL stages
Stages sit under Transformations and run from top to bottom. Add stage puts a new stage at the end, and the + on the line below any stage inserts one after it. Click a stage's name to rename it. The icons in its header move it up or down, duplicate it or delete it.
A stage can read any source by alias and any earlier stage by name, and autocomplete offers both
along with each source's columns. Typing {{ starts completion for run., vars. and fn..
| Action | Shortcut | Button |
|---|---|---|
| Preview the stage | Cmd/Ctrl+Enter | Preview, in the stage header |
| Format the SQL | Shift+Alt+F | Format SQL, bottom right of the cell |
| Open completion | Ctrl+Space |
Compact SQL shrinks a long cell to a fixed height and Expand SQL restores it. The ⋯ menu lists the shortcuts and links to the DataFusion SQL reference.
One stage per step
Give each logical step its own stage, as you would write one CTE per step, and refer to earlier stages by name. You can then preview every intermediate result, and a failing preview points at the step that broke.
-- Stage: clean_orders
SELECT order_id, customer_id, amount, dt
FROM orders
WHERE order_id IS NOT NULL AND amount > 0-- Stage: daily_revenue
SELECT dt, SUM(amount) AS revenue
FROM clean_orders
GROUP BY dtPreview a stage
Preview runs the stage and every stage before it on your current draft, unsaved edits included. It shows up to 50 rows with their column types under the cell and writes nothing. While it runs, the button turns into Cancel, and editing pauses until the preview finishes or you cancel it.
If the pipeline uses run.* values, a Provide run values dialog asks for them first. When a stage
fails, Preview failed and the error message appear under the cell.
Add a destination
Without a destination you can preview but not deploy or run. Choose Add destination below the stages. It starts with the pipeline's name as the Data set name, Gold as the Storage tier and Append as the Write mode.
| Field | What to set |
|---|---|
| Data set name | The dataset the last stage writes. |
| Storage tier | Bronze, Silver or Gold. |
| Write mode | Append, Overwrite or Merge. Overwrite asks for a Replace scope (Full table or a predicate) and Merge for merge keys. The editor shows the predicate it builds. |
| Partition columns | Preview the last stage to pick from its columns, or type them comma-separated. |
Write modes and partitioning explains each choice.
Variables and the guide
The Variables panel at the top holds values stored with the pipeline. Choose Add variable, enter a
name and a value, and use it as {{ vars.name }}; see Template variables.
Guide, in the header, summarizes how sources, triggers and template values fit together.
Save, deploy and run
Your edits stay in the browser until you press Save, which stores the draft as the Working version. If you close the editor or leave the page with unsaved changes, Beetl asks before discarding them.
If someone else, the assistant or an MCP client saved the pipeline after you opened it, Beetl rejects your save instead of overwriting their change. Copy the SQL you want to keep, reload the page and apply your edits to the latest version.
Deploy sits in the Working view and works once everything is saved. It promotes the Working version and starts a first run. To run the deployed version again, switch to Deployed and press Run, or right-click the pipeline's node on Home and choose Run pipeline. When Run is unavailable, hovering over it tells you why. Deploy and trigger has the details.
Edit with the assistant
With the editor on its own page, or open beside a conversation in AI Chat, you can ask the assistant to change the pipeline. It edits your draft and leaves the saved pipeline alone. The stages it changed flash, and its reply ends with a Draft updated row where Dismiss undoes that turn's edits and Save saves the draft. Preview the changed stages before you save.
Last updated on