Write modes

A sync's write mode decides what a successful run does to the sync's dataset. The person who sets up the sync chooses it under Write mode and can change it later. A package declares neither a write mode nor a primary key: its code emits records, and a checkpoint if it declares one. For how syncs, output links and datasets fit together, see Connection types, connections and syncs.

Each run commits as one transaction. A failed run leaves the dataset as it was, and its checkpoint does not advance.

Tell the people who set up your sync which mode it is built for. A sync that stops early to fit a run, like the example below, is built for Append: in Replace it would publish only the part it read.

Append

An Append run adds every record it yields to the dataset. Existing rows are never updated or removed, and Beetl does not deduplicate or merge records by key. A sync that rereads everything stores it again on every run, so declare a checkpoint and read only what is new, or choose Replace.

A run that yields no records succeeds without writing.

Rows that change at the source arrive again as new rows. Include the source's stable ID and a last-modified time in each record, so that a pipeline can keep the latest row per ID. Rows deleted at the source stay in the dataset. If you need to know about deletions, read a deleted flag or a deletions feed and emit it as a field.

Replace

A Replace run starts from the beginning every time, even when the sync declares a checkpoint, and writes a new version of the dataset that holds only that run's records. The new version is published only after the run succeeds, so readers see the old contents until then. Read the complete set on every run: a Replace sync that skips records it read before ends up with only the new ones.

A Replace run that yields no records leaves the dataset empty. If an empty response from the provider can also mean an outage, throw an error instead. The run then fails and the previous contents stay.

Checkpoints

A sync that declares a checkpoint resumes from its last saved checkpoint, so each run reads only what is new.

Changing a sync's settings, including its write mode, resets its checkpoint. The next run reads everything again and replaces the dataset's contents, and later runs continue from there. Changing only the schedule resets nothing. See Checkpoints.

This sync reads changes from a CRM and stops after a bounded number of pages, so that each run fits the limits below. Every run continues from the checkpoint the last run saved.

import { auth, defineIntegration, z } from "@beetlio/connect";

const Contact = z.object({
  id: z.string(),
  email: z.string().nullable(),
  updatedAt: z.string(),
  deleted: z.boolean(),
});

const Changes = z.object({
  changed: z.array(Contact),
  cursor: z.string(),
  hasMore: z.boolean(),
});

export default defineIntegration({
  key: "example-crm",
  displayName: "Example CRM",
  connection: { origin: "https://api.example.com", auth: auth.bearer() },
  syncs: (sync) => ({
    contacts: sync({
      records: Contact,
      checkpoint: z.object({ cursor: z.string() }),
      async *run(ctx) {
        let cursor = ctx.checkpoint?.cursor;
        // Stop after 200 pages; the next run continues from the checkpoint.
        for (let page = 0; page < 200; page++) {
          const query = cursor ? `?${new URLSearchParams({ cursor })}` : "";
          const changes = await ctx.json(`/contacts/changes${query}`, Changes);
          yield {
            records: changes.changed,
            checkpoint: { cursor: changes.cursor },
          };
          if (!changes.hasMore) return;
          cursor = changes.cursor;
        }
      },
    }),
  }),
});

Each contact carries its id, its updatedAt time and a deleted flag, so a pipeline can keep the latest version of each contact and drop the deleted ones.

Partitioning

Partitioning is chosen when a user creates the sync and cannot change later.

Sync time adds a dt date column holding the day, week (starting Monday), month or year of the run, in the timezone chosen for the sync. A scheduled run uses its scheduled time, and retries keep the same period, so a daily Replace sync partitioned by day keeps one snapshot per day. This option needs a record schema without a field named dt. An output field can be any top-level string, integer, number or boolean field of the record schema.

Per-run limits

A run that exceeds a limit fails without writing anything. Transient failures are retried, and each attempt starts from the same checkpoint, so a run that is too large fails on every attempt.

Design each run to fit. For an Append sync, end the generator after a bounded amount of work, as the example above does, and let the next run continue from the checkpoint. A Replace run has to fit the whole set into one run.