MongoDB CDC - Firebolt Documentation

Private Preview FeatureThis feature is currently in private preview. Contact support@firebolt.io to request early access.

Firebolt connects to your MongoDB deployment and keeps a CDC table continuously in sync with a collection: every insert, update, replace, and delete arrives within seconds, with nothing to deploy in between. Under the hood, the connection is a native MongoDB change stream. Documents do not need a fixed schema to become queryable. Fields with stable types become typed columns, and every other field stays reachable by name through a JSON fallback.

Prepare the source deployment

On the MongoDB side you need:

A replica set or sharded cluster. Change streams read the oplog; a standalone mongod has none.
MongoDB 6.0 or later, with document pre- and post-images enabled on the collection. Firebolt reads exact point-in-time images rather than re-fetching documents, so updates are never observed out of order or half-applied:
```
db.runCommand({
  collMod: "orders",
  changeStreamPreAndPostImages: { enabled: true }
})
```
A role with find and changeStream privileges on the collection.
Oplog and pre-image retention sized so the feed’s history outlives any planned ingestion pause. If the stored position ages out of the oplog, ingestion cannot resume from it and the table must be re-bootstrapped.
Network reachability from the Firebolt engine to the cluster (VPC peering, PrivateLink, or an Atlas access list).

Create the location

CREATE LOCATION mongo_prod WITH
    SOURCE = MONGODB
    URI = 'mongodb+srv://cluster0.example.mongodb.net/'
    USER = 'firebolt_cdc'
    PASSWORD = '********';

TLS follows the URI: mongodb+srv:// connections use TLS, and plain mongodb:// URIs can request it with tls=true. Credentials can also be embedded in the URI instead of USER and PASSWORD. See CREATE LOCATION (MongoDB).

Create the stream

A stream binds to one collection, named as <database>.<collection>. You can declare the columns explicitly:

CREATE STREAM orders_changes (
    _id    TEXT,
    status TEXT,
    amount BIGINT
)
LOCATION = 'mongo_prod'
COLLECTION = 'app.orders'
KEY = '_id';

or omit the column list and let Firebolt infer it:

CREATE STREAM orders_changes
LOCATION = 'mongo_prod'
COLLECTION = 'app.orders';

KEY names the column used as the merge key downstream and defaults to '_id'.

Schema inference

Inference reads the collection’s $jsonSchema validator. Every top-level field the validator declares with a single scalar type (or a homogeneous array of scalars) becomes a typed column:

BSON type	Firebolt type
`bool`	`BOOLEAN`
`int`	`INT`
`long`	`BIGINT`
`double`	`DOUBLE`
`string`, `objectId`	`TEXT`
`date`	`TIMESTAMPTZ`
array of one scalar type	`ARRAY(<type>)`

Fields with polymorphic types, nested objects, decimal128, and anything the validator does not declare are deliberately left out of the typed schema; they stay fully queryable through the overflow fallback described below. Inference runs once, at CREATE STREAM; a collection without a $jsonSchema validator requires an explicit column list.

Read raw change events

SELECT "$op", "$wall_time", _id, status, amount
FROM READ_STREAM(STREAM orders_changes, consume => false);

Change rows carry the declared columns plus pseudo-columns (reference them double-quoted):

Pseudo-column	Type	Description
`$op`	TEXT	Operation: `I` insert, `U` update, `R` replace, `D` delete, `T` collection drop, rename, or invalidation.
`$doc`	JSON	Exact post-image of the document; `NULL` for deletes.
`$before`	JSON	Exact pre-image; `NULL` for inserts.
`$update_desc`	JSON	Updated fields, removed fields, and truncated arrays, for updates.
`$key`	JSON	The document key (`_id`, plus the shard key on sharded clusters).
`$resume_token`	TEXT	The change stream position of this event.
`$cluster_time`	BIGINT	Cluster timestamp, packed into a 64-bit integer; the ordering authority for the feed.
`$wall_time`	TIMESTAMPTZ	Wall-clock time of the operation on the source.
`$ns`	TEXT	Namespace, as `database.collection`.
`$stream_row_number`	BIGINT	1-based position of the row within this statement’s read, in delivery order.

The first consuming read with no stored position starts at the current end of the oplog; it does not replay history. With snapshot => true, READ_STREAM instead scans the collection and returns every document as a synthetic insert, pinning the change-stream position first so the subsequent feed continues with no gap. The CDC table’s initial backfill uses this path. Consuming reads advance the stream’s stored position in the same Firebolt transaction that writes the data, so ingestion is exactly-once with no deduplication step.

Create the CDC table and start ingestion

The inference form mirrors the stream’s schema, so a complete continuously updated mirror of a collection is two statements:

CREATE CDC TABLE orders FROM STREAM orders_changes;

ALTER CDC TABLE orders RESUME;

The worker backfills from a collection snapshot, then applies the change feed continuously, ordered by $cluster_time. Semantics, merge modes, and lifecycle commands are covered in CDC tables; progress is visible in information_schema.cdc_ingests.

Query any field, declared or not

A CDC table created through inference carries a hidden JSON overflow column holding each document’s fields that are not typed columns. Name resolution falls back to it automatically, so undeclared fields and nested paths work in ordinary SQL without schema changes:

-- status and amount are typed columns; coupon and customer.city live in the overflow
SELECT _id, status, coupon
FROM orders
WHERE CAST(customer.city AS TEXT) = 'Berlin';

New fields that start appearing in documents after the stream was created land in the overflow as well, so source schema drift never breaks ingestion or requires DDL. When a field stabilizes and earns a typed column (for query performance or for use in the merge key), recreate the stream and table with the field declared.

Operations

Delivery and recovery. The resume token commits atomically with the data it produced. After a crash or engine restart, ingestion resumes from the last committed token; the apply is additionally guarded by $cluster_time, so a replayed event is a no-op.
Feed expiry. If ingestion is paused long enough for the stored token to age out of the oplog (or the needed pre-images to expire), the stream reports that its history is lost. Nothing on the source is harmed; recreate the CDC table to re-baseline from a fresh snapshot.
Collection lifecycle events. A collection drop, rename, or stream invalidation emits a terminal $op = 'T' and ends the feed; recreate the stream and table against the new collection.
DROP STREAM removes only Firebolt metadata. MongoDB change streams hold no server-side resource comparable to a replication slot, so there is nothing to clean up on the source.

Limitations

One collection per stream.
Inference requires a $jsonSchema validator; without one, declare columns explicitly.
ALTER STREAM is not supported; schema changes are drop-and-recreate.
decimal128 and nested-object fields are not inferred as typed columns; they are available through the overflow fallback.

​Prepare the source deployment

​Create the location

​Create the stream

​Schema inference

​Read raw change events

​Create the CDC table and start ingestion

​Query any field, declared or not

​Operations

​Limitations

Prepare the source deployment

Create the location

Create the stream

Schema inference

Read raw change events

Create the CDC table and start ingestion

Query any field, declared or not

Operations

Limitations