CDC ingests

Nightly Feature

This feature is currently in private preview. Contact support@firebolt.io to request early access.

The information_schema.cdc_ingests view reports CDC table ingestion: one row per CDC table with an attached stream. Identity comes from metadata. Progress and retry diagnostics come from the matching worker on the node serving the query (see has_worker), so query the node where ingestion runs to observe them.

SELECT table_name, stream_name, state, phase,
       has_worker, apply_count, last_commit_time, live_lag_ms, last_error, error_count
FROM information_schema.cdc_ingests;

Retry diagnostics are tracked independently for initial load, ingestion, cascade, and bookkeeping. Bookkeeping covers the worker’s catalog and source-position checks, including progress accounting; losing the stream position to retention counts as an ingestion failure. A successful operation clears only its own outstanding failure. last_error reports the latest outstanding failure, and error_count counts failure incidents across all pipeline steps since the worker started: a step that keeps failing counts once until it succeeds again. Initial-load retries back off exponentially to one minute; other work follows its worker or maintenance cadence. A failure identified as permanent, or an initial-load, ingestion or cascade operation that keeps failing for at least 10 attempts over at least 4 hours by default, stops source intake for that attachment and changes state to error. Bookkeeping failures never change state. Failure details preserve the underlying exception text, truncated to at most 2,000 bytes.

Columns in information_schema.cdc_ingests

Column Name	Data Type	Description
database_name	TEXT	The database containing the CDC table.
schema_name	TEXT	The schema containing the CDC table.
table_name	TEXT	The CDC table.
stream_schema	TEXT	The schema containing the attached stream.
stream_name	TEXT	The stream feeding the table.
state	TEXT	Persisted attachment state: `enabled` after `RESUME`, `paused` after `ATTACH` or `SUSPEND` (an attachment in `error` stays in `error`), or `error` after a permanent stop. `enabled` does not prove that a worker is running; see `has_worker`.
phase	TEXT	The matching worker’s phase, `backfilling` or `streaming`. A MongoDB table whose load has not caught up with the change feed yet, and so returns no rows to queries, reports `backfilling` on every Engine and in every `state`. Otherwise `NULL` without a matching worker (`has_worker = false`), while paused, after a permanent stop (`state = 'error'`, even while `has_worker` is `true` during the final drain), or until the worker has determined its phase. A retried failure does not change it.
apply_count	BIGINT	Progress ticks since the worker started, not rows. For a Postgres stream one tick can cover several source transactions. `0` without a matching worker.
last_commit_time	TIMESTAMPTZ	When the matching worker last committed a batch or completed the initial load.
live_lag_ms	BIGINT	Milliseconds since `last_commit_time`. It grows on an idle source that has nothing to send, so treat it as time since the last applied batch rather than as a measure of lag.
source_lag_ms	BIGINT	Always `NULL`. Source-side lag measurement is not implemented.
last_error	TEXT	The recorded reason when `state` is `error`; otherwise the most recent failure not yet cleared by success of the same pipeline step. `NULL` when neither exists.
error_count	BIGINT	Failure incidents recorded by the matching worker, across all pipeline steps; a step that keeps failing counts once. `0` without one.
last_cascade_time	TIMESTAMPTZ	When background maintenance last applied staged changes into the base table.
cascade_lag_ms	BIGINT	Milliseconds since `last_cascade_time`. It grows on an idle table whose base is already current.
cascade_leftover_tablets	BIGINT	Tablets the last maintenance run left for the next one.
cascade_count	BIGINT	Maintenance runs that applied something, since the worker started. `NULL` without a matching worker.
last_cascade_error	TEXT	The most recent maintenance failure not yet cleared by a successful run.
has_worker	BOOLEAN	Whether the node serving this query runs the worker started by the latest `RESUME`. A worker still winding down after a later `RESUME` or `DETACH STREAM` does not count. After a `SUSPEND` run on another Engine, its worker still counts, with `state = 'paused'`, until it stops at its next reconcile; on the Engine that ran `SUSPEND`, it no longer counts once `SUSPEND` returns.

Examples

Check ingestion health

SELECT table_name, phase, has_worker, live_lag_ms, last_error
FROM information_schema.cdc_ingests
WHERE state = 'error' OR (state = 'enabled' AND (NOT has_worker OR last_error IS NOT NULL));

has_worker describes the node serving the query, so run this on the Engine where RESUME ran; elsewhere every enabled table reports has_worker = false. A worker that stalls without an error does not match; on a source you know is busy, also check live_lag_ms.

Inspect one table’s ingest

SELECT table_name, state, phase, last_commit_time, last_error
FROM information_schema.cdc_ingests
WHERE table_name = 'orders';

CDC tables: lifecycle and monitoring.
CREATE CDC TABLE: DDL reference, including ALTER CDC TABLE RESUME | SUSPEND | CASCADE.
information_schema.streams: stream metadata.

Overview

Performance and Observability

Security

Self-Managed

Managed service

Guides

SQL reference

Release notes

API reference

Legal

Columns in information_schema.cdc_ingests

Examples

Check ingestion health

Inspect one table’s ingest

​Columns in information_schema.cdc_ingests

​Examples

​Check ingestion health

​Inspect one table’s ingest

​Related

Columns in information_schema.cdc_ingests

Examples

Check ingestion health

Inspect one table’s ingest

Related