CDC Environment Variables

This is the per-approach environment variable reference for the ClickHouse CDC system, with descriptions, default values, and validation rules, plus a complete working configuration example per approach. It satisfies Requirement 9.3.

These definitions are generated by, and must stay consistent with, apps/cms/src/modules/cdc/ai-flow/config-generator.ts (the generateConfig(approach, target) function). If you change a variable there, update this page in the same change. Every key matches the pattern ^[A-Z_][A-Z0-9_]*$.

The variables you need depend on two axes:

approach — debezium_kafka, materialized_engine, or airbyte.
target — docker_compose (the full stateful stack) or cloudflare_workers (edge components only).

You can fetch the exact set for any combination from the API:

bash

curl -sS -X POST https://your-cms-host/api/v1/cdc/deploy/validate-env \
  -H "Authorization: Bearer $LUMI_ADMIN_TOKEN" -H "X-Lumi-Site: $SITE_ID" \
  -H "Content-Type: application/json" \
  -d '{"approach":"debezium_kafka","target":"docker_compose","env":{}}'
# → 400 ENV_VALIDATION_ERROR listing every missing required variable

Universal variables (all approaches and targets)

These identity variables are always present so the two halves of a split deployment can be correlated.

Variable	Description	Default	Required	Validation rule
`CDC_PIPELINE_NAME`	Human-readable name of the CDC pipeline this deployment serves.	—	Yes	Non-empty string, at most 128 characters
`CDC_APPROACH`	The CDC replication approach this deployment implements.	`<approach>`	Yes	One of: `debezium_kafka`, `materialized_engine`, `airbyte`
`CDC_DEPLOYMENT_TARGET`	The deployment target hosting these components.	`<target>`	Yes	One of: `docker_compose`, `cloudflare_workers`

Stateful stack — common variables (`docker_compose`)

Shared by all three approaches when the target is docker_compose.

Variable	Description	Default	Required	Validation rule
`SOURCE_DATABASE_URL`	PostgreSQL Source_Database connection string the connector replicates from.	—	Yes	Valid `postgres://` or `postgresql://` connection URL
`CLICKHOUSE_SINK_URL`	ClickHouse_Sink connection string that receives replicated data.	—	Yes	Valid `clickhouse://`, `http://`, or `https://` connection URL
`CLICKHOUSE_DATABASE`	ClickHouse database name that holds the replicated tables.	`default`	No	Non-empty string
`CDC_REPLICATION_TABLES`	Comma-separated list of fully-qualified PostgreSQL tables to replicate.	—	Yes	Comma-separated list of one or more table names

Debezium + Kafka (`docker_compose`)

Universal + stateful-common variables (above) plus:

Variable	Description	Default	Required	Validation rule
`KAFKA_BOOTSTRAP_SERVERS`	Comma-separated Kafka_Broker bootstrap servers (`host:port`).	`kafka:9092`	Yes	Comma-separated list of `host:port` entries
`DEBEZIUM_CONNECT_URL`	Debezium Kafka Connect REST endpoint used to register the connector.	`http://debezium:8083`	Yes	Valid `http://` or `https://` URL
`KAFKA_TOPIC_PREFIX`	Prefix applied to per-table Kafka topics created by Debezium.	`lumibase_cdc`	No	Non-empty string
`DEBEZIUM_SLOT_NAME`	PostgreSQL replication slot name Debezium creates on the Source_Database.	`lumibase_debezium`	No	Non-empty string
`DEBEZIUM_PUBLICATION_NAME`	PostgreSQL publication name Debezium uses for logical replication.	`lumibase_cdc_pub`	No	Non-empty string
`KAFKA_BUFFER_MAX_BYTES`	Local buffer size cap (bytes) retained during a Kafka outage.	`524288000`	No	Positive integer (≥ 1)
`KAFKA_BUFFER_MAX_AGE_MS`	Maximum age (ms) of locally buffered events during a Kafka outage.	`3600000`	No	Positive integer (≥ 1)
`DEBEZIUM_MAX_SLOT_FAILURES`	Consecutive replication-slot failures tolerated before the pipeline is marked `error`.	`3`	No	Positive integer (≥ 1)

Complete working configuration example — Debezium + Kafka

bash

# Universal
CDC_PIPELINE_NAME=orders-analytics
CDC_APPROACH=debezium_kafka
CDC_DEPLOYMENT_TARGET=docker_compose

# Stateful common
SOURCE_DATABASE_URL=postgresql://cdc_user:secret@postgres:5432/lumibase
CLICKHOUSE_SINK_URL=clickhouse://clickhouse:9000/analytics
CLICKHOUSE_DATABASE=analytics
CDC_REPLICATION_TABLES=public.orders,public.customers

# Debezium + Kafka
KAFKA_BOOTSTRAP_SERVERS=kafka:9092
DEBEZIUM_CONNECT_URL=http://debezium:8083
KAFKA_TOPIC_PREFIX=lumibase_cdc
DEBEZIUM_SLOT_NAME=lumibase_debezium
DEBEZIUM_PUBLICATION_NAME=lumibase_cdc_pub
KAFKA_BUFFER_MAX_BYTES=524288000
KAFKA_BUFFER_MAX_AGE_MS=3600000
DEBEZIUM_MAX_SLOT_FAILURES=3

Materialized Engine (`docker_compose`)

Universal + stateful-common variables (above) plus:

Variable	Description	Default	Required	Validation rule
`MATERIALIZED_DATABASE_NAME`	ClickHouse `MaterializedPostgreSQL` database name created for replication.	`lumibase_cdc_mat`	No	Non-empty string
`MATERIALIZED_SLOT_NAME`	PostgreSQL replication slot name the Materialized Engine owns.	`lumibase_mat`	No	Non-empty string
`MATERIALIZED_RECONNECT_MAX_RETRIES`	Maximum reconnection attempts before erroring.	`5`	No	Positive integer (≥ 1)
`MATERIALIZED_RECONNECT_BASE_DELAY_MS`	Initial exponential-backoff delay (ms) for reconnection.	`1000`	No	Positive integer (≥ 1)
`MATERIALIZED_SCHEMA_DRIFT_INTERVAL_MS`	Interval (ms) at which source schema drift is checked.	`60000`	No	Positive integer (≥ 1)

Complete working configuration example — Materialized Engine

bash

# Universal
CDC_PIPELINE_NAME=catalog-analytics
CDC_APPROACH=materialized_engine
CDC_DEPLOYMENT_TARGET=docker_compose

# Stateful common
SOURCE_DATABASE_URL=postgresql://cdc_user:secret@postgres:5432/lumibase
CLICKHOUSE_SINK_URL=clickhouse://clickhouse:9000/analytics
CLICKHOUSE_DATABASE=analytics
CDC_REPLICATION_TABLES=public.products,public.categories

# Materialized Engine
MATERIALIZED_DATABASE_NAME=lumibase_cdc_mat
MATERIALIZED_SLOT_NAME=lumibase_mat
MATERIALIZED_RECONNECT_MAX_RETRIES=5
MATERIALIZED_RECONNECT_BASE_DELAY_MS=1000
MATERIALIZED_SCHEMA_DRIFT_INTERVAL_MS=60000

Airbyte (`docker_compose`)

Universal + stateful-common variables (above) plus:

Variable	Description	Default	Required	Validation rule
`AIRBYTE_API_URL`	Airbyte_Connector API base URL used to provision source/destination/connection.	`http://airbyte:8001/api`	Yes	Valid `http://` or `https://` URL
`AIRBYTE_WORKSPACE_ID`	Airbyte workspace ID that owns the provisioned resources.	—	Yes	Non-empty string
`AIRBYTE_SYNC_MODE`	Airbyte sync mode for the connection.	`incremental_cdc`	No	One of: `full_refresh`, `incremental_cdc`
`AIRBYTE_SYNC_INTERVAL_SECONDS`	Sync schedule interval in seconds; enforces the 5-minute Airbyte minimum.	`300`	No	Integer between 300 and 86400 (inclusive)
`AIRBYTE_PROVISION_TIMEOUT_MS`	Timeout (ms) for provisioning Airbyte resources.	`120000`	No	Positive integer (≥ 1)
`AIRBYTE_SYNC_MAX_RETRIES`	Number of times a failed sync is retried with backoff.	`3`	No	Positive integer (≥ 1)

Complete working configuration example — Airbyte

bash

# Universal
CDC_PIPELINE_NAME=reporting-sync
CDC_APPROACH=airbyte
CDC_DEPLOYMENT_TARGET=docker_compose

# Stateful common
SOURCE_DATABASE_URL=postgresql://cdc_user:secret@postgres:5432/lumibase
CLICKHOUSE_SINK_URL=clickhouse://clickhouse:9000/analytics
CLICKHOUSE_DATABASE=analytics
CDC_REPLICATION_TABLES=public.invoices,public.payments

# Airbyte
AIRBYTE_API_URL=http://airbyte:8001/api
AIRBYTE_WORKSPACE_ID=00000000-0000-0000-0000-000000000000
AIRBYTE_SYNC_MODE=incremental_cdc
AIRBYTE_SYNC_INTERVAL_SECONDS=300
AIRBYTE_PROVISION_TIMEOUT_MS=120000
AIRBYTE_SYNC_MAX_RETRIES=3

Edge components (`cloudflare_workers`)

When the target is cloudflare_workers, the deployment includes only the universal variables plus the edge variables below. Stateful connection variables (SOURCE_DATABASE_URL, CLICKHOUSE_SINK_URL, Kafka/Materialized/Airbyte variables) are not part of a Workers deployment — those live in the companion docker_compose/managed-services deployment, reached over HTTPS. The edge variables are identical across all three approaches.

Variable	Description	Default	Required	Validation rule
`CDC_STATEFUL_STACK_URL`	HTTPS endpoint of the companion `docker_compose`/managed-services stateful deployment the Workers runtime communicates with.	—	Yes	Valid `https://` connection URL (HTTPS required for the Workers runtime)
`CDC_API_AUTH_TOKEN`	Bearer token securing the CDC control-plane API endpoints exposed by the Worker.	—	Yes	Non-empty string
`REDIS_URL`	Redis connection string the Cache_Invalidator refreshes.	—	Yes	Valid `redis://` or `rediss://` connection URL
`CACHE_KEY_NAMESPACE`	CacheProvider key namespace prefix used when deriving cache keys.	`lumibase`	No	Non-empty string
`CACHE_INVALIDATOR_QUEUE_MAX`	Bounded queue size for buffering events during a Redis outage.	`10000`	No	Positive integer (≥ 1)
`CACHE_INVALIDATOR_DEDUP_WINDOW_MS`	Deduplication window (ms) for collapsing consecutive UPDATE events for the same key.	`1000`	No	Positive integer (≥ 1)

Complete working configuration example — Cloudflare Workers edge

bash

# Universal (the approach matches the companion stateful deployment)
CDC_PIPELINE_NAME=orders-analytics
CDC_APPROACH=debezium_kafka
CDC_DEPLOYMENT_TARGET=cloudflare_workers

# Edge components
CDC_STATEFUL_STACK_URL=https://cdc-stack.internal.example.com
CDC_API_AUTH_TOKEN=replace-with-a-strong-random-token
REDIS_URL=rediss://default:token@us1-abc-12345.upstash.io:6379
CACHE_KEY_NAMESPACE=lumibase
CACHE_INVALIDATOR_QUEUE_MAX=10000
CACHE_INVALIDATOR_DEDUP_WINDOW_MS=1000

Security: CDC_API_AUTH_TOKEN and the credentials embedded in REDIS_URL / CDC_STATEFUL_STACK_URL are secrets. On Cloudflare Workers set them with wrangler secret put rather than committing them to wrangler.toml. The CDC control-plane API is admin-gated; never expose it without CDC_API_AUTH_TOKEN.

Validation behavior

When you submit env values (via /deploy or /deploy/validate-env), each value is checked against the rule above. Invalid values return 400 ENV_VALIDATION_ERROR with the list of invalid fields and the specific violated constraint for each (Requirements 7.4, 7.5). Each entry in invalidFields carries the offending key, a short stable rule identifier (one of required, key_format, unknown_key, type, min_length, max_length, pattern, min, max, enum, url, protocol), and a human-readable reason:

json

{
  "errors": [
    {
      "code": "ENV_VALIDATION_ERROR",
      "message": "Environment variable validation failed.",
      "invalidFields": [
        { "key": "AIRBYTE_SYNC_INTERVAL_SECONDS", "rule": "max", "reason": "value must be <= 86400" }
      ]
    }
  ]
}

CDC Environment Variables

Universal variables (all approaches and targets)

Stateful stack — common variables (docker_compose)

Debezium + Kafka (docker_compose)

Complete working configuration example — Debezium + Kafka

Materialized Engine (docker_compose)

Complete working configuration example — Materialized Engine

Airbyte (docker_compose)

Complete working configuration example — Airbyte

Edge components (cloudflare_workers)

Complete working configuration example — Cloudflare Workers edge

Validation behavior

See also

Stateful stack — common variables (`docker_compose`)

Debezium + Kafka (`docker_compose`)

Materialized Engine (`docker_compose`)

Airbyte (`docker_compose`)

Edge components (`cloudflare_workers`)