Azure Cosmos DB change data capture and real-time data replication

• Azure platform admins and DBAs who must approve a change feed consumer, with the hostname, Azure Key, monitored container, and lease container written down before anything runs, and TLS on port 443 confirmed for the network path
• Data platform leads feeding Snowflake, SQL Server, PostgreSQL, data lakes, or Kafka from Cosmos DB, who want one Core Hub for every source and target instead of a separate pipeline per engine
• Application engineers who wrote change feed consumers or Azure Functions triggers and now own leases, retries, and monitoring for a pipeline the business treats as infrastructure
• Teams replacing Kafka Connect pipelines that read Cosmos DB, or weighing a commercial path with best-in-class enterprise support, rated 4.9/5 by customers and heterogeneous targets against building and operating their own consumer

Native change feed capture

The agent implements the Azure Cosmos DB change feed processor, which is part of the Azure V4 SDK, and it embeds the official azure-cosmos driver. Changes are read from the monitored container.

• Inserts and updates to the monitored container are reflected in the change feed and captured.
• Leases track progress. The change feed processor stores its state in a lease container, so processing resumes where it left off and can be coordinated across workers.
• Snapshots for the first load. Full snapshot tasks seed the target before CDC takes over. Change data capture docs ↗

What to prepare before the first run

Two things must be in place before the agent starts.

1. Lease container, prepared up front. The lease container stores the state of the change feed processor and coordinates processing across workers. Create it in the same database with the partition key path /id, which is the default, using the Azure CLI command in the setup guide. Use the Advanced settings tab if you need a different partition key.
2. Service key and TLS. Gluesync connects with the Azure Cosmos DB service key, entered as the Azure Key, over HTTPS to port 443 with TLS. The network path needs outbound access on that port.

Regions and connectivity

• Regions. The agent works with Azure Cosmos DB DBaaS in any region.
• Connections. You enter the hostname, the port (default 443), the database name, the Azure Key, the monitored container, and the lease container name. Connections use TLS.

Architecture around Core Hub

Agents run close to the systems they read and write, and Core Hub coordinates them: it hosts the web UI and REST APIs and routes every change from the Cosmos DB source agent to its targets. A pipeline groups the source agent, its target agents, and the entities (containers) they replicate. The agent also supports full snapshot tasks that seed a Cosmos DB container before CDC starts, and it supports the target role, so Cosmos DB can be the destination as well as the origin. Core Hub and agents deploy with Docker, Docker Compose, or Kubernetes.

Explore the general CDC streaming architecture →