Apache CouchDB change data capture and real-time data replication

• Application teams whose data lives in CouchDB and who need it in PostgreSQL, SQL Server, or a warehouse for reporting, without exporting databases on a timer
• CouchDB administrators who want a source connection that only reads: a database member with read access, no design documents to deploy, no filter functions, and no change to the applications that write
• Data platform leads who feed analytics, search, or event streams from CouchDB and want one control plane for sources and targets, backed by best-in-class enterprise support, rated 4.9/5 by customers
• Engineers who wrote a _changes listener or a Kafka connector for CouchDB and now own checkpoints, restarts, type mapping, and every target writer

The changes feed, CouchDB's native change log

The agent polls the _changes feed of every replicated database in long-poll mode, from the position Core Hub saved. It creates nothing in your databases and needs no change to your applications.

• Inserts, updates, and deletes: a new document is an insert, a later revision is an update, and a deleted document is a delete keyed on its _id.
• Whole documents: every change carries the current document, read with the feed, so targets receive complete rows.
• Catch-up in one pass: a poll keeps reading while pages come back full, so a backlog drains without waiting for the next interval.
• At-least-once, applied safely: the position moves only after a page is delivered, and targets apply a repeated change without harm.
• State in Core Hub: the position of each database lives in Core Hub metadata. CouchDB source setup guide ↗

A parallel snapshot that does not page _all_docs

_all_docs is served as one stream and does not get faster with more requests. The agent lists document ids from the changes feed instead and reads the documents with many parallel _bulk_get requests.

• Parallel reads: the number of _bulk_get requests in flight is configurable. In our lab, 16 parallel requests read 94,000 documents per second, against 28,000 for _all_docs, and a 10-million-document database reached PostgreSQL end to end in 274 seconds.
• Resumable: an interrupted snapshot resumes from the feed position it had safely reached instead of starting over.
• No gap with CDC: changes made while the snapshot runs are replayed from the position the snapshot started at, so the target ends up with every document in its latest state.
• Bounded memory: a slow target slows the readers down, and a database is never held in memory.

Databases, entities, and fields

• One entity is one database: discovery lists the databases of the server and leaves the system databases out.
• Fields inferred from the documents: CouchDB declares no schema, so the agent samples each database's documents and infers every field and its type: string, integer, number, boolean, object, array, or JSON. The sample size is configurable.
• Keyed on _id: the entity key is the document id, and design documents are skipped in the snapshot and in CDC.
• Exact numbers: numbers are read from the literal JSON text, so integers keep every digit.

What your CouchDB administrator sets up

The agent reads documents and the changes feed, and nothing else. A database member covers it.

1. Create a user for Gluesync in the _users database.
2. Add that user to the members of every database to replicate, in the database's _security object.
3. Open the CouchDB HTTP port to Core Hub, 5984 by default, or the TLS port you use.
4. In Core Hub, enter one node address, the port, and the credentials. With a user that is not a server administrator, list the databases to replicate in the connection.
5. Keep _id as the key of each entity, so deletes reach the target.

Clusters, TLS, and reverse proxies

• Single node or cluster: list the other nodes of a cluster and the agent tries them in order when the node it talks to is down.
• TLS: with TLS on, the server certificate is always validated. A certificate from a private certificate authority is trusted by giving the agent that CA certificate.
• Authentication: HTTP Basic authentication, or anonymous access for a server that allows it.
• Behind a proxy: set the path prefix the server answers under, for example /couchdb.

Architecture around Core Hub

Agents sit close to each database, and Core Hub is the control plane above them: web UI, REST APIs, and routing from the CouchDB source agent to its targets. A pipeline groups the source agent, its target agents, and the entities (databases) they replicate. Core Hub and agents deploy with Docker, Docker Compose, or Kubernetes.

Explore the general CDC streaming architecture →

Open the integrations finder with CouchDB pre-selected to see every target you can pair with it. CouchDB can be a source, a target, or both.

Gluesync keeps pace with your change volume at any scale. MOLO17 Professional Services can help map your document shapes to relational tables with your team.

Target agents write in optimized batches, never row by row, and switch to native bulk load for both snapshots and CDC on targets such as Snowflake, Databricks, Google BigQuery, Amazon Redshift, Microsoft SQL Server, and PostgreSQL.

• CouchDB to PostgreSQL, SQL Server, Oracle, or MySQL, with document fields mapped to typed columns: see Replicate to PostgreSQL
• Feed CouchDB documents to a cloud warehouse, data lake, or object store: see warehouse sync
• Publish CouchDB changes to Apache Kafka or another event stream for downstream services: see Replicate to Kafka
• Move CouchDB data into MongoDB, Couchbase, or Elasticsearch: see Replicate to MongoDB
• Copy CouchDB databases to another CouchDB server with the same document ids: see Replicate to CouchDB