Canonical: https://molo17.com/solutions/replicate-to-couchdb/ Markdown mirror: https://molo17.com/solutions/replicate-to-couchdb/index.md Title: Replicate to Apache CouchDB in real time | MOLO17 Description: Real-time replication to Apache CouchDB 3.x from Oracle, SQL Server, PostgreSQL, and MySQL, with parallel _bulk_docs writes across shards. Start a trial. [Solutions](/solutions/) Replicate to CouchDB SOLUTION · REPLICATE TO COUCHDB # Real-time data replication to Apache CouchDB from your operational databases **Rows from your systems of record land in CouchDB as JSON documents, continuously, written in parallel and merged into the documents already there.** Gluesync by MOLO17 captures changes from Oracle, SQL Server, PostgreSQL, MySQL, IBM i, MongoDB, and other [heterogeneous sources](/integrations/?target=CouchDB#integration-finder) with a dedicated agent per database, then writes them into Apache CouchDB 3.3, 3.4, and 3.5. The CouchDB agent talks to CouchDB over its HTTP JSON API, writes every batch with parallel `_bulk_docs` requests, handles revisions and conflicts for you, and creates databases with the number of shards your write volume needs. The same agent also reads from CouchDB as a [CDC source](/solutions/couchdb-cdc/). Core Hub, the Gluesync control plane, runs every pipeline from one web UI and REST API. [Start a Gluesync trial](/get-gluesync/) [Talk to us](/contacts/) WHO THIS IS FOR ## Teams that need CouchDB to reflect operations now, not after the next export - Application teams building offline-first or document-centric apps on CouchDB who need customers, orders, or catalogue data from a relational system of record kept current in their databases - CouchDB administrators who want writes that respect revisions, keep design documents, and spread across shards, instead of a script that overwrites documents and fails on the first conflict - Platform leads who want one Core Hub for Oracle, SQL Server, PostgreSQL, MySQL, IBM i, and MongoDB instead of one loader per source, backed by [best-in-class enterprise support, rated 4.9/5 by customers](/support/#customer-ratings) - Engineers replacing `_bulk_docs` import jobs, Kafka Connect sinks, or nightly JSON exports that fill CouchDB, who want snapshots and monitoring built in THE PROBLEM ## CouchDB data loaded in batches is already behind, and writing it right takes more than a POST CouchDB's own replication only moves data between CouchDB servers, so data from relational and other NoSQL systems usually arrives through scheduled imports. Apps then show yesterday's prices or stock, and every import job has to fetch revisions, retry conflicts, turn rows into documents, and avoid field names CouchDB rejects. Teams looking for real-time replication into CouchDB usually weigh a Kafka sink connector, an ETL tool that exports JSON, or code their own team maintains. Gluesync addresses that with **per-agent CDC into CouchDB**. A source agent reads each database's native change log, journal, or change stream; Core Hub routes the changes; the CouchDB agent applies them to your databases as they arrive. Whether the data comes from Oracle, IBM i, or another CouchDB, the pipeline model, the snapshot, and the operations stay the same. HOW IT WORKS ## How Gluesync writes to CouchDB ### Parallel \_bulk\_docs writes over the HTTP JSON API The agent talks to CouchDB over its HTTP JSON API with no vendor SDK, on a single node or a cluster. Each batch collapses to the last change of every document and goes out as parallel `_bulk_docs` requests, gzip compressed, with the number of requests in flight configurable. - **Revisions handled for you:** updates and deletes look up the live revision of each document, and documents that change under the write are looked up and written again. - **Deletes as tombstones:** a delete becomes a CouchDB tombstone, and deleting a document that is already gone is not an error. - **Idempotent by document id:** a busy or restarting server, an unreachable node, or a timeout is retried safely, and a change applied twice leaves the same document. - **Fast upsert snapshots:** a snapshot into a new or empty database skips the revision lookup and writes documents straight away. [CouchDB target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/couchdb-target.html) ### Shards set the write throughput, and Gluesync sets the shards The number of shards (`q`) of a CouchDB database is the main lever of its write throughput. When a target database does not exist, the agent creates it with the shards and replicas you choose, and you can raise the requests in flight with them. - **Measured in our lab:** on a single CouchDB 3.5.1 node, writes went from 15,000 documents per second on a database with 2 shards to 81,000 with 32 shards. - **Through the agent:** about 34,000 documents per second for inserts and upsert snapshots on a database with 8 shards and 8 requests in flight. - **Your layout, your call:** create the databases beforehand with the `q` you want, or let the agent create them with the shards set in the agent configuration. ### Partial changes merge into the stored document - **Merged, not replaced:** a change that carries only some columns is laid over the stored document at its current revision, so the other fields, and attachment stubs, keep their values. - **Duplicate keys by policy:** an insert over an existing document is reported as a duplicate key, so the per-entity insert conflict strategy of Core Hub decides what happens. - **Truncate your way:** before a snapshot, delete the documents and keep the design documents, recreate the database with the same shards and security, or leave the data as it is. ### Rows become documents with predictable ids - **One entity is one database:** the database is named after the table or an override, with an optional prefix, and made valid for CouchDB. - **Document ids from the key:** a single `_id` key is used as it is, a composite key is joined with `::`, and a custom key mapping composes any id you need. Documents from a CouchDB source keep their ids. - **JSON types:** numbers, booleans, strings, arrays, and objects are written as their JSON form. Decimals are written as strings by default to keep every digit, and dates as ISO text or epoch milliseconds, per column. - **Valid field names:** CouchDB reserves names that start with an underscore, so the agent checks target column names when the entity is set up, before any write. ### What your CouchDB administrator sets up The agent needs a user that can read and write the target databases. Full settings are in the [CouchDB target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/couchdb-target.html). 1. Create a user for Gluesync and make it a member of every target database. 2. To let Gluesync create missing databases, use a server administrator, or create the databases beforehand with the shards you want. 3. Open the CouchDB HTTP port to Core Hub, `5984` by default, or the TLS port you use. 4. In Core Hub, enter one node address, the port, and the credentials, list the other nodes of a cluster, and choose the shards for the databases Gluesync creates. ### Clusters, TLS, and reverse proxies - **Cluster failover:** list the other nodes of a cluster and the agent tries them in order when the node it talks to is down. - **TLS:** the server certificate is always validated, and a private certificate authority is trusted by giving the agent its CA certificate. - **Behind a proxy:** set the path prefix the server answers under, for example `/couchdb`. ### Architecture around Core Hub Source agents sit close to each database, and Core Hub orchestrates them through its web UI and REST APIs. A pipeline groups a source agent with the CouchDB agent and the entities it replicates, and pipeline state lives in Core Hub. Core Hub and agents deploy with Docker, Docker Compose, or Kubernetes. See [CDC streaming](/solutions/cdc-streaming/) for the wider pattern. [Explore the general CDC streaming architecture →](/solutions/cdc-streaming/) WRITE OPTIONS ## The CouchDB target agent Apache CouchDB has one Gluesync agent, for source and target. Every Gluesync source agent can feed it, and each entity follows the same snapshot-then-CDC path. | Agent | Write technique | Versions | Best for | | --- | --- | --- | --- | | [Apache CouchDB agent ↗](https://docs.molo17.com/gluesync/latest/agents/couchdb-intro.html) | HTTP JSON API with no vendor SDK; parallel, gzip-compressed `_bulk_docs` requests with revision lookup, conflict retries, tombstones for deletes, and partial changes merged into the stored document | Apache CouchDB 3.3, 3.4, and 3.5; single node or cluster; tested on 3.5.1 | CouchDB databases fed from Oracle, SQL Server, PostgreSQL, MySQL, IBM i, MongoDB, or another CouchDB, where documents should stay current and writes should scale with the shards of each database. Pick it whenever CouchDB is the destination. | SOURCES AND TOPOLOGIES ## Feed CouchDB from the systems of record you already run Any Gluesync source agent can feed CouchDB. Open the [integrations finder with CouchDB pre-selected](/integrations/?target=CouchDB#integration-finder) to see every source you can pair with it. One Core Hub runs PostgreSQL to CouchDB, Oracle to CouchDB, and SQL Server to CouchDB side by side, with the same snapshot, monitoring, and duplicate-key settings for each. Gluesync keeps pace with your change volume at any scale, and [MOLO17 Professional Services](/solutions/professional-services/) can help design the document ids and database layout with your team. - PostgreSQL and MySQL to CouchDB from the WAL and the binlog: see [PostgreSQL CDC](/solutions/postgresql-cdc/) and [MySQL CDC](/solutions/mysql-cdc/) - Oracle to CouchDB from the redo logs: see [Oracle CDC](/solutions/oracle-cdc/) - SQL Server to CouchDB through Change Data Capture or Change Tracking: see [SQL Server CDC](/solutions/sql-server-cdc/) - IBM i (AS/400) and MongoDB to CouchDB through the native journal APIs and Change Streams: see [IBM i CDC](/solutions/ibm-i-cdc/) and [MongoDB CDC](/solutions/mongodb-cdc/) - CouchDB to CouchDB, with the same document ids, from the changes feed: see [CouchDB CDC](/solutions/couchdb-cdc/) FAIR, HIGH-LEVEL COMPARISON ## Where Gluesync fits among CouchDB ingestion approaches | Approach | What buyers usually get | Where Gluesync fits | | --- | --- | --- | | CouchDB replication (\_replicator) | Built-in, reliable replication between CouchDB servers and PouchDB clients; the source is always CouchDB | Brings data from relational and NoSQL systems of record into CouchDB through each source's native change log; see [database offload](/solutions/database-offload/) | | Kafka Connect and a CouchDB sink connector | Capture into Kafka topics, written by a community sink connector; you run Kafka, Connect, offsets, and the revision and conflict handling | Changes written to CouchDB by key with revisions and conflicts handled, and no Kafka cluster in the path. See the [Debezium alternative](/solutions/debezium-alternative/) page | | ETL tools and JSON imports | Scheduled extracts turned into JSON and posted to `_bulk_docs`; CouchDB as a target is rare and usually custom | Continuous delivery from log-based capture, with snapshots seeding each database; read [batch ETL vs real-time data replication](/blog/batch-etl-vs-real-time-data-replication/) | | DIY scripts | Full control; your team owns extract queries, document mapping, revisions, conflict retries, and failures | Log-based capture, parallel writes, and Core Hub monitoring without pipeline code to maintain; see [migrating to Gluesync](/migrate-to-gluesync/) | RELATED CONTENT ## CouchDB replication research and implementation detail - [CouchDB CDC from the changes feed](/solutions/couchdb-cdc/) - [Batch ETL vs real-time data replication: how to choose](/blog/batch-etl-vs-real-time-data-replication/) - [Log-based CDC explained](/blog/real-time-replication-log-based-cdc/) - [Debezium alternative: managed CDC versus Kafka + Debezium](/solutions/debezium-alternative/) - [Replicate to MongoDB](/solutions/replicate-to-mongodb/) - [Apache CouchDB agent overview ↗](https://docs.molo17.com/gluesync/latest/agents/couchdb-intro.html) - [CouchDB target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/couchdb-target.html) - [Custom document keys ↗](https://docs.molo17.com/gluesync/latest/core-hub/custom-document-keys.html) - [Insert conflict strategy ↗](https://docs.molo17.com/gluesync/latest/core-hub/insert-conflict-strategy.html) FAQ ## CouchDB replication questions What does replicating to CouchDB with Gluesync involve? A source agent captures committed changes from your database through its native change mechanism, Core Hub routes them, and the CouchDB agent writes them to CouchDB databases as JSON documents, continuously, after a snapshot seeds each database. How does Gluesync write data into CouchDB? Over the CouchDB HTTP JSON API, with parallel, gzip-compressed \_bulk\_docs requests. The agent looks up revisions for updates and deletes, retries documents that changed under the write, writes deletes as tombstones, and merges partial changes into the stored document. Which CouchDB versions are supported? Apache CouchDB 3.3, 3.4, and 3.5, on a single node or on a cluster. MOLO17 tests the agent against CouchDB 3.5.1. How fast can Gluesync write to CouchDB? Write throughput follows the number of shards of the target database. In our lab, a single CouchDB 3.5.1 node took 15,000 documents per second on 2 shards and 81,000 on 32 shards. The agent creates missing databases with the shards you choose. How are document ids built? A single \_id key is used as it is, a composite key is joined with ::, and a custom key mapping can compose any id. Documents replicated from a CouchDB source keep their ids. What permissions does the CouchDB user need? Membership of every target database, for reading revisions and writing documents. Creating missing databases needs a server administrator; otherwise create them beforehand with the shards you want. Is the traffic encrypted? Yes. With TLS on, the server certificate is always validated, and a certificate from a private certificate authority is trusted by giving the agent that CA certificate. Can CouchDB also be a source? Yes. The same agent captures changes from the CouchDB \_changes feed, with a parallel snapshot. The CouchDB CDC page has the details. Which sources can replicate to CouchDB? Any Gluesync source agent, including Oracle, SQL Server, PostgreSQL, MySQL, IBM Db2 for i, Db2 for LUW, SAP HANA, MongoDB, and CouchDB itself. The integrations finder on our website lists every pairing. REPLICATE TO A TARGET ## Other targets Gluesync delivers to - [Replicate to Aerospike](/solutions/replicate-to-aerospike/) - [Replicate to DynamoDB](/solutions/replicate-to-dynamodb/) - [Replicate to Redshift](/solutions/replicate-to-redshift/) - [Replicate to Amazon S3 & S3-compatible](/solutions/replicate-to-amazon-s3/) - [Replicate to Cassandra](/solutions/replicate-to-cassandra/) - [Replicate to Kafka](/solutions/replicate-to-kafka/) - [Replicate to Cosmos DB](/solutions/replicate-to-cosmos-db/) - [Replicate to Azure Data Lake](/solutions/replicate-to-azure-data-lake/) - [Replicate to ClickHouse](/solutions/replicate-to-clickhouse/) - [Replicate to CockroachDB](/solutions/replicate-to-cockroachdb/) - [Replicate to Couchbase](/solutions/replicate-to-couchbase/) - [Replicate to Databricks](/solutions/replicate-to-databricks/) - [Replicate to Elasticsearch and OpenSearch](/solutions/replicate-to-elasticsearch/) - [Replicate to file stores](/solutions/replicate-to-file-stores/) - [Replicate to BigQuery](/solutions/replicate-to-bigquery/) - [Replicate to Google Cloud Storage](/solutions/replicate-to-google-cloud-storage/) - [Replicate to Google Pub/Sub](/solutions/replicate-to-google-pubsub/) - [Replicate to Greenplum](/solutions/replicate-to-greenplum/) - [Replicate to GridGain](/solutions/replicate-to-gridgain/) - [Replicate to Db2](/solutions/replicate-to-db2/) - [Replicate to Informix](/solutions/replicate-to-informix/) - [Replicate to MariaDB](/solutions/replicate-to-mariadb/) - [Replicate to Microsoft Fabric](/solutions/replicate-to-microsoft-fabric/) - [Replicate to SQL Server](/solutions/replicate-to-sql-server/) - [Replicate to MongoDB](/solutions/replicate-to-mongodb/) - [Replicate to MySQL](/solutions/replicate-to-mysql/) - [Replicate to Oracle](/solutions/replicate-to-oracle/) - [Replicate to PostgreSQL](/solutions/replicate-to-postgresql/) - [Replicate to RavenDB](/solutions/replicate-to-ravendb/) - [Replicate to Redis](/solutions/replicate-to-redis/) - [Replicate to Redis Streams](/solutions/replicate-to-redis-streams/) - [Replicate to SAP ASE](/solutions/replicate-to-sap-ase/) - [Replicate to SAP HANA](/solutions/replicate-to-sap-hana/) - [Replicate to SAP SQL Anywhere](/solutions/replicate-to-sap-sql-anywhere/) - [Replicate to ScyllaDB](/solutions/replicate-to-scylladb/) - [Replicate to SingleStore](/solutions/replicate-to-singlestore/) - [Replicate to Snowflake](/solutions/replicate-to-snowflake/) - [Replicate to Solace PubSub+](/solutions/replicate-to-solace/) - [Replicate to Supabase](/solutions/replicate-to-supabase/) - [Replicate to Vertica](/solutions/replicate-to-vertica/) - [Replicate to WarehousePG](/solutions/replicate-to-warehousepg/) - [Replicate to YugabyteDB](/solutions/replicate-to-yugabytedb/) ## Evaluate Gluesync with your CouchDB databases Start a trial against your CouchDB server, or talk to MOLO17 about your sources, the document ids you need, and the shard layout each database should use. [Start a Gluesync trial](/get-gluesync/) [Talk to MOLO17](/contacts/)