Canonical: https://molo17.com/solutions/dynamodb-cdc/ Markdown mirror: https://molo17.com/solutions/dynamodb-cdc/index.md Title: DynamoDB CDC and replication with Gluesync | MOLO17 Description: DynamoDB CDC from DynamoDB Streams, replicated in real time to warehouses, data lakes, databases, and event streams with Gluesync by MOLO17. Start a trial. [Solutions](/solutions/) DynamoDB CDC SOLUTION · DYNAMODB CDC # Amazon DynamoDB change data capture and real-time data replication **Real-time DynamoDB CDC from DynamoDB Streams, the native change stream, with no scans to find the changes.** Gluesync by MOLO17 captures item changes from Amazon DynamoDB tables through a dedicated source agent that reads DynamoDB Streams, the native change stream API for DynamoDB. The agent delivers those changes continuously to the databases, warehouses, lakes, and event streams your teams already use, and it can also write to DynamoDB as a target. Manage every pipeline from the Core Hub web UI, the Gluesync control plane. [Start a Gluesync trial](/get-gluesync/) [Talk to us](/contacts/) WHO THIS IS FOR ## DynamoDB teams moving operational data into analytics, databases, and streams - Cloud and platform engineers who own DynamoDB tables in AWS and must approve an IAM access key, enable DynamoDB Streams on each table, and accept the AWS charges that Streams can add - Data platform leads feeding Google BigQuery, Google Cloud Storage, Amazon S3 data lakes, or event streams from DynamoDB, who want one control plane for every source and target - Teams migrating off DynamoDB or running it alongside Aerospike, Couchbase, or MongoDB, who need a snapshot for the initial load and CDC until cutover - Engineers who built a DynamoDB Streams consumer in Lambda, or a Kinesis export, and now maintain the consumer, its retries, and recovery after a gap themselves THE PROBLEM ## Batch copies of DynamoDB tables go stale between runs DynamoDB is usually the operational store behind an application, so a scheduled scan or export hands analytics and downstream systems a copy that is already behind. A full-table scan reads every item on each run, and it does not record which items changed in between. Buyers searching for DynamoDB CDC, DynamoDB replication, or DynamoDB to a warehouse usually compare four options: a Lambda consumer they write and run, Kinesis Data Streams for DynamoDB, AWS-native integrations such as AWS Glue or zero-ETL, and managed connectors whose capture method they need to confirm before they compare. Gluesync addresses that with **per-agent CDC**. The DynamoDB source agent reads DynamoDB Streams for each table you enable, and target agents write to the destination you choose. Both run under Core Hub, so one pipeline model covers DynamoDB as a source, as a target, or both. HOW IT WORKS ## How Gluesync does DynamoDB CDC ### Read DynamoDB Streams, the native change API CDC is implemented on the native DynamoDB Streams APIs. The agent connects through the AWS SDK for Java and Kotlin, which Gluesync embeds, and it forwards changes from the stream in near real time. - **Source role:** consumes DynamoDB Streams to capture changes from tables in any AWS region. - **Initial load:** Core Hub snapshot tasks scan DynamoDB tables, and the stream carries ongoing changes. - **State:** pipeline state is tracked in Core Hub metadata. - [DynamoDB CDC streams setup docs ↗](https://docs.molo17.com/gluesync/latest/agents/dynamodb-change-data-capture-streams.html) ### What your team enables in AWS before the first pipeline 1. Create an IAM access key ID and secret access key for the runtime user. The AWS managed policy `AmazonDynamoDBFullAccess` covers both the source and target roles. 2. Enable DynamoDB Streams on each table you replicate. Streams are off by default. In the AWS console, open the table's _Exports and streams_ tab, enable _DynamoDB stream details_, and choose **New and old images**. Gluesync does not turn streams on for you. 3. Enter the region, access key ID, and secret access key in Core Hub, or set them through the credentials REST endpoint. ### Regions, connectivity, and TLS - **Regions:** the agent works in any AWS region where DynamoDB is deployed. - **TLS:** encryption is always on for the source, and TLS is enforced by default for the target. - **Target writes:** the target role writes through the AWS SDK. ### Architecture around Core Hub The DynamoDB agent runs next to your AWS estate, and Core Hub coordinates it with every target agent through its web UI and REST APIs. A pipeline groups the source agent, its target agents, and the entities (tables) they replicate. Core Hub and agents deploy with Docker, Docker Compose, or Kubernetes. [Explore the general CDC streaming architecture →](/solutions/cdc-streaming/) CAPTURE OPTIONS ## One DynamoDB source agent, built on DynamoDB Streams DynamoDB has one Gluesync source agent. It captures changes from DynamoDB Streams, so there is no log or journal choice to make; the decisions are IAM and per-table stream enablement. The agent also covers the target role. | Agent | Capture technique | Versions | Best for | | --- | --- | --- | --- | | [Amazon DynamoDB agent ↗](https://docs.molo17.com/gluesync/latest/agents/dynamodb-intro.html) | DynamoDB Streams with new and old images, enabled per table | All AWS regions where DynamoDB is deployed | Teams that run DynamoDB in AWS and need its changes delivered to databases, warehouses, lakes, or streams, with source and target roles under one Core Hub. | TARGETS AND TOPOLOGIES ## Keep DynamoDB as hot storage and feed the rest of your estate Use the [integrations directory](/integrations/?source=Amazon+DynamoDB#integration-finder) to pair Amazon DynamoDB as source or target with databases, cloud warehouses, object and lake storage, or event streams, subject to the documented role of each agent. The DynamoDB agent also supports the target role. It writes through the AWS SDK, and a write to an existing key overwrites the item. Bulk load and TRUNCATE do not apply to DynamoDB targets. Gluesync keeps pace with your change volume at any scale. [MOLO17 Professional Services](/solutions/professional-services/) can size the deployment with your team. Target agents write in optimized batches, never row by row, and switch to native bulk load for both snapshots and CDC on targets such as [Snowflake](/solutions/replicate-to-snowflake/), [Google BigQuery](/solutions/replicate-to-bigquery/), [Amazon Redshift](/solutions/replicate-to-redshift/), [Microsoft SQL Server](/solutions/replicate-to-sql-server/), and [PostgreSQL](/solutions/replicate-to-postgresql/). - Keep DynamoDB as hot storage and send its changes to Amazon S3 or Google Cloud Storage for a lake: see [data lake](/solutions/data-lake/) - Feed a cloud warehouse such as Google BigQuery continuously, without a custom pipeline per target: see [warehouse sync](/solutions/warehouse-sync/) - Migrate off DynamoDB to Aerospike, Couchbase, or MongoDB with a snapshot and CDC until cutover: see [cloud migration](/solutions/cloud-migration/) - Publish DynamoDB changes to an event-streaming platform for downstream services, as in our [Solace PubSub+ CDC post](/blog/now-available-cdc-for-solace-pubsub-via-gluesync/) FAIR, HIGH-LEVEL COMPARISON ## Where Gluesync fits among DynamoDB CDC approaches | Approach | What buyers usually get | Where Gluesync fits | | --- | --- | --- | | Lambda consumers on DynamoDB Streams | AWS-native and close to the data. You write, deploy, and operate the consumer, its retries, and its target logic. | A productized source agent with snapshots for the initial load, Core Hub monitoring, and targets beyond AWS, with no consumer code to maintain. See [CDC streaming](/solutions/cdc-streaming/). | | Kinesis Data Streams for DynamoDB | AWS-managed delivery of item-level changes into a Kinesis stream, which you then consume and transform yourself. | One control plane for capture, snapshots, and delivery to the target, instead of a Kinesis stream plus your own consumers. | | AWS Glue and zero-ETL integrations | Managed inside AWS, with the destinations AWS supports. A strong fit when the destination is one AWS service. | Agent-based delivery to the targets in the [integrations directory](/integrations/?source=Amazon+DynamoDB#integration-finder), on-premises or in any cloud, managed from one Core Hub. | | Managed SaaS connectors | Quick to set up for common warehouses. Pricing, schedules, and capture behavior differ by vendor, so confirm how DynamoDB changes are captured before comparing. | Streams-based capture from agents you deploy on your own infrastructure, with Core Hub operations and [best-in-class MOLO17 enterprise support (rated 4.9/5 by customers)](/support/#customer-ratings). | | Scheduled scans or DynamoDB exports | Simple for periodic refreshes. Each run reads the whole table or export, and changes made between runs wait for the next run. | Stream-based CDC for changes between runs, with snapshots for the initial load. See [CDC streaming](/solutions/cdc-streaming/). | RELATED CONTENT ## DynamoDB CDC research and implementation detail - [DynamoDB as a CDC source and target for Gluesync](/blog/general-availability-of-dynamodb-cdc-as-a-replication-source-and-target-for-gluesync/) - [Gluesync for Google BigQuery: the fastest path to seamless data synchronization](/blog/google-big-query-data-synchronization-without-jdbc/) - [Now available: CDC for Solace PubSub+ via Gluesync](/blog/now-available-cdc-for-solace-pubsub-via-gluesync/) - [CDC streaming solution](/solutions/cdc-streaming/) - [Cloud migration solution](/solutions/cloud-migration/) - [DynamoDB agent overview ↗](https://docs.molo17.com/gluesync/latest/agents/dynamodb-intro.html) - [DynamoDB CDC streams setup ↗](https://docs.molo17.com/gluesync/latest/agents/dynamodb-change-data-capture-streams.html) FAQ ## DynamoDB CDC questions What is DynamoDB CDC with Gluesync? Change data capture from Amazon DynamoDB through the DynamoDB Streams API. A Gluesync source agent reads the stream for each table you enable and delivers the changes to the configured targets through Core Hub. Does Gluesync turn on DynamoDB Streams for us? No. Streams are off by default, and you turn them on per table in the AWS console, choosing New and old images. Gluesync does not enable streams through the SDK. How long does DynamoDB keep stream records? DynamoDB keeps stream records for 24 hours, the maximum available value, so the agent always reads from a window that covers a full day of changes. Core Hub snapshot tasks give you a fresh baseline for a table whenever you want one. Which regions does the DynamoDB agent support? The agent works in any AWS region where DynamoDB is deployed, and it reads tables in any region as a source, with the same agent covering the target role. What IAM permissions does the agent need? The AWS managed policy AmazonDynamoDBFullAccess for both roles, along with an IAM access key ID and secret access key. Can Gluesync write to DynamoDB, not only read from it? Yes. The target role writes through the AWS SDK with TLS enforced by default. A write to an existing key overwrites the item, and bulk load and TRUNCATE do not apply to DynamoDB targets. Is the connection to DynamoDB encrypted? Yes. TLS is always on for the source agent, and it is enforced by default for the target agent. How do we run the initial load? Core Hub snapshot tasks scan DynamoDB tables for the initial load, and the Streams capture carries ongoing changes. CDC BY SOURCE DATABASE ## Other sources Gluesync captures from - [Cosmos DB CDC](/solutions/cosmos-db-cdc/) - [Couchbase CDC](/solutions/couchbase-cdc/) - [Db2 LUW CDC](/solutions/db2-luw-cdc/) - [IBM i CDC](/solutions/ibm-i-cdc/) - [Informix CDC](/solutions/informix-cdc/) - [MariaDB CDC](/solutions/mariadb-cdc/) - [MongoDB CDC](/solutions/mongodb-cdc/) - [MySQL CDC](/solutions/mysql-cdc/) - [Oracle CDC](/solutions/oracle-cdc/) - [PostgreSQL CDC](/solutions/postgresql-cdc/) - [SAP ASE (Sybase) CDC](/solutions/sap-ase-cdc/) - [SAP HANA CDC](/solutions/sap-hana-cdc/) - [ScyllaDB CDC](/solutions/scylladb-cdc/) - [SQL Server CDC](/solutions/sql-server-cdc/) - [YugabyteDB CDC](/solutions/yugabytedb-cdc/) ## Evaluate Gluesync with your DynamoDB streams Start a trial on your infrastructure, or talk to MOLO17 about stream setup, IAM, snapshots, and the targets you need. [Start a Gluesync trial](/get-gluesync/) [Talk to MOLO17](/contacts/)