Canonical: https://molo17.com/solutions/vertica-cdc/ Markdown mirror: https://molo17.com/solutions/vertica-cdc/index.md Title: Vertica CDC and real-time replication with Gluesync | MOLO17 Description: Vertica CDC from database epochs with no triggers or objects added, replicated in real time to warehouses, NoSQL, and streams with Gluesync. Start a trial. [Solutions](/solutions/) Vertica CDC SOLUTION · VERTICA CDC # Vertica change data capture and real-time data replication **Epoch-based CDC for Vertica with SELECT-only access: no triggers, no shadow tables, and no objects added to your database.** Gluesync by MOLO17 reads a consistent snapshot of your Vertica tables and then captures their changes as the difference between two epochs of the database. Every committed row in Vertica carries the epoch of its commit, so the agent delivers inserts, updates, and deletes with historical `SELECT` statements alone, and streams them continuously to the databases, warehouses, lakes, and event streams your teams already use. Manage every pipeline from the Core Hub web UI, the Gluesync control plane. [Start a Gluesync trial](/get-gluesync/) [Talk to us](/contacts/) WHO THIS IS FOR ## Vertica teams putting analytical data to work in other platforms - Vertica DBAs who approve what touches the cluster: the agent runs `SELECT` only, needs no superuser and no role, and adds no triggers, shadow tables, or other objects - Data platform leads feeding Snowflake, BigQuery, Kafka, data lakes, or operational databases from curated Vertica tables, who want one Core Hub for sources and targets instead of a stack of export jobs - Architects who need a consistent first load of large Vertica tables, followed by changes from the exact same point, with nothing lost and nothing delivered twice - Engineers who built timestamp polling or full re-extracts against Vertica and now own the schedules, the diffing queries, and the on-call rotation for a pipeline the business treats as infrastructure THE PROBLEM ## Vertica data that leaves in batches is stale before it arrives Vertica often holds years of curated analytics, and downstream teams want that data in other systems: a warehouse in another cloud, a search index, an operational database, a Kafka topic. Vertica offers no change stream and no row triggers, so the usual answer is a scheduled export or a polling query that rereads tables to work out what changed. Each run adds read load to the cluster, and the copy is only as fresh as the last run. Buyers searching for Vertica CDC usually weigh managed ELT tools, a Debezium and Kafka stack that needs a custom source for Vertica, or scripts their own team maintains. Gluesync addresses that with **per-agent CDC**. The Vertica agent uses what Vertica does record, the epoch of every commit, to compute an exact change set, and target agents write to the destination you choose, all under one Core Hub. HOW IT WORKS ## How Gluesync does Vertica CDC ### Epoch-based CDC: the difference between two epochs Every committed row in Vertica carries the epoch of its commit, and `AT EPOCH n SELECT …` reads the database as it was at epoch `n`, with no locks. That is enough to compute an exact change set, without a log reader. - **One position per entity:** the agent records the last epoch it delivered and the row count of the table at that epoch, and keeps that position in Core Hub. - **Net change per window:** each poll delivers the inserts, updates, and deletes committed since the position, and updates carry the full row. - **Key changes:** an update that changes a key column reaches the target as a delete of the old key and an insert of the new one. - **Restart-safe:** the new position is saved with the last chunk of a window, so after a crash the agent replays the same window and produces the same result. - **Nothing added to Vertica:** no triggers, no shadow tables, and no objects of any kind. The agent never writes to the source, and every read is a historical query that takes no locks. ### Snapshot at one pinned epoch, then CDC from the same epoch The snapshot reads every table at one epoch, pinned when it starts, so you get one consistent picture while applications keep writing. CDC starts from that same epoch: a change committed during the snapshot is delivered by CDC afterwards, so nothing is lost and nothing arrives twice. - **Parallel hash buckets:** each table is read in parallel hash buckets, each on its own connection, and the number of buckets is configurable. - **Resume per bucket:** after a restart, every bucket continues where it stopped. - **Partitions:** configured partitions replace the hash buckets and are still read at the pinned epoch. - **Filters:** a SQL `WHERE` clause per table limits the rows the snapshot and CDC read. ### Truncates and lost history are handled for you When the history a table needs is gone, for example after a `TRUNCATE` or a purge, the agent reloads the table on the target automatically, so the target converges to the source without anyone intervening. ### What your Vertica admin sets up A few short steps, all with standard Vertica statements. Full details are in the [Vertica source setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/vertica-source.html). 1. Use Vertica 24.1, the tested source version. 2. Keep at least an hour of history. A superuser sets it once with `ALTER DATABASE DEFAULT SET HistoryRetentionTime = 86400;`, which keeps a day, and a longer value lets a pipeline ride through a longer pause without a reload. 3. Create a login for Gluesync with `USAGE` on the schemas to replicate, `SELECT` on their tables, and `USAGE` on `public`. It needs no superuser and no role. 4. Make sure every CDC entity has a key: the primary key of the table, or the keys you choose on the entity in Core Hub. 5. Keep `MaxClientSessions` above the sessions your pipelines use, and allow Core Hub to reach Vertica on its client port, 5433 by default. ### Connectivity and operations - **Connection:** host, port (default `5433`), database name, username, and password, over the Vertica JDBC driver bundled with Gluesync, with optional TLS. - **Technical fields:** add the transaction ID, timestamp, and operation to the replicated rows when the target needs them. - **Same agent, both roles:** the Vertica agent is also a target. See [Replicate to Vertica](/solutions/replicate-to-vertica/). ### Architecture around Core Hub Lightweight agents sit next to each system; Core Hub is the control plane for the web UI and the REST APIs, and it routes changes between source and target agents. Pipelines group the Vertica source agent, its target agents, and the entities you replicate, and the position of every entity lives in Core Hub. Core Hub and agents deploy with Docker, Docker Compose, or Kubernetes. [Explore the general CDC streaming architecture →](/solutions/cdc-streaming/) CAPTURE OPTIONS ## One Vertica agent: epoch-based capture, as source and as target Gluesync has one Vertica agent. It captures changes from your tables with epoch reads, and it also writes to Vertica, so a single agent covers both roles in a pipeline. | Agent | Capture technique | Versions | Best for | | --- | --- | --- | --- | | [Vertica agent ↗](https://docs.molo17.com/gluesync/latest/agents/vertica-intro.html) | Epoch-based CDC: `AT EPOCH` historical reads compute the net change between two epochs, with a snapshot at one pinned epoch read in parallel hash buckets | Vertica 24.1 tested as a source; any Vertica deployment as a target | Vertica estates where changes must reach other platforms without triggers, shadow tables, or any object added to the cluster, using a login with `USAGE` and `SELECT` only. Pick it whenever Vertica is the source. | TARGETS AND TOPOLOGIES ## Send Vertica data to the platforms that need it Use the [integrations directory](/integrations/?source=Vertica#integration-finder) to pair Vertica as a source with relational engines, NoSQL stores, cloud warehouses, object and lake storage, or event streams. The same Vertica agent also writes to Vertica: see [Replicate to Vertica](/solutions/replicate-to-vertica/) for the target side. Gluesync keeps pace with your change volume at any scale. [MOLO17 Professional Services](/solutions/professional-services/) can size the deployment and the history retention with your team. Target agents write in optimized batches, never row by row, and switch to native bulk load for both snapshots and CDC on targets such as [Snowflake](/solutions/replicate-to-snowflake/), [Databricks](/solutions/replicate-to-databricks/), [Google BigQuery](/solutions/replicate-to-bigquery/), [Amazon Redshift](/solutions/replicate-to-redshift/), [Microsoft SQL Server](/solutions/replicate-to-sql-server/), and [PostgreSQL](/solutions/replicate-to-postgresql/). - Feed another warehouse or a data lake continuously from Vertica: see [warehouse sync](/solutions/warehouse-sync/) - Offload reads from Vertica to PostgreSQL, MongoDB, or another database: see [database offload](/solutions/database-offload/) - Publish Vertica changes to Apache Kafka or another event-streaming platform for downstream services - Run a snapshot plus CDC to keep Vertica and a new platform aligned until cutover: see [cloud migration](/solutions/cloud-migration/) FAIR, HIGH-LEVEL COMPARISON ## Where Gluesync fits among Vertica CDC approaches | Approach | What buyers usually get | Where Gluesync fits | | --- | --- | --- | | Scheduled exports and timestamp polling | Native export commands and queries driven by a scheduler; deciding which rows changed is a query you write, and each run rereads data on the cluster | Epoch reads that deliver exactly the rows that changed, including deletes, with no polling query to maintain and no objects added to Vertica | | Fivetran, Airbyte, and other ELT tools | Managed or open-source connectors with broad catalogs; Vertica source support, change detection, and sync schedule vary by connector and product | A commercial agent for Vertica with net-change CDC, a consistent pinned-epoch snapshot, and MOLO17 enterprise support under one Core Hub | | Debezium plus Kafka | Open-source Kafka Connect stack that you run yourselves; Debezium does not list a Vertica connector, so this path usually means a custom or third-party source | Productized Vertica agent with Core Hub operations and [MOLO17 enterprise support](/support/#customer-ratings); no Kafka mesh required. Read the [Debezium alternative](/solutions/debezium-alternative/) comparison | | Qlik Replicate, AWS DMS, and similar replication services | Mature replication products with broad source and target lists; Vertica as a source, and its capture method, differ by product and version | Agent-based Vertica capture under Core Hub with [heterogeneous targets](/integrations/?source=Vertica#integration-finder); see [migrating to Gluesync](/migrate-to-gluesync/) | | DIY scripts and full re-extracts | Full control with high build and operations cost; every extract reads production-sized tables and lags behind the cluster | Productized capture with pinned-epoch snapshots, restart-safe positions, and Core Hub monitoring; setup is described in the [Vertica source setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/vertica-source.html) | RELATED CONTENT ## Vertica CDC research and implementation detail - [Replicate to Vertica](/solutions/replicate-to-vertica/) - [Warehouse sync: keep cloud warehouses synchronized with operations](/solutions/warehouse-sync/) - [CDC streaming architecture](/solutions/cdc-streaming/) - [Debezium alternative: managed CDC versus Kafka and Debezium](/solutions/debezium-alternative/) - [Batch ETL vs real-time data replication: how to choose](/blog/batch-etl-vs-real-time-data-replication/) - [Vertica agent overview ↗](https://docs.molo17.com/gluesync/latest/agents/vertica-intro.html) - [Vertica source setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/vertica-source.html) - [Vertica target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/vertica-target.html) FAQ ## Vertica CDC questions What is Vertica CDC with Gluesync? Change data capture from Vertica using epochs. Every committed row carries the epoch of its commit, so the agent reads a snapshot of your tables and then computes the inserts, updates, and deletes between two epochs with historical SELECT statements, and Core Hub delivers them to the configured targets. How can Gluesync capture changes from Vertica without a log? Vertica records the epoch of every commit, and AT EPOCH reads the database as it was at a given epoch, with no locks. The agent compares two epochs to deliver the net change of each window, so it needs no triggers, shadow tables, or other objects in your database. Does Gluesync add anything to the Vertica database? No. The agent runs SELECT statements only and never writes to the source. It creates no triggers, no shadow tables, and no objects of its own. What does the Vertica admin need to set up? Vertica 24.1, a HistoryRetentionTime of at least one hour, which a superuser sets with ALTER DATABASE, a login with USAGE on the schemas and on public and SELECT on the tables, a key on every CDC entity, room in MaxClientSessions, and network access from Core Hub to the client port, 5433 by default. No superuser or role is needed for the Gluesync login. Which Vertica versions does Gluesync support? MOLO17 tests the Vertica source against Vertica 24.1. As a target, the same agent writes to any Vertica deployment, on premises or on AWS, Google Cloud, or Azure. How does the snapshot stay consistent while applications keep writing? The snapshot reads every table at one epoch pinned when it starts, in parallel hash buckets, each on its own connection. CDC then starts from that same epoch, so nothing is lost and nothing is delivered twice, and a restart resumes every bucket where it stopped. What happens after a TRUNCATE or when the history is gone? The agent reloads the table on the target automatically, so the target converges to the source without intervention. Can Gluesync write to Vertica, not only read from it? Yes. The same Vertica agent is also a target, written over the Vertica JDBC driver with TLS and load-balanced connections. See the Replicate to Vertica page. CDC BY SOURCE DATABASE ## Other sources Gluesync captures from - [Cosmos DB CDC](/solutions/cosmos-db-cdc/) - [Couchbase CDC](/solutions/couchbase-cdc/) - [Db2 LUW CDC](/solutions/db2-luw-cdc/) - [DynamoDB CDC](/solutions/dynamodb-cdc/) - [IBM i CDC](/solutions/ibm-i-cdc/) - [Informix CDC](/solutions/informix-cdc/) - [MariaDB CDC](/solutions/mariadb-cdc/) - [MongoDB CDC](/solutions/mongodb-cdc/) - [MySQL CDC](/solutions/mysql-cdc/) - [Oracle CDC](/solutions/oracle-cdc/) - [PostgreSQL CDC](/solutions/postgresql-cdc/) - [SAP ASE (Sybase) CDC](/solutions/sap-ase-cdc/) - [SAP HANA CDC](/solutions/sap-hana-cdc/) - [ScyllaDB CDC](/solutions/scylladb-cdc/) - [SQL Server CDC](/solutions/sql-server-cdc/) - [YugabyteDB CDC](/solutions/yugabytedb-cdc/) ## Evaluate Gluesync with your Vertica tables Start a trial on your infrastructure, or talk to MOLO17 about history retention, snapshot sizing, and targets for your Vertica estate. [Start a Gluesync trial](/get-gluesync/) [Talk to MOLO17](/contacts/)