Gluesync PostgreSQL WAL-based CDC source connector

Introducing the New PostgreSQL CDC (WAL-based) Connector in Gluesync 1.5

PostgreSQL WAL-based CDC is now GA with Gluesync 1.5

We at MOLO17 are excited to announce our latest source connector release: the introduction of the PostgreSQL CDC (Change Data Capture). Previously available only through a trigger-based auditing structure, this update marks a significant milestone, especially for businesses leveraging the robustness and versatility of PostgreSQL databases. The new connector is designed to streamline your data processes, ensuring that your data-driven insights and operations are more efficient, timely, and accurate than ever before.

Background

Before that release, Gluesync users got only one option for sourcing real-time data from PostgreSQL databases. While earlier versions relied on trigger-based auditing technology for PostgreSQL, they faced limitations, especially in cases where the database itself was under high pressure driven by the number of operations per second hitting its table records.

Native integration with your PostgreSQL database via WAL

We’re now introducing a dual approach to data capture, ensuring comprehensive compatibility and high performance across PostgreSQL versions:

  • Change Data Capture (CDC): For PostgreSQL versions 10 and above, the connector leverages native replication slots, optimizing workload isolation and data capture efficiency.
  • Gluesync Data Capture (GCD): Ensuring complete coverage, including for earlier PostgreSQL versions, the connector employs Gluesync’s proprietary trigger-based auditing.

Central to this system is the Write-Ahead Logging (WAL), a fundamental component in PostgreSQL that guarantees data integrity and consistency. By recording changes before they are committed to the database, WAL provides a reliable foundation for the replication process, enhancing the robustness and reliability of real-time data synchronization.

A diagram showcasing Gluesync's PostgreSQL source connector architecture based on WAL CDC or GDC

Seamless Integration and Performance

The PostgreSQL CDC connector is meticulously engineered to integrate seamlessly with your existing PostgreSQL setups, prioritizing minimal downtime and maximizing data integrity to ensure reliable and efficient data operations.

Details about implementing PostgreSQL CDC in Gluesync 1.5 can be found inside the official Gluesync documentation, which offers comprehensive guidance on setting up WAL-based Change Data Capture. You can find detailed information there, including technical setup, configuration steps, and other essential guidelines. Documentation is available here at the following link: Gluesync 1.5 – Setting up CDC for PostgreSQL​​.

Real-world Applications

In the realm of complex data integration projects, Gluesync 1.5’s CDC capability shines by ensuring that every incremental change made in the source PostgreSQL database is accurately captured and replicated in the target system by reading the transactions log based on the write-ahead-logs (WAL) technique. This level of detail is crucial for maintaining data consistency during migrations and/or replication projects, especially in dynamic environments where data changes frequently. Additionally, for real-time analytics, CDC’s comprehensive data capture enables more nuanced and up-to-date analysis, as every change in the database is immediately available for analytical processing, providing businesses with real-time insights that can drive informed decision-making.

Challenges and Solutions

Our CDC engine does not limit itself to sourcing real-time data changes; we designed it to outperform in bi-directional sync processes. You may already know that, unlike other data replication solutions, Gluesync provides a fully integrated high-performant bi-directional sync capability, and this new source connector also implements the same proving recursion loop barriers that are not so easy to achieve right out of the box. You can learn more about Gluesync’s bi-directional sync capabilities by visiting our product page here at this link.

Who’s MOLO17 and Why choosing us

MOLO17 is the reference partner for NoSQL solutions and has developed several projects using such technologies. Our mission is to ease the adoption of NoSQL databases with a safe & smooth approach providing a comprehensive set of features out-of-the-box: real-time event-driven, no third-party tools dependency, high-availability, and high-throughput approach.

MOLO17 securely unlocks enterprise data potential with no downtime and no impact on business continuity. MOLO17 products and solutions offer the best NoSQL database technologies to guarantee always-available consistent data and quality outcomes for mission-critical applications, whether in the cloud, on-premise, or for all mobile devices.

Want to get in touch with us? Drop us a line writing at hello@molo17.com.

Scroll to Top