Real-time data replication to Greenplum from your operational databases

• Data warehouse teams running Greenplum who model current-state tables for reporting and need them keyed like the source, with duplicate keys resolved by a per-entity setting instead of a hand-written merge job
• Platform leads who want one Core Hub for Oracle, SQL Server, PostgreSQL, MySQL, IBM i, and MongoDB instead of one loader per source, backed by best-in-class enterprise support, rated 4.9/5 by customers
• Greenplum DBAs who want loads that use the cluster the way it is designed: every primary segment pulling its share of the data in parallel, tables distributed by key, and append-optimized column storage with compression
• Engineers replacing gpload scripts, Kafka Connect sinks, or scheduled ELT jobs that fill Greenplum, who want every source delivered by one product with snapshots and monitoring built in

The PostgreSQL write path, with Greenplum's connection defaults

Greenplum speaks the PostgreSQL wire protocol, so the Greenplum agent is the PostgreSQL target with Greenplum's connection defaults, a bulk path built for segments, and table creation that sets distribution and storage. It connects to the coordinator through the bundled PostgreSQL JDBC driver, on Greenplum Database 6.x and 7.x, and applies the change stream from any Gluesync source agent after a snapshot seeds each table.

• Same type mapping: values are written with the PostgreSQL target's type mapping, including enum types, arrays, and hstore.
• TRUNCATE before snapshot: the Core Hub option for a clean reload is honored on the target tables.
• Database connection only: the agent needs a connection to the coordinator and calls no Greenplum management API. Greenplum agent overview ↗

Two write paths: optimized batches and parallel bulk loads

The write path is chosen per entity, so a small reference table and a billion-row fact table can use the one that suits them. Both run on Greenplum 6 and 7 alike.

• Batch path: optimized JDBC batches of INSERT and DELETE statements. An upsert is an UPDATE … FROM followed by an INSERT of the keys that were not updated.
• Bulk path: turned on per entity for the snapshot and for CDC. Rows are encoded as tab-delimited text in a spool file on Core Hub and loaded into a staging table in the GLUESYNC schema, then merged into the target table.

Every segment loads its share with gpfdist

The bulk load method decides how the file reaches Greenplum, and the default chooses for you.

• gpfdist: the agent serves the file from an embedded gpfdist server and loads it through a temporary readable external table. Every primary segment pulls its share of the data in parallel, so the load grows with the number of segments.
• COPY: the file streams through COPY … FROM STDIN on the coordinator.
• Automatic choice: the default uses gpfdist when the segments can reach Core Hub and the load is large enough to benefit, and COPY otherwise. At setup the agent checks that every primary segment can reach the server.

Merged in one transaction, loaded straight when the key is enforced

• One transaction per flush: the staging table is merged into the target table in the same transaction as the load. If the load fails, the transaction rolls back and the batch is retried without writing any row twice.
• Co-located merges: the staging table is distributed by the key, so the merge joins rows that sit on the same segment as the target table's rows.
• Snapshot straight into the table: a snapshot of a table with an enforced unique key, such as a primary key, is loaded directly into the table with no staging step, so every row is written once.

Tables that look like Greenplum tables

When a target table does not exist, Core Hub proposes a CREATE TABLE from the source definition that you can edit before it runs. The agent also writes any table in the database that the connection user can write to, with the distribution policy and storage it already has.

• Distribution: tables are distributed by their primary key, and randomly when they have none. Edit the statement to distribute by another key.
• Storage: choose HEAP, AO_ROW, or AO_COLUMN append-optimized storage for the tables the agent creates, with ZSTD or ZLIB compression and a configurable compression level.
• Greenplum's own schemas stay out of the way: gp_toolkit and the other schemas Greenplum keeps for its tools, plus external and foreign tables, are not listed in discovery.

Data types

The type coverage is the PostgreSQL target's, matched to the types Greenplum has, and every column can be mapped to the alternatives Core Hub lists.

• Numbers and booleans: smallint, integer, bigint, the serial types, boolean, numeric and decimal, money, real, and double precision.
• Dates and times: date, time, timetz, timestamp, timestamptz, and interval.
• Text and documents: varchar, char, text, json, jsonb, xml, uuid, enum types, tsvector, and hstore when the extension is installed.
• Binary, network, geometric, and ranges: bytea and bit strings, inet, cidr, and macaddr, the geometric types, and the range types, plus arrays.

What your Greenplum admin sets up

The agent does not need a superuser. It needs a role that can log in to the coordinator and create its helper schema and tables. Full statements and settings are in the Greenplum target setup guide ↗.

1. Create a login role for Gluesync and grant CONNECT and CREATE on the database, and USAGE and CREATE on the target schema. The CREATE on the database lets the agent create its GLUESYNC helper schema.
2. For the gpfdist load, add one role attribute, with the same syntax on Greenplum 6 and 7: ALTER ROLE gluesync CREATEEXTTABLE (type='readable', protocol='gpfdist');
3. Allow the Core Hub host in the coordinator's pg_hba.conf and open the coordinator port, 5432 by default.
4. For the gpfdist load, open port 8580, the default, from every segment host to Core Hub, and set the advertised host to the name or address of the Core Hub machine that the segment hosts resolve.
5. In Core Hub, enter the coordinator's host, port, database, username, and password, and choose the storage type for the tables Gluesync creates.

TLS, Docker, and Kubernetes

• TLS for the connection: turn on TLS for the coordinator connection and either trust the server certificate or verify it against a trust store you provide.
• TLS for gpfdist: serve gpfdists from a PKCS12 keystore, with optional client certificates presented by the segments, signed by the same CA.
• Docker: publish the gpfdist port of the Core Hub container, 8580 by default, and set the advertised host to the Docker host address that the segments can reach.
• Kubernetes: expose the gpfdist port of the Core Hub pod with a Service or a host port and allow it in the network policies between the segment hosts and the pod.

Architecture around Core Hub

Source agents sit close to each database, and Core Hub orchestrates them through its web UI and REST APIs. A pipeline groups a source agent with the Greenplum agent and the tables it replicates, and pipeline state lives in Core Hub. Core Hub and agents deploy with Docker, Docker Compose, or Kubernetes. See CDC streaming for the wider pattern.

Explore the general CDC streaming architecture →

Any Gluesync source agent can feed Greenplum. Open the integrations finder with Greenplum pre-selected to see every source you can pair with it.

One Core Hub runs Oracle to Greenplum, SQL Server to Greenplum, and PostgreSQL to Greenplum side by side, with the same snapshot, monitoring, and duplicate-key settings for each. Gluesync keeps pace with your change volume at any scale, and MOLO17 Professional Services can help plan distribution, storage, and the write path for each table with your team.

• Oracle to Greenplum from the redo logs: see Oracle CDC
• SQL Server to Greenplum through Change Data Capture or Change Tracking: see SQL Server CDC
• PostgreSQL and MySQL to Greenplum from the WAL and the binlog: see PostgreSQL CDC and MySQL CDC
• IBM i (AS/400) and MongoDB to Greenplum through the native journal APIs and Change Streams: see IBM i CDC and MongoDB CDC
• WarehousePG, EDB's open-source Greenplum fork, written by the same agent: see Replicate to WarehousePG