New in Gluesync 2.3.1: the Greenplum target agent

Gluesync 2.3.1.0 adds a NEW Greenplum target agent. It writes snapshots and CDC changes from any Gluesync source into Greenplum Database 6.x and 7.x, and its bulk path loads through an embedded gpfdist server so every primary segment pulls its share of the data in parallel. Greenplum speaks the PostgreSQL wire protocol, so the agent is the Gluesync PostgreSQL target with Greenplum's connection defaults, a bulk path built for segments, and table creation that sets distribution and storage.

Docs: Greenplum agent · Target setup · 2.3.1.0 release notes.

Where it fits

Greenplum is a target in Gluesync. Any supported source feeds it through a pipeline in Core Hub: the snapshot first, then the source agent's CDC stream. The agent needs a database connection to the coordinator and, for the gpfdist load, network access from the segment hosts to Core Hub. It does not need a superuser and does not call any Greenplum management API.

Two write paths

Batch path. With bulk operations off, the agent writes JDBC batches: INSERT, DELETE, and for an upsert an UPDATE … FROM followed by an INSERT of the keys that were not updated. Greenplum 6 has no INSERT … ON CONFLICT and neither version has MERGE, so this is the statement shape that runs on both 6 and 7.

Bulk path. Toggle it per entity, for the snapshot and for CDC. Rows are encoded as tab-delimited text in a spool file on Core Hub, then loaded into a staging table in the GLUESYNC schema in one transaction and merged into the target table. The bulk load method decides how the file reaches Greenplum:

• GPFDIST: the agent serves the file from its embedded gpfdist server and runs INSERT INTO … SELECT … FROM a temporary readable external table. Every primary segment reads in parallel, so the load grows with the number of segments
• COPY: the file streams through COPY … FROM STDIN on the coordinator
• AUTO (default): gpfdist when the segments can reach Core Hub and the flush is at least 8 MiB, COPY otherwise

A snapshot of a table with an enforced unique key is loaded straight into the table.

Tables that look like Greenplum tables

Tables the agent creates are distributed by their primary key, randomly when they have none. Pick the storage per target: HEAP (default), AO_ROW or AO_COLUMN append-optimized, with ZSTD (default) or ZLIB compression and a configurable level. Enum types, arrays and hstore are written too.

Setup in a few lines

Point the agent at the coordinator (port 5432 by default) with a user that has CREATE on the database and on the target schema. For the gpfdist load, add one role attribute, same syntax on Greenplum 6 and 7:

ALTER ROLE gluesync CREATEEXTTABLE (type='readable', protocol='gpfdist');

The embedded gpfdist server listens on port 8580 by default, with an advertised host the segments resolve to Core Hub. TLS is available both for the database connection and for gpfdist (gpfdists, with optional client certificates for the segments). The docs cover Docker and Kubernetes networking and a reachability check.

Tested on

The integration suite runs against Greenplum Database 6.27 and 7.1, each with two primary segments.

Greenplum ships with Gluesync 2.3.1.0 alongside the Vertica source and the SAP ASE log reader. Full release: Gluesync 2.3.1. Point your pipelines at your Greenplum cluster and let every segment do the loading. Questions on the setup? I am around.