Canonical: https://molo17.com/solutions/replicate-to-google-pubsub/ Markdown mirror: https://molo17.com/solutions/replicate-to-google-pubsub/index.md Title: Replicate to Google Pub/Sub in real time | MOLO17 Description: Real-time replication to Google Pub/Sub from Cloud SQL, Oracle, SQL Server, and MongoDB, with batched Java SDK publishing after a snapshot. Start a trial. [Solutions](/solutions/) Replicate to Google Pub/Sub SOLUTION · REPLICATE TO GOOGLE PUB/SUB # Real-time data replication to Google Pub/Sub topics **Every committed change in your databases is published to a Pub/Sub topic as it happens, authenticated with your Google Cloud service account, instead of waiting for a batch job.** Gluesync by MOLO17 captures changes from Google Cloud SQL, Oracle, SQL Server, PostgreSQL, IBM i, MongoDB, and other [heterogeneous sources](/integrations/?target=Google+Pub%2FSub#integration-finder) with a dedicated agent per database. The Pub/Sub target agent publishes each change through Google's official Java SDK, after a snapshot loads every entity, and Core Hub, the Gluesync control plane, runs every pipeline from one web UI and REST API. [Start a Gluesync trial](/get-gluesync/) [Talk to us](/contacts/) WHO THIS IS FOR ## Teams publishing database changes into Google Cloud messaging - Platform engineers on Google Cloud who want changes from Cloud SQL, Oracle, or SQL Server published to Pub/Sub topics that services subscribe to, with the project, topics, and IAM roles managed in the Google Cloud console - Microservice teams that need a feed of table changes without a polling job against the database, with each consumer reading from its own subscription - Data platform leads who feed BigQuery and Pub/Sub from the same source list through one Core Hub, backed by [best-in-class enterprise support, rated 4.9/5 by customers](/support/#customer-ratings) - Engineers who run a custom publisher or a connector chain that moves changes into Pub/Sub today, and want [every source](/integrations/?target=Google+Pub%2FSub#integration-finder) published through one agent model THE PROBLEM ## Pub/Sub subscribers that wait on batch jobs see old data Services that subscribe to Pub/Sub usually receive their events from a job that queries a database on a schedule, or from application code that publishes after each write. Scheduled queries trail the source by the length of their window and add load to production, and application-level publishing misses every change made outside that code path. Buyers weighing a fix usually compare Google Cloud native change capture with Dataflow pipelines, Kafka Connect with a Pub/Sub sink, Debezium-based chains, or custom publishers they maintain. Gluesync addresses that with **per-agent CDC into Pub/Sub**. A source agent reads each database's native change mechanism, Core Hub routes each change, and the Pub/Sub target agent publishes it to the topic for that entity. The message model stays the same whatever the source is, with the row's key and an operation type on every message. HOW IT WORKS ## How Gluesync publishes to Google Pub/Sub ### The write path: Google's official Java SDK, publishing in optimized batches The Pub/Sub target agent publishes through Google's official Java SDK. It is a target agent: it receives changes from any Gluesync source agent and publishes one message per change to the topic configured for that entity, straight to Pub/Sub with no local buffering on the agent. - **Optimized batches, never row by row:** Core Hub groups changes before they reach the agent, and the SDK publishes them to Pub/Sub in batches, so each request carries many messages and the publish path stays efficient as change volume grows. - **Service account authentication:** the agent signs in with a Google Cloud service account, so IAM decides which topics it can publish to. - **Port 443:** the agent reaches the Pub/Sub API over HTTPS on port 443 by default. [Google Pub/Sub agent overview ↗](https://docs.molo17.com/gluesync/latest/agents/google-pubsub-intro.html) ### Snapshot first, then change messages on the same topic - **Seed, then stream:** each entity is loaded from its full table, then the same topic receives changes from the source agent's change mechanism in real time. - **Operation type on every message:** inserts, updates, and deletes publish with an `operation` attribute of `i`, `u`, or `d`, so subscribers can filter or route them without parsing the body. - **Deletes:** a delete publishes with operation `d` and the key of the removed row, so subscribers remove the record from their own state. - **Parallel and resumable snapshots:** logical partitioning reads large source tables in parallel ranges, snapshot writing concurrency is configurable per entity, and an interrupted snapshot resumes from its last saved state. ### Topics, keys, and the JSON message body - **Topics you own:** each entity publishes to a topic you create in your Google Cloud project, so retention, subscriptions, and IAM stay under your platform team's control. - **Message key:** every message carries a `key` attribute with the row's document key, built from the fields you choose with the document key builder, so subscribers match each change to the record it belongs to. - **JSON body:** the message body is JSON, so subscribers parse one format whether the change came from Cloud SQL, Oracle, or MongoDB. - **Shaping before publish:** Custom Field Functions and UDFs reshape each payload, and Allowed Operations chooses which operations reach a topic. See [data transformation](/data-transformation/). ### Setup: a project, a service account, and topics Each Pub/Sub pipeline needs a project, its topics, and a service account that can publish to them. The full field list and REST calls are in the [target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/google-pubsub-target.html). 1. Create the topics in the Google Cloud console, one for each entity you publish. 2. Create a service account with publish rights on those topics, and assign its IAM roles in the console. 3. Download the service account's JSON key file. 4. Upload the JSON file in the Core Hub UI, or mount it into the agent as a volume, then enter the Project Id. The port defaults to `443`. ### Architecture around Core Hub Lightweight agents sit close to each source. Core Hub orchestrates them through its web UI and REST APIs and routes every change to the Pub/Sub agent. A pipeline groups the source agents, the Pub/Sub agent, and the entities they replicate, and Core Hub and the agents deploy with Docker, Docker Compose, or Kubernetes, on Google Cloud, on-premises, or in any other cloud. The same capture agent can feed Pub/Sub for services and BigQuery for analytics from one Core Hub. See [CDC streaming](/solutions/cdc-streaming/) and [snapshot tasks ↗](https://docs.molo17.com/gluesync/latest/core-hub/snapshot-tasks.html). [Explore the general CDC streaming architecture →](/solutions/cdc-streaming/) WRITE OPTIONS ## The Pub/Sub target agent: one publisher for every source Google Pub/Sub has one Gluesync target agent. The source agent decides how each change is captured, and the Pub/Sub agent publishes it in optimized batches to the topic for that entity. | Agent | Write technique | Versions | Best for | | --- | --- | --- | --- | | [Google Pub/Sub agent ↗](https://docs.molo17.com/gluesync/latest/agents/google-pubsub-intro.html) | Google Cloud Pub/Sub Java SDK with optimized batched publishing, keyed JSON messages, and service account authentication | Any region where Google Cloud Pub/Sub is available | Google Cloud teams that want change messages on Pub/Sub topics next to their Cloud SQL, Oracle, or SQL Server sources, with IAM controlling who can publish and who can subscribe. | SOURCES AND TOPOLOGIES ## Publish to Pub/Sub from the databases your teams run Any Gluesync source agent can publish to Pub/Sub, each with its own native capture technique. Open the [integrations finder with Google Pub/Sub pre-selected](/integrations/?target=Google+Pub%2FSub#integration-finder) to see every source you can pair with it, from Cloud SQL and Oracle to IBM i and Couchbase. One Core Hub can run Cloud SQL for PostgreSQL to Pub/Sub next to Oracle to Pub/Sub and SQL Server to Pub/Sub, with one set of snapshot and monitoring controls. Gluesync keeps pace with your change volume at any scale; [MOLO17 Professional Services](/solutions/professional-services/) can help design the topics and subscriptions with your team. - PostgreSQL and Cloud SQL for PostgreSQL to Pub/Sub from the write-ahead log: see [PostgreSQL CDC](/solutions/postgresql-cdc/) - MySQL and Cloud SQL for MySQL to Pub/Sub from the binary log: see [MySQL CDC](/solutions/mysql-cdc/) - Oracle to Pub/Sub from the redo logs through LogMiner or XStream: see [Oracle CDC](/solutions/oracle-cdc/) - SQL Server to Pub/Sub through Change Data Capture or Change Tracking: see [SQL Server CDC](/solutions/sql-server-cdc/) FAIR, HIGH-LEVEL COMPARISON ## Where Gluesync fits among Pub/Sub publishing approaches | Approach | What buyers usually get | Where Gluesync fits | | --- | --- | --- | | Application-level publishing | Full control of the message format. Your team writes the publish call next to each write, and owns retries, ordering, and any change to the code path that publishes | Changes come from the database log, so writes made outside the application are published too, and the message model stays the same across sources; see [CDC streaming](/solutions/cdc-streaming/) | | Scheduled queries and batch jobs | Simple to schedule on tables with a timestamp column. Each run queries production, and subscribers are only as current as the schedule | Log-based capture with no polling queries against production, and changes published as they arrive; see [warehouse sync](/solutions/warehouse-sync/) for the same capture feeding analytics | | Kafka Connect with a Pub/Sub sink, or Debezium-based chains | Open-source connectors that move changes through Kafka or a Connect runtime before they reach Pub/Sub. Your team runs each hop | One Core Hub from source to Pub/Sub, with no Kafka cluster in the path. Read the [Debezium alternative](/solutions/debezium-alternative/) comparison | | Google Cloud native change capture and Dataflow pipelines | Google Cloud services for capturing database changes and transforming them, configured per Google Cloud product | Agents that run next to each source, on premises or in any cloud, with Pub/Sub as one target among many; see [cloud migration](/solutions/cloud-migration/) | | Commercial replication suites | Mature replication products with broad target lists. Pub/Sub support and setup differ by product | A commercial product with a dedicated agent per source, Core Hub operations, and MOLO17 enterprise support behind every pipeline | RELATED CONTENT ## Google Pub/Sub background and implementation detail - [Google Pub/Sub target connector: data streaming with Gluesync](/blog/google-pub-sub-target-connector-data-intergration-with-gluesync/) - [CDC streaming without source overhead](/solutions/cdc-streaming/) - [Warehouse sync: keep cloud warehouses synchronized with operations](/solutions/warehouse-sync/) - [Log-based CDC explained](/blog/real-time-replication-log-based-cdc/) - [Data transformation before targets](/data-transformation/) - [Google Pub/Sub agent overview ↗](https://docs.molo17.com/gluesync/latest/agents/google-pubsub-intro.html) - [Google Pub/Sub target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/google-pubsub-target.html) FAQ ## Google Pub/Sub replication questions What does replicating to Google Pub/Sub with Gluesync involve? A source agent captures committed changes from your database through its native change mechanism, Core Hub routes them, and the Pub/Sub agent publishes each change to the topic for its entity, after a snapshot loads the entity. How are messages built? Each message carries a key attribute with the row's document key and an operation attribute: i for insert, u for update, and d for delete. The message body is JSON. How does Gluesync authenticate to Google Cloud? With a service account JSON key. Upload the file in Core Hub or mount it into the agent as a volume, and grant the service account publish rights on the topics in IAM. Does Gluesync create the topics? You create the topics in the Google Cloud console before the pipeline starts, which keeps naming, retention, and IAM in your hands. Gluesync then publishes each entity's changes to its topic. Which sources can publish to Pub/Sub? Any Gluesync source agent, including Cloud SQL for PostgreSQL, Cloud SQL for MySQL, Oracle, SQL Server, IBM i, MongoDB, and Couchbase. The integrations finder lists every pairing. Which regions does the Pub/Sub agent support? Any region where Google Cloud Pub/Sub is available. Publishing uses Google's official Java SDK. Does Gluesync publish one message at a time? No. Changes are grouped and the Java SDK publishes them in optimized batches. Gluesync keeps pace with your change volume at any scale. REPLICATE TO A TARGET ## Other targets Gluesync delivers to - [Replicate to Aerospike](/solutions/replicate-to-aerospike/) - [Replicate to DynamoDB](/solutions/replicate-to-dynamodb/) - [Replicate to Redshift](/solutions/replicate-to-redshift/) - [Replicate to Amazon S3 & S3-compatible](/solutions/replicate-to-amazon-s3/) - [Replicate to Cassandra](/solutions/replicate-to-cassandra/) - [Replicate to Kafka](/solutions/replicate-to-kafka/) - [Replicate to Cosmos DB](/solutions/replicate-to-cosmos-db/) - [Replicate to Azure Data Lake](/solutions/replicate-to-azure-data-lake/) - [Replicate to ClickHouse](/solutions/replicate-to-clickhouse/) - [Replicate to CockroachDB](/solutions/replicate-to-cockroachdb/) - [Replicate to Couchbase](/solutions/replicate-to-couchbase/) - [Replicate to file stores](/solutions/replicate-to-file-stores/) - [Replicate to BigQuery](/solutions/replicate-to-bigquery/) - [Replicate to Google Cloud Storage](/solutions/replicate-to-google-cloud-storage/) - [Replicate to GridGain](/solutions/replicate-to-gridgain/) - [Replicate to Db2](/solutions/replicate-to-db2/) - [Replicate to Informix](/solutions/replicate-to-informix/) - [Replicate to MariaDB](/solutions/replicate-to-mariadb/) - [Replicate to SQL Server](/solutions/replicate-to-sql-server/) - [Replicate to MongoDB](/solutions/replicate-to-mongodb/) - [Replicate to MySQL](/solutions/replicate-to-mysql/) - [Replicate to Oracle](/solutions/replicate-to-oracle/) - [Replicate to PostgreSQL](/solutions/replicate-to-postgresql/) - [Replicate to RavenDB](/solutions/replicate-to-ravendb/) - [Replicate to Redis](/solutions/replicate-to-redis/) - [Replicate to SAP ASE](/solutions/replicate-to-sap-ase/) - [Replicate to SAP HANA](/solutions/replicate-to-sap-hana/) - [Replicate to ScyllaDB](/solutions/replicate-to-scylladb/) - [Replicate to SingleStore](/solutions/replicate-to-singlestore/) - [Replicate to Snowflake](/solutions/replicate-to-snowflake/) - [Replicate to Solace PubSub+](/solutions/replicate-to-solace/) - [Replicate to Vertica](/solutions/replicate-to-vertica/) - [Replicate to YugabyteDB](/solutions/replicate-to-yugabytedb/) ## Evaluate Gluesync with your Google Cloud project Start a trial on your infrastructure, or talk to MOLO17 about your sources, your topics, and the service accounts that publish to them. [Start a Gluesync trial](/get-gluesync/) [Talk to MOLO17](/contacts/)