Real-time data replication to BigQuery from your operational databases

• Analytics engineers who model on BigQuery and need current-state tables keyed like the source, with a primary key on every replicated table so models read the same keys the application uses
• Data platform leads feeding BigQuery from Oracle, SQL Server, PostgreSQL, or MongoDB who want one Core Hub for every source instead of one connector product per source, backed by best-in-class enterprise support, rated 4.9/5 by customers
• Architects who want BigQuery ingestion to follow BigQuery's own load model: changes collapsed per key, staged in Cloud Storage, and applied with one load job per table per cycle, with replication that keeps moving through quota windows on its own
• Engineers running Debezium with a Kafka Connect BigQuery sink, or scheduled extract scripts, who want every source delivered to BigQuery by one product with snapshots and CDC under one control plane

The write path: native SDK, Cloud Storage staging, and load jobs

The Google BigQuery agent uses the official Google BigQuery SDK, which Gluesync embeds, so there is no JDBC driver to license or tune. It is a target agent: it applies the change stream from any Gluesync source agent to your datasets in real time, after a snapshot seeds each table.

• Optimized batches, never row by row: every write is grouped. Changes are gathered into highly optimized batches, with a batch size configurable on the agent, so BigQuery receives a few large operations instead of a stream of single-row calls.
• Snapshots: batches are loaded with WRITE_TRUNCATE load jobs, which replace the table contents in one atomic refresh. Truncate before snapshot is on by default and can be switched off in the target settings.
• Through quota windows: when a load job reaches a BigQuery quota, the agent switches to streaming writes for a cool-down window, then returns to load jobs on its own, so replication keeps moving. Google BigQuery agent overview ↗

Native bulk load for snapshots and CDC

Bulk load uses BigQuery's own load jobs and is switched on per entity with two independent settings: Bulk for Snapshot for the initial load and Bulk for CDC for the ongoing change stream. Both can be changed on an existing entity without recreating it.

In each mirroring cycle Core Hub collects the change events and collapses those on the same primary key: an insert followed by a delete is skipped, and consecutive updates become one update. The batch is written as Parquet, uploaded to your Cloud Storage bucket, and loaded into a persistent staging table with a WRITE_TRUNCATE load job. Columns the source log omitted can be backfilled from the live table, then deletes for the changed keys and inserts of their current values are applied in one pass. Each changed key ends as one current row, and staged files are cleared once the load completes.

Snapshot first, then continuous CDC

• Seed, then stream: each table loads in full first, then follows changes from its source agent as they arrive, in commit order per entity.
• Scheduled reloads: the Chronos scheduler runs a snapshot, or a snapshot followed by CDC, on a cadence for tables you refresh on a timetable. See schedules and events.
• Truncate control: a snapshot can replace the table contents with one load job, or leave existing rows in place when truncation is disabled.

Keys, tables, and setup in your Google Cloud project

Gluesync writes to datasets you own. The agent authenticates with a service account that can write to those datasets and read and write a staging bucket in the same project. Full statements are in the Google BigQuery target setup guide ↗.

1. Create each target table in the dataset with its primary key, for example PRIMARY KEY (ID) NOT ENFORCED. Every replicated table needs one.
2. Create a Cloud Storage bucket for staging in the same project as the datasets, close to them to keep egress low. An optional folder prefix keeps staged files grouped.
3. Create a service account with roles/bigquery.dataEditor and roles/bigquery.jobUser on the datasets, and roles/storage.objectAdmin on the staging bucket.
4. Create a JSON key for that service account and upload it in Core Hub, or mount it into the agent as a volume.
5. Enter the Google Cloud project ID, the staging bucket, and the dataset location, which defaults to US.

Architecture around Core Hub

Source agents sit close to each database, and Core Hub orchestrates them through its web UI and REST APIs. A pipeline groups a source agent with the BigQuery agent and the tables it replicates. Query Studio, the SQL workbench in Core Hub, opens BigQuery in a read-only session with autocomplete, so you can check landed rows without another client. See Query Studio, and CDC streaming for the wider pattern.

Explore the general CDC streaming architecture →

Any Gluesync source agent can feed BigQuery, each with its own native capture technique. Open the integrations finder with Google BigQuery pre-selected to see every source you can pair with it.

One Core Hub runs Oracle to BigQuery, PostgreSQL to BigQuery, and MongoDB to BigQuery side by side, with the same snapshot, scheduling, and write settings for each. Gluesync keeps pace with your change volume at any scale, and MOLO17 Professional Services can help lay out datasets and staging with your team.

• Oracle to BigQuery through its native capture: see Oracle CDC
• SQL Server to BigQuery through Change Data Capture or Change Tracking: see SQL Server CDC
• PostgreSQL and MySQL to BigQuery from the WAL and the binlog: see PostgreSQL CDC and MySQL CDC
• MongoDB and DynamoDB to BigQuery: see MongoDB CDC and DynamoDB CDC