SOLUTION · REPLICATE TO DATABRICKS

Real-time data replication to Databricks from your operational databases

Committed changes from your operational databases reach Delta tables in Unity Catalog continuously, applied by key with Delta MERGE, instead of waiting for the next ingestion job.

Gluesync by MOLO17 captures changes from PostgreSQL, Oracle, SQL Server, MySQL, MongoDB, and other heterogeneous sources with a dedicated agent per database. The Databricks agent runs SQL on a Databricks SQL warehouse through the Statement Execution API, so it needs only outbound HTTPS to the workspace: no JDBC driver, no cloud-storage credentials, and no external location. Core Hub, the Gluesync control plane, runs snapshots and CDC for every pipeline from one web UI and REST API.

Start a Gluesync trial Talk to us

WHO THIS IS FOR

Teams that need the lakehouse to reflect operations now, not after the next job

Data engineers building on Delta tables who need current-state tables keyed like the source, with inserts, updates, and deletes applied by key and column types generated from the source definition
Platform leads running Databricks on AWS, Azure, or Google Cloud who want one Core Hub for every source, backed by best-in-class enterprise support, rated 4.9/5 by customers
Architects who want ingestion governed by Unity Catalog: a service principal with grants on one catalog, a SQL warehouse for the writes, and a volume for bulk snapshot files
Engineers replacing notebooks, Auto Loader jobs, or Kafka pipelines that feed Databricks, who want every source delivered by one product with snapshots and CDC under one control plane

THE PROBLEM

Lakehouse tables that are reloaded on a schedule are already stale

Most Databricks ingestion starts as a job: extract the source on a timer, drop files in cloud storage, and merge them into Delta tables with a notebook. Reports then trail the last run, each extract adds read load to production, and every new source adds a job, a storage path, and the credentials to reach it. Teams looking for Databricks CDC or real-time Delta Lake ingestion usually weigh Lakeflow Connect, Debezium and Kafka with a streaming job, managed ELT services such as Fivetran or Airbyte, or their own notebooks.

Gluesync addresses that with per-agent CDC into Databricks. A source agent reads each database's native change log or change stream, Core Hub routes the changes, and the Databricks agent applies them to Delta tables with key-matched MERGE statements. PostgreSQL to Databricks and Oracle to Databricks follow the same model, with the same snapshot and operations.

HOW IT WORKS

How Gluesync writes to Databricks

Key-matched Delta MERGE, in the order the source made the changes

The Databricks agent is a target agent: it applies the change stream from any Gluesync source agent to Delta tables in real time, after a snapshot seeds each table. Inserts and updates become MERGE statements matched on the key, and deletes become MERGE ... WHEN MATCHED THEN DELETE, so the table stays equal to the source and a batch replayed after a restart leaves it as it was.

Last change wins: each batch is cut into statements in which every key appears once, kept in source order, so repeated updates of a row and key changes in one transaction land as the source made them.
Key changes: rows are matched on the key value before the change, so an update that changes the key moves the row.
Typed values: every value is cast to its column type, with decimals at their precision, microsecond timestamps, and JSON for complex columns.
Batch size: the rows per statement are configurable per entity, and statements stay below the API size limit. Databricks agent overview ↗

Bulk snapshots through a Unity Catalog volume

Large tables load in bulk. With bulk loading on for the snapshot, the agent stages rows as JSON Lines, uploads the file to a Unity Catalog volume, and loads it with one MERGE ... USING read_files(...) per million rows and at the end of the snapshot. The load applies the same casts as the CDC path and keeps the last row of each key, and the checkpoint waits until the rows are in the table.

Seed, then stream: the snapshot populates each table, then changes follow from the source agent as they arrive, in commit order per entity.
Scheduled reloads: the Chronos scheduler runs a snapshot, or a snapshot followed by CDC, on a cadence for tables you refresh on a timetable. See schedules and events.

Tables, keys, and column types

Schemas, Delta tables, columns, and informational primary keys are read from the catalog's information_schema. When a target table does not exist, the agent creates it with CREATE TABLE ... USING DELTA, and the source key becomes an informational PRIMARY KEY, so the key is found again when an entity is mapped onto the table.

Integers and floats map to SMALLINT, INT, BIGINT, FLOAT, and DOUBLE, and decimals keep their source precision and scale as DECIMAL(p, s), up to 38 digits.
Dates and times map to DATE, TIMESTAMP_NTZ for wall-clock values, and TIMESTAMP for offset timestamps, which keep the instant.
Booleans, strings, and binaries map to BOOLEAN, STRING, and BINARY; arrays and maps are written as JSON text by default, or into ARRAY, MAP, and VARIANT columns.

Setup in your Databricks workspace

The agent needs a SQL warehouse, a Unity Catalog catalog, and an identity with rights to write it. Full grants are in the Databricks target setup guide ↗.

Create a service principal with an OAuth secret, or use a personal access token, and give it CAN USE on the SQL warehouse.
Grant it USE CATALOG, USE SCHEMA, CREATE TABLE, SELECT, and MODIFY on the target catalog, plus volume rights when bulk snapshots are on.
In Core Hub, enter the workspace host, the warehouse ID, the catalog, and the credentials of the service principal or the token.
Map each entity to a schema and table, or let the agent create the Delta tables from the source definition.

Architecture around Core Hub

Source agents sit close to each database, and Core Hub orchestrates them through its web UI and REST APIs. A pipeline groups a source agent with the Databricks agent and the tables it replicates. A stopped serverless warehouse does not fail the connection check: it starts with the first statement. Query Studio runs SQL on the same warehouse, and data comparison checks the Delta tables against the source, so you can verify a table from the same console. See Query Studio, and CDC streaming for the wider pattern.

Explore the general CDC streaming architecture →

WRITE OPTIONS

The Databricks target agent: one agent, Delta MERGE by key

Databricks has one Gluesync target agent. CDC writes key-matched Delta MERGE statements through a SQL warehouse, and snapshots can load in bulk from a Unity Catalog volume, so one agent covers every table in the pipeline.

Agent	Write technique	Versions	Best for
Databricks agent ↗	Databricks SQL Statement Execution API over HTTPS, applying key-matched Delta MERGE statements, with bulk snapshot loads through `read_files` from a Unity Catalog volume	Any workspace with Unity Catalog and a SQL warehouse, on AWS, Azure, or Google Cloud	Teams feeding Delta tables from transactional databases who want inserts, updates, and deletes applied by key, governed by Unity Catalog, with OAuth service principal authentication under one Core Hub.

SOURCES AND TOPOLOGIES

Feed Databricks from the databases you already run

Any Gluesync source agent can feed Databricks, each with its own native capture technique. Open the integrations finder with Databricks pre-selected to see every source you can pair with it.

One Core Hub runs PostgreSQL to Databricks, Oracle to Databricks, and SQL Server to Databricks side by side, with the same snapshot and write settings for each. Gluesync keeps pace with your change volume at any scale, and MOLO17 Professional Services can help with table design and warehouse sizing.

PostgreSQL to Databricks from the write-ahead log: see PostgreSQL CDC
Oracle to Databricks from the redo logs: see Oracle CDC
SQL Server and MySQL to Databricks: see SQL Server CDC and MySQL CDC
MongoDB to Databricks from change streams: see MongoDB CDC

FAIR, HIGH-LEVEL COMPARISON

Where Gluesync fits among Databricks ingestion approaches

Approach	What buyers usually get	Where Gluesync fits
Lakeflow Connect	Ingestion managed inside Databricks, scoped to its connector list and the sources it reaches	Agents you run next to each database, with one Core Hub across every source and target; see CDC streaming
Kafka with a streaming job	Event streams read by Structured Streaming into Delta; your team runs the brokers, the jobs, and the merge logic	Changes applied straight from each database agent to Delta tables by key, with no Kafka cluster or streaming job in the path. Kafka stays available as another target when you need it
Debezium with Kafka Connect	Open-source log-based capture, usually run through Kafka Connect; keys, offsets, deletes, and restarts are yours to operate	A native capture agent per database under one Core Hub, so operations sit in one control plane; read the Debezium alternative comparison
Fivetran, Airbyte, and other ELT tools	Managed or open-source connectors that load Databricks on a sync schedule; change capture varies by source connector	Log-based capture with a commercial agent per source and MOLO17 enterprise support, writing continuously rather than on a sync schedule
Notebooks and scheduled extracts	Full control; your team owns the extract queries, the files, the merge code, and the load each run puts on the source database	Log-based capture with no polling queries against production and no merge code to maintain; see migrating to Gluesync

Databricks replication research and implementation detail

What does replicating to Databricks with Gluesync involve?

A source agent captures committed changes from your database through its native change mechanism, Core Hub routes them, and the Databricks agent applies them to Delta tables in Unity Catalog. Each table is seeded with a snapshot first and then follows the change stream.

How does Gluesync write data into Databricks?

Through the Databricks SQL Statement Execution API on a SQL warehouse, over HTTPS. Inserts and updates become Delta MERGE statements matched on the key, and deletes become MERGE statements that delete the matched row. Large snapshots can load in bulk from a Unity Catalog volume.

Does the agent need a JDBC driver or cloud-storage credentials?

No. The agent only needs outbound HTTPS to the workspace. Bulk snapshot files go to a Unity Catalog volume through the Databricks Files API, so no external location or storage keys are involved.

Which sources can replicate to Databricks?

Any Gluesync source agent. PostgreSQL, Oracle, SQL Server, MySQL, and MongoDB can all feed Databricks, and each one uses its own native capture technique.

Does Gluesync create the Delta tables?

Yes. When a target table does not exist, the agent creates it as a Delta table from the source definition, with the source key as an informational primary key. Existing Delta tables in the catalog can be targets too.

How does the agent authenticate to Databricks?

With an OAuth service principal, which is the recommended setup, or with a personal access token. The agent refreshes the OAuth token on its own.

Can I query the Databricks tables from Gluesync?

Yes. Query Studio runs SQL on the agent's SQL warehouse, with read-only mode, EXPLAIN plans, and the table DDL, and data comparison checks the Delta tables against the source.

Which Databricks workspaces are supported?

Any workspace with Unity Catalog and a SQL warehouse, serverless, pro, or classic, on AWS, Azure, or Google Cloud.

REPLICATE TO A TARGET