Canonical: https://molo17.com/solutions/replicate-to-databricks/ Markdown mirror: https://molo17.com/solutions/replicate-to-databricks/index.md Title: Replicate to Databricks in real time with Gluesync | MOLO17 Description: Real-time replication to Databricks Delta tables from PostgreSQL, Oracle, SQL Server, MySQL, and MongoDB, with key-matched MERGE and bulk snapshots. Try it. [Solutions](/solutions/) Replicate to Databricks SOLUTION · REPLICATE TO DATABRICKS # Real-time data replication to Databricks from your operational databases **Committed changes from your operational databases reach Delta tables in Unity Catalog continuously, applied by key with Delta MERGE, instead of waiting for the next ingestion job.** Gluesync by MOLO17 captures changes from PostgreSQL, Oracle, SQL Server, MySQL, MongoDB, and other [heterogeneous sources](/integrations/?target=Databricks#integration-finder) with a dedicated agent per database. The Databricks agent runs SQL on a Databricks SQL warehouse through the Statement Execution API, so it needs only outbound HTTPS to the workspace: no JDBC driver, no cloud-storage credentials, and no external location. Core Hub, the Gluesync control plane, runs snapshots and CDC for every pipeline from one web UI and REST API. [Start a Gluesync trial](/get-gluesync/) [Talk to us](/contacts/) WHO THIS IS FOR ## Teams that need the lakehouse to reflect operations now, not after the next job - Data engineers building on Delta tables who need current-state tables keyed like the source, with inserts, updates, and deletes applied by key and column types generated from the source definition - Platform leads running Databricks on AWS, Azure, or Google Cloud who want one Core Hub for every source, backed by [best-in-class enterprise support, rated 4.9/5 by customers](/support/#customer-ratings) - Architects who want ingestion governed by Unity Catalog: a service principal with grants on one catalog, a SQL warehouse for the writes, and a volume for bulk snapshot files - Engineers replacing notebooks, Auto Loader jobs, or Kafka pipelines that feed Databricks, who want [every source](/integrations/?target=Databricks#integration-finder) delivered by one product with snapshots and CDC under one control plane THE PROBLEM ## Lakehouse tables that are reloaded on a schedule are already stale Most Databricks ingestion starts as a job: extract the source on a timer, drop files in cloud storage, and merge them into Delta tables with a notebook. Reports then trail the last run, each extract adds read load to production, and every new source adds a job, a storage path, and the credentials to reach it. Teams looking for Databricks CDC or real-time Delta Lake ingestion usually weigh Lakeflow Connect, Debezium and Kafka with a streaming job, managed ELT services such as Fivetran or Airbyte, or their own notebooks. Gluesync addresses that with **per-agent CDC into Databricks**. A source agent reads each database's native change log or change stream, Core Hub routes the changes, and the Databricks agent applies them to Delta tables with key-matched `MERGE` statements. PostgreSQL to Databricks and Oracle to Databricks follow the same model, with the same snapshot and operations. HOW IT WORKS ## How Gluesync writes to Databricks ### Key-matched Delta MERGE, in the order the source made the changes The Databricks agent is a target agent: it applies the change stream from any Gluesync source agent to Delta tables in real time, after a snapshot seeds each table. Inserts and updates become `MERGE` statements matched on the key, and deletes become `MERGE ... WHEN MATCHED THEN DELETE`, so the table stays equal to the source and a batch replayed after a restart leaves it as it was. - **Last change wins:** each batch is cut into statements in which every key appears once, kept in source order, so repeated updates of a row and key changes in one transaction land as the source made them. - **Key changes:** rows are matched on the key value before the change, so an update that changes the key moves the row. - **Typed values:** every value is cast to its column type, with decimals at their precision, microsecond timestamps, and JSON for complex columns. - **Batch size:** the rows per statement are configurable per entity, and statements stay below the API size limit. [Databricks agent overview ↗](https://docs.molo17.com/gluesync/latest/agents/databricks-intro.html) ### Bulk snapshots through a Unity Catalog volume Large tables load in bulk. With bulk loading on for the snapshot, the agent stages rows as JSON Lines, uploads the file to a Unity Catalog volume, and loads it with one `MERGE ... USING read_files(...)` per million rows and at the end of the snapshot. The load applies the same casts as the CDC path and keeps the last row of each key, and the checkpoint waits until the rows are in the table. - **Seed, then stream:** the snapshot populates each table, then changes follow from the source agent as they arrive, in commit order per entity. - **Scheduled reloads:** the Chronos scheduler runs a snapshot, or a snapshot followed by CDC, on a cadence for tables you refresh on a timetable. See [schedules and events](/schedules-and-events/). ### Tables, keys, and column types Schemas, Delta tables, columns, and informational primary keys are read from the catalog's `information_schema`. When a target table does not exist, the agent creates it with `CREATE TABLE ... USING DELTA`, and the source key becomes an informational `PRIMARY KEY`, so the key is found again when an entity is mapped onto the table. - **Integers and floats** map to `SMALLINT`, `INT`, `BIGINT`, `FLOAT`, and `DOUBLE`, and **decimals** keep their source precision and scale as `DECIMAL(p, s)`, up to 38 digits. - **Dates and times** map to `DATE`, `TIMESTAMP_NTZ` for wall-clock values, and `TIMESTAMP` for offset timestamps, which keep the instant. - **Booleans, strings, and binaries** map to `BOOLEAN`, `STRING`, and `BINARY`; arrays and maps are written as JSON text by default, or into `ARRAY`, `MAP`, and `VARIANT` columns. ### Setup in your Databricks workspace The agent needs a SQL warehouse, a Unity Catalog catalog, and an identity with rights to write it. Full grants are in the [Databricks target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/databricks-target.html). 1. Create a service principal with an OAuth secret, or use a personal access token, and give it `CAN USE` on the SQL warehouse. 2. Grant it `USE CATALOG`, `USE SCHEMA`, `CREATE TABLE`, `SELECT`, and `MODIFY` on the target catalog, plus volume rights when bulk snapshots are on. 3. In Core Hub, enter the workspace host, the warehouse ID, the catalog, and the credentials of the service principal or the token. 4. Map each entity to a schema and table, or let the agent create the Delta tables from the source definition. ### Architecture around Core Hub Source agents sit close to each database, and Core Hub orchestrates them through its web UI and REST APIs. A pipeline groups a source agent with the Databricks agent and the tables it replicates. A stopped serverless warehouse does not fail the connection check: it starts with the first statement. Query Studio runs SQL on the same warehouse, and data comparison checks the Delta tables against the source, so you can verify a table from the same console. See [Query Studio](/query-studio/), and [CDC streaming](/solutions/cdc-streaming/) for the wider pattern. [Explore the general CDC streaming architecture →](/solutions/cdc-streaming/) WRITE OPTIONS ## The Databricks target agent: one agent, Delta MERGE by key Databricks has one Gluesync target agent. CDC writes key-matched Delta MERGE statements through a SQL warehouse, and snapshots can load in bulk from a Unity Catalog volume, so one agent covers every table in the pipeline. | Agent | Write technique | Versions | Best for | | --- | --- | --- | --- | | [Databricks agent ↗](https://docs.molo17.com/gluesync/latest/agents/databricks-intro.html) | Databricks SQL Statement Execution API over HTTPS, applying key-matched Delta MERGE statements, with bulk snapshot loads through `read_files` from a Unity Catalog volume | Any workspace with Unity Catalog and a SQL warehouse, on AWS, Azure, or Google Cloud | Teams feeding Delta tables from transactional databases who want inserts, updates, and deletes applied by key, governed by Unity Catalog, with OAuth service principal authentication under one Core Hub. | SOURCES AND TOPOLOGIES ## Feed Databricks from the databases you already run Any Gluesync source agent can feed Databricks, each with its own native capture technique. Open the [integrations finder with Databricks pre-selected](/integrations/?target=Databricks#integration-finder) to see every source you can pair with it. One Core Hub runs PostgreSQL to Databricks, Oracle to Databricks, and SQL Server to Databricks side by side, with the same snapshot and write settings for each. Gluesync keeps pace with your change volume at any scale, and [MOLO17 Professional Services](/solutions/professional-services/) can help with table design and warehouse sizing. - PostgreSQL to Databricks from the write-ahead log: see [PostgreSQL CDC](/solutions/postgresql-cdc/) - Oracle to Databricks from the redo logs: see [Oracle CDC](/solutions/oracle-cdc/) - SQL Server and MySQL to Databricks: see [SQL Server CDC](/solutions/sql-server-cdc/) and [MySQL CDC](/solutions/mysql-cdc/) - MongoDB to Databricks from change streams: see [MongoDB CDC](/solutions/mongodb-cdc/) FAIR, HIGH-LEVEL COMPARISON ## Where Gluesync fits among Databricks ingestion approaches | Approach | What buyers usually get | Where Gluesync fits | | --- | --- | --- | | Lakeflow Connect | Ingestion managed inside Databricks, scoped to its connector list and the sources it reaches | Agents you run next to each database, with one Core Hub across every source and target; see [CDC streaming](/solutions/cdc-streaming/) | | Kafka with a streaming job | Event streams read by Structured Streaming into Delta; your team runs the brokers, the jobs, and the merge logic | Changes applied straight from each database agent to Delta tables by key, with no Kafka cluster or streaming job in the path. Kafka stays available as another target when you need it | | Debezium with Kafka Connect | Open-source log-based capture, usually run through Kafka Connect; keys, offsets, deletes, and restarts are yours to operate | A native capture agent per database under one Core Hub, so operations sit in one control plane; read the [Debezium alternative](/solutions/debezium-alternative/) comparison | | Fivetran, Airbyte, and other ELT tools | Managed or open-source connectors that load Databricks on a sync schedule; change capture varies by source connector | Log-based capture with a commercial agent per source and MOLO17 enterprise support, writing continuously rather than on a sync schedule | | Notebooks and scheduled extracts | Full control; your team owns the extract queries, the files, the merge code, and the load each run puts on the source database | Log-based capture with no polling queries against production and no merge code to maintain; see [migrating to Gluesync](/migrate-to-gluesync/) | RELATED CONTENT ## Databricks replication research and implementation detail - [Batch ETL vs real-time data replication: how to choose](/blog/batch-etl-vs-real-time-data-replication/) - [Log-based CDC: how real-time replication works](/blog/real-time-replication-log-based-cdc/) - [CDC streaming without source overhead](/solutions/cdc-streaming/) - [Debezium alternative: managed CDC versus Kafka + Debezium](/solutions/debezium-alternative/) - [Keep cloud warehouses synchronized with operations](/solutions/warehouse-sync/) - [Databricks agent overview ↗](https://docs.molo17.com/gluesync/latest/agents/databricks-intro.html) - [Databricks target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/databricks-target.html) - [Query Studio ↗](https://docs.molo17.com/gluesync/latest/gs-modules/query-studio.html) FAQ ## Databricks replication questions What does replicating to Databricks with Gluesync involve? A source agent captures committed changes from your database through its native change mechanism, Core Hub routes them, and the Databricks agent applies them to Delta tables in Unity Catalog. Each table is seeded with a snapshot first and then follows the change stream. How does Gluesync write data into Databricks? Through the Databricks SQL Statement Execution API on a SQL warehouse, over HTTPS. Inserts and updates become Delta MERGE statements matched on the key, and deletes become MERGE statements that delete the matched row. Large snapshots can load in bulk from a Unity Catalog volume. Does the agent need a JDBC driver or cloud-storage credentials? No. The agent only needs outbound HTTPS to the workspace. Bulk snapshot files go to a Unity Catalog volume through the Databricks Files API, so no external location or storage keys are involved. Which sources can replicate to Databricks? Any Gluesync source agent. PostgreSQL, Oracle, SQL Server, MySQL, and MongoDB can all feed Databricks, and each one uses its own native capture technique. Does Gluesync create the Delta tables? Yes. When a target table does not exist, the agent creates it as a Delta table from the source definition, with the source key as an informational primary key. Existing Delta tables in the catalog can be targets too. How does the agent authenticate to Databricks? With an OAuth service principal, which is the recommended setup, or with a personal access token. The agent refreshes the OAuth token on its own. Can I query the Databricks tables from Gluesync? Yes. Query Studio runs SQL on the agent's SQL warehouse, with read-only mode, EXPLAIN plans, and the table DDL, and data comparison checks the Delta tables against the source. Which Databricks workspaces are supported? Any workspace with Unity Catalog and a SQL warehouse, serverless, pro, or classic, on AWS, Azure, or Google Cloud. REPLICATE TO A TARGET ## Other targets Gluesync delivers to - [Replicate to Aerospike](/solutions/replicate-to-aerospike/) - [Replicate to DynamoDB](/solutions/replicate-to-dynamodb/) - [Replicate to Redshift](/solutions/replicate-to-redshift/) - [Replicate to Amazon S3 & S3-compatible](/solutions/replicate-to-amazon-s3/) - [Replicate to Cassandra](/solutions/replicate-to-cassandra/) - [Replicate to Kafka](/solutions/replicate-to-kafka/) - [Replicate to Cosmos DB](/solutions/replicate-to-cosmos-db/) - [Replicate to Azure Data Lake](/solutions/replicate-to-azure-data-lake/) - [Replicate to ClickHouse](/solutions/replicate-to-clickhouse/) - [Replicate to CockroachDB](/solutions/replicate-to-cockroachdb/) - [Replicate to Couchbase](/solutions/replicate-to-couchbase/) - [Replicate to Elasticsearch and OpenSearch](/solutions/replicate-to-elasticsearch/) - [Replicate to file stores](/solutions/replicate-to-file-stores/) - [Replicate to BigQuery](/solutions/replicate-to-bigquery/) - [Replicate to Google Cloud Storage](/solutions/replicate-to-google-cloud-storage/) - [Replicate to Google Pub/Sub](/solutions/replicate-to-google-pubsub/) - [Replicate to GridGain](/solutions/replicate-to-gridgain/) - [Replicate to Db2](/solutions/replicate-to-db2/) - [Replicate to Informix](/solutions/replicate-to-informix/) - [Replicate to MariaDB](/solutions/replicate-to-mariadb/) - [Replicate to Microsoft Fabric](/solutions/replicate-to-microsoft-fabric/) - [Replicate to SQL Server](/solutions/replicate-to-sql-server/) - [Replicate to MongoDB](/solutions/replicate-to-mongodb/) - [Replicate to MySQL](/solutions/replicate-to-mysql/) - [Replicate to Oracle](/solutions/replicate-to-oracle/) - [Replicate to PostgreSQL](/solutions/replicate-to-postgresql/) - [Replicate to RavenDB](/solutions/replicate-to-ravendb/) - [Replicate to Redis](/solutions/replicate-to-redis/) - [Replicate to Redis Streams](/solutions/replicate-to-redis-streams/) - [Replicate to SAP ASE](/solutions/replicate-to-sap-ase/) - [Replicate to SAP HANA](/solutions/replicate-to-sap-hana/) - [Replicate to ScyllaDB](/solutions/replicate-to-scylladb/) - [Replicate to SingleStore](/solutions/replicate-to-singlestore/) - [Replicate to Snowflake](/solutions/replicate-to-snowflake/) - [Replicate to Solace PubSub+](/solutions/replicate-to-solace/) - [Replicate to Supabase](/solutions/replicate-to-supabase/) - [Replicate to Vertica](/solutions/replicate-to-vertica/) - [Replicate to YugabyteDB](/solutions/replicate-to-yugabytedb/) ## Evaluate Gluesync with your Databricks workspace Start a trial against your Databricks workspace, or talk to MOLO17 about your sources, table design, and change volume. [Start a Gluesync trial](/get-gluesync/) [Talk to MOLO17](/contacts/)