Canonical: https://molo17.com/blog/gluesync-databricks-target-agent-delta-merge-unity-catalog/ Markdown mirror: https://molo17.com/blog/gluesync-databricks-target-agent-delta-merge-unity-catalog/index.md Title: New Databricks target agent in Gluesync 2.3 · MOLO17 Description: New Databricks target agent in Gluesync 2.3: key-matched Delta MERGE, bulk snapshots from a Unity Catalog volume, OAuth service principal or PAT. [← All articles](/blog/) Article # New in Gluesync 2.3: the Databricks target agent  22 Oct 2026 [Gluesync](/blog/?category=gluesync) 6 min read Gluesync 2.3 adds a new Databricks target agent. Snapshots and CDC changes land in Delta tables in Unity Catalog as key-matched MERGE statements on a SQL warehouse, with bulk snapshots from a Unity Catalog volume and OAuth service principal or PAT authentication. Gluesync 2.3 adds a new **Databricks target agent**. It lands snapshots and CDC changes in Delta Lake tables registered in Unity Catalog, through a Databricks SQL warehouse and the Databricks SQL Statement Execution API. Every change is a Delta `MERGE` matched on the key, so inserts, updates and deletes keep the table equal to the source. Databricks is a target in Gluesync. Any supported source feeds it through a pipeline in Core Hub: snapshot first, then the source agent's CDC stream. Core Hub needs only outbound HTTPS to the workspace. No JDBC driver, no cloud-storage credentials, no external location. Docs: [Databricks agent](https://docs.molo17.com/gluesync/latest/agents/databricks-intro.html) · [Target setup](https://docs.molo17.com/gluesync/latest/agents/databricks-target.html) · [Changelog](https://docs.molo17.com/gluesync/latest/agents/databricks-agent-changelog.html). Overview: [Replicate to Databricks with Gluesync](https://molo17.com/solutions/replicate-to-databricks/). ## What the agent does - Writes snapshots and CDC changes into Delta tables in Unity Catalog, as key-matched `MERGE` statements on a SQL warehouse - Loads large snapshots in bulk from a Unity Catalog volume, one `MERGE` per load - Authenticates with an OAuth service principal (machine-to-machine) or a personal access token - Discovers schemas, Delta tables, columns and primary keys from the Unity Catalog `information_schema` - Runs Query Studio and data comparison on the same warehouse It works with any Databricks workspace that has Unity Catalog and a SQL warehouse (serverless, pro or classic), on AWS, Azure or Google Cloud, including Databricks Free Edition. ## Why MERGE by key Delta tables take changes best as `MERGE`. So every batch is applied that way: - **Inserts and updates:** `MERGE … WHEN MATCHED THEN UPDATE SET … WHEN NOT MATCHED THEN INSERT …`. Rows are matched null-safely (`<=>`) on the key they had **before** the change, so an update that changes the key moves the row. - **Deletes:** `MERGE … WHEN MATCHED THEN DELETE`. - **Truncate**, and the first snapshot batch: `TRUNCATE TABLE` on every table of the entity, unless _Disable truncate before snapshot_ is set. A Delta `MERGE` fails when two source rows match the same target row, and a CDC batch can touch a row many times. Gluesync keeps the order of the changes and cuts them into statements in which every key appears once. A later image of a row replaces the earlier one when both carry the same columns, and a key change or a different set of columns starts a new statement. The last change of each row is the one left on the table. Every statement the connector writes can run twice with the same result. Rate limiting (HTTP 429), an unavailable warehouse (HTTP 502 to 504) and dropped connections are retried up to five times with backoff, following `Retry-After`. A revoked OAuth token is replaced once. ## Bulk snapshots from a Unity Catalog volume Enable _bulk operations while snapshot_ on an entity and its snapshot loads in bulk: - Rows are staged in a JSON Lines file and loaded from a Unity Catalog volume with one `MERGE … USING read_files(…)` every _Rows per bulk snapshot load_ rows (default 1,000,000) and at the end of the snapshot. - Gluesync confirms the staged transactions only once they are loaded, so after a restart they are sent again and written once. - Staged files are deleted after each load. - The default staging volume is `.gluesync_staging.bulk_files`, created by the connector when missing. Point _Staging volume for bulk snapshots_ at your own `catalog.schema.volume` if you prefer. CDC and multi-table entities use the per-transaction `MERGE`. ## Tables Gluesync generates `CREATE TABLE IF NOT EXISTS .. (…) USING DELTA`. Key columns are `NOT NULL` and become an informational `PRIMARY KEY` constraint named `pk_`. Schema discovery reads it back, so the key is found again when an entity is mapped onto the table. ## Tables you create Any Delta table in the catalog can be a target. Schemas, tables and columns come from `.information_schema`. A column is a key when the table declares a `PRIMARY KEY` constraint on it; otherwise the key comes from the entity mapping. Columns are mapped explicitly, so keep the table in step with the source: a column the change carries and the table lacks fails the write with `UNRESOLVED_COLUMN`. ## Write settings - **Maximum rows per MERGE statement** (`maxRowsPerStatement`): default 1000 per entity. A statement is also cut before it reaches 3,000,000 characters, well below the 16 MiB a Databricks statement may take. Raise it for large snapshots. - **Statement timeout in seconds** (`statementTimeoutSeconds`): default 600. A statement still running after this time is cancelled on the warehouse. - **SQL pre/post snapshot and CDC commands** per entity, run on the warehouse. Keep them idempotent. ## Data types Every value is written as a literal cast to the type of its column, with microsecond timestamps. The defaults: | Gluesync type | Default Databricks type | | --- | --- | | `SHORT` / `INT` / `LONG` | `SMALLINT` / `INT` / `BIGINT` | | `FLOAT` / `DOUBLE` | `FLOAT` / `DOUBLE`, NaN and infinities kept | | `BIG_DECIMAL` | `DECIMAL(p, s)` up to 38 digits. Unsized is `DECIMAL(38, 18)` | | `BOOLEAN` | `BOOLEAN` | | `STRING` | `STRING`, or `VARIANT` parsed from JSON | | `BYTE_ARRAY` | `BINARY` | | `LOCAL_DATE` | `DATE` | | `LOCAL_TIME` / `OFFSET_TIME` | `STRING` (Databricks has no time-of-day type) | | `LOCAL_DATE_TIME` | `TIMESTAMP_NTZ` | | `OFFSET_DATE_TIME` | `TIMESTAMP` | | `ARRAY` / `MAP` | `STRING` with the JSON text, or native `ARRAY<…>` / `MAP<…>`, or `VARIANT` | Alternatives per type are in the [target setup docs](https://docs.molo17.com/gluesync/latest/agents/databricks-target.html). ## Connection setup Prerequisites: a workspace with Unity Catalog reachable over HTTPS from Core Hub, a SQL warehouse, and a service principal with an OAuth secret (recommended) or a personal access token. Grants for that identity: - `CAN USE` on the SQL warehouse - `USE CATALOG` on the catalog and `USE SCHEMA` on the target schemas - `SELECT` and `MODIFY` on the target tables - `CREATE TABLE` on the schema when Gluesync creates the tables - `READ VOLUME` and `WRITE VOLUME` on the staging volume for bulk snapshots, or `CREATE SCHEMA` and `CREATE VOLUME` on the catalog for the default one In the Control Plane, add Databricks as the **target** of a pipeline: - **Workspace URL:** `https://`, for example `https://adb-1234567890123456.7.azuredatabricks.net` or `https://dbc-1a2b3c4d-5e6f.cloud.databricks.com` - **Port:** `443` - **Authentication mode:** `SERVICE_PRINCIPAL` (default, OAuth machine-to-machine) or `PAT` - **Service principal client ID** (empty with a PAT) and **Client secret or personal access token** - **SQL warehouse ID:** the last part of the warehouse HTTP path, `/sql/1.0/warehouses/` - **Unity Catalog catalog:** the catalog that holds the target schemas The connection check reads the warehouse state without starting it, so a stopped warehouse passes. Serverless warehouses start on the first statement. ## Query Studio and data comparison The agent is SQL-capable. [Query Studio](https://docs.molo17.com/gluesync/latest/gs-modules/query-studio.html) runs queries on the configured warehouse and catalog with the same identity as the pipeline: - One statement per query, through the Statement Execution API. Read-only mode accepts `SELECT`, `WITH`, `SHOW`, `DESCRIBE` and `EXPLAIN`. - Backtick names and backslash-escaped strings, as Databricks SQL reads them. - _Explain_ shows the `EXPLAIN` plan, the DDL viewer shows `SHOW CREATE TABLE`. - A cancelled query is cancelled on the warehouse too. - Data comparison reads timestamps with every microsecond and checks the Delta tables against the source. The result grid shows milliseconds; cast a column to `STRING` to see the microseconds. ## Where it fits Operational databases on one side, the lakehouse on the other. PostgreSQL, Oracle, SQL Server, MySQL, MongoDB and the rest of the Gluesync catalog land their snapshot and CDC changes in Delta tables governed by Unity Catalog, kept equal to the source key by key. The full list of sources and targets is on the [Gluesync](https://molo17.com/gluesync/) page. ## Next steps 1. Read the [Databricks target setup](https://docs.molo17.com/gluesync/latest/agents/databricks-target.html) and set up the grants. 2. Deploy **Gluesync 2.3** and add Databricks as the target of a pipeline in Core Hub. 3. Turn on bulk snapshots for your largest tables, then watch CDC keep each Delta table current. Want to see it on your workspace? [Request a free trial](https://molo17.com/get-gluesync/) or [book a demo](https://molo17.com/book-a-demo/). Questions on a concrete build: I am around. [← Older article New in Gluesync 2.3: the ClickHouse target agent](/blog/gluesync-clickhouse-target-agent-replacingmergetree/) ## Keep reading 1. [Gluesync New in Gluesync 2.3: the ClickHouse target agent Gluesync 2.3 adds a new ClickHouse target agent. Every change lands as a new version of its key in a ReplacingMergeTree(\_version, \_is\_deleted) table, deletes become tombstones, and Query Studio reads the current row with FINAL. 15 Oct 2026](/blog/gluesync-clickhouse-target-agent-replacingmergetree/) 2. [Gluesync Query Forge: federated SQL across agents, without a warehouse One JDBC URL into Core Hub. Schemas map to the agents you already run. Cross-agent joins without standing up a warehouse. 24 Sep 2026](/blog/query-forge-federated-sql-why-when-what-how/) 3. [Gluesync Gluesync Couchbase DCP source agent: native CDC without Eventing If you’ve ever wanted Couchbase CDC without standing up Eventing, Gluesync 2.2.11.2 adds that path: a source-only DCP agent. No Eventing service, nothing written to the source bucket, CE and EE including Capella. It sits next to the Eventing source, it doesn’t replace it. 17 Sep 2026](/blog/gluesync-couchbase-dcp-source-agent/) [Back to all articles](/blog/) [More in Gluesync →](/blog/?category=gluesync)