New in Gluesync 2.3: the Databricks target agent

Gluesync 2.3 adds a new Databricks target agent. It lands snapshots and CDC changes in Delta Lake tables registered in Unity Catalog, through a Databricks SQL warehouse and the Databricks SQL Statement Execution API. Every change is a Delta MERGE matched on the key, so inserts, updates and deletes keep the table equal to the source.

Databricks is a target in Gluesync. Any supported source feeds it through a pipeline in Core Hub: snapshot first, then the source agent's CDC stream. Core Hub needs only outbound HTTPS to the workspace. No JDBC driver, no cloud-storage credentials, no external location.

Docs: Databricks agent · Target setup · Changelog. Overview: Replicate to Databricks with Gluesync.

What the agent does

• Writes snapshots and CDC changes into Delta tables in Unity Catalog, as key-matched MERGE statements on a SQL warehouse
• Loads large snapshots in bulk from a Unity Catalog volume, one MERGE per load
• Authenticates with an OAuth service principal (machine-to-machine) or a personal access token
• Discovers schemas, Delta tables, columns and primary keys from the Unity Catalog information_schema
• Runs Query Studio and data comparison on the same warehouse

It works with any Databricks workspace that has Unity Catalog and a SQL warehouse (serverless, pro or classic), on AWS, Azure or Google Cloud, including Databricks Free Edition.

Why MERGE by key

Delta tables take changes best as MERGE. So every batch is applied that way:

• Inserts and updates: MERGE … WHEN MATCHED THEN UPDATE SET … WHEN NOT MATCHED THEN INSERT …. Rows are matched null-safely (<=>) on the key they had before the change, so an update that changes the key moves the row.
• Deletes: MERGE … WHEN MATCHED THEN DELETE.
• Truncate, and the first snapshot batch: TRUNCATE TABLE on every table of the entity, unless Disable truncate before snapshot is set.

A Delta MERGE fails when two source rows match the same target row, and a CDC batch can touch a row many times. Gluesync keeps the order of the changes and cuts them into statements in which every key appears once. A later image of a row replaces the earlier one when both carry the same columns, and a key change or a different set of columns starts a new statement. The last change of each row is the one left on the table.

Every statement the connector writes can run twice with the same result. Rate limiting (HTTP 429), an unavailable warehouse (HTTP 502 to 504) and dropped connections are retried up to five times with backoff, following Retry-After. A revoked OAuth token is replaced once.

Bulk snapshots from a Unity Catalog volume

Enable bulk operations while snapshot on an entity and its snapshot loads in bulk:

• Rows are staged in a JSON Lines file and loaded from a Unity Catalog volume with one MERGE … USING read_files(…) every Rows per bulk snapshot load rows (default 1,000,000) and at the end of the snapshot.
• Gluesync confirms the staged transactions only once they are loaded, so after a restart they are sent again and written once.
• Staged files are deleted after each load.
• The default staging volume is <catalog>.gluesync_staging.bulk_files, created by the connector when missing. Point Staging volume for bulk snapshots at your own catalog.schema.volume if you prefer.

CDC and multi-table entities use the per-transaction MERGE.

Tables Gluesync generates

CREATE TABLE IF NOT EXISTS <catalog>.<schema>.<table> (…) USING DELTA. Key columns are NOT NULL and become an informational PRIMARY KEY constraint named pk_<table>. Schema discovery reads it back, so the key is found again when an entity is mapped onto the table.

Tables you create

Any Delta table in the catalog can be a target. Schemas, tables and columns come from <catalog>.information_schema. A column is a key when the table declares a PRIMARY KEY constraint on it; otherwise the key comes from the entity mapping.

Columns are mapped explicitly, so keep the table in step with the source: a column the change carries and the table lacks fails the write with UNRESOLVED_COLUMN.

Write settings

• Maximum rows per MERGE statement (maxRowsPerStatement): default 1000 per entity. A statement is also cut before it reaches 3,000,000 characters, well below the 16 MiB a Databricks statement may take. Raise it for large snapshots.
• Statement timeout in seconds (statementTimeoutSeconds): default 600. A statement still running after this time is cancelled on the warehouse.
• SQL pre/post snapshot and CDC commands per entity, run on the warehouse. Keep them idempotent.

Data types

Every value is written as a literal cast to the type of its column, with microsecond timestamps. The defaults:

Gluesync type	Default Databricks type
SHORT / INT / LONG	SMALLINT / INT / BIGINT
FLOAT / DOUBLE	FLOAT / DOUBLE, NaN and infinities kept
BIG_DECIMAL	DECIMAL(p, s) up to 38 digits. Unsized is DECIMAL(38, 18)
BOOLEAN	BOOLEAN
STRING	STRING, or VARIANT parsed from JSON
BYTE_ARRAY	BINARY
LOCAL_DATE	DATE
LOCAL_TIME / OFFSET_TIME	STRING (Databricks has no time-of-day type)
LOCAL_DATE_TIME	TIMESTAMP_NTZ
OFFSET_DATE_TIME	TIMESTAMP
ARRAY / MAP	STRING with the JSON text, or native ARRAY<…> / MAP<…>, or VARIANT

Alternatives per type are in the target setup docs.

Connection setup

Prerequisites: a workspace with Unity Catalog reachable over HTTPS from Core Hub, a SQL warehouse, and a service principal with an OAuth secret (recommended) or a personal access token. Grants for that identity:

• CAN USE on the SQL warehouse
• USE CATALOG on the catalog and USE SCHEMA on the target schemas
• SELECT and MODIFY on the target tables
• CREATE TABLE on the schema when Gluesync creates the tables
• READ VOLUME and WRITE VOLUME on the staging volume for bulk snapshots, or CREATE SCHEMA and CREATE VOLUME on the catalog for the default one

In the Control Plane, add Databricks as the target of a pipeline:

• Workspace URL: https://<workspace-host>, for example https://adb-1234567890123456.7.azuredatabricks.net or https://dbc-1a2b3c4d-5e6f.cloud.databricks.com
• Port: 443
• Authentication mode: SERVICE_PRINCIPAL (default, OAuth machine-to-machine) or PAT
• Service principal client ID (empty with a PAT) and Client secret or personal access token
• SQL warehouse ID: the last part of the warehouse HTTP path, /sql/1.0/warehouses/<id>
• Unity Catalog catalog: the catalog that holds the target schemas

The connection check reads the warehouse state without starting it, so a stopped warehouse passes. Serverless warehouses start on the first statement.

Query Studio and data comparison

The agent is SQL-capable. Query Studio runs queries on the configured warehouse and catalog with the same identity as the pipeline:

• One statement per query, through the Statement Execution API. Read-only mode accepts SELECT, WITH, SHOW, DESCRIBE and EXPLAIN.
• Backtick names and backslash-escaped strings, as Databricks SQL reads them.
• Explain shows the EXPLAIN plan, the DDL viewer shows SHOW CREATE TABLE.
• A cancelled query is cancelled on the warehouse too.
• Data comparison reads timestamps with every microsecond and checks the Delta tables against the source. The result grid shows milliseconds; cast a column to STRING to see the microseconds.

Where it fits

Operational databases on one side, the lakehouse on the other. PostgreSQL, Oracle, SQL Server, MySQL, MongoDB and the rest of the Gluesync catalog land their snapshot and CDC changes in Delta tables governed by Unity Catalog, kept equal to the source key by key. The full list of sources and targets is on the Gluesync page.

Next steps

1. Read the Databricks target setup and set up the grants.
2. Deploy Gluesync 2.3 and add Databricks as the target of a pipeline in Core Hub.
3. Turn on bulk snapshots for your largest tables, then watch CDC keep each Delta table current.

Want to see it on your workspace? Request a free trial or book a demo. Questions on a concrete build: I am around.