New in Gluesync 2.3: the Microsoft Fabric Open Mirroring target agent

Gluesync 2.3 adds a NEW Microsoft Fabric Open Mirroring target agent. It lands snapshots and CDC changes as Parquet files in the landing zone of a Fabric mirrored database, and Fabric turns them into Delta tables, ready for the SQL analytics endpoint, semantic models and Spark. Gluesync writes no Delta files itself: it writes the change files and the small metadata files Open Mirroring expects, and Fabric does the rest.

Microsoft Fabric Open Mirroring is a target in Gluesync. Any supported source feeds it through a pipeline in Core Hub: the snapshot first, then the source agent's CDC stream. Core Hub talks to OneLake over HTTPS through the Azure Data Lake Storage Gen2 API and signs in with a Microsoft Entra service principal.

Docs: Microsoft Fabric Open Mirroring agent · Target setup · Changelog.

Why Open Mirroring

Open Mirroring is the way Microsoft Fabric lets any application replicate data into a mirrored database. The application writes Parquet files into a landing zone folder in OneLake, Fabric reads them and keeps the Delta tables of the mirrored database up to date. That makes the contract between Gluesync and Fabric a folder of files, and a strict one: Fabric reads a file as soon as it appears under its final name and never accepts it changing.

So the agent is built around that contract:

• Every row carries a row marker (insert, update or delete), so the mirrored table stays equal to the source
• Files are published atomically and in sequence, so Fabric never reads a half-written file
• A change is confirmed to the source only once the file that holds it is on OneLake

When to use it

When the data your teams need in Fabric lives in operational databases, and you want it in a mirrored database with the changes following in near real time, instead of a copy refreshed by batch jobs. The mirrored tables appear in the SQL analytics endpoint: a table landed in the root of the landing zone shows in the dbo schema, a table landed in a <schema>.schema folder shows in that schema.

What is written

For a table Orders of schema sales, the agent writes:

<mirrored database>/Files/LandingZone/_partnerEvents.json
<mirrored database>/Files/LandingZone/sales.schema/Orders/_metadata.json
<mirrored database>/Files/LandingZone/sales.schema/Orders/00000000000000000001.parquet
<mirrored database>/Files/LandingZone/sales.schema/Orders/00000000000000000002.parquet

• Parquet files have a 20-digit name, strictly increasing and without gaps. Gluesync does not store the sequence: after a start it lists the table folder and continues after the highest name there. A retried upload takes the same name and leaves no gap.
• _metadata.json holds the primary key columns, written once per table before the first file. Fabric does not allow the key columns to change, so an existing file is never rewritten.
• _partnerEvents.json sits at the root of the landing zone with partner name MOLO17 and source type Gluesync. It can be switched off.

Atomic publication

Each file is uploaded under an underscore-prefixed temporary name (_00000000000000000042.parquet), which Open Mirroring ignores. It is closed with a single flush and renamed to its final name in one operation that refuses to replace an existing file. On a retry the agent first looks at the final name: a file of the right size there was published by an earlier attempt and is kept, a different one is reported as a conflict and never overwritten.

How changes become rows in Fabric

Change files end with an __rowMarker__ column:

Value	Meaning
0	Insert
1	Update, with the full row as the source delivers it
2	Delete, carrying the key columns only
4	Upsert, used for inserts and updates when Use the upsert row marker is on

• Snapshot files have no __rowMarker__ column: Fabric loads them as plain inserts, as Open Mirroring recommends for the initial load.
• Key changes: an update that changes a key column is written as a delete of the old key followed by an insert of the new row.
• Tables without a primary key: Fabric cannot apply updates or deletes to them, so Gluesync writes no _metadata.json, appends inserts and updates as new rows and skips deletes with a warning.
• Truncate: Open Mirroring has none. A truncate request is skipped with a warning, and a new snapshot does not clear the table folder: delete the table folder in the landing zone to start a table over.

Change files and throughput

The changes of a table are collected and written as one Parquet file when the first of three limits is reached: 1 second, 100,000 rows or 16 MB by default. A closed file uploads while the next changes are already being collected. Because a change is confirmed only once its file is on OneLake, a restart or a crash loses nothing: the changes of a file not yet published are read and applied again, and inserts and updates of a collected file carry the upsert marker (4), so applying them twice leaves the table as it was.

The window is a trade-off. With the default of 1 second, the number of files stays at about one per table per second whatever the load. Every file costs Fabric a Delta commit, so fewer, larger files mean less work for the replication. Setting the window to 0 writes one file per commit.

Snapshots have no separate bulk mode: each snapshot streams its pages into one Parquet file per table, closed and published at Maximum row count in Parquet file (default 1,000,000 rows) or when the snapshot ends. Files of different tables upload in parallel, files of one table in sequence order.

Parquet defaults: ZSTD compression (SNAPPY, GZIP and UNCOMPRESSED are also available, and Fabric reads all of them), 10,000-row row groups, 8 MB upload blocks with 4 blocks in flight per file.

As reference figures measured in the MOLO17 lab against a real landing zone, not guarantees: a 1,000,000-row snapshot ran at about 155,000 rows per second on a narrow table and 41,000 rows per second on a 50-column table, and CDC reached up to 12,000 rows per second from a PostgreSQL source with large commits. Your rate depends on your network, the width of the rows and the settings above.

Data types and schema evolution

Values are written as Parquet columns with microsecond timestamps. Decimals keep precision and scale, offset times and timestamps are written in UTC, and arrays and maps travel as JSON strings, as Open Mirroring asks for complex values.

Adding columns works: a column that appears in later Parquet files is added to the Delta table by Fabric. Dropping, renaming or retyping a column, renaming a table or changing its key columns needs the table folder deleted and recreated with a new snapshot, so that Fabric rebuilds the Delta table.

Connection setup

Prerequisites:

• A Fabric workspace with a mirrored database that uses Open Mirroring, reachable over HTTPS from Core Hub (onelake.dfs.fabric.microsoft.com)
• A Microsoft Entra app registration with a client secret, added to the workspace with the Contributor role or higher
• The Fabric tenant setting that lets service principals use Fabric APIs, enabled for that app registration

In Core Hub, add Microsoft Fabric Open Mirroring as the target of a pipeline:

• Client ID, Client Secret and Tenant ID of the app registration
• Landing zone URL, pasted as shown on the Home page of the mirrored database (https://onelake.dfs.fabric.microsoft.com/<workspace id>/<mirrored database id>/Files/LandingZone). The abfss:// form and the blob host are understood too
• Optionally a custom path prefix under the landing zone

On setup Gluesync signs in to Entra, asks for a OneLake token and checks that the landing zone folder exists. A failure comes back as one error that names what to change, such as a wrong client secret or a service principal with no access to the workspace.

Monitoring

In Fabric, Monitor replication on the mirrored database shows the status of each table, the rows replicated and the time of the last update, and Fabric writes its own status files under Files/Monitoring. In the Gluesync logs, every upload reports its path, duration and the uploads active and queued.

Where it fits

Analytics in Microsoft Fabric on live operational data. The sources in the Gluesync catalog land their snapshot and CDC changes in a Fabric mirrored database, kept current row by row. The full list of sources and targets is on the Gluesync page.

Next steps

1. Read the Microsoft Fabric Open Mirroring target setup.
2. Deploy Gluesync 2.3.0.3 and add Microsoft Fabric Open Mirroring as the target of a pipeline in Core Hub.
3. Paste the landing zone URL of your mirrored database and watch the tables appear in Fabric.

Want to see it on your Fabric workspace? Request a free trial or book a demo.