Gluesync Core Hub · Built into every pipeline
Reshape data while it moves. Pick a pre-built field function per column, write a Java UDF when a record needs real logic, publish reusable Custom Field Functions at pipeline level, keep only the rows that match your filters, and decide how documents and messages are keyed on NoSQL, event-streaming and object-store targets. Spark helps you write and review all of it.
Why transform in transit
01 / Shape mismatch
Oracle dates, SQL Server decimals and MySQL booleans do not land cleanly in Couchbase documents, Kafka messages or BigQuery columns. Fixing them after the fact means another job, another schedule and another copy.
02 / Compliance at the edge
If a target team only needs the last four digits of a card, the full number should never reach them. Masking, filtering and anonymization belong in the pipeline, not in a downstream view somebody may forget to apply.
03 / The platform outcome
Gluesync applies the same field functions, UDFs, filters and document keys to the initial snapshot and to every change captured afterwards, from the Control Plane, with no extra component to run.
How it works
Map
In the Fields Editor, map source to target columns and attach a pre-built field function where the shape has to change: casts, date patterns, trims, constants, technical fields and masking. Parameters come with ready-to-use defaults.
Extend
When a record needs business logic, add a Java UDF to the entity. When several pipelines need the same custom conversion, publish it once as a Custom Field Function and bind it to target columns like any catalog entry.
Scope
Add inclusive filter clauses that decide which rows are copied on snapshot and kept in sync on CDC, and compose the document or message key your NoSQL, Kafka or object-store target expects.
Capabilities
Every capability on this page is configured in the Control Plane and stored with the pipeline, so Bootstrapper exports and REST calls carry the same configuration. The simulations below mirror the Core Hub screens where each one lives.
Field functions
Field functions are applied in transit to a mapped column. Regular transformations change business data on its way to the target; technical field functions populate target-only columns with timestamps or constants, or override the incoming value, which is how data masking and anonymization work without code.
Data pipelines / P-MSSQL-SNOW Customer 360 / dbo.orders / Fields
| Target | Field function |
|---|---|
| ORDER_TSorder_ts · DATETIME2(6) → VARCHAR(32) | Convert Date and Time to String |
| IBANiban · NVARCHAR(34) → VARCHAR(34) | Direct mapping |
| IS_ACTIVEis_active · BIT → CHAR(1) | Direct mapping |
| SYNCED_ATTarget only → TIMESTAMP_TZ | — |
Pick the mask function, keep the defaults or set the mask character, and the target only ever receives the redacted value. Discovery of which columns need it is covered on the PII discovery and masking page.
User Defined Functions
A UDF receives each insert, update and delete after it is read from the source and before it is written to the target. It returns the record to write, or null to skip it, and can raise an error that goes through the pipeline's error handling. Code is checked and compiled in Core Hub before it can run.
Data pipelines / P-MSSQL-SNOW Customer 360 / dbo.orders / UDF
public class UDF_orders { public Pair<MappingFunctionOperation, Map<String, Object>> onChange( Map<String, Object> newValues, Map<String, Object> oldValues, MappingFunctionOperation operation, Logger logger) { Map<String, Object> row = operation == MappingFunctionOperation.Delete ? oldValues : newValues; if ("TEST".equals(row.get("CHANNEL"))) return null; // skip the operation if (operation == MappingFunctionOperation.Delete) { Map<String, Object> soft = new HashMap<>(oldValues); soft.put("IS_DELETED", true); soft.put("DELETED_AT", OffsetDateTime.now(ZoneOffset.UTC).toString()); return new Pair<>(MappingFunctionOperation.Update, soft); } return new Pair<>(operation, newValues); }}Custom Field Functions
When the catalog does not have the conversion you need, or a target column has to be built from several source columns at once, write it once as a Custom Field Function. It belongs to the pipeline, shows up in the Fields Editor next to the pre-built catalog, and can be bound to as many target columns and entities as you like.
Data pipelines / P-ORA-PG Plant data / Custom Field Functions
package com.molo17.gluesync.collections;import java.time.*;import com.molo17.gluesync.commons.function.GsFunction;public class Format { @GsFunction(name = "compose_timestamp", description = "A date column and a time column into one timestamp") public static OffsetDateTime composeTimestamp(LocalDate date, LocalTime time, String zone) { if (date == null) return null; LocalTime t = time == null ? LocalTime.MIDNIGHT : time; return date.atTime(t).atZone(ZoneId.of(zone)).toOffsetDateTime(); }}Ask Spark to explain or improve these custom functions… · grounded on this pipeline's sources, columns and compiled signatures
Filters
Filters decide which rows belong on the target. A row is replicated only if it matches all clauses. During the snapshot only matching rows are copied; during CDC a row that stops matching is converted into a DELETE on the target, so the target always reflects the filter. Filters are evaluated in Core Hub after the read, so they add no load on the source.
Data pipelines / P-MSSQL-SNOW Customer 360 / dbo.orders / Settings
Applied to CDC changes and to the initial snapshot. Clauses behave like a SQL WHERE and a row is kept only when all of them match.
Right before a snapshot, Gluesync runs DELETE FROM ORDERS WHERE REGION != 'EU' on the target. Snapshot only; CDC is untouched.
EUSelective deletion ahead of snapshot removes rows from the target table immediately before a snapshot starts, with the same filter controls. It is a destructive operation on the target and is documented as such.
Custom document keys
Document stores, Kafka topics and object-store targets do not have a primary key the way a table does. Gluesync lets you build the key each record is written with: choose the source fields, order them, pick a separator, add a prefix or a suffix, and preview the result before saving. The same key is used for inserts, updates and deletes, so a change on the source always finds its document or message.
Data pipelines / P-PG-CB Retail / sales.transactions / Settings
tx_ - region - store_id - transaction_id - _gluesync
Sample · tx_-USA-NY-1001-998877-_gluesync
Spark across all of it
Spark reads the pipeline the same way the Control Plane does. In the UDF and Custom Field Functions editors it explains or improves the code in front of you. In AI Studio it can read an entity's mapping, filters and keys, propose a change such as a Mask Email field function on a target column, and apply it only after you approve the plan.
AI Studio / Spark / Pipeline helper
dbo.customers.email to CUSTOMERS.EMAIL with a direct mapping and carries a PII:email tag. I propose Mask Email with *: john.doe@example.com → j*******@example.com. This changes the entity, so it needs your approval.Explore the platform
Transformation is one part of the pipeline. The pages below cover what it connects to: PII discovery so you know what to mask, Spark and the MCP server that drive the helpers, and Query Studio to check the result.
Columns classified as PII carry a tag in the Fields Editor, UDF, filter and document key screens, so you know what to mask before it leaves the pipeline.
Explore PII discovery & masking →The agents behind the Spark helpers in the UDF and Custom Field Functions editors, and the pipeline helper in AI Studio.
Explore Spark Agents →The same field functions, mapping compiler and entity upserts, exposed as tools to Spark and to your own AI clients.
Explore Core Hub MCP Server →Check source values before you write a filter, and read the target after the transformation lands, from the same Control Plane.
Explore Query Studio →Resources
Technical guides, API reference, deployment tutorials.
Read documentation ↗What we're building next. Submit feature requests.
View roadmap ↗Access technical support and operational assistance.
Get support →Real-time platform availability and incident history.
Check status ↗Request a free trial or talk with the MOLO17 team about field functions, UDFs, filters and document keys on your own sources and targets.
