SOLUTION · REPLICATE TO ELASTICSEARCH AND OPENSEARCH

Real-time data replication to Elasticsearch and OpenSearch from your operational databases

Committed changes from your systems of record reach your search indexes as they happen, written through the bulk API with the source key as the document ID, instead of waiting for the next reindex job.

Gluesync by MOLO17 captures changes from Oracle, SQL Server, PostgreSQL, MySQL, MongoDB, and other heterogeneous sources with a dedicated agent per database. One target agent serves both Elasticsearch and OpenSearch over the standard HTTP JSON API, so there is no vendor SDK to install, and it detects which one it is talking to. Core Hub, the Gluesync control plane, runs snapshots and CDC for every pipeline from one web UI and REST API.

Start a Gluesync trial Talk to us

VENDOR COMPATIBILITY

Battle-tested on every Elasticsearch and OpenSearch you deliver to

The same Gluesync Elasticsearch and OpenSearch agent is tested against each vendor offering below, self-managed or fully managed, so delivery behaves the same wherever Elasticsearch and OpenSearch runs.

Elasticsearch Target Tested
OpenSearch Target Tested

WHO THIS IS FOR

Teams that need search results to match the system of record

Search and application engineers who serve queries from Elasticsearch or OpenSearch and need each document keyed like the source row, with inserts, updates, and deletes reflected in the index
Platform leads feeding Elasticsearch or OpenSearch clusters from Oracle, SQL Server, PostgreSQL, MySQL, or MongoDB through one Core Hub, backed by best-in-class enterprise support, rated 4.9/5 by customers
Architects who want indexes created with an explicit mapping derived from the source column types, so field types are decided by design and not guessed from the first document
Engineers replacing polling jobs, Logstash pipelines, or Kafka chains that exist only to fill an index, who want every source delivered by one product with snapshots and monitoring built in

THE PROBLEM

Search indexes refreshed on a schedule return yesterday's data

Most search indexes are fed by a batch: a job queries the source on a timer, reshapes the rows, and bulk-loads them. Between runs, users search a copy that trails the business, deleted rows linger as stale hits, every poll adds read load to production tables, and each source brings its own script. Teams looking for Elasticsearch CDC or real-time OpenSearch ingestion usually weigh Logstash with a JDBC input, Kafka Connect with an Elasticsearch sink, Debezium, or scripts they write and run themselves.

Gluesync addresses that with per-agent CDC into Elasticsearch and OpenSearch. A source agent reads each database's native change log, journal, or change stream; Core Hub routes the changes; the target agent applies them to indexes as they arrive. PostgreSQL to Elasticsearch, Oracle to OpenSearch, and MongoDB to Elasticsearch share the same snapshot, index handling, and monitoring.

HOW IT WORKS

How Gluesync writes to Elasticsearch and OpenSearch

One bulk path for snapshots and CDC, keyed by document ID

The agent is a target agent: it receives the snapshot and the change stream from any Gluesync source agent and writes both through POST /_bulk. The Gluesync document ID, built from the row's key, becomes the document _id, so a change always lands on the same document.

Inserts: indexed with the INDEX action by default, which writes the document under its _id, or with CREATE when you want an existing document to be reported as a duplicate key and Core Hub's insert-conflict strategies applied.
Updates: sent as partial updates with doc_as_upsert, so a document that is missing is created. A column set to null clears its field.
Deletes: sent as a bulk delete by _id, and a document that is already gone is not an error.
Acknowledged before confirmed: the transaction is confirmed only after the cluster acknowledges the bulk request. A replay after a restart writes the same document IDs again, so the index converges to the source. Elasticsearch agent overview ↗

Snapshot first, then change events into the same index

Seed, then stream: each entity is loaded from its full table through the bulk API, then the same index receives the change stream from the source agent in real time.
Truncate and reload: a truncate, or the deletion that precedes a snapshot, runs _delete_by_query with match_all and keeps the index and its mapping. You can also choose to skip it and leave the documents in place.
Batch size: documents per bulk request and bytes per request are configurable per agent, and the document count can be overridden per entity.
Scheduled reloads: the Chronos scheduler runs a snapshot, or a snapshot followed by CDC, on a cadence. See schedules and events.

Indexes, names, and explicit mappings

When an index does not exist, the agent creates it, with an explicit mapping generated from the source column types. An index that already exists is left as it is, mapping included.

Index names: by default prefix-scope-collection. A template with {prefix}, {scope}, and {collection} placeholders reshapes the name for every entity, and a per-entity override names one index exactly. Names are normalized to what the cluster accepts: lowercased, with reserved characters replaced by underscores.
Strings and numbers: strings map to keyword, or text when you pick full-text search for that field. Integers map to short, integer, and long, floats to float and double, and decimals to double, or scaled_float or keyword by choice.
Dates, booleans, and binaries: dates are written in ISO-8601, booleans map to boolean, and byte arrays map to binary as Base64. Times without a date map to keyword.
Maps and arrays: maps map to object, or nested, flattened, or dynamic mapping by choice. Arrays are mapped as keyword, object, or nested when you select one, and otherwise follow dynamic mapping.
Ingest pipelines: name an ingest pipeline on an entity and its bulk requests run through it, so enrichment and analysis stay where the cluster already does them.

Setup: cluster address, authentication, and TLS

The agent needs network reach to the cluster and an account that can bulk-index, update, and delete documents, run _delete_by_query when truncates are applied, and create indexes. Every field is listed in the Elasticsearch target setup guide ↗.

Create a user, an API key, or a token with the index privileges above on the indexes the pipeline writes.
Choose the authentication: Basic with a username and password, an API key, a Bearer token, or none for a closed network.
Enter the host and the port, which defaults to 9200, turn on TLS for anything beyond a private network, and add the other cluster nodes as additional nodes. Set a path prefix when a reverse proxy sits in front of the cluster.
Leave the server flavor on automatic, which reads it from the cluster's root response, or pin it to Elasticsearch or OpenSearch.
Map each entity to an index, or let the index name pattern create the indexes from the source definition.

Architecture around Core Hub

Lightweight agents sit close to each source, and Core Hub orchestrates them through its web UI and REST APIs. A pipeline groups the source agents, the Elasticsearch agent, and the entities they replicate, and Core Hub and the agents deploy with Docker, Docker Compose, or Kubernetes, on-premises or in any cloud. See CDC streaming and snapshot tasks ↗.

Because the search index is one target among many on the same Core Hub, a source table can feed an index for search and land in a database or warehouse in another pipeline, from the same capture agent and with the same monitoring.

Explore the general CDC streaming architecture →

WRITE OPTIONS

The Elasticsearch target agent: one HTTP JSON agent for both products

Elasticsearch and OpenSearch share one Gluesync target agent. It detects the flavor from the cluster, or you pin it, and the write path is the same for both.

Agent	Write technique	Versions	Best for
Elasticsearch agent ↗	HTTP JSON client writing snapshots and CDC through `POST /_bulk`, with `doc_as_upsert` partial updates, deletes by `_id`, and explicit index mappings generated from the source types	Elasticsearch and OpenSearch clusters that expose the standard HTTP JSON API	Teams that serve search from Elasticsearch or OpenSearch and want indexes that follow operational databases. Pick it when documents should be keyed by the source key, with mappings generated for you and Basic, API key, or Bearer authentication.

SOURCES AND TOPOLOGIES

Feed your search indexes from the databases you already run

Any Gluesync source agent can feed Elasticsearch and OpenSearch, each with its own native capture technique. Open the integrations finder with Elasticsearch pre-selected to see every source you can pair with it.

One Core Hub runs Oracle to Elasticsearch, PostgreSQL to OpenSearch, and MongoDB to Elasticsearch side by side, each pipeline with its own snapshot and monitoring. Gluesync keeps pace with your change volume at any scale, and MOLO17 Professional Services can plan index design and mappings with your team.

Oracle to Elasticsearch from the redo logs through LogMiner or XStream: see Oracle CDC
SQL Server to OpenSearch through Change Data Capture or Change Tracking: see SQL Server CDC
PostgreSQL and MySQL to Elasticsearch from the WAL and the binlog: see PostgreSQL CDC and MySQL CDC
MongoDB to Elasticsearch from change streams: see MongoDB CDC

FAIR, HIGH-LEVEL COMPARISON

Where Gluesync fits among search indexing approaches

Approach	What buyers usually get	Where Gluesync fits
Logstash with a JDBC input	Scheduled queries against the source that load results into an index; your team owns the polling statements, the tracking column, and how deletes are handled	Log-based capture with no polling queries against production, and deletes applied by document ID; read log-based CDC explained
Kafka Connect with an Elasticsearch sink	Events flow through Kafka and a sink connector into the index; your team runs the brokers, Connect, and the connector	Changes go straight from each source agent to the index, with no Kafka cluster in the path. Kafka stays available as another target; see Replicate to Kafka
Debezium	Open-source log-based capture, usually paired with Kafka and a sink; offsets, schema history, and restarts are yours to operate	A native capture agent per database under one Core Hub, so operations sit in one control plane; read the Debezium alternative comparison
Elastic connectors, Beats, and OpenSearch Data Prepper	Ingestion tools from the Elastic and OpenSearch projects, built around their own ecosystems and source lists	One control plane for every source and target, so the same capture also feeds databases, warehouses, and Kafka, with MOLO17 enterprise support
Custom indexing scripts	Full control; your team owns the extract queries, the bulk code, the mappings, and the load each run puts on the source database	Snapshot seeding, continuous CDC, and generated mappings from one console; see migrating to Gluesync

Elasticsearch and OpenSearch replication research and implementation detail

What does replicating to Elasticsearch or OpenSearch with Gluesync involve?

A source agent captures committed changes from your database through its native change mechanism, Core Hub routes them, and the Elasticsearch agent writes them to indexes through the bulk API. Each table is seeded with a snapshot first and then follows the change stream.

Is OpenSearch a separate agent?

No. Elasticsearch and OpenSearch share one agent and one HTTP JSON client. The agent reads the flavor from the cluster's root response, or you can pin it to Elasticsearch or OpenSearch in the connection settings.

How are inserts, updates, and deletes written?

Everything goes through the bulk API. Inserts are indexed under the document ID built from the source key, updates are partial updates with doc_as_upsert, and deletes remove the document by ID. A missing document on delete is not an error.

Does Gluesync create the indexes and mappings?

Yes. When an index does not exist, the agent creates it with an explicit mapping generated from the source column types, and you can choose alternatives such as text for a string field or nested for a map. An existing index is left untouched, mapping included.

How are index names chosen?

By default from a prefix, the entity's namespace, and its collection. A name template or a per-entity override replaces that, and the name is normalized to what the cluster accepts, for example lowercased with reserved characters replaced.

What happens on a truncate or before a snapshot?

By default the agent runs a delete by query that matches every document and keeps the index and its mapping. You can choose to skip it and leave the documents in place.

How does the agent authenticate and connect?

With Basic authentication, an API key, a Bearer token, or none for a closed network, optionally over TLS. It supports additional cluster nodes and a path prefix for a reverse proxy in front of the cluster.

Can documents run through an ingest pipeline?

Yes. Name an ingest pipeline on an entity and its bulk requests are sent through it.

Which clusters are supported?

Elasticsearch and OpenSearch clusters that expose the standard HTTP JSON API.

REPLICATE TO A TARGET