Canonical: https://molo17.com/solutions/replicate-to-elasticsearch/ Markdown mirror: https://molo17.com/solutions/replicate-to-elasticsearch/index.md Title: Replicate to Elasticsearch and OpenSearch | MOLO17 Description: Real-time replication to Elasticsearch and OpenSearch indexes from Oracle, SQL Server, PostgreSQL, MySQL, and MongoDB, with bulk snapshots and CDC. Try it. [Solutions](/solutions/) Replicate to Elasticsearch and OpenSearch SOLUTION · REPLICATE TO ELASTICSEARCH AND OPENSEARCH # Real-time data replication to Elasticsearch and OpenSearch from your operational databases **Committed changes from your systems of record reach your search indexes as they happen, written through the bulk API with the source key as the document ID, instead of waiting for the next reindex job.** Gluesync by MOLO17 captures changes from Oracle, SQL Server, PostgreSQL, MySQL, MongoDB, and other [heterogeneous sources](/integrations/?target=Elasticsearch#integration-finder) with a dedicated agent per database. One target agent serves both Elasticsearch and OpenSearch over the standard HTTP JSON API, so there is no vendor SDK to install, and it detects which one it is talking to. Core Hub, the Gluesync control plane, runs snapshots and CDC for every pipeline from one web UI and REST API. [Start a Gluesync trial](/get-gluesync/) [Talk to us](/contacts/) VENDOR COMPATIBILITY ## Battle-tested on every Elasticsearch and OpenSearch you deliver to The same Gluesync Elasticsearch and OpenSearch agent is tested against each vendor offering below, self-managed or fully managed, so delivery behaves the same wherever Elasticsearch and OpenSearch runs. - **Elasticsearch** Target Tested - **OpenSearch** Target Tested WHO THIS IS FOR ## Teams that need search results to match the system of record - Search and application engineers who serve queries from Elasticsearch or OpenSearch and need each document keyed like the source row, with inserts, updates, and deletes reflected in the index - Platform leads feeding Elasticsearch or OpenSearch clusters from Oracle, SQL Server, PostgreSQL, MySQL, or MongoDB through one Core Hub, backed by [best-in-class enterprise support, rated 4.9/5 by customers](/support/#customer-ratings) - Architects who want indexes created with an explicit mapping derived from the source column types, so field types are decided by design and not guessed from the first document - Engineers replacing polling jobs, Logstash pipelines, or Kafka chains that exist only to fill an index, who want [every source](/integrations/?target=Elasticsearch#integration-finder) delivered by one product with snapshots and monitoring built in THE PROBLEM ## Search indexes refreshed on a schedule return yesterday's data Most search indexes are fed by a batch: a job queries the source on a timer, reshapes the rows, and bulk-loads them. Between runs, users search a copy that trails the business, deleted rows linger as stale hits, every poll adds read load to production tables, and each source brings its own script. Teams looking for Elasticsearch CDC or real-time OpenSearch ingestion usually weigh Logstash with a JDBC input, Kafka Connect with an Elasticsearch sink, Debezium, or scripts they write and run themselves. Gluesync addresses that with **per-agent CDC into Elasticsearch and OpenSearch**. A source agent reads each database's native change log, journal, or change stream; Core Hub routes the changes; the target agent applies them to indexes as they arrive. PostgreSQL to Elasticsearch, Oracle to OpenSearch, and MongoDB to Elasticsearch share the same snapshot, index handling, and monitoring. HOW IT WORKS ## How Gluesync writes to Elasticsearch and OpenSearch ### One bulk path for snapshots and CDC, keyed by document ID The agent is a target agent: it receives the snapshot and the change stream from any Gluesync source agent and writes both through `POST /_bulk`. The Gluesync document ID, built from the row's key, becomes the document `_id`, so a change always lands on the same document. - **Inserts:** indexed with the `INDEX` action by default, which writes the document under its `_id`, or with `CREATE` when you want an existing document to be reported as a duplicate key and Core Hub's insert-conflict strategies applied. - **Updates:** sent as partial updates with `doc_as_upsert`, so a document that is missing is created. A column set to null clears its field. - **Deletes:** sent as a bulk delete by `_id`, and a document that is already gone is not an error. - **Acknowledged before confirmed:** the transaction is confirmed only after the cluster acknowledges the bulk request. A replay after a restart writes the same document IDs again, so the index converges to the source. [Elasticsearch agent overview ↗](https://docs.molo17.com/gluesync/latest/agents/elasticsearch-intro.html) ### Snapshot first, then change events into the same index - **Seed, then stream:** each entity is loaded from its full table through the bulk API, then the same index receives the change stream from the source agent in real time. - **Truncate and reload:** a truncate, or the deletion that precedes a snapshot, runs `_delete_by_query` with `match_all` and keeps the index and its mapping. You can also choose to skip it and leave the documents in place. - **Batch size:** documents per bulk request and bytes per request are configurable per agent, and the document count can be overridden per entity. - **Scheduled reloads:** the Chronos scheduler runs a snapshot, or a snapshot followed by CDC, on a cadence. See [schedules and events](/schedules-and-events/). ### Indexes, names, and explicit mappings When an index does not exist, the agent creates it, with an explicit mapping generated from the source column types. An index that already exists is left as it is, mapping included. - **Index names:** by default `prefix-scope-collection`. A template with `{prefix}`, `{scope}`, and `{collection}` placeholders reshapes the name for every entity, and a per-entity override names one index exactly. Names are normalized to what the cluster accepts: lowercased, with reserved characters replaced by underscores. - **Strings and numbers:** strings map to `keyword`, or `text` when you pick full-text search for that field. Integers map to `short`, `integer`, and `long`, floats to `float` and `double`, and decimals to `double`, or `scaled_float` or `keyword` by choice. - **Dates, booleans, and binaries:** dates are written in ISO-8601, booleans map to `boolean`, and byte arrays map to `binary` as Base64. Times without a date map to `keyword`. - **Maps and arrays:** maps map to `object`, or `nested`, `flattened`, or dynamic mapping by choice. Arrays are mapped as `keyword`, `object`, or `nested` when you select one, and otherwise follow dynamic mapping. - **Ingest pipelines:** name an ingest pipeline on an entity and its bulk requests run through it, so enrichment and analysis stay where the cluster already does them. ### Setup: cluster address, authentication, and TLS The agent needs network reach to the cluster and an account that can bulk-index, update, and delete documents, run `_delete_by_query` when truncates are applied, and create indexes. Every field is listed in the [Elasticsearch target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/elasticsearch-target.html). 1. Create a user, an API key, or a token with the index privileges above on the indexes the pipeline writes. 2. Choose the authentication: Basic with a username and password, an API key, a Bearer token, or none for a closed network. 3. Enter the host and the port, which defaults to `9200`, turn on TLS for anything beyond a private network, and add the other cluster nodes as additional nodes. Set a path prefix when a reverse proxy sits in front of the cluster. 4. Leave the server flavor on automatic, which reads it from the cluster's root response, or pin it to Elasticsearch or OpenSearch. 5. Map each entity to an index, or let the index name pattern create the indexes from the source definition. ### Architecture around Core Hub Lightweight agents sit close to each source, and Core Hub orchestrates them through its web UI and REST APIs. A pipeline groups the source agents, the Elasticsearch agent, and the entities they replicate, and Core Hub and the agents deploy with Docker, Docker Compose, or Kubernetes, on-premises or in any cloud. See [CDC streaming](/solutions/cdc-streaming/) and [snapshot tasks ↗](https://docs.molo17.com/gluesync/latest/core-hub/snapshot-tasks.html). Because the search index is one target among many on the same Core Hub, a source table can feed an index for search and land in a database or warehouse in another pipeline, from the same capture agent and with the same monitoring. [Explore the general CDC streaming architecture →](/solutions/cdc-streaming/) WRITE OPTIONS ## The Elasticsearch target agent: one HTTP JSON agent for both products Elasticsearch and OpenSearch share one Gluesync target agent. It detects the flavor from the cluster, or you pin it, and the write path is the same for both. | Agent | Write technique | Versions | Best for | | --- | --- | --- | --- | | [Elasticsearch agent ↗](https://docs.molo17.com/gluesync/latest/agents/elasticsearch-intro.html) | HTTP JSON client writing snapshots and CDC through `POST /_bulk`, with `doc_as_upsert` partial updates, deletes by `_id`, and explicit index mappings generated from the source types | Elasticsearch and OpenSearch clusters that expose the standard HTTP JSON API | Teams that serve search from Elasticsearch or OpenSearch and want indexes that follow operational databases. Pick it when documents should be keyed by the source key, with mappings generated for you and Basic, API key, or Bearer authentication. | SOURCES AND TOPOLOGIES ## Feed your search indexes from the databases you already run Any Gluesync source agent can feed Elasticsearch and OpenSearch, each with its own native capture technique. Open the [integrations finder with Elasticsearch pre-selected](/integrations/?target=Elasticsearch#integration-finder) to see every source you can pair with it. One Core Hub runs Oracle to Elasticsearch, PostgreSQL to OpenSearch, and MongoDB to Elasticsearch side by side, each pipeline with its own snapshot and monitoring. Gluesync keeps pace with your change volume at any scale, and [MOLO17 Professional Services](/solutions/professional-services/) can plan index design and mappings with your team. - Oracle to Elasticsearch from the redo logs through LogMiner or XStream: see [Oracle CDC](/solutions/oracle-cdc/) - SQL Server to OpenSearch through Change Data Capture or Change Tracking: see [SQL Server CDC](/solutions/sql-server-cdc/) - PostgreSQL and MySQL to Elasticsearch from the WAL and the binlog: see [PostgreSQL CDC](/solutions/postgresql-cdc/) and [MySQL CDC](/solutions/mysql-cdc/) - MongoDB to Elasticsearch from change streams: see [MongoDB CDC](/solutions/mongodb-cdc/) FAIR, HIGH-LEVEL COMPARISON ## Where Gluesync fits among search indexing approaches | Approach | What buyers usually get | Where Gluesync fits | | --- | --- | --- | | Logstash with a JDBC input | Scheduled queries against the source that load results into an index; your team owns the polling statements, the tracking column, and how deletes are handled | Log-based capture with no polling queries against production, and deletes applied by document ID; read [log-based CDC explained](/blog/real-time-replication-log-based-cdc/) | | Kafka Connect with an Elasticsearch sink | Events flow through Kafka and a sink connector into the index; your team runs the brokers, Connect, and the connector | Changes go straight from each source agent to the index, with no Kafka cluster in the path. Kafka stays available as another target; see [Replicate to Kafka](/solutions/replicate-to-kafka/) | | Debezium | Open-source log-based capture, usually paired with Kafka and a sink; offsets, schema history, and restarts are yours to operate | A native capture agent per database under one Core Hub, so operations sit in one control plane; read the [Debezium alternative](/solutions/debezium-alternative/) comparison | | Elastic connectors, Beats, and OpenSearch Data Prepper | Ingestion tools from the Elastic and OpenSearch projects, built around their own ecosystems and source lists | One control plane for every source and target, so the same capture also feeds databases, warehouses, and Kafka, with MOLO17 enterprise support | | Custom indexing scripts | Full control; your team owns the extract queries, the bulk code, the mappings, and the load each run puts on the source database | Snapshot seeding, continuous CDC, and generated mappings from one console; see [migrating to Gluesync](/migrate-to-gluesync/) | RELATED CONTENT ## Elasticsearch and OpenSearch replication research and implementation detail - [CDC streaming without source overhead](/solutions/cdc-streaming/) - [Replicate to Kafka](/solutions/replicate-to-kafka/) - [Batch ETL vs real-time data replication: how to choose](/blog/batch-etl-vs-real-time-data-replication/) - [Log-based CDC explained](/blog/real-time-replication-log-based-cdc/) - [Debezium alternative: managed CDC versus Kafka + Debezium](/solutions/debezium-alternative/) - [Elasticsearch agent overview ↗](https://docs.molo17.com/gluesync/latest/agents/elasticsearch-intro.html) - [Elasticsearch target setup guide ↗](https://docs.molo17.com/gluesync/latest/agents/elasticsearch-target.html) - [Snapshot tasks ↗](https://docs.molo17.com/gluesync/latest/core-hub/snapshot-tasks.html) FAQ ## Elasticsearch and OpenSearch replication questions What does replicating to Elasticsearch or OpenSearch with Gluesync involve? A source agent captures committed changes from your database through its native change mechanism, Core Hub routes them, and the Elasticsearch agent writes them to indexes through the bulk API. Each table is seeded with a snapshot first and then follows the change stream. Is OpenSearch a separate agent? No. Elasticsearch and OpenSearch share one agent and one HTTP JSON client. The agent reads the flavor from the cluster's root response, or you can pin it to Elasticsearch or OpenSearch in the connection settings. How are inserts, updates, and deletes written? Everything goes through the bulk API. Inserts are indexed under the document ID built from the source key, updates are partial updates with doc\_as\_upsert, and deletes remove the document by ID. A missing document on delete is not an error. Does Gluesync create the indexes and mappings? Yes. When an index does not exist, the agent creates it with an explicit mapping generated from the source column types, and you can choose alternatives such as text for a string field or nested for a map. An existing index is left untouched, mapping included. How are index names chosen? By default from a prefix, the entity's namespace, and its collection. A name template or a per-entity override replaces that, and the name is normalized to what the cluster accepts, for example lowercased with reserved characters replaced. What happens on a truncate or before a snapshot? By default the agent runs a delete by query that matches every document and keeps the index and its mapping. You can choose to skip it and leave the documents in place. How does the agent authenticate and connect? With Basic authentication, an API key, a Bearer token, or none for a closed network, optionally over TLS. It supports additional cluster nodes and a path prefix for a reverse proxy in front of the cluster. Can documents run through an ingest pipeline? Yes. Name an ingest pipeline on an entity and its bulk requests are sent through it. Which clusters are supported? Elasticsearch and OpenSearch clusters that expose the standard HTTP JSON API. REPLICATE TO A TARGET ## Other targets Gluesync delivers to - [Replicate to Aerospike](/solutions/replicate-to-aerospike/) - [Replicate to DynamoDB](/solutions/replicate-to-dynamodb/) - [Replicate to Redshift](/solutions/replicate-to-redshift/) - [Replicate to Amazon S3 & S3-compatible](/solutions/replicate-to-amazon-s3/) - [Replicate to Cassandra](/solutions/replicate-to-cassandra/) - [Replicate to Kafka](/solutions/replicate-to-kafka/) - [Replicate to Cosmos DB](/solutions/replicate-to-cosmos-db/) - [Replicate to Azure Data Lake](/solutions/replicate-to-azure-data-lake/) - [Replicate to ClickHouse](/solutions/replicate-to-clickhouse/) - [Replicate to CockroachDB](/solutions/replicate-to-cockroachdb/) - [Replicate to Couchbase](/solutions/replicate-to-couchbase/) - [Replicate to Databricks](/solutions/replicate-to-databricks/) - [Replicate to file stores](/solutions/replicate-to-file-stores/) - [Replicate to BigQuery](/solutions/replicate-to-bigquery/) - [Replicate to Google Cloud Storage](/solutions/replicate-to-google-cloud-storage/) - [Replicate to Google Pub/Sub](/solutions/replicate-to-google-pubsub/) - [Replicate to GridGain](/solutions/replicate-to-gridgain/) - [Replicate to Db2](/solutions/replicate-to-db2/) - [Replicate to Informix](/solutions/replicate-to-informix/) - [Replicate to MariaDB](/solutions/replicate-to-mariadb/) - [Replicate to Microsoft Fabric](/solutions/replicate-to-microsoft-fabric/) - [Replicate to SQL Server](/solutions/replicate-to-sql-server/) - [Replicate to MongoDB](/solutions/replicate-to-mongodb/) - [Replicate to MySQL](/solutions/replicate-to-mysql/) - [Replicate to Oracle](/solutions/replicate-to-oracle/) - [Replicate to PostgreSQL](/solutions/replicate-to-postgresql/) - [Replicate to RavenDB](/solutions/replicate-to-ravendb/) - [Replicate to Redis](/solutions/replicate-to-redis/) - [Replicate to Redis Streams](/solutions/replicate-to-redis-streams/) - [Replicate to SAP ASE](/solutions/replicate-to-sap-ase/) - [Replicate to SAP HANA](/solutions/replicate-to-sap-hana/) - [Replicate to ScyllaDB](/solutions/replicate-to-scylladb/) - [Replicate to SingleStore](/solutions/replicate-to-singlestore/) - [Replicate to Snowflake](/solutions/replicate-to-snowflake/) - [Replicate to Solace PubSub+](/solutions/replicate-to-solace/) - [Replicate to Supabase](/solutions/replicate-to-supabase/) - [Replicate to Vertica](/solutions/replicate-to-vertica/) - [Replicate to YugabyteDB](/solutions/replicate-to-yugabytedb/) ## Evaluate Gluesync with your Elasticsearch or OpenSearch cluster Start a trial against your cluster, or talk to MOLO17 about your sources, index design, and change volume. [Start a Gluesync trial](/get-gluesync/) [Talk to MOLO17](/contacts/)