← All articles

Article

Courier: Gluesync pipelines now speak REST in both directions

Courier: Gluesync pipelines now speak REST in both directions featured image

Courier brings REST APIs into Gluesync as sources and targets: describe an API once in Core Hub, then send changes out or take records in with the same pipelines.

Gluesync 2.3.2.1 brings Courier: REST APIs become sources and targets of your pipelines. A pipeline can deliver every captured change to an API, or take records from one, with the same entities, mappings and checkpoints you already use for databases. Courier is the REST editor of Core Hub, and the HTTP agent (courier-agent) ships inside Core Hub. Nothing to deploy: open Courier in the menu and start describing your API.

Every detail of what ships in 2.3.2.1 comes from the Gluesync release notes and the HTTP Source and HTTP Target documentation. The previous release is covered in Gluesync 2.3.2: point-in-time restart, disaster recovery, lake file queries and Podman.

Why Courier

Every integration project has one system with no database to read, only an API. Until now that meant a script, a cron job and a long debate about who owns it when it stops. Courier turns that API into a first-class part of the pipeline: described once in the web UI, reusable by every pipeline, monitored and checkpointed like any other agent.

When to use it

  • Changes out to an API: push every insert, update and delete captured from a database to a SaaS application, an internal service or a partner endpoint, using HTTP Target
  • Records in over HTTP: let a SaaS webhook, a device or your own application push records into a pipeline, and deliver them to any Gluesync target, using HTTP Source
  • One row, more APIs: add one HTTP Target per collection and the same row reaches every API you need

What it does

HTTP Target: changes out to any API

  • Collections and endpoints: a collection holds the base URL and the authentication (None, Basic, Bearer, or an API key in a header or in the query string). Its endpoints hold the verb (GET, POST, PUT, PATCH, DELETE, HEAD, OPTIONS), the path template, the headers, the body type (NONE, JSON, FORM_URLENCODED) and typed bindings (PATH, QUERY, BODY, HEADER, typed as STRING, NUMBER or BOOLEAN)
  • Same model as a database: the pipeline sees a collection as a schema and an endpoint as a table. Pointing a source table at an API is the same as pointing it at a database: pick the endpoint, map the columns. Map to pipeline builds the mapping from the endpoint side
  • One call per change: every captured insert, update and delete becomes one HTTP call. Each endpoint has one verb, and the operation travels in the body field operation and in the {__operation} placeholder. Snapshot rows are sent as updates
  • Sent twice, applied once: every request carries an Idempotency-Key, a SHA-256 of the source position, table, operation, values and before-images. It stays the same when Core Hub hands the change over again, after a retry or a resume
  • Clear delivery rules: a non-2xx answer, 3xx included, fails the write and it is retried from the checkpoint. Redirects are never followed. A DELETE answered 404 or 410 counts as delivered. Calls time out after 10 seconds to connect and 30 seconds overall
  • Filters first: pipeline filters apply before delivery, so only the rows you want reach the API

HTTP Source: records in over HTTP

  • Inbound endpoints: publish CREATE, UPDATE and DELETE endpoints on the agent's own listener (port 8090 by default), at POST /api-inbound/<basePath>/<path>
  • Durable before acknowledged: each authenticated request becomes one pipeline row on a disk queue, forced to disk before the client gets 202 Accepted with its sequence number. The pipeline delivers it to any target, at least once and in order per endpoint
  • Typed rows: inbound STRING maps to VARCHAR(255), NUMBER to NUMERIC(18,2), plus BOOLEAN. An endpoint without a key gets a synthetic _seq key
  • Calm under pressure: limits on body size, requests in flight, queue size and free disk answer 408, 413 or 503 with Retry-After instead of falling over. The defaults are 1024 KB per body, 64 requests in flight, 2048 MB of backlog per endpoint, 8192 MB in total, and 1 GB or 5% of free disk
  • Observable: the http_source_requests_total{endpoint,status} metric counts every request by endpoint and status

Secure before it's convenient

  • Inbound tokens: inbound calls need a Personal API Token with the gluesync.https_agent scope, owned by a Manager or a Super Admin and admitted by the collection's allowlist and blocklist. Viewer and Monitor tokens get 403
  • TLS by default: the inbound listener needs a keystore. Plain HTTP is accepted on loopback only, unless you enable inboundAllowPlainHttp
  • Secrets protected: credentials and header values are encrypted at rest and masked in every answer
  • Egress guard: outbound calls never reach loopback, link-local, cloud metadata, multicast or NAT64 addresses, nor Core Hub's own port, unless you set GS_COURIER_ALLOW_LOCAL_EGRESS=true. Private RFC 1918 ranges are allowed

Coming next: stored procedures over HTTP

An upcoming Gluesync release brings stored procedure invocation to Courier. A new EXECUTE operation lets an inbound HTTP request call a stored procedure or a function on the database of a SQL agent, whether that agent is the source or the target of your pipeline. The business logic stays where it already lives, in the database, and any client that speaks HTTP can run it with one request.

  • Seven databases: PostgreSQL, YugabyteDB, MySQL, MariaDB, SQL Server, Oracle and Db2 LUW
  • Typed parameters: IN, OUT and INOUT parameters bind from the path, the query string or the JSON body, and every value is converted to the SQL type the routine declares
  • The full answer back: the response carries the OUT values, up to 16 result sets and the row count, in JSON or XML
  • Clear errors: failures come back as 400, 409, 422 or 504
  • Bounded and transactional: a call times out after 30 seconds by default, up to a maximum of 300 seconds, and runs in a single transaction unless the routine commits on its own
  • Same security model: calls use the same Personal API Token as HTTP Source, and only the procedures you configure as endpoints can be called

Setup lives in a new Procedures tab in Courier. Every call starts from an inbound HTTP request to a configured endpoint.

How to start

  1. Upgrade Core Hub to 2.3.2.1 or later. It ships the HTTP agent and the Courier module. To automate the setup, the courier block of the bootstrapper creates or updates the Courier definitions before the agent is configured
  2. Open Courier in the Core Hub menu and create a collection: base URL and authentication
  3. Add the endpoints: verb, path template, headers, body type and bindings. Use Send test request to check what the API answers: the latest 50 results of every endpoint are kept
  4. For changes out, create a pipeline with HTTP Target, pick the endpoint as the target table and map the columns, or use Map to pipeline. Follow HTTP Target setup
  5. For records in, publish inbound endpoints with HTTP Source, issue a Personal API Token with the HTTPS agent scope and point your client at the endpoint. Follow HTTP Source setup

HTTP Source accepts any HTTP/1.1 client able to send a bearer token, and HTTP Target reaches any HTTP/1.1 API over http:// or https:// that answers 2xx to accepted deliveries. HTTP Target has no bulk load, UDF or truncate: it delivers changes one call at a time, exactly as the API expects them. The full list of changes is in the HTTP agent changelog.

Your pipelines already spoke SQL, NoSQL and streaming. Now they speak HTTP too. Get Gluesync and try Courier today.