Installation steps

Gluesync SQL to NoSQL for Apache Kafka

Prerequisites

To have Gluesync working on your Kafka instance you will need to have:

  • as many topics created one per each of the entities you’re looking to source changes from;

  • valid user credentials with permission to post to the above-mentioned topics;

  • valid certificate in case of SASL authentication.

Basic configuration example

This module can be customized by using a configuration file, in JSON format. The file name to use must be specified as a parameter when launching the app, with the -f or --file tokens. The file should be composed of a union of the common configuration files (see here Installation steps) and source-specific configuration:

{
  ...
  // (source section omitted)
  ...
  "targetHost": "KAFKAIPADDRESS",
  "targetPort": "9092",
  "targetUsername": "KafkaUsername",
  "targetPassword": "KafkaPassword",
  ...

  "kafka": {
    "topicPrefix": "MYTOPICPREFIX_",
    "useTransactions": true,
    "saslMechanism": "SCRAM-SHA-256",
    "securityProtocol": "SASL_SSL",
    "certificatesType": "PEM",
    "certificatesPath": "/path/to/certificate.pem",
    "certificatesPassword": "certificatePassword",
    "disableAuth": false
  },
}

The following are the parameters used to configure your Kafka target:

  • targetHost: the host address of your Kafka cluster;

  • targetPort: the port exposed by your Kafka cluster;

  • targetUsername: username to use to connect against Kafka;

  • targetPassword: password to use to connect against Kafka.

Apache Kafka-specific configurations are listed under the kafka property:

  • topicPrefix (optional): default to NULL. String value that defines a prefix used in conjunction with the entity name to build the Kafka topic name;

  • useTransactions (optional): defaults to false. If set to true it enables the usage of Kafka transactions, look at this documentation in order to know more about it;

  • saslMechanism (required if auth in place): defaults to SCRAM-SHA-256. Accepts the following parameters:

    • GSSAPI,

    • OAUTHBEARER,

    • PLAIN,

    • SCRAM-SHA-256,

    • SCRAM-SHA-512.

  • securityProtocol (required if auth in place): defaults to SASL_SSL. Accepts the following parameters:

    • PLAINTEXT,

    • SASL_PLAINTEXT,

    • SASL_SSL,

    • SSL.

  • certificatesType (required if auth in place): defaults to PEM. Accepts the following parameters:

    • PEM,

    • JKS.

  • certificatesPath (required if auth in place): defaults to NULL. The path to your certificates;

  • certificatesPassword (required if auth in place): defaults to NULL. The password used to lock your certificate;

  • disableAuth (optional): defaults to false. Disable authentication over Kafka broker.