KafkaSender for Azure Section

The "KafkaSender" for Azure section provides configuration settings for a Kafka client using the Azure Event Hubs Kafka Surface endpoint for sending Events to Azure Event Hubs. It requires an Event Hubs namespace with Standard tier or higher.

"KafkaSender": { 
  "Properties": {
    "bootstrap.servers": "<namespace-name>.servicebus.windows.net:9093",
    "sasl.mechanisms": "PLAIN",
    "sasl.password": "Endpoint=sb://<namespace‑name>.servicebus.windows.net/;
     SharedAccessKeyName=<policy-name>;
     SharedAccessKey=<shared-access-key>",
    "sasl.username": "$ConnectionString",
    "security.protocol": "SASL_SSL"
  }
}

The "KafkaSender" for Azure "Properties" parameters are listed in the following table.

Parameter	Description
bootstrap.servers	The Event Hubs namespace name followed by .servicebus.windows.net:9093
sasl.mechanisms	Always "PLAIN" for Event Hubs
sasl.password	The full connection string for the Event Hubs namespace
sasl.username	Always "$ConnectionString" for Event Hubs
security.protocol	Always "SASL_SSL" for Event Hubs

Tuning Hints for KafkaSender for Azure

Event Hubs (Standard tier) enforces limits on message size, data rate, and message rates. Surpassing these limits could result in failed event delivery. To mitigate this, tune the KafkaSender settings as follows.

{
   "KafkaSender": {
      "Properties": {
         "bootstrap.servers": "example‑eventhubs‑namespace.
          servicebus.windows.net:9093",
         "connections.max.idle.ms": "180000",
         "enable.idempotence": "true",
         "max.request.size": "1000000",
         "metadata.max.age.ms": "180000",
         "request.timeout.ms": "30000",
         "retries": "2147483647",
         "retry.backoff.max.ms": "300000",
         "retry.backoff.ms": "1000",
         "sasl.mechanisms": "PLAIN",
         "sasl.password": "Endpoint=sb://example‑eventhubs‑namespace.
          servicebus.windows.net/;SharedAccessKeyName=KeyName;
          SharedAccessKey=abcdef2321312312=",
         "sasl.username": "$ConnectionString",
         "security.protocol": "SASL_SSL",
         "socket.keepalive.enable": "true"
      }
   }
}

Key settings are explained in the following table.

Setting	Explanation
retries	Set to a large number (for example, 2147483647) to ensure continuous retries if Event Hubs throttles requests.
retry.backoff.ms and retry.backoff.max.ms	Introduces a delay between retries to prevent overwhelming the server. Initial retry delay starts at 1000 ms, increasing to a maximum of 300000 ms.
enable.idempotence	Ensures events are delivered in order, even after retries, which is crucial when Event Hubs throttles requests.
max.request.size	Configures the maximum size of a Kafka request. Set to avoid exceeding the limits imposed by Event Hubs.
request.timeout.ms	Timeout value to allow for longer waits during retries, preventing premature failures.

Event Hubs Standard Tier Considerations

Without the appropriate settings, Event Hubs (Standard tier) may reject events that exceed its message or data rate limits. In such cases, manual intervention is needed to move unsent event data files from the Undeliverable folder to the ChangeCapture folder for retry.