Camel Kafka Connector

Troubleshooting the Camel Kafka Connector

This page describes some common pitfalls when deploying the connectors.

Table of Contents

Kafka Connect does not load the Camel connector

If the Kafka Connect runtime is not loading the Camel connector, the cause could be an incorrect directory structure. There is a specific directory structure that needs to be followed when installing the connectors. They are meant to be installed in a subdirectory of the Kafka Connect plugin.path and every connector should be within its own directory.

For example, if you have set the value of plugin.path to /camel-kafka-connectors/ then every connector should be within a directory inside it:

  • /camel-kafka-connectors/camel-<component>-kafka-connector

It’s possible to have multiple connectors in the plugin.path, therefore in a setup with both amqp and hdfs connectors, it would be possible to have a directory structure like this:

  • /camel-kafka-connectors/camel-hdfs-kafka-connector

  • /camel-kafka-connectors/camel-amqp-kafka-connector

Missing or Incorrect Converters

A common root cause for Kafka Connect failing to serialize or deserialize the data in transit is incorrect values for the key.converter or the value.converter (or both). There are usually verifiable in the logs by looking for Caused by: org.apache.kafka.common.errors.SerializationException or Caused by: org.apache.kafka.connect.errors.DataException.

Here’s one example with trying to exchange data using the AMQP connector without setting the key.converter and value.converter correctly:

Caused by: org.apache.kafka.connect.errors.DataException: JsonConverter with schemas.enable requires "schema" and "payload" fields and may not contain additional fields. If you are trying to deserialize plain JSON data, set schemas.enable=false in your converter configuration.

Missing Dependencies

When deploying the connector manually (ie.: without the assistance of tools such as Strimzi) you may need to make sure to provide the required runtime dependencies and their dependencies. Some notable examples are:

  • AMQP connector requires a JMS-compliant client (such as qpid-jms-client)

  • From version 0.7.0 onward, the ActiveMQ and Artemis JMS clients are packaged along with the SJMS2 connector. If you need to use other JMS clients, then you have to repackage including all required libraries.

  • JDBC connector requires the JDBC client for the database

Known Incompatibilities

  • Do not use activemq-all 5.16.0 or older with the SJMS2 connector: the activemq-all jar contains packages older versions of Camel that cause conflict when trying to run the connector. You may find very interesting errors such as java.lang.NoSuchMethodError: org.apache.camel.main.Main.configure()Lorg/apache/camel/main/MainConfigurationProperties when trying to do so.

AWS connectors on Confluent Platform images

Since 4.22.0, an AWS connector (for example camel-aws-s3-source-kafka-connector) running on a Confluent Platform cp-kafka-connect image can fail when its task starts, while it creates the AWS SDK client:

java.util.ServiceConfigurationError: software.amazon.awssdk.http.SdkHttpService: software.amazon.awssdk.http.apache.ApacheSdkHttpService not a subtype

The image puts its own copy of the AWS SDK on the worker classpath (/usr/share/java/kafka-serde-tools), with the apache-client HTTP client. Kafka Connect lets the connector see the HTTP clients registered on the worker classpath, and the AWS SDK fails when one of them belongs to another copy of the SDK. Up to 4.18.x the connectors shipped apache-client themselves, which hid the problem. Since Camel 4.22.0 they ship apache5-client instead.

Any of the following avoids the error:

  • Add apache-client, at the AWS SDK version of the connector, and its dependencies to the connector directory. The connector keeps using the Apache 5 client. For the 4.22.0 connectors, which use AWS SDK 2.50.2:

    cd <plugin.path>/camel-aws-s3-source-kafka-connector
for a in software/amazon/awssdk/apache-client/2.50.2/apache-client-2.50.2.jar \
         org/apache/httpcomponents/httpclient/4.5.14/httpclient-4.5.14.jar \
         org/apache/httpcomponents/httpcore/4.4.16/httpcore-4.4.16.jar \
         commons-logging/commons-logging/1.2/commons-logging-1.2.jar; do
  curl -sSfLO "https://repo1.maven.org/maven2/$a"
done

  • Name the HTTP client with the AWS SDK system property, for example KAFKA_OPTS=-Dsoftware.amazon.awssdk.http.service.impl=software.amazon.awssdk.http.apache5.Apache5SdkHttpService. The property applies to the whole worker JVM, including other plugins that use the AWS SDK.

  • Remove apache-client-*.jar from /usr/share/java/kafka-serde-tools in the image. Do this only if nothing else on the worker uses that copy of the AWS SDK, such as Schema Registry client-side field level encryption with AWS KMS.

CAMEL-26343 makes the Camel AWS components set their HTTP client themselves: connectors built on a Camel release that contains it do not need these workarounds.

Known Limitations

Camel File Kafka Connector

When using append chars, it must be UTF-8 encoded so that they can be transmitted over the internet (i.e.: for new line use use %0A instead of \n and so on). You can check the W3 Schools encoding reference for a reference list of characters and their encoded strings.