Kafka Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Stable"
Receive data from Kafka topics through Plain Login Module.
Configuration Options
The following table summarizes the configuration options available for the kafka-source
Kamelet:
Property | Name | Description | Type | Default | Example |
---|---|---|---|---|---|
Bootstrap Servers | Required Comma separated list of Kafka Broker URLs. | string | |||
Password | Required Password to authenticate to kafka. | string | |||
Topic Names | Required Comma separated list of Kafka topic names. | string | |||
Username | Required Username to authenticate to Kafka. | string | |||
Allow Manual Commit | Whether to allow doing manual commits. | boolean | false | ||
Auto Commit Enable | If true, periodically commit to ZooKeeper the offset of messages already fetched by the consumer. | boolean | true | ||
Auto Offset Reset | What to do when there is no initial offset. There are 3 enums and the value can be one of latest, earliest, none. | string | latest | ||
Consumer Group | A string that uniquely identifies the group of consumers to which this source belongs. | string | my-group-id | ||
Automatically Deserialize Headers | When enabled the Kamelet source will deserialize all message headers to String representation. | boolean | true | ||
Poll On Error Behavior | What to do if kafka threw an exception while polling for new messages. There are 5 enums and the value can be one of | string | ERROR_HANDLER | ||
SASL Mechanism | The Simple Authentication and Security Layer (SASL) Mechanism used. | string | PLAIN | ||
Security Protocol | Protocol used to communicate with brokers. SASL_PLAINTEXT, PLAINTEXT, SASL_SSL and SSL are supported. | string | SASL_SSL | ||
Topic Is Pattern | Whether the topic is a pattern (regular expression). This can be used to subscribe to dynamic number of topics matching the pattern. | boolean | false |
Dependencies
At runtime, the kafka-source
Kamelet relies upon the presence of the following dependencies:
-
camel:core
-
camel:kafka
-
camel:kamelet
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel
Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:kafka-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink"
You can now run it directly through the following command
camel run route.yaml
Kafka Source Kamelet Description
Authentication
This Kamelet requires SASL/PLAIN authentication to connect to Kafka through a Plain Login Module. The credentials are configured through the user
and password
properties.
Configuration
The Kafka Source Kamelet supports the following configurations:
-
Topic: Comma-separated list of Kafka topic names to consume from (required)
-
Bootstrap Servers: Comma-separated list of Kafka bootstrap servers (required)
-
User: Username for SASL/PLAIN authentication (required)
-
Password: Password for SASL/PLAIN authentication (required)
-
Consumer Group: Kafka consumer group ID for managing offsets
-
Auto Offset Reset: What to do when there is no initial offset (earliest, latest, none)
-
Allow Manual Commit: Enable manual commit for better control over message processing
Output Format
The Kamelet outputs Kafka message content and includes Kafka headers and metadata such as topic, partition, offset, and timestamp.
Usage Example
- route:
from:
uri: "kamelet:kafka-source"
parameters:
topic: "orders,payments"
bootstrapServers: "kafka.example.com:9092"
user: "kafka-user"
password: "kafka-password"
steps:
- to:
uri: "kamelet:log-sink"
Example with Consumer Group
- route:
from:
uri: "kamelet:kafka-source"
parameters:
topic: "user-events"
bootstrapServers: "kafka1.example.com:9092,kafka2.example.com:9092"
user: "kafka-user"
password: "kafka-password"
consumerGroup: "my-consumer-group"
autoOffsetReset: "earliest"
steps:
- to:
uri: "kamelet:log-sink"