The Apache Camel community introduces a new subproject in the ecosystem: Camel-Kafka-Connector. This project born as a PoC exploring the possibility of leveraging the Apache Camel components as Kafka source and sink connectors. The main idea behind the project is reusing the Camel Components’ flexibility in a simple way, through a configuration file mixing Kafka Connect configuration and Camel route definitions and options.
What is Kafka Connect?
It is an Apache Kafka’s framework that defines a standardized way to stream data in and out a kafka broker. In Kafka connect a user can define source and sink connectors to stream data, respectively, in and out a kafka broker. This framework features:
- Distributed and standalone mode
- Rest interface
- Automatic offset management
Kafka connect key concepts
- Connector: generally refer to a source/sink implemented using kafka connect framework.
- SinkConnector / SourceConnector: are responsible to setting up the connector and partitioning the work by creating SinkTask / SourceTask.
- SinkTask / SourceTask: are responsible to handle the actual work.
- Key / value converter: a component able to convert the key or value of a kafka message from a format to another.
- Transformer: a component able to manipulate a message.
This is just an introduction to the new subproject: we’ll add more information and more posts while the project will go ahead. For the moment stay tuned and as always contributions are welcome. Show your love for the new arrival in the Camel family: https://github.com/apache/camel-kafka-connector: