Chunking is a crucial aspect of data processing that can significantly impact retrieval quality, query latency, costs, and even the accuracy of Large Language Model (LLM) outputs. In this blog post, we’ll explore what chunking is, its importance, and how the new Chunking DSL in Apache Camel 4.8.0 improves data processing workflows.

The Problem with Traditional Chunking Approaches

Before Camel 4.8.0, applications using Camel would have to implement custom chunking logic or rely on external libraries. This approach not only added complexity but also made it challenging to handle data cleaning, deduplication, and other tasks. Developers had to write code to tokenize, which involved implementing their own chunking logic or using features like Document Splitters (from LangChain4J) via Camel Processors.

Introducing the New Chunking DSL in Camel 4.8.0

Starting with Camel 4.8.0, we’ve introduced a new Chunking DSL that allows you to describe the desired chunking as part of the route. This feature makes routes more elegant and reduces the need for custom code. With the new Chunking DSL, you can focus on processing your data without worrying about the intricacies of chunking.

Example Use Case: Chunking Documents by Paragraph

Here’s an example of using the Chunking DSL to tokenize documents by paragraph:

from("kafka:documents")
	.tokenize(tokenizer()
			.byParagraph()
			.maxTokens(1024)
			.maxOverlap(10)
			.using(LangChain4jTokenizerDefinition.TokenizerType.OPEN_AI)
			.end())
	.split().body()
	.to("kafka:ingestion");

This code snippet demonstrates the new chunking capabilities in Camel 4.8.0, making it easier to process and manage large datasets.

But that’s not all - the new Chunking DSL also integrates seamlessly with existing features like embedding. By combining these features, you can create even more efficient data processing workflows. For example:

from("kafka:ingestion")
	.to("langchain4j-embeddings:myEmbedding")
	.setHeader(Qdrant.Headers.ACTION).constant(QdrantAction.UPSERT)
	.transform(new DataType("qdrant:embeddings"))
	.toF("qdrant:%s", configuration.qdrant().collection().name());

In this example, we’re using the new Chunking DSL to tokenize documents by paragraph, and then passing the resulting chunks through an embedding pipeline. The langchain4j-embeddings endpoint is used to embed the chunked data into a vector space, which can be stored in Qdrant for efficient querying.

By combining these features, you can create routes that are not only more elegant but also more efficient and scalable. With the new Chunking DSL, you can focus on processing your data without worrying about the intricacies of chunking or embedding.

Conclusion

The new Chunking DSL in Apache Camel 4.8.0 is a game-changer for data processing workflows. By providing an elegant way to describe chunking as part of the route, this feature reduces complexity and improves efficiency. We encourage you to explore this feature and experience the benefits it can bring to your data processing workflows.