LangChain4j Ingest
Ingest documents into a LangChain4j EmbeddingStore: split, embed and store the message body.
What’s inside
-
LangChain4j Ingest component, URI syntax:
langchain4j-ingest:pipelineName
Please refer to the above links for usage and configuration details.
Maven coordinates
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-langchain4j-ingest-starter</artifactId>
</dependency>Spring Boot Auto-Configuration
The starter supports 13 options, which are listed below.
| Name | Description | Default | Type |
|---|---|---|---|
camel.component.langchain4j-ingest.autowired-enabled | Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. | true | Boolean |
camel.component.langchain4j-ingest.configuration | The configuration. The option is a org.apache.camel.component.langchain4j.ingest.LangChain4jIngestConfiguration type. | LangChain4jIngestConfiguration | |
camel.component.langchain4j-ingest.document-id-header | Name of the header carrying the document id, such as CamelAwsS3Key for an S3 consumer or CamelKafkaKey for a Kafka one. The CamelLangChain4jIngestDocumentId exchange property, when set, takes precedence - a route that parses documents captures the id into that property before the parse, so a document cannot forge its own identity. An exchange without an id fails. | CamelLangChain4jIngestDocumentId | String |
camel.component.langchain4j-ingest.document-splitter | The DocumentSplitter deciding how a document becomes segments, referenced as #bean:name - LangChain4j ships alternatives beside the default recursive one. When set, maxSegmentSize and maxOverlapSize are ignored (they parameterize the default splitter only). Segments returned without the identity metadata are re-stamped, so a custom splitter cannot break citation. Not looked up by type on purpose - an application may hold unrelated splitters. The option is a dev.langchain4j.data.document.DocumentSplitter type. | DocumentSplitter | |
camel.component.langchain4j-ingest.embedding-batch-size | How many segments are embedded per request to the embedding model. Providers with generous per-request limits ingest large documents faster with a bigger batch; a batch carries at most embeddingBatchSize x maxSegmentSize characters, so tune the two together against the provider’s token limits. | 32 | Integer |
camel.component.langchain4j-ingest.embedding-model | The EmbeddingModel to embed segments with. When not set, the single bean of that type in the registry is used; zero or several beans fail the endpoint start with an error naming this option. The option is a dev.langchain4j.model.embedding.EmbeddingModel type. | EmbeddingModel | |
camel.component.langchain4j-ingest.embedding-store | The EmbeddingStore to write segments to. When not set, the single bean of that type in the registry is used; zero or several beans fail the endpoint start with an error naming this option. The option is a dev.langchain4j.store.embedding.EmbeddingStore<dev.langchain4j.data.segment.TextSegment> type. | TextSegment> | |
camel.component.langchain4j-ingest.enabled | Whether to enable auto configuration of the langchain4j-ingest component. This is enabled by default. | Boolean | |
camel.component.langchain4j-ingest.idempotent-repository | The IdempotentRepository remembering already ingested document ids, referenced as #bean:name. When set, a delivery whose id was already written is answered with a skipped result instead of being re-ingested: first write wins per id. A blank document releases its claim, so a later, populated delivery under the same id still ingests. The claim is eager: a duplicate racing an in-flight first delivery is answered skipped even if that delivery then fails - with an at-least-once source the skipped duplicate is acknowledged and the failed original may be the only other copy, so pair eager deduplication with a source that redelivers on failure. Not looked up by type on purpose - an application may hold unrelated idempotent repositories. The repository is started but never stopped by the endpoint (it may be shared); a persistent repository’s lifecycle belongs to whoever created it. The option is a org.apache.camel.spi.IdempotentRepository type. | IdempotentRepository | |
camel.component.langchain4j-ingest.lazy-start-producer | Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. | false | Boolean |
camel.component.langchain4j-ingest.max-document-size | Maximum size of one document in characters, applied to the text about to be split; 0, the default, means no limit. The pipeline holds a document in memory whole, so the cap is the protection against oversized - on a consumer-fed pipeline, attacker-sized - payloads. An oversized document fails the exchange cleanly and, with a repository configured, releases its dedup claim. | 0 | Integer |
camel.component.langchain4j-ingest.max-overlap-size | How much of the previous segment each segment repeats, in characters. Overlap keeps a sentence split across a boundary retrievable from either side. | 50 | Integer |
camel.component.langchain4j-ingest.max-segment-size | Maximum size of one segment, in characters. | 500 | Integer |