Camel Components

MongoDB GridFS

Since Camel 2.18

Both producer and consumer are supported

Maven users will need to add the following dependency to their pom.xml for this component:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-mongodb-gridfs</artifactId>
    <version>x.y.z</version>
    <!-- use the same version as your Camel core version -->
</dependency>

URI format

mongodb-gridfs:connectionBean?database=databaseName&bucket=bucketName[&moreOptions...]

Configuring Options

Camel components are configured on two separate levels:

  • component level

  • endpoint level

Configuring Component Options

At the component level, you set general and shared configurations that are, then, inherited by the endpoints. It is the highest configuration level.

For example, a component may have security settings, credentials for authentication, urls for network connection and so forth.

Some components only have a few options, and others may have many. Because components typically have pre-configured defaults that are commonly used, then you may often only need to configure a few options on a component; or none at all.

You can configure components using:

  • the Component DSL.

  • in a configuration file (application.properties, *.yaml files, etc).

  • directly in the Java code.

Configuring Endpoint Options

You usually spend more time setting up endpoints because they have many options. These options help you customize what you want the endpoint to do. The options are also categorized into whether the endpoint is used as a consumer (from), as a producer (to), or both.

Configuring endpoints is most often done directly in the endpoint URI as path and query parameters. You can also use the Endpoint DSL and DataFormat DSL as a type safe way of configuring endpoints and data formats in Java.

A good practice when configuring options is to use Property Placeholders.

Property placeholders provide a few benefits:

  • They help prevent using hardcoded urls, port numbers, sensitive information, and other settings.

  • They allow externalizing the configuration from the code.

  • They help the code to become more flexible and reusable.

The following two sections list all the options, firstly for the component followed by the endpoint.

Component Options

The MongoDB GridFS component supports the following options which are listed below.

Name Description Default Type

bridgeErrorHandler (consumer)

Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions (if possible) occurred while the Camel consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. Important: This is only possible if the 3rd party component allows Camel to be alerted if an exception was thrown. Some components handle this internally only, and therefore bridgeErrorHandler is not possible. In other situations we may improve the Camel component to hook into the 3rd party component and make this possible for future releases. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored.

false

boolean

lazyStartProducer (producer)

Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing.

false

boolean

autowiredEnabled (advanced)

Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc.

true

boolean

Endpoint Options

The MongoDB GridFS endpoint is configured using URI syntax:

mongodb-gridfs:connectionBean

With the following path and query parameters:

Path Parameters

Name Description Default Type

connectionBean (common)

Required Name of com.mongodb.client.MongoClient to use.

String

Query Parameters

Name Description Default Type

bucket (common)

Sets the name of the GridFS bucket within the database. Default is fs.

fs

String

database (common)

Required Sets the name of the MongoDB database to target.

String

readPreference (common)

Sets a MongoDB ReadPreference on the Mongo connection. Read preferences set directly on the connection will be overridden by this setting. The com.mongodb.ReadPreference#valueOf(String) utility method is used to resolve the passed readPreference value. Some examples for the possible values are nearest, primary or secondary etc.

ReadPreference

writeConcern (common)

Set the WriteConcern for write operations on MongoDB using the standard ones. Resolved from the fields of the WriteConcern class by calling the WriteConcern#valueOf(String) method.

Enum values:

  • ACKNOWLEDGED

  • W1

  • W2

  • W3

  • UNACKNOWLEDGED

  • JOURNALED

  • MAJORITY

WriteConcern

delay (consumer)

Sets the delay between polls within the Consumer. Default is 500ms.

500

long

fileAttributeName (consumer)

If the QueryType uses a FileAttribute, this sets the name of the attribute that is used. Default is camel-processed.

camel-processed

String

initialDelay (consumer)

Sets the initialDelay before the consumer will start polling. Default is 1000ms.

1000

long

persistentTSCollection (consumer)

If the QueryType uses a persistent timestamp, this sets the name of the collection within the DB to store the timestamp.

camel-timestamps

String

persistentTSObject (consumer)

If the QueryType uses a persistent timestamp, this is the ID of the object in the collection to store the timestamp.

camel-timestamp

String

query (consumer)

Additional query parameters (in JSON) that are used to configure the query used for finding files in the GridFsConsumer.

String

queryStrategy (consumer)

Sets the QueryStrategy that is used for polling for new files. Default is Timestamp.

Enum values:

  • TimeStamp

  • PersistentTimestamp

  • FileAttribute

  • TimeStampAndFileAttribute

  • PersistentTimestampAndFileAttribute

TimeStamp

QueryStrategy

bridgeErrorHandler (consumer (advanced))

Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions (if possible) occurred while the Camel consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. Important: This is only possible if the 3rd party component allows Camel to be alerted if an exception was thrown. Some components handle this internally only, and therefore bridgeErrorHandler is not possible. In other situations we may improve the Camel component to hook into the 3rd party component and make this possible for future releases. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored.

false

boolean

exceptionHandler (consumer (advanced))

To let the consumer use a custom ExceptionHandler. Notice if the option bridgeErrorHandler is enabled then this option is not in use. By default the consumer will deal with exceptions, that will be logged at WARN or ERROR level and ignored.

ExceptionHandler

exchangePattern (consumer (advanced))

Sets the exchange pattern when the consumer creates an exchange.

Enum values:

  • InOnly

  • InOut

ExchangePattern

operation (producer)

Sets the operation this endpoint will execute against GridFs.

String

lazyStartProducer (producer (advanced))

Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing.

false

boolean

Message Headers

The MongoDB GridFS component supports the following message header(s), which is/are listed below:

Name Description Default Type

CamelFileContentType (consumer)

Constant: FILE_CONTENT_TYPE

The content type of the file.

String

CamelFileLength (consumer)

Constant: FILE_LENGTH

The size of the file.

long

CamelFileLastModified (consumer)

Constant: FILE_LAST_MODIFIED

The upload date of the file.

Date

CamelFileName (common)

Constant: FILE_NAME

(producer) The name of the file: the name that create gives the file, and the file that findOne, remove, listAll and count look for. (consumer) The name of the file that was found.

String

Content-Type (producer)

Constant: CONTENT_TYPE

The content type of the file.

String

CamelFileNameProduced (producer)

Constant: FILE_NAME_PRODUCED

The file name produced.

String

CamelGridFsMetadata (common)

Constant: GRIDFS_METADATA

Any additional metadata stored along with the file in JSON format.

String

CamelGridFsOperation (producer)

Constant: GRIDFS_OPERATION

The operation to perform.

String

CamelGridFsChunkSize (producer)

Constant: GRIDFS_CHUNKSIZE

The number of bytes per chunk for the uploaded file.

Integer

CamelGridFsFileId (producer)

Constant: GRIDFS_FILE_ID_PRODUCED

The ObjectId of the file produced.

ObjectId

CamelGridFsObjectId (producer)

Constant: GRIDFS_OBJECT_ID

The ObjectId of the file.

ObjectId

Message Body

The message body at each side of the MongoDB GridFS component: Consumer is the body of the messages the consumer creates, Consumer reply the bodies a consumer that answers its caller can reply with, Producer input the bodies the producer can send or write, and Producer output the body after the producer.

Side Body Description

Consumer

stream

One message per new file: a stream of the content of the file (the stream cache when stream caching is on), closed when the exchange is done, with its name, length, upload date, content type and metadata in headers.

Producer input

text, bytes, stream or file

The content of the file that create, the default operation, uploads under the name in the CamelFileName header: text, bytes, a stream or a file. A Map, a POJO or no body fail. The other operations do not use the body; the CamelGridFsOperation header sets the operation when the option is not set.

  • With operation=findOne: any

  • With operation=listAll: any

  • With operation=count: any

  • With operation=remove: any

Producer output

unchanged

create, the default, leaves the body unchanged, with the id of the file in the CamelGridFsFileId header. findOne gives a stream of the content of the file named by CamelFileName; listAll a Reader of text, one line per file with its name and id (the hex string of an ObjectId) separated by a tab; count the number of files as a Long; remove leaves the body unchanged.

  • With operation=findOne: stream

  • With operation=listAll: stream

  • With operation=count: value (java.lang.Long)

  • With operation=remove: unchanged

The bodies are the ones with the default options. A side that is not listed is not known, and tooling does not guess it. The carriers, the sides and the tools that use them are described in Message Body Types.

Usage

GridFS operations - producer endpoint

The operation option, or the CamelGridFsOperation header when the option is not set, chooses the operation: create (the default), remove, findOne, listAll or count. Another operation fails the exchange with an IllegalArgumentException that lists the supported operations.

count

Returns the total number of files in the collection, returning a Long as the message body.

Java-only: using ProducerTemplate to invoke the count operation
// from("direct:count").to("mongodb-gridfs?database=tickets&operation=count");
Long result = template.requestBody("direct:count", "irrelevantBody", Long.class);

You can provide a filename header to provide a count of files matching that filename.

Java-only: using ProducerTemplate with filename header
Map<String, Object> headers = new HashMap<String, Object>();
headers.put(Exchange.FILE_NAME, "filename.txt");
Long count = template.requestBodyAndHeaders("direct:count", "irrelevantBody", headers, Long.class);

listAll

Returns a Reader that lists all the filenames and their IDs in a tab separated stream. The ID of a file is the hex string of its ObjectId.

// from("direct:listAll").to("mongodb-gridfs?database=tickets&operation=listAll");
Reader result = template.requestBodyAndHeader("direct:listAll", "irrelevantBody");

filename1.txt   665f1c2e8b3e4a5d6c7b8a90
filename2.txt   665f1c2e8b3e4a5d6c7b8a91

findOne

Finds a file in the GridFS system and sets the body to an InputStream of the content. Also provides the metadata has headers. It uses Exchange.FILE_NAME from the incoming headers to determine the file to find.

Java-only: using ProducerTemplate to find a file by name
// from("direct:findOne").to("mongodb-gridfs?database=tickets&operation=findOne");
Map<String, Object> headers = new HashMap<String, Object>();
headers.put(Exchange.FILE_NAME, "filename.txt");
InputStream result = template.requestBodyAndHeaders("direct:findOne", "irrelevantBody", headers);

create

Create a new file in the GridFs database. It uses the Exchange.FILE_NAME from the incoming headers for the name and the body contents (as an InputStream) as the content.

Java-only: using ProducerTemplate to create a file
// from("direct:create").to("mongodb-gridfs?database=tickets&operation=create");
Map<String, Object> headers = new HashMap<String, Object>();
headers.put(Exchange.FILE_NAME, "filename.txt");
InputStream stream = ... the data for the file ...
template.requestBodyAndHeaders("direct:create", stream, headers);

remove

Removes a file from the GridFS database.

Java-only: using ProducerTemplate to remove a file
// from("direct:remove").to("mongodb-gridfs?database=tickets&operation=remove");
Map<String, Object> headers = new HashMap<String, Object>();
headers.put(Exchange.FILE_NAME, "filename.txt");
template.requestBodyAndHeaders("direct:remove", "", headers);

GridFS consumer

The consumer polls the bucket for new files, as the queryStrategy option chooses, and routes one message per file. The body is a stream of the content of the file, which is closed when the exchange is done. The CamelFileName, CamelFileLength, CamelFileLastModified (the upload date), CamelFileContentType and CamelGridFsMetadata headers describe the file.

When the exchange of a file fails, the consumer reports the failure to its exception handler (which logs it, unless the exceptionHandler or bridgeErrorHandler option is set) and leaves the file as it was, so that the next poll offers it again:

  • With a file attribute (FileAttribute, TimeStampAndFileAttribute and PersistentTimestampAndFileAttribute), the file is not marked as processed, and the poll goes on with the other files.

  • With TimeStamp and PersistentTimestamp, the files are routed in the order of their upload date, and the poll stops at the file that failed: the timestamp stays before that file, so the next poll offers it again, and then the files uploaded after it. A file that always fails holds back the files after it.

Only the first failure of a file is reported to the exception handler: when the file fails again at the next poll, the consumer logs the failure at DEBUG. A file is reported again when it fails after a poll that routed it or did not offer it.

A failure that the route handles, such as with a dead letter channel or onException with handled(true), does not fail the exchange: the file is then done.

Examples

Example route

The following route defined in Spring XML executes the operation findOne on a collection.

Get a file from GridFS

  • Java

  • XML

  • YAML

from("direct:start")
    .to("mongodb-gridfs:mongoBean?database={{mongodb.database}}&operation=findOne")
    .to("direct:result");
<route>
  <from uri="direct:start" />
  <!-- using bean 'mongoBean' defined above -->
  <to uri="mongodb-gridfs:mongoBean?database={{mongodb.database}}&amp;operation=findOne" />
  <to uri="direct:result" />
</route>
- route:
    from:
      uri: direct:start
      steps:
        - to:
            uri: mongodb-gridfs:mongoBean
            parameters:
              database: "{{mongodb.database}}"
              operation: findOne
        - to:
            uri: direct:result

Configuration of a database in Spring XML

The following Spring XML creates a bean defining the connection to a MongoDB instance.

<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
    <bean id="mongoBean" class="com.mongodb.Mongo">
        <constructor-arg name="host" value="${mongodb.host}" />
        <constructor-arg name="port" value="${mongodb.port}" />
    </bean>
</beans>