Google Storage

Since Camel 3.9

Both producer and consumer are supported

The Google Storage component provides access to Google Cloud Storage via the google java storage library.

Maven users will need to add the following dependency to their pom.xml for this component:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-google-storage</artifactId>
    <!-- use the same version as your Camel core version -->
    <version>x.x.x</version>
</dependency>

Authentication Configuration

Google Storage component authentication is targeted for use with the GCP Service Accounts. For more information please refer to Google Storage Auth Guide.

When you have the service account key you can provide authentication credentials to your application code. Google security credentials can be set through the component endpoint:

String endpoint = "google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json";

Or by providing the path to the GCP credentials file location:

Provide authentication credentials to your application code by setting the environment variable GOOGLE_APPLICATION_CREDENTIALS :

export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/my-key.json"

URI Format

google-storage://bucketNameOrArn?[options]

By default the bucket will be created if it don’t already exists.
You can append query options to the URI in the following format, ?options=value&option2=value&…​

For example in order to read file hello.txt from bucket myCamelBucket, use the following snippet:

from("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json&objectName=hello.txt")
  .to("file:/var/downloaded");

URI Options

The Google Storage component supports 16 options, which are listed below.

Name Description Default Type

autoCreateBucket (common)

Setting the autocreation of the bucket bucketName.

true

boolean

configuration (common)

The component configuration

GoogleCloudStorageConfiguration

serviceAccountKey (common)

Service account key to authenticate an application as a service account

String

storageClass (common)

The Cloud Storage class to use when creating the new buckets

STANDARD

StorageClass

storageClient (common)

Autowired The storage client

Storage

storageLocation (common)

The Cloud Storage location to use when creating the new buckets

US-EAST1

String

bridgeErrorHandler (consumer)

Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored.

false

boolean

deleteAfterRead (consumer)

Delete objects from the bucket after they have been retrieved. The delete is only performed if the Exchange is committed. If a rollback occurs, the object is not deleted. If this option is false, then the same objects will be retrieve over and over again on the polls.

true

boolean

destinationBucket (consumer)

Define the destination bucket where an object must be moved when moveAfterRead is set to true.

String

includeBody (consumer)

If it is true, the Object exchange will be consumed and put into the body. If false the Object stream will be put raw into the body and the headers will be set with the object metadata.

true

boolean

includeFolders (consumer)

If it is true, the folders/directories will be consumed. If it is false, they will be ignored, and Exchanges will not be created for those

true

boolean

moveAfterRead (consumer)

Move objects from the origin bucket to a different bucket after they have been retrieved. To accomplish the operation the destinationBucket option must be set. The copy bucket operation is only performed if the Exchange is committed. If a rollback occurs, the object is not moved.

false

boolean

lazyStartProducer (producer)

Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing.

false

boolean

objectName (producer)

The Object name inside the bucket

String

operation (producer)

Set the operation for the producer. There are 7 enums and the value can be one of: copyObject, listObjects, deleteObject, deleteBucket, listBuckets, getObject, createDownloadLink

GoogleCloudStorageOperations

autowiredEnabled (advanced)

Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc.

true

boolean

The Google Storage endpoint is configured using URI syntax:

google-storage:bucketName

with the following path and query parameters:

Path Parameters (1 parameters):

Name Description Default Type

bucketName

Required Bucket name or ARN

String

Query Parameters (32 parameters):

Name Description Default Type

autoCreateBucket (common)

Setting the autocreation of the bucket bucketName.

true

boolean

serviceAccountKey (common)

Service account key to authenticate an application as a service account

String

storageClass (common)

The Cloud Storage class to use when creating the new buckets

STANDARD

StorageClass

storageClient (common)

Autowired The storage client

Storage

storageLocation (common)

The Cloud Storage location to use when creating the new buckets

US-EAST1

String

bridgeErrorHandler (consumer)

Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored.

false

boolean

deleteAfterRead (consumer)

Delete objects from the bucket after they have been retrieved. The delete is only performed if the Exchange is committed. If a rollback occurs, the object is not deleted. If this option is false, then the same objects will be retrieve over and over again on the polls.

true

boolean

destinationBucket (consumer)

Define the destination bucket where an object must be moved when moveAfterRead is set to true.

String

includeBody (consumer)

If it is true, the Object exchange will be consumed and put into the body. If false the Object stream will be put raw into the body and the headers will be set with the object metadata.

true

boolean

includeFolders (consumer)

If it is true, the folders/directories will be consumed. If it is false, they will be ignored, and Exchanges will not be created for those

true

boolean

moveAfterRead (consumer)

Move objects from the origin bucket to a different bucket after they have been retrieved. To accomplish the operation the destinationBucket option must be set. The copy bucket operation is only performed if the Exchange is committed. If a rollback occurs, the object is not moved.

false

boolean

sendEmptyMessageWhenIdle (consumer)

If the polling consumer did not poll any files, you can enable this option to send an empty message (no body) instead.

false

boolean

exceptionHandler (consumer)

To let the consumer use a custom ExceptionHandler. Notice if the option bridgeErrorHandler is enabled then this option is not in use. By default the consumer will deal with exceptions, that will be logged at WARN or ERROR level and ignored.

ExceptionHandler

exchangePattern (consumer)

Sets the exchange pattern when the consumer creates an exchange. There are 3 enums and the value can be one of: InOnly, InOut, InOptionalOut

ExchangePattern

pollStrategy (consumer)

A pluggable org.apache.camel.PollingConsumerPollingStrategy allowing you to provide your custom implementation to control error handling usually occurred during the poll operation before an Exchange have been created and being routed in Camel.

PollingConsumerPollStrategy

lazyStartProducer (producer)

Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing.

false

boolean

objectName (producer)

The Object name inside the bucket

String

operation (producer)

Set the operation for the producer. There are 7 enums and the value can be one of: copyObject, listObjects, deleteObject, deleteBucket, listBuckets, getObject, createDownloadLink

GoogleCloudStorageOperations

backoffErrorThreshold (scheduler)

The number of subsequent error polls (failed due some error) that should happen before the backoffMultipler should kick-in.

int

backoffIdleThreshold (scheduler)

The number of subsequent idle polls that should happen before the backoffMultipler should kick-in.

int

backoffMultiplier (scheduler)

To let the scheduled polling consumer backoff if there has been a number of subsequent idles/errors in a row. The multiplier is then the number of polls that will be skipped before the next actual attempt is happening again. When this option is in use then backoffIdleThreshold and/or backoffErrorThreshold must also be configured.

int

delay (scheduler)

Milliseconds before the next poll.

500

long

greedy (scheduler)

If greedy is enabled, then the ScheduledPollConsumer will run immediately again, if the previous run polled 1 or more messages.

false

boolean

initialDelay (scheduler)

Milliseconds before the first poll starts.

1000

long

repeatCount (scheduler)

Specifies a maximum limit of number of fires. So if you set it to 1, the scheduler will only fire once. If you set it to 5, it will only fire five times. A value of zero or negative means fire forever.

0

long

runLoggingLevel (scheduler)

The consumer logs a start/complete log line when it polls. This option allows you to configure the logging level for that. There are 6 enums and the value can be one of: TRACE, DEBUG, INFO, WARN, ERROR, OFF

TRACE

LoggingLevel

scheduledExecutorService (scheduler)

Allows for configuring a custom/shared thread pool to use for the consumer. By default each consumer has its own single threaded thread pool.

ScheduledExecutorService

scheduler (scheduler)

To use a cron scheduler from either camel-spring or camel-quartz component. Use value spring or quartz for built in scheduler

none

Object

schedulerProperties (scheduler)

To configure additional properties when using a custom scheduler or any of the Quartz, Spring based scheduler.

Map

startScheduler (scheduler)

Whether the scheduler should be auto started.

true

boolean

timeUnit (scheduler)

Time unit for initialDelay and delay options. There are 7 enums and the value can be one of: NANOSECONDS, MICROSECONDS, MILLISECONDS, SECONDS, MINUTES, HOURS, DAYS

MILLISECONDS

TimeUnit

useFixedDelay (scheduler)

Controls if fixed delay or fixed rate is used. See ScheduledExecutorService in JDK for details.

true

boolean

Usage

Message headers evaluated by the Google Storage Producer

Header Type Description

CamelGoogleCloudStorageBucketName

String

The bucket Name which this object will be stored or which will be used for the current operation

CamelGoogleCloudStorageObjectName

String

The object Name which will be used for the current operation

CamelGoogleCloudStorageDestinationBucketName

String

The bucket Destination Name which will be used for the current operation

CamelGoogleCloudStorageDestinationObjectName

String

The object Destination Name which will be used for the current operation

CamelGoogleCloudStorageContentLength

Long

The content length of this object.

CamelGoogleCloudStorageContentType

String

The content type of this object.

CamelGoogleCloudStorageContentDisposition

String

The content disposition of this object.

CamelGoogleCloudStorageContentEncoding

String

The content encoding of this object.

CamelGoogleCloudStorageContentMd5

String

The md5 checksum of this object.

CamelGoogleCloudStorageOperation

String

The operation to perform. Permitted values are copyObject, listObjects, deleteObject, deleteBucket, listBuckets, getObject, createDownloadLink

CamelGoogleCloudStorageDownloadLinkExpirationTime

Long

The time in millisecond the download link will be valid.

Message headers set by the Google Storage Producer

Header Type Description

CamelGoogleCloudStorageETag

String

The ETag value for the newly uploaded object.

Message headers set by the Google Storage Consumer

Header Type Description

CamelGoogleCloudStorageBucketName

String

The bucket Name which this object will be stored or which will be used for the current operation

CamelGoogleCloudStorageObjectName

String

The object Name which will be used for the current operation

CamelGoogleCloudStorageCacheControl

String

The Cache-Control metadata can specify two different aspects of how data is served from Cloud Storage: whether the data can be cached and whether the data can be transformed

CamelGoogleCloudStorageComponentCount

String

The component count of this object

CamelGoogleCloudStorageContentDisposition

String

The content disposition of this object.

CamelGoogleCloudStorageContentEncoding

String

The content encoding of this object.

CamelGoogleCloudStorageContentLanguage

String

The Content-Language metadata indicates the language(s) that the object is intended for.

CamelGoogleCloudStorageContentType

String

The content type of this object.

CamelGoogleCloudStorageCustomTime

String

The Custom-Time metadata is a user-specified date and time represented in the RFC 3339 format YYYY-MM-DD’T’HH:MM:SS.SS’Z' or YYYY-MM-DD’T’HH:MM:SS’Z' when milliseconds are zero. This metadata is typically set in order to use the DaysSinceCustomTime condition in Object Lifecycle Management.

CamelGoogleCloudStorageCrc32cHex

String

The CRC32c of the object

CamelGoogleCloudStorageETag

String

The ETag for the Object.

CamelGoogleCloudStorageGeneration

String

Is the generation number of the object for which you are retrieving information.

CamelGoogleCloudStorageBlobId

String

The blob id of the object

CamelGoogleCloudStorageKmsKeyName

String

The KMS key name

CamelGoogleCloudStorageContentMd5

String

The md5 checksum of this object.

CamelGoogleCloudStorageMediaLink

String

The media link

CamelGoogleCloudStorageMetageneration

String

The metageneration of the object

CamelGoogleCloudStorageContentLength

Long

The content length of this object.

CamelGoogleCloudStorageStorageClass

String

The storage class of the object

CamelGoogleCloudStorageCreateTime

String

The creation time of the object

CamelGoogleCloudStorageLastUpdate

String

The last update of the object

Google Storage Producer operations

Google Storage component provides the following operation on the producer side:

  • copyObject

  • listObjects

  • deleteObject

  • deleteBucket

  • listBuckets

  • getObject

  • createDownloadLink

If you don’t specify an operation explicitly the producer will a file upload.

Advanced component configuration

If you need to have more control over the storageClient instance configuration, you can create your own instance and refer to it in your Camel google-storage component configuration:

from("google-storage://myCamelBucket?storageClient=#client")
.to("mock:result");

Google Storage Producer Operation examples

  • File Upload: This operation will upload a file to the Google Storage based on the body content

//upload a file
byte[] payload = "Camel rocks!".getBytes();
ByteArrayInputStream bais = new ByteArrayInputStream(payload);
from("direct:start")
.process( exchange -> {
    exchange.getIn().setHeader(GoogleCloudStorageConstants.OBJECT_NAME, "camel.txt");
    exchange.getIn().setBody(bais);
})
.to("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json")
.log("uploaded file object:${header.CamelGoogleCloudStorageObjectName}, body:${body}");

This operation will upload the file camel.txt with the content "Camel rocks!" in the myCamelBucket bucket

  • CopyObject: this operation copy an object from one bucket to a different one

  from("direct:start").process( exchange -> {
    exchange.getIn().setHeader(GoogleCloudStorageConstants.OPERATION, GoogleCloudStorageOperations.copyObject);
    exchange.getIn().setHeader(GoogleCloudStorageConstants.OBJECT_NAME, "camel.txt" );
    exchange.getIn().setHeader(GoogleCloudStorageConstants.DESTINATION_BUCKET_NAME, "myCamelBucket_dest");
    exchange.getIn().setHeader(GoogleCloudStorageConstants.DESTINATION_OBJECT_NAME, "camel_copy.txt");
  })
  .to("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json")
  .to("mock:result");

This operation will copy the object with the name expressed in the header DESTINATION_OBJECT_NAME to the DESTINATION_BUCKET_NAME bucket, from the bucket myCamelBucket.

  • DeleteObject: this operation deletes an object from a bucket

  from("direct:start").process( exchange -> {
    exchange.getIn().setHeader(GoogleCloudStorageConstants.OPERATION, GoogleCloudStorageOperations.deleteObject);
    exchange.getIn().setHeader(GoogleCloudStorageConstants.OBJECT_NAME, "camel.txt" );
  })
  .to("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json")
  .to("mock:result");

This operation will delete the object from the bucket myCamelBucket.

  • ListBuckets: this operation list the buckets for this account in this region

from("direct:start")
.to("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json&operation=listBuckets")
.to("mock:result");

This operation will list the buckets for this account.

  • DeleteBucket: this operation delete the bucket specified as URI parameter or header

from("direct:start")
.to("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json&operation=deleteBucket")
.to("mock:result");

This operation will delete the bucket myCamelBucket.

  • ListObjects: this operation list object in a specific bucket

from("direct:start")
.to("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json&operation=listObjects")
.to("mock:result");

This operation will list the objects in the myCamelBucket bucket.

  • GetObject: this operation get a single object in a specific bucket

from("direct:start")
.process( exchange -> {
  exchange.getIn().setHeader(GoogleCloudStorageConstants.OBJECT_NAME, "camel.txt");
})
.to("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json&operation=getObject")
.to("mock:result");

This operation will return an Blob objct instance related to the OBJECT_NAME object in myCamelBucket bucket.

  • CreateDownloadLink: this operation will return a download link

from("direct:start")
.process( exchange -> {
  exchange.getIn().setHeader(GoogleCloudStorageConstants.OBJECT_NAME, "camel.txt" );
  exchange.getIn().setHeader(GoogleCloudStorageConstants.DOWNLOAD_LINK_EXPIRATION_TIME, 86400000L); //1 day
})
.to("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json&operation=createDownloadLink")
.to("mock:result");

This operation will return a download link url for the file OBJECT_NAME in the bucket myCamelBucket. It’s possible to specify the expiration time for the created link through the header DOWNLOAD_LINK_EXPIRATION_TIME. If not specified, by default it is 5 minutes.

Bucket Autocreation

With the option autoCreateBucket users are able to avoid the autocreation of a Bucket in case it doesn’t exist. The default for this option is true. If set to false any operation on a not-existent bucket won’t be successful and an error will be returned.

MoveAfterRead consumer option

In addition to deleteAfterRead it has been added another option, moveAfterRead. With this option enabled the consumed object will be moved to a target destinationBucket instead of being only deleted. This will require specifying the destinationBucket option. As example:

  from("google-storage://myCamelBucket?serviceAccountKey=/home/user/Downloads/my-key.json"
    + "&autoCreateBucket=true"
    + "&destinationBucket=myCamelProcessedBucket"
    + "&moveAfterRead=true"
    + "&deleteAfterRead=true"
    + "&includeBody=true"
  )
  .to("mock:result");

In this case the objects consumed will be moved to myCamelProcessedBucket bucket and deleted from the original one (because of deleteAfterRead).