Azure Files

Since Camel 3.22

Both producer and consumer are supported

This component provides access to Azure Files.

A preview component so anything can change in a next release or it could be even dropped. At the same time it is consolidated enough, sparingly documented, a few users reported it was working in their environment, and it is ready for wider feedback.

When consuming from remote files server make sure you read the section titled Consuming Files further below for details related to consuming files.

Maven users will need to add the following dependency to their pom.xml for this component:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-azure-files</artifactId>
    <version>x.y.z</version>
    <!-- use the same version as your Camel core version -->
</dependency>

Endpoint URI Format

azure-files://account[.file.core.windows.net][:port]/share[/directory]

Where directory represents the underlying directory. The directory is a relative path and does not include the share name. The relative path can contain nested folders, such as inbox/spam. It defaults to the share root directory.

The autoCreate option is supported for the directory, when consumer or producer starts, there’s an additional operation performed to create the directory configured for the endpoint. The default value for autoCreate is true. On contrary, the share must exist, it is not automatically created.

If no port number is provided, Camel will provide default values according to the protocol (https 443).

You can append query options to the URI in the following format ?option=value&option2=value&…​.

The SAS token or shared key based auth is supported, the options for selected one must be present.

Configuring Options

Camel components are configured on two separate levels:

  • component level

  • endpoint level

Configuring Component Options

The component level is the highest level which holds general and common configurations that are inherited by the endpoints. For example a component may have security settings, credentials for authentication, urls for network connection and so forth.

Some components only have a few options, and others may have many. Because components typically have pre configured defaults that are commonly used, then you may often only need to configure a few options on a component; or none at all.

Configuring components can be done with the Component DSL, in a configuration file (application.properties|yaml), or directly with Java code.

Configuring Endpoint Options

Where you find yourself configuring the most is on endpoints, as endpoints often have many options, which allows you to configure what you need the endpoint to do. The options are also categorized into whether the endpoint is used as consumer (from) or as a producer (to), or used for both.

Configuring endpoints is most often done directly in the endpoint URI as path and query parameters. You can also use the Endpoint DSL and DataFormat DSL as a type safe way of configuring endpoints and data formats in Java.

A good practice when configuring options is to use Property Placeholders, which allows to not hardcode urls, port numbers, sensitive information, and other settings. In other words placeholders allows to externalize the configuration from your code, and gives more flexibility and reuse.

The following two sections lists all the options, firstly for the component followed by the endpoint.

Component Options

The Azure Files component supports 5 options, which are listed below.

Name Description Default Type

bridgeErrorHandler (consumer)

Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored.

false

boolean

lazyStartProducer (producer)

Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing.

false

boolean

autowiredEnabled (advanced)

Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc.

true

boolean

healthCheckConsumerEnabled (health)

Used for enabling or disabling all consumer based health checks from this component.

true

boolean

healthCheckProducerEnabled (health)

Used for enabling or disabling all producer based health checks from this component. Notice: Camel has by default disabled all producer based health-checks. You can turn on producer checks globally by setting camel.health.producersEnabled=true.

true

boolean

Endpoint Options

The Azure Files endpoint is configured using URI syntax:

azure-files:account/share

with the following path and query parameters:

Path Parameters (4 parameters)

Name Description Default Type

account (common)

Required The account to use.

String

share (common)

Required The share to use.

String

port (common)

Port of the FTP server.

int

directoryName (common)

The starting directory.

String

Query Parameters (97 parameters)

Name Description Default Type

disconnect (common)

Whether or not to disconnect from remote FTP server right after use. Disconnect will only disconnect the current connection to the FTP server. If you have a consumer which you want to stop, then you need to stop the consumer/route instead.

false

boolean

doneFileName (common)

Producer: If provided, then Camel will write a 2nd done file when the original file has been written. The done file will be empty. This option configures what file name to use. Either you can specify a fixed name. Or you can use dynamic placeholders. The done file will always be written in the same folder as the original file. Consumer: If provided, Camel will only consume files if a done file exists. This option configures what file name to use. Either you can specify a fixed name. Or you can use dynamic placeholders.The done file is always expected in the same folder as the original file. Only $\{file.name} and $\{file.name.next} is supported as dynamic placeholders.

String

fileName (common)

Use Expression such as File Language to dynamically set the filename. For consumers, it’s used as a filename filter. For producers, it’s used to evaluate the filename to write. If an expression is set, it take precedence over the CamelFileName header. (Note: The header itself can also be an Expression). The expression options support both String and Expression types. If the expression is a String type, it is always evaluated using the File Language. If the expression is an Expression type, the specified Expression type is used - this allows you, for instance, to use OGNL expressions. For the consumer, you can use it to filter filenames, so you can for instance consume today’s file using the File Language syntax: mydata-$\{date:now:yyyyMMdd}.txt. The producers support the CamelOverruleFileName header which takes precedence over any existing CamelFileName header; the CamelOverruleFileName is a header that is used only once, and makes it easier as this avoids to temporary store CamelFileName and have to restore it afterwards.

String

delete (consumer)

If true, the file will be deleted after it is processed successfully.

false

boolean

moveFailed (consumer)

Sets the move failure expression based on Simple language. For example, to move files into a .error subdirectory use: .error. Note: When moving the files to the fail location Camel will handle the error and will not pick up the file again.

String

noop (consumer)

If true, the file is not moved or deleted in any way. This option is good for readonly data, or for ETL type requirements. If noop=true, Camel will set idempotent=true as well, to avoid consuming the same files over and over again.

false

boolean

preMove (consumer)

Expression (such as File Language) used to dynamically set the filename when moving it before processing. For example to move in-progress files into the order directory set this value to order.

String

preSort (consumer)

When pre-sort is enabled then the consumer will sort the file and directory names during polling, that was retrieved from the file system. You may want to do this in case you need to operate on the files in a sorted order. The pre-sort is executed before the consumer starts to filter, and accept files to process by Camel. This option is default=false meaning disabled.

false

boolean

recursive (consumer)

If a directory, will look for files in all the sub-directories as well.

false

boolean

resumeDownload (consumer)

Configures whether resume download is enabled. In addition the options localWorkDirectory must be configured so downloaded files are stored in a local directory, which is required to support resuming of downloads.

false

boolean

sendEmptyMessageWhenIdle (consumer)

If the polling consumer did not poll any files, you can enable this option to send an empty message (no body) instead.

false

boolean

streamDownload (consumer)

Sets the download method to use when not using a local working directory. If set to true, the remote files are streamed to the route as they are read. When set to false, the remote files are loaded into memory before being sent into the route. If enabling this option then you must set stepwise=false as both cannot be enabled at the same time.

false

boolean

bridgeErrorHandler (consumer (advanced))

Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored.

false

boolean

download (consumer (advanced))

Whether the FTP consumer should download the file. If this option is set to false, then the message body will be null, but the consumer will still trigger a Camel Exchange that has details about the file such as file name, file size, etc. It’s just that the file will not be downloaded.

false

boolean

exceptionHandler (consumer (advanced))

To let the consumer use a custom ExceptionHandler. Notice if the option bridgeErrorHandler is enabled then this option is not in use. By default the consumer will deal with exceptions, that will be logged at WARN or ERROR level and ignored.

ExceptionHandler

exchangePattern (consumer (advanced))

Sets the exchange pattern when the consumer creates an exchange.

Enum values:

  • InOnly

  • InOut

ExchangePattern

inProgressRepository (consumer (advanced))

A pluggable in-progress repository org.apache.camel.spi.IdempotentRepository. The in-progress repository is used to account the current in progress files being consumed. By default a memory based repository is used.

IdempotentRepository

localWorkDirectory (consumer (advanced))

When consuming, a local work directory can be used to store the remote file content directly in local files, to avoid loading the content into memory. This is beneficial, if you consume a very big remote file and thus can conserve memory.

String

onCompletionExceptionHandler (consumer (advanced))

To use a custom org.apache.camel.spi.ExceptionHandler to handle any thrown exceptions that happens during the file on completion process where the consumer does either a commit or rollback. The default implementation will log any exception at WARN level and ignore.

ExceptionHandler

pollStrategy (consumer (advanced))

A pluggable org.apache.camel.PollingConsumerPollingStrategy allowing you to provide your custom implementation to control error handling usually occurred during the poll operation before an Exchange have been created and being routed in Camel.

PollingConsumerPollStrategy

processStrategy (consumer (advanced))

A pluggable org.apache.camel.component.file.GenericFileProcessStrategy allowing you to implement your own readLock option or similar. Can also be used when special conditions must be met before a file can be consumed, such as a special ready file exists. If this option is set then the readLock option does not apply.

GenericFileProcessStrategy

fileExist (producer)

What to do if a file already exists with the same name. Override, which is the default, replaces the existing file. - Append - adds content to the existing file. - Fail - throws a GenericFileOperationException, indicating that there is already an existing file. - Ignore - silently ignores the problem and does not override the existing file, but assumes everything is okay. - Move - option requires to use the moveExisting option to be configured as well. The option eagerDeleteTargetFile can be used to control what to do if an moving the file, and there exists already an existing file, otherwise causing the move operation to fail. The Move option will move any existing files, before writing the target file. - TryRename is only applicable if tempFileName option is in use. This allows to try renaming the file from the temporary name to the actual name, without doing any exists check. This check may be faster on some file systems and especially FTP servers.

Enum values:

  • Override

  • Append

  • Fail

  • Ignore

  • Move

  • TryRename

Override

GenericFileExist

flatten (producer)

Flatten is used to flatten the file name path to strip any leading paths, so it’s just the file name. This allows you to consume recursively into sub-directories, but when you eg write the files to another directory they will be written in a single directory. Setting this to true on the producer enforces that any file name in CamelFileName header will be stripped for any leading paths.

false

boolean

jailStartingDirectory (producer)

Used for jailing (restricting) writing files to the starting directory (and sub) only. This is enabled by default to not allow Camel to write files to outside directories (to be more secured out of the box). You can turn this off to allow writing files to directories outside the starting directory, such as parent or root folders.

true

boolean

tempFileName (producer)

The same as tempPrefix option but offering a more fine grained control on the naming of the temporary filename as it uses the File Language. The location for tempFilename is relative to the final file location in the option 'fileName', not the target directory in the base uri. For example if option fileName includes a directory prefix: dir/finalFilename then tempFileName is relative to that subdirectory dir.

String

tempPrefix (producer)

This option is used to write the file using a temporary name and then, after the write is complete, rename it to the real name. Can be used to identify files being written and also avoid consumers (not using exclusive read locks) reading in progress files. Is often used by FTP when uploading big files.

String

allowNullBody (producer (advanced))

Used to specify if a null body is allowed during file writing. If set to true then an empty file will be created, when set to false, and attempting to send a null body to the file component, a GenericFileWriteException of 'Cannot write null body to file.' will be thrown. If the fileExist option is set to 'Override', then the file will be truncated, and if set to append the file will remain unchanged.

false

boolean

disconnectOnBatchComplete (producer (advanced))

Whether or not to disconnect from remote FTP server right after a Batch upload is complete. disconnectOnBatchComplete will only disconnect the current connection to the FTP server.

false

boolean

eagerDeleteTargetFile (producer (advanced))

Whether or not to eagerly delete any existing target file. This option only applies when you use fileExists=Override and the tempFileName option as well. You can use this to disable (set it to false) deleting the target file before the temp file is written. For example you may write big files and want the target file to exists during the temp file is being written. This ensure the target file is only deleted until the very last moment, just before the temp file is being renamed to the target filename. This option is also used to control whether to delete any existing files when fileExist=Move is enabled, and an existing file exists. If this option copyAndDeleteOnRenameFails false, then an exception will be thrown if an existing file existed, if its true, then the existing file is deleted before the move operation.

true

boolean

keepLastModified (producer (advanced))

Will keep the last modified timestamp from the source file (if any). Will use the FileConstants.FILE_LAST_MODIFIED header to located the timestamp. This header can contain either a java.util.Date or long with the timestamp. If the timestamp exists and the option is enabled it will set this timestamp on the written file. Note: This option only applies to the file producer. You cannot use this option with any of the ftp producers.

false

boolean

lazyStartProducer (producer (advanced))

Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing.

false

boolean

moveExistingFileStrategy (producer (advanced))

Strategy (Custom Strategy) used to move file with special naming token to use when fileExist=Move is configured. By default, there is an implementation used if no custom strategy is provided.

FileMoveExistingStrategy

autoCreate (advanced)

Automatically create missing directories in the file’s pathname. For the file consumer, that means creating the starting directory. For the file producer, it means the directory the files should be written to.

true

boolean

connectTimeout (advanced)

Sets the connect timeout for waiting for a connection to be established Used by both FTPClient and JSCH.

10000

int

maximumReconnectAttempts (advanced)

Specifies the maximum reconnect attempts Camel performs when it tries to connect to the remote FTP server. Use 0 to disable this behavior.

int

reconnectDelay (advanced)

Delay in millis Camel will wait before performing a reconnect attempt.

1000

long

throwExceptionOnConnectFailed (advanced)

Should an exception be thrown if connection failed (exhausted)By default exception is not thrown and a WARN is logged. You can use this to enable exception being thrown and handle the thrown exception from the org.apache.camel.spi.PollingConsumerPollStrategy rollback method.

false

boolean

timeout (advanced)

Sets the data timeout for waiting for reply Used only by FTPClient.

30000

int

sharedKey (both)

Shared key (storage account key).

String

antExclude (filter)

Ant style filter exclusion. If both antInclude and antExclude are used, antExclude takes precedence over antInclude. Multiple exclusions may be specified in comma-delimited format.

String

antFilterCaseSensitive (filter)

Sets case sensitive flag on ant filter.

true

boolean

antInclude (filter)

Ant style filter inclusion. Multiple inclusions may be specified in comma-delimited format.

String

eagerMaxMessagesPerPoll (filter)

Allows for controlling whether the limit from maxMessagesPerPoll is eager or not. If eager then the limit is during the scanning of files. Where as false would scan all files, and then perform sorting. Setting this option to false allows for sorting all files first, and then limit the poll. Mind that this requires a higher memory usage as all file details are in memory to perform the sorting.

true

boolean

exclude (filter)

Is used to exclude files, if filename matches the regex pattern (matching is case in-sensitive). Notice if you use symbols such as plus sign and others you would need to configure this using the RAW() syntax if configuring this as an endpoint uri. See more details at configuring endpoint uris.

String

excludeExt (filter)

Is used to exclude files matching file extension name (case insensitive). For example to exclude bak files, then use excludeExt=bak. Multiple extensions can be separated by comma, for example to exclude bak and dat files, use excludeExt=bak,dat. Note that the file extension includes all parts, for example having a file named mydata.tar.gz will have extension as tar.gz. For more flexibility then use the include/exclude options.

String

filter (filter)

Pluggable filter as a org.apache.camel.component.file.GenericFileFilter class. Will skip files if filter returns false in its accept() method.

GenericFileFilter

filterDirectory (filter)

Filters the directory based on Simple language. For example to filter on current date, you can use a simple date pattern such as $\{date:now:yyyMMdd}.

String

filterFile (filter)

Filters the file based on Simple language. For example to filter on file size, you can use $\{file:size} 5000.

String

idempotent (filter)

Option to use the Idempotent Consumer EIP pattern to let Camel skip already processed files. Will by default use a memory based LRUCache that holds 1000 entries. If noop=true then idempotent will be enabled as well to avoid consuming the same files over and over again.

false

Boolean

idempotentKey (filter)

To use a custom idempotent key. By default the absolute path of the file is used. You can use the File Language, for example to use the file name and file size, you can do: idempotentKey=$\{file:name}-$\{file:size}.

String

idempotentRepository (filter)

A pluggable repository org.apache.camel.spi.IdempotentRepository which by default use MemoryIdempotentRepository if none is specified and idempotent is true.

IdempotentRepository

include (filter)

Is used to include files, if filename matches the regex pattern (matching is case in-sensitive). Notice if you use symbols such as plus sign and others you would need to configure this using the RAW() syntax if configuring this as an endpoint uri. See more details at configuring endpoint uris.

String

includeExt (filter)

Is used to include files matching file extension name (case insensitive). For example to include txt files, then use includeExt=txt. Multiple extensions can be separated by comma, for example to include txt and xml files, use includeExt=txt,xml. Note that the file extension includes all parts, for example having a file named mydata.tar.gz will have extension as tar.gz. For more flexibility then use the include/exclude options.

String

maxDepth (filter)

The maximum depth to traverse when recursively processing a directory.

2147483647

int

maxMessagesPerPoll (filter)

To define a maximum messages to gather per poll. By default no maximum is set. Can be used to set a limit of e.g. 1000 to avoid when starting up the server that there are thousands of files. Set a value of 0 or negative to disabled it. Notice: If this option is in use then the File and FTP components will limit before any sorting. For example if you have 100000 files and use maxMessagesPerPoll=500, then only the first 500 files will be picked up, and then sorted. You can use the eagerMaxMessagesPerPoll option and set this to false to allow to scan all files first and then sort afterwards.

int

minDepth (filter)

The minimum depth to start processing when recursively processing a directory. Using minDepth=1 means the base directory. Using minDepth=2 means the first sub directory.

int

move (filter)

Expression (such as Simple Language) used to dynamically set the filename when moving it after processing. To move files into a .done subdirectory just enter .done.

String

exclusiveReadLockStrategy (lock)

Pluggable read-lock as a org.apache.camel.component.file.GenericFileExclusiveReadLockStrategy implementation.

GenericFileExclusiveReadLockStrategy

readLock (lock)

Used by consumer, to only poll the files if it has exclusive read-lock on the file (i.e. the file is not in-progress or being written). Camel will wait until the file lock is granted. This option provides the build in strategies: - none - No read lock is in use - markerFile - Camel creates a marker file (fileName.camelLock) and then holds a lock on it. This option is not available for the FTP component - changed - Changed is using file length/modification timestamp to detect whether the file is currently being copied or not. Will at least use 1 sec to determine this, so this option cannot consume files as fast as the others, but can be more reliable as the JDK IO API cannot always determine whether a file is currently being used by another process. The option readLockCheckInterval can be used to set the check frequency. - fileLock - is for using java.nio.channels.FileLock. This option is not avail for Windows OS and the FTP component. This approach should be avoided when accessing a remote file system via a mount/share unless that file system supports distributed file locks. - rename - rename is for using a try to rename the file as a test if we can get exclusive read-lock. - idempotent - (only for file component) idempotent is for using a idempotentRepository as the read-lock. This allows to use read locks that supports clustering if the idempotent repository implementation supports that. - idempotent-changed - (only for file component) idempotent-changed is for using a idempotentRepository and changed as the combined read-lock. This allows to use read locks that supports clustering if the idempotent repository implementation supports that. - idempotent-rename - (only for file component) idempotent-rename is for using a idempotentRepository and rename as the combined read-lock. This allows to use read locks that supports clustering if the idempotent repository implementation supports that.Notice: The various read locks is not all suited to work in clustered mode, where concurrent consumers on different nodes is competing for the same files on a shared file system. The markerFile using a close to atomic operation to create the empty marker file, but its not guaranteed to work in a cluster. The fileLock may work better but then the file system need to support distributed file locks, and so on. Using the idempotent read lock can support clustering if the idempotent repository supports clustering, such as Hazelcast Component or Infinispan.

Enum values:

  • none

  • markerFile

  • fileLock

  • rename

  • changed

  • idempotent

  • idempotent-changed

  • idempotent-rename

none

String

readLockCheckInterval (lock)

Interval in millis for the read-lock, if supported by the read lock. This interval is used for sleeping between attempts to acquire the read lock. For example when using the changed read lock, you can set a higher interval period to cater for slow writes. The default of 1 sec. may be too fast if the producer is very slow writing the file. Notice: For FTP the default readLockCheckInterval is 5000. The readLockTimeout value must be higher than readLockCheckInterval, but a rule of thumb is to have a timeout that is at least 2 or more times higher than the readLockCheckInterval. This is needed to ensure that amble time is allowed for the read lock process to try to grab the lock before the timeout was hit.

1000

long

readLockDeleteOrphanLockFiles (lock)

Whether or not read lock with marker files should upon startup delete any orphan read lock files, which may have been left on the file system, if Camel was not properly shutdown (such as a JVM crash). If turning this option to false then any orphaned lock file will cause Camel to not attempt to pickup that file, this could also be due another node is concurrently reading files from the same shared directory.

true

boolean

readLockLoggingLevel (lock)

Logging level used when a read lock could not be acquired. By default a DEBUG is logged. You can change this level, for example to OFF to not have any logging. This option is only applicable for readLock of types: changed, fileLock, idempotent, idempotent-changed, idempotent-rename, rename.

Enum values:

  • TRACE

  • DEBUG

  • INFO

  • WARN

  • ERROR

  • OFF

DEBUG

LoggingLevel

readLockMarkerFile (lock)

Whether to use marker file with the changed, rename, or exclusive read lock types. By default a marker file is used as well to guard against other processes picking up the same files. This behavior can be turned off by setting this option to false. For example if you do not want to write marker files to the file systems by the Camel application.

true

boolean

readLockMinAge (lock)

This option is applied only for readLock=changed. It allows to specify a minimum age the file must be before attempting to acquire the read lock. For example use readLockMinAge=300s to require the file is at last 5 minutes old. This can speedup the changed read lock as it will only attempt to acquire files which are at least that given age.

0

long

readLockMinLength (lock)

This option is applied only for readLock=changed. It allows you to configure a minimum file length. By default Camel expects the file to contain data, and thus the default value is 1. You can set this option to zero, to allow consuming zero-length files.

1

long

readLockRemoveOnCommit (lock)

This option is applied only for readLock=idempotent. It allows to specify whether to remove the file name entry from the idempotent repository when processing the file is succeeded and a commit happens. By default the file is not removed which ensures that any race-condition do not occur so another active node may attempt to grab the file. Instead the idempotent repository may support eviction strategies that you can configure to evict the file name entry after X minutes - this ensures no problems with race conditions. See more details at the readLockIdempotentReleaseDelay option.

false

boolean

readLockRemoveOnRollback (lock)

This option is applied only for readLock=idempotent. It allows to specify whether to remove the file name entry from the idempotent repository when processing the file failed and a rollback happens. If this option is false, then the file name entry is confirmed (as if the file did a commit).

true

boolean

readLockTimeout (lock)

Optional timeout in millis for the read-lock, if supported by the read-lock. If the read-lock could not be granted and the timeout triggered, then Camel will skip the file. At next poll Camel, will try the file again, and this time maybe the read-lock could be granted. Use a value of 0 or lower to indicate forever. Currently fileLock, changed and rename support the timeout. Notice: For FTP the default readLockTimeout value is 20000 instead of 10000. The readLockTimeout value must be higher than readLockCheckInterval, but a rule of thumb is to have a timeout that is at least 2 or more times higher than the readLockCheckInterval. This is needed to ensure that amble time is allowed for the read lock process to try to grab the lock before the timeout was hit.

10000

long

backoffErrorThreshold (scheduler)

The number of subsequent error polls (failed due some error) that should happen before the backoffMultipler should kick-in.

int

backoffIdleThreshold (scheduler)

The number of subsequent idle polls that should happen before the backoffMultipler should kick-in.

int

backoffMultiplier (scheduler)

To let the scheduled polling consumer backoff if there has been a number of subsequent idles/errors in a row. The multiplier is then the number of polls that will be skipped before the next actual attempt is happening again. When this option is in use then backoffIdleThreshold and/or backoffErrorThreshold must also be configured.

int

delay (scheduler)

Milliseconds before the next poll.

500

long

greedy (scheduler)

If greedy is enabled, then the ScheduledPollConsumer will run immediately again, if the previous run polled 1 or more messages.

false

boolean

initialDelay (scheduler)

Milliseconds before the first poll starts.

1000

long

repeatCount (scheduler)

Specifies a maximum limit of number of fires. So if you set it to 1, the scheduler will only fire once. If you set it to 5, it will only fire five times. A value of zero or negative means fire forever.

0

long

runLoggingLevel (scheduler)

The consumer logs a start/complete log line when it polls. This option allows you to configure the logging level for that.

Enum values:

  • TRACE

  • DEBUG

  • INFO

  • WARN

  • ERROR

  • OFF

TRACE

LoggingLevel

scheduledExecutorService (scheduler)

Allows for configuring a custom/shared thread pool to use for the consumer. By default each consumer has its own single threaded thread pool.

ScheduledExecutorService

scheduler (scheduler)

To use a cron scheduler from either camel-spring or camel-quartz component. Use value spring or quartz for built in scheduler.

none

Object

schedulerProperties (scheduler)

To configure additional properties when using a custom scheduler or any of the Quartz, Spring based scheduler.

Map

startScheduler (scheduler)

Whether the scheduler should be auto started.

true

boolean

timeUnit (scheduler)

Time unit for initialDelay and delay options.

Enum values:

  • NANOSECONDS

  • MICROSECONDS

  • MILLISECONDS

  • SECONDS

  • MINUTES

  • HOURS

  • DAYS

MILLISECONDS

TimeUnit

useFixedDelay (scheduler)

Controls if fixed delay or fixed rate is used. See ScheduledExecutorService in JDK for details.

true

boolean

sdd (security)

part of service SAS token.

String

se (security)

part of SAS token.

String

si (security)

part of service SAS token.

String

sig (security)

part of SAS token.

String

sip (security)

part of SAS token.

String

sp (security)

part of SAS token.

String

spr (security)

part of SAS token.

String

sr (security)

part of service SAS token.

String

srt (security)

part of SAS token.

String

ss (security)

part of account SAS token.

String

st (security)

part of SAS token.

String

sv (security)

part of SAS token.

String

shuffle (sort)

To shuffle the list of files (sort in random order).

false

boolean

sortBy (sort)

Built-in sort by using the File Language. Supports nested sorts, so you can have a sort by file name and as a 2nd group sort by modified date.

String

sorter (sort)

Pluggable sorter as a java.util.Comparator class.

Comparator

Message Headers

The Azure Files component supports 8 message header(s), which is/are listed below:

Name Description Default Type

CamelFileLength (both)

Constant: FILE_LENGTH

A long value containing the file size. For producer, known length helps if the body converts to InputStream more efficiently than to bytes array.

long

CamelFileLastModified (consumer)

Constant: FILE_LAST_MODIFIED

A Long value containing the last modified timestamp of the file.

long

CamelFileName (common)

Constant: FILE_NAME

Specifies the output file name (relative to the endpoint directory) to be used for the output message when sending to the endpoint. If this is not present and no expression either, then a generated message ID is used as the filename instead.

String

CamelFileNameOnly (common)

Constant: FILE_NAME_ONLY

Only the file name (the name with no leading paths).

String

CamelFileParent (common)

Constant: FILE_PARENT

The parent path.

String

CamelRemoteFileInputStream (common)

Constant: REMOTE_FILE_INPUT_STREAM

The remote file input stream.

InputStream

CamelFileLocalWorkPath (common)

Constant: FILE_LOCAL_WORK_PATH

Path to the local work file, if local work directory is used.

String

CamelFileHost (common)

Constant: FILE_HOST

The remote hostname.

String

Endpoint URI Examples

azure-files://camelazurefiles.file.core.windows.net/samples?sv=2022-11-02&ss=f&srt=sco&sp=rwdlc&se=2023-06-18T22:29:13Z&st=2023-06-05T14:29:13Z&spr=https&sig=MPsMh8zci0v3To7IT9SKdaFGZV8ezno63m9C8s9bdVQ%3D
azure-files://camelazurefiles/samples/inbox/spam?sharedKey=FAKE502UyuBD...3Z%2BASt9dCmJg%3D%3D&delete=true

Paths

The path separator is /. The absolute paths start with the path separator. The absolute paths do not include the share name and they are relative to the share root rather than to the endpoint starting directory.

NOTE: At some places, namely logs of used libraries, OS-specific path separator appears, and the relative paths are relative to the share root (rather than to the current working directory or to the endpoint starting directory) so interpret them with a grain of salt.

Concurrency

This component does not support concurrency on its endpoints.

More Information

This component mimics the FTP component. So, there are more samples and details on the FTP component page.

This component uses the Azure Java SDK libraries for the actual work.

Consuming Files

The remote consumer will by default leave the consumed files untouched on the remote cloud files server. You have to configure it explicitly if you want it to delete the files or move them to another location. For example you can use delete=true to delete the files, or use move=.done to move the files into .done sub directory.

In Camel, the .-prefixed folders are excluded from recursive polling.

The regular File consumer is different as it will by default move files to a .camel sub directory. The reason Camel does not do this by default for the remote consumer is that it may lack permissions by default to be able to move or delete files.

Body Type Options

For each matching file, the consumer sends to the Camel exchange a message with a selected body type:

  • byte[] by default

  • java.io.InputStream if streamDownload=true is configured

  • java.io.File if localWorkDirectory is configured

The body type configuration should be tuned to fit available resources, performance targets, route processors, caching, resuming, etc.

Limitations

The option readLock can be used to force Camel not to consume files that is currently in the progress of being written. However, this option is turned off by default, as it requires that the user has write access. See the options table at File2 for more details about read locks.
There are other solutions to avoid consuming files that are currently being written; for instance, you can write to a temporary destination and move the file after it has been written.

For the readLock=changed, it relies only on the last modified, furthermore a precision finer than 5 seconds might be problematic.

When moving files using move or preMove option, the files are restricted to the share. That prevents consumer from moving files outside of the endpoint share.

Exchange Properties

The consumer sets the following exchange properties

Header Description

CamelBatchIndex

Current index out of total number of files being consumed in this batch.

CamelBatchSize

Total number of files being consumed in this batch.

CamelBatchComplete

True if there are no more files in this batch.

Producing Files

The Files producer is optimized for two body types:

  • java.io.InputStream if CamelFileLength header is set

  • byte[]

In either case the remote file size is allocated and then rewritten with body content. Any inconsistency between declared file length and stream length results in a corrupted remote file.

Limitations

The underlying Azure Files service does not allow to grow files. The file length must be known at its creation time, consequently:

  • CamelFileLength header has an important meaning even for producers.

  • No append mode is supported.

About Timeouts

You can use the connectTimeout option to set a timeout in millis to connect or disconnect.

The timeout option only applies as the data timeout in millis.

The meta-data operations timeout is minimum of: readLockCheckInterval, timeout and 20_000 millis.

For now the files upload has no timeout. During the upload, the underlying library could log timeout warnings. They are recoverable and upload could continue.

Using Local Work Directory

Camel supports consuming from remote files servers and downloading the files directly into a local work directory. This avoids reading the entire remote file content into memory as it is streamed directly into the local file using FileOutputStream.

Camel will store to a local file with the same name as the remote file, though with .inprogress as extension while the file is being downloaded. Afterwards, the file is renamed to remove the .inprogress suffix. And finally, when the Exchange is complete the local file is deleted.

So if you want to download files from a remote files server and store it as local files then you need to route to a file endpoint such as:

from("azure-files://...&localWorkDirectory=/tmp").to("file://inbox");

The route above is ultra efficient as it avoids reading the entire file content into memory. It will download the remote file directly to a local file stream. The java.io.File handle is then used as the Exchange body. The file producer leverages this fact and can work directly on the work file java.io.File handle and perform a java.io.File.rename to the target filename. As Camel knows it’s a local work file, it can optimize and use a rename instead of a file copy, as the work file is meant to be deleted anyway.

Custom Filtering

Camel supports pluggable filtering strategies. This strategy it to use the build in org.apache.camel.component.file.GenericFileFilter in Java. You can then configure the endpoint with such a filter to skip certain filters before being processed.

In the sample we have built our own filter that only accepts files starting with report in the filename.

And then we can configure our route using the filter attribute to reference our filter (using # notation) that we have defined in the spring XML file:

The accept(file) file argument has properties:

  • endpoint path: the share name such as /samples

  • relative path: a path to the file such as subdir/a file

  • directory: true if a directory

  • file length: if not a directory then a length of the file in bytes

Filtering using ANT path matcher

The ANT path matcher is a filter that is shipped out-of-the-box in the camel-spring jar. So you need to depend on camel-spring if you are using Maven.
The reason is that we leverage Spring’s AntPathMatcher to do the actual matching.

The file paths are matched with the following rules:

  • ? matches one character

  • * matches zero or more characters

  • ** matches zero or more directories in a path

The sample below demonstrates how to use it:

from("azure-files://...&antInclude=**/*.txt").to("...");

Using a Proxy

Consult the underlying library documentation.

Consuming a single file using a fixed name

Unlike FTP component that features special combination of options:

  • useList=false

  • fileName=myFileName.txt

  • ignoreFileNotFoundOrPermissionError=true

to optimize the single file using a fixed name use case, it is necessary to fallback to regular filters (i.e. the list permission is needed).

Debug logging

This component has log level TRACE that can be helpful if you have problems.