camel-azure-storage-datalake-kafka-connector source configuration

Connector description: Camel Azure Datalake Gen2 Component

When using camel-azure-storage-datalake-kafka-connector as source make sure to use the following Maven dependency to have support for the connector:

<dependency>
  <groupId>org.apache.camel.kafkaconnector</groupId>
  <artifactId>camel-azure-storage-datalake-kafka-connector</artifactId>
  <version>x.x.x</version>
  <!-- use the same version as your Camel Kafka connector version -->
</dependency>

To use this Source connector in Kafka connect you’ll need to set the following connector.class

connector.class=org.apache.camel.kafkaconnector.azurestoragedatalake.CamelAzurestoragedatalakeSourceConnector

The camel-azure-storage-datalake source connector supports 65 options, which are listed below.

Name Description Default Required Priority

camel.source.path.accountName

name of the azure account

null

false

MEDIUM

camel.source.path.fileSystemName

name of filesystem to be used

null

false

MEDIUM

camel.source.endpoint.accountKey

account key for authentication

null

false

MEDIUM

camel.source.endpoint.clientId

client id for azure account

null

false

MEDIUM

camel.source.endpoint.clientSecret

client secret for azure account

null

false

MEDIUM

camel.source.endpoint.clientSecretCredential

client secret credential for authentication

null

false

MEDIUM

camel.source.endpoint.close

Whether or not a file changed event raised indicates completion (true) or modification (false)

null

false

MEDIUM

camel.source.endpoint.closeStreamAfterRead

check for closing stream after read

null

false

MEDIUM

camel.source.endpoint.dataCount

count number of bytes to download

null

false

MEDIUM

camel.source.endpoint.dataLakeServiceClient

service client of datalake

null

false

MEDIUM

camel.source.endpoint.directoryName

directory of the file to be handled in component

null

false

MEDIUM

camel.source.endpoint.downloadLinkExpiration

download link expiration time

null

false

MEDIUM

camel.source.endpoint.expression

expression for queryInputStream

null

false

MEDIUM

camel.source.endpoint.fileDir

directory of file to do operations in the local system

null

false

MEDIUM

camel.source.endpoint.fileName

name of file to be handled in component

null

false

MEDIUM

camel.source.endpoint.fileOffset

offset position in file for different operations

null

false

MEDIUM

camel.source.endpoint.maxResults

maximum number of results to show at a time

null

false

MEDIUM

camel.source.endpoint.maxRetryRequests

no of retries to a given request

null

false

MEDIUM

camel.source.endpoint.openOptions

set open options for creating file

null

false

MEDIUM

camel.source.endpoint.path

path in azure datalake for operations

null

false

MEDIUM

camel.source.endpoint.permission

permission string for the file

null

false

MEDIUM

camel.source.endpoint.position

This parameter allows the caller to upload data in parallel and control the order in which it is appended to the file.

null

false

MEDIUM

camel.source.endpoint.recursive

recursively include all paths

null

false

MEDIUM

camel.source.endpoint.regex

regular expression for matching file names

null

false

MEDIUM

camel.source.endpoint.retainUncommitedData

Whether or not uncommitted data is to be retained after the operation

null

false

MEDIUM

camel.source.endpoint.serviceClient

datalake service client for azure storage datalake

null

false

MEDIUM

camel.source.endpoint.sharedKeyCredential

shared key credential for azure datalake gen2

null

false

MEDIUM

camel.source.endpoint.tenantId

tenant id for azure account

null

false

MEDIUM

camel.source.endpoint.timeout

Timeout for operation

null

false

MEDIUM

camel.source.endpoint.umask

umask permission for file

null

false

MEDIUM

camel.source.endpoint.userPrincipalNameReturned

whether or not to use upn

null

false

MEDIUM

camel.source.endpoint.bridgeErrorHandler

Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored.

false

false

MEDIUM

camel.source.endpoint.exceptionHandler

To let the consumer use a custom ExceptionHandler. Notice if the option bridgeErrorHandler is enabled then this option is not in use. By default the consumer will deal with exceptions, that will be logged at WARN or ERROR level and ignored.

null

false

MEDIUM

camel.source.endpoint.exchangePattern

Sets the exchange pattern when the consumer creates an exchange. One of: [InOnly] [InOut] [InOptionalOut]

null

false

MEDIUM

camel.component.azure-storage-datalake.accountKey

account key for authentication

null

false

MEDIUM

camel.component.azure-storage-datalake.clientId

client id for azure account

null

false

MEDIUM

camel.component.azure-storage-datalake.client Secret

client secret for azure account

null

false

MEDIUM

camel.component.azure-storage-datalake.client SecretCredential

client secret credential for authentication

null

false

MEDIUM

camel.component.azure-storage-datalake.close

Whether or not a file changed event raised indicates completion (true) or modification (false)

null

false

MEDIUM

camel.component.azure-storage-datalake.closeStream AfterRead

check for closing stream after read

null

false

MEDIUM

* camel.component.azure-storage-datalake.configuration*

configuration object for datalake

null

false

MEDIUM

camel.component.azure-storage-datalake.dataCount

count number of bytes to download

null

false

MEDIUM

camel.component.azure-storage-datalake.directory Name

directory of the file to be handled in component

null

false

MEDIUM

camel.component.azure-storage-datalake.download LinkExpiration

download link expiration time

null

false

MEDIUM

camel.component.azure-storage-datalake.expression

expression for queryInputStream

null

false

MEDIUM

camel.component.azure-storage-datalake.fileDir

directory of file to do operations in the local system

null

false

MEDIUM

camel.component.azure-storage-datalake.fileName

name of file to be handled in component

null

false

MEDIUM

camel.component.azure-storage-datalake.fileOffset

offset position in file for different operations

null

false

MEDIUM

camel.component.azure-storage-datalake.maxResults

maximum number of results to show at a time

null

false

MEDIUM

camel.component.azure-storage-datalake.maxRetry Requests

no of retries to a given request

null

false

MEDIUM

camel.component.azure-storage-datalake.openOptions

set open options for creating file

null

false

MEDIUM

camel.component.azure-storage-datalake.path

path in azure datalake for operations

null

false

MEDIUM

camel.component.azure-storage-datalake.permission

permission string for the file

null

false

MEDIUM

camel.component.azure-storage-datalake.position

This parameter allows the caller to upload data in parallel and control the order in which it is appended to the file.

null

false

MEDIUM

camel.component.azure-storage-datalake.recursive

recursively include all paths

null

false

MEDIUM

camel.component.azure-storage-datalake.regex

regular expression for matching file names

null

false

MEDIUM

camel.component.azure-storage-datalake.retain UncommitedData

Whether or not uncommitted data is to be retained after the operation

null

false

MEDIUM

camel.component.azure-storage-datalake.service Client

datalake service client for azure storage datalake

null

false

MEDIUM

camel.component.azure-storage-datalake.sharedKey Credential

shared key credential for azure datalake gen2

null

false

MEDIUM

camel.component.azure-storage-datalake.tenantId

tenant id for azure account

null

false

MEDIUM

camel.component.azure-storage-datalake.timeout

Timeout for operation

null

false

MEDIUM

camel.component.azure-storage-datalake.umask

umask permission for file

null

false

MEDIUM

camel.component.azure-storage-datalake.user PrincipalNameReturned

whether or not to use upn

null

false

MEDIUM

camel.component.azure-storage-datalake.bridgeError Handler

Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored.

false

false

MEDIUM

camel.component.azure-storage-datalake.autowired Enabled

Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc.

true

false

MEDIUM

The camel-azure-storage-datalake source connector has no converters out of the box.

The camel-azure-storage-datalake source connector has no transforms out of the box.

The camel-azure-storage-datalake source connector has no aggregation strategies out of the box.