AWS S3 Event Based Source
Provided by: "Apache Software Foundation"
Support Level for this Kamelet is: "Preview"
Receives the events of an Amazon S3 bucket, such as a new or a deleted file, from the Amazon SQS queue that Amazon EventBridge sends them to, one message per event.
Use it to react to new files in a bucket as soon as they arrive, without listing the bucket again and again, for example to process uploaded documents or imported data files. To read the files of a bucket by polling it, use aws-s3-source.
Output: the body is the EventBridge event, as JSON text: detail-type says what happened (such as Object Created or Object Deleted), detail.bucket.name is the bucket and detail.object.key the key of the file. With getObject set to true, the event of a file uploaded with a simple upload (PutObject) carries the content of the file instead, as a stream, with the CamelAwsS3ProducedBucketName and CamelAwsS3ProducedKey headers; the other events then come as parsed JSON (a Jackson JsonNode).
Good to know: it needs a setup in AWS. In the properties of the bucket, turn on sending events to Amazon EventBridge; then create an EventBridge rule for the events of source aws.s3 and that bucket, with the queue as its target. Each event is deleted from the queue once it has been processed, unless deleteAfterRead is false. The files stay in the bucket. getObject skips files created by a multipart upload or a copy, and never connects through uriEndpointOverride.
Configuration Options
The following table summarizes the configuration options available for the aws-s3-event-based-source Kamelet:
| Property | Name | Description | Type | Default | Example |
|---|---|---|---|---|---|
Access Key | Required The access key ID of an IAM user, which you create in the AWS console under IAM, Users, Security credentials. Use it together with | string | |||
Queue Name | Required The name of the SQS queue that the EventBridge rule sends the S3 events to, such as | string | s3-events | ||
AWS Region | Required The AWS region of the queue and the bucket, as a region code such as Enum values: * ap-south-1 * eu-south-1 * us-gov-east-1 * me-central-1 * ca-central-1 * eu-central-1 * us-iso-west-1 * us-west-1 * us-west-2 * af-south-1 * eu-north-1 * eu-west-3 * eu-west-2 * eu-west-1 * ap-northeast-3 * ap-northeast-2 * ap-northeast-1 * me-south-1 * sa-east-1 * ap-east-1 * cn-north-1 * us-gov-west-1 * ap-southeast-1 * ap-southeast-2 * us-iso-east-1 * ap-southeast-3 * us-east-1 * us-east-2 * cn-northwest-1 * us-isob-east-1 * aws-global * aws-cn-global * aws-us-gov-global * aws-iso-global * aws-iso-b-global | string | eu-west-1 | ||
Secret Key | Required The secret access key that belongs to | string | |||
AWS Host | The host name of AWS. Leave it as | string | amazonaws.com | ||
Autocreate Queue | If true, the queue is created when it does not exist. A name that ends in | boolean | false | ||
Delay | How long to wait between two polls of the queue, in milliseconds. | integer | 500 | ||
Auto-delete Messages | If true, each event is deleted from the queue once it has been processed. Set it to false to keep the events; they are then received again each time the visibility timeout of the queue ends. | boolean | true | ||
Get Object | If true, for an event of a file uploaded with a simple upload (PutObject), it fetches the file and sends its content instead of the event. It connects to S3 with | boolean | false | ||
Greedy Scheduler | If true, it polls again right away when the last poll received messages, instead of waiting for | boolean | false | ||
Endpoint Overwrite | If true, it connects to the address in | boolean | false | ||
Protocol |
| string | https | http | |
Queue URL | The full address of the queue, as the SQS console shows it. When it is set, it is used as it is, and the queue is not looked up by name. | string | https://sqs.eu-west-1.amazonaws.com/123456789012/s3-events | ||
Overwrite Endpoint URI | The address to connect to instead of AWS, as a URL such as | string | http://localhost:4566 |
Dependencies
At runtime, the aws-s3-event-based-source Kamelet relies upon the presence of the following dependencies:
-
camel:aws2-sqs
-
camel:aws2-s3
-
camel:jsonpath
-
camel:kamelet
-
camel:jackson
Camel JBang usage
Prerequisites
-
You’ve installed JBang.
-
You have executed the following command:
jbang app install camel@apache/camel Supposing you have a file named route.yaml with this content:
- route:
from:
uri: "kamelet:aws-s3-event-based-source"
parameters:
.
.
.
steps:
- to:
uri: "kamelet:log-sink" You can now run it directly through the following command
camel run route.yaml AWS S3 Event based Source Kamelet Description
Authentication methods
Access Key/Secret Key is the basic method for authenticating to the AWS SQS Service.
Required Setup
To use this Kamelet you’ll need to set up Eventbridge on your bucket and subscribe Eventbridge bus to an SQS Queue.
For doing this you’ll need to enable Eventbridge notification on your bucket and creating a rule on Eventbridge console related to all the events on S3 bucket and pointing to the SQS Queue specified as parameter in this Kamelet.