Camel Components

Tokenize

Since Camel 2.0

The tokenizer language is a built-in language in camel-core, which is most often used with the Split EIP to split a message using a token-based strategy.

The tokenizer language is intended to tokenize text documents using a specified delimiter pattern. It can also be used to tokenize XML documents with some limited capability. For a truly XML-aware tokenization, the use of the XML Tokenize language is recommended as it offers a faster, more efficient tokenization specifically for XML documents.

Tokenize Options

The Tokenize language supports the following options which are listed below.

Name Default Java Type Description

token (common)

String

Required The (start) token to use as tokenizer, for example you can use the new line token. You can use simple language as the token to support dynamic tokens.

endToken (common)

String

The end token to use as tokenizer if using start/end token pairs. You can use simple language as the token to support dynamic tokens.

inheritNamespaceTagName (advanced)

String

To inherit namespaces from a root/parent tag name when using XML. You can use simple language as the tag name to support dynamic names.

regex (advanced)

false

Boolean

If the token is a regular expression pattern.

xml (common)

false

Boolean

Whether the input is XML messages. This option must be set to true if working with XML payloads.

includeTokens (common)

false

Boolean

Whether to include the tokens in the parts when using pairs. When including tokens then the endToken property must also be configured (to use pair mode).

group (advanced)

String

To group N parts together, for example to split big files into chunks of 1000 lines. You can use simple language as the group to support dynamic group sizes.

groupDelimiter (advanced)

String

Sets the delimiter to use when grouping. If this has not been set then token will be used as the delimiter.

skipFirst (advanced)

false

Boolean

To skip the very first element.

source (common)

String

Source to use, instead of message body. You can prefix with variable:, header:, or property: to specify kind of source. Otherwise, the source is assumed to be a variable. Use empty or null to use default source, which is the message body.

resultType (common)

String

The class of the result type (type from output).

trim (advanced)

true

Boolean

Whether to trim the source code to remove leading and trailing whitespaces and line breaks.

resolveResource (advanced)

false

Boolean

Whether a result of the expression that is a String starting with resource: is loaded as a resource and its content becomes the result, e.g. a script that returns resource:file:order.json or resource:classpath:templates/order.json (a name without a scheme is a classpath resource). Off by default; the resource: prefix on the expression text itself is always resolved. Applies to the expression used as a value, not as a predicate.

Message Body

The message body at each side of the Tokenize language: Expression input is the body its expressions can read, and Expression result what they return.

Side Body Description

Expression input

text, bytes, stream or file

Text as text, bytes, a stream or a file. Another body is tokenized as its toString(), and fails with endToken or xml: marshal it first.

Not known when source is set.

Expression result

iterator

An Iterator of the tokens as text, read once: in setBody the body is the Iterator, for the Split EIP to walk. With group, each N tokens are joined into one text.

The bodies are the ones with the default options. A side that is not listed is not known, and tooling does not guess it. The carriers, the sides and the tools that use them are described in Message Body Types.

Example

The following example shows how to take a request from the direct:a endpoint then split it into pieces using an Expression, then forward each piece to direct:b:

When using new-line (\n) as tokenizer, then XML and YAML DSL must use the token option\ as done in the following example.

  • Java

  • XML

  • YAML

from("direct:a")
    .split(body().tokenize("\n"))
        .to("direct:b");
<route>
  <from uri="direct:a"/>
  <split>
    <tokenize token="\n"/>
    <to uri="direct:b"/>
  </split>
</route>
- route:
    from:
      uri: direct:a
      steps:
        - split:
            expression:
              tokenize:
                token: "\n"
            steps:
              - to:
                  uri: direct:b

See Also

For more examples see Split EIP.

Dependencies

The Tokenize language is part of camel-core.