Spark
JVM since1.3.0 Nativeunsupported ⚠️Deprecated
Send RDD or DataFrame jobs to Apache Spark clusters.
What’s inside
-
Spark component, URI syntax:
spark:endpointType
Please refer to the above link for usage and configuration details.
Maven coordinates
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-spark</artifactId>
</dependency>
Check the User guide for more information about writing Camel Quarkus applications.
Usage
Special BOM
camel-quarkus-spark
is special compared to other Camel Quarkus extensions which are managed in camel-quarkus-bom
together with their important dependencies. This is because the Camel Spark extension depends on rather old versions of various libraries, that may conflict with their newer versions required by other Quarkus and Camel Quarkus extensions. Therefore, camel-quarkus-spark
extension and its dependencies are managed in a special BOM called camel-quarkus-spark-bom
.
For camel-quarkus-spark
to work properly in end user applications, camel-quarkus-spark-bom
need to be imported before any other Quarkus or Camel Quarkus BOMs.
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-spark-bom</artifactId>
<version>${camel-quarkus.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<!-- Other BOMs -->
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-bom</artifactId>
<version>${camel-quarkus.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
The above setup will make |
Camel Quarkus limitations
-
spark:dataframe
does not work on Java 9+ because Spark 2.x does not support Java 9+. This will be solved when Spark is upgraded to 3.x in Camel, see issue #1955. -
spark:hive
is does not work, see issue #1956.