The Connector for Apache Spark – Community Edition allows you to configure a connection to Spark server, and then use activities to run Spark SQL, execute spark Scala code and submit spark job.
It provides the following features:
- Spark Connection Shared Resource: You can use the Spark connection shared resource to connect Spark Livy server. The shared resource is used by the Spark activities.
- Spark SQL Activity: You can use this activity to run spark sql.
- Spark Execution Activity: You can use this activity to submit spark code and jobs.
- Wait for Completion Activity: You can use this activity to wait for spark execution completion.