SparkLivy Operator

The SparkLivy operator uses Livy to run Spark applications on a Spark instance.

Operator arguments

Name	type	Value restrictions	Is required	Default	Description
file	String	It should be a path to a valid java or python spark application file	Yes		Spark application to submit (jar or python file)
args	List of Strings, Numbers or Booleans		Yes		Additional application arguments
className	String		No. Required for java applications.		The entry point of the java application.
name	String		No		Name of the task. When submitted the id of the study, experiment and trial will be appended.
queue	String		No		The name of the YARN queue to which submit a Spark application
pyFiles	List of Strings	Each item of the list should be a path that matches an existing python file	No		A list of python scripts to be added to the PYTHONPATH
proxyUser	String		No		The user to be used to launch Spark applications
pollingInterval	Number	pollingInterval > 0	No	10	The number of seconds to wait before checking if a launched Spark application has finished
component	String	It should match the name of an existing Component of the System under test	Yes		The name of the component whose properties can be used as arguments of the operator

Parameters applied from Experiments

The operator fetches the following parameters from the current Experiment to apply them to the System under test.

Name	Description	Restrictions
spark_driver_memory	Memory for the driver
spark_executor_memory	Memory per executor
spark_total_executor_cores	Total cores used by the application	Spark standalone and Mesos only
spark_executor_cores	Cores per executor	Spark standalone and YARN only
spark_num_executors	The number of executors	YARN only

Examples

Execute with Livy

- name: Run spark application
  operator: SparkLivy
  arguments:
    component: sparkemr
    file: /spark-examples.jar