SparkLivy Operator
Last updated
Last updated
The SparkLivy operator uses Livy to run Spark applications on a Spark instance.
| Name | type | Value restrictions | Is required | Default | Description |
|---|---|---|---|---|---|
The operator fetches the following parameters from the current Experiment to apply them to the System under test.
| Name | Description | Restrictions |
|---|---|---|
file
String
It should be a path to a valid java or python spark application file
Yes
Spark application to submit (jar or python file)
args
List of Strings, Numbers or Booleans
Yes
Additional application arguments
className
String
No. Required for java applications.
The entry point of the java application.
name
String
No
Name of the task. When submitted the id of the study, experiment and trial will be appended.
queue
String
No
The name of the YARN queue to which submit a Spark application
pyFiles
List of Strings
Each item of the list should be a path that matches an existing python file
No
A list of python scripts to be added to the PYTHONPATH
proxyUser
String
No
The user to be used to launch Spark applications
pollingInterval
Number
pollingInterval > 0
No
10
The number of seconds to wait before checking if a launched Spark application has finished
component
String
It should match the name of an existing Component of the System under test
Yes
The name of the component whose properties can be used as arguments of the operator
spark_driver_memory
Memory for the driver
spark_executor_memory
Memory per executor
spark_total_executor_cores
Total cores used by the application
Spark standalone and Mesos only
spark_executor_cores
Cores per executor
Spark standalone and YARN only
spark_num_executors
The number of executors
YARN only