Comment on page
SparkLivy Operator
The SparkLivy operator uses Livy to run Spark applications on a Spark instance.
Name | type | Value restrictions | Is required | Default | Description |
---|---|---|---|---|---|
file | String | It should be a path to a valid java or python spark application file | Yes | | Spark application to submit (jar or python file) |
args | List of Strings, Numbers or Booleans | | Yes | | Additional application arguments |
className | String | | No. Required for java applications. | | The entry point of the java application. |
name | String | | No | | Name of the task. When submitted the id of the study, experiment and trial will be appended. |
queue | String | | No | | The name of the YARN queue to which submit a Spark application |
pyFiles | List of Strings | Each item of the list should be a path that matches an existing python file | No | | A list of python scripts to be added to the PYTHONPATH |
proxyUser | String | | No | | The user to be used to launch Spark applications |
pollingInterval | Number | pollingInterval > 0 | No | 10 | The number of seconds to wait before checking if a launched Spark application has finished |
component | String | It should match the name of an existing Component of the System under test | Yes | | The name of the component whose properties can be used as arguments of the operator |
The operator fetches the following parameters from the current Experiment to apply them to the System under test.
Name | Description | Restrictions |
---|---|---|
spark_driver_memory | Memory for the driver | |
spark_executor_memory | Memory per executor | |
spark_total_executor_cores | Total cores used by the application | Spark standalone and Mesos only |
spark_executor_cores | Cores per executor | Spark standalone and YARN only |
spark_num_executors | The number of executors | YARN only |
- name: Run spark application
operator: SparkLivy
arguments:
component: sparkemr
file: /spark-examples.jar
Last modified 3d ago