Comment on page

SparkLivy Operator

The SparkLivy operator uses Livy to run Spark applications on a Spark instance.
Operator arguments
Name
type
Value restrictions
Is required
Default
Description
file
String
It should be a path to a valid java or python spark application file
Yes
​
Spark application to submit (jar or python file)
args
List of Strings, Numbers or Booleans
​
Yes
​
Additional application arguments
className
String
​
No. Required for java applications.
​
The entry point of the java application.
name
String
​
No
​
Name of the task. When submitted the id of the study, experiment and trial will be appended.
queue
String
​
No
​
The name of the YARN queue to which submit a Spark application
pyFiles
List of Strings
Each item of the list should be a path that matches an existing python file
No
​
A list of python scripts to be added to the PYTHONPATH
proxyUser
String
​
No
​
The user to be used to launch Spark applications
pollingInterval
Number
pollingInterval > 0
No
10
The number of seconds to wait before checking if a launched Spark application has finished
component
String
It should match the name of an existing Component of the System under test
Yes
​
The name of the component whose properties can be used as arguments of the operator
Parameters applied from Experiments
The operator fetches the following parameters from the current Experiment to apply them to the System under test.
Name
Description
Restrictions
spark_driver_memory
Memory for the driver
​
spark_executor_memory
Memory per executor
​
spark_total_executor_cores
Total cores used by the application
Spark standalone and Mesos only
spark_executor_cores
Cores per executor
Spark standalone and YARN only
spark_num_executors
The number of executors
YARN only
Examples
Execute with Livy
- name: Run spark application
  operator: SparkLivy
  arguments:
    component: sparkemr
    file: /spark-examples.jar

Name	type	Value restrictions	Is required	Default	Description
`file`	String	It should be a path to a valid java or python spark application file	Yes		Spark application to submit (jar or python file)
`args`	List of Strings, Numbers or Booleans		Yes		Additional application arguments
`className`	String		No. Required for java applications.		The entry point of the java application.
`name`	String		No		Name of the task. When submitted the id of the study, experiment and trial will be appended.
`queue`	String		No		The name of the YARN queue to which submit a Spark application
`pyFiles`	List of Strings	Each item of the list should be a path that matches an existing python file	No		A list of python scripts to be added to the PYTHONPATH
`proxyUser`	String		No		The user to be used to launch Spark applications
`pollingInterval`	Number	`pollingInterval` > 0	No	10	The number of seconds to wait before checking if a launched Spark application has finished
`component`	String	It should match the name of an existing Component of the System under test	Yes		The name of the component whose properties can be used as arguments of the operator

Name	Description	Restrictions
`spark_driver_memory`	Memory for the driver
`spark_executor_memory`	Memory per executor
`spark_total_executor_cores`	Total cores used by the application	Spark standalone and Mesos only
`spark_executor_cores`	Cores per executor	Spark standalone and YARN only
`spark_num_executors`	The number of executors	YARN only

SparkSubmit Operator

NeoLoadWeb Operator

Last modified 3d ago