Comment on page

SparkSubmit Operator

The SparkSubmit operator connects to a Spark instance and invokes a local spark-submit to schedule a job.
Operator arguments
Name
Type
Value Restrictions
Required
Default
Description
file
String
It should be a path to a valid java or python spark application file
Yes
​
Spark application to submit (jar or python file)
args
List of Strings, Numbers or Booleans
​
Yes
​
Additional application arguments
master
String
It should be a valid supported Master URL:
local
local[K]
local[K,F]
local[]
local[,F]
spark://HOST:PORT
spark://HOST1:PORT1, HOST2:PORT2
yarn
Yes
​
The master URL for the Spark cluster
deployMode
client cluster
​
No
cluster
Whether to launch the driver locally (client) or in the cluster (cluster)
className
String
​
No
​
The entry point of the java application. Required for java applications.
name
String
​
No
​
Name of the task. When submitted the id of the study, experiment and trial will be appended.
jars
List of Strings
Each item of the list should be a path that matches an existing jar file
No
​
A list of jars to be added in the classpath.
pyFiles
List of Strings
Each item of the list should be a path that matches an existing python file
No
​
A list of python scripts to be added to the PYTHONPATH
files
List of Strings
Each item of the list should be a path that matches an existing file
No
​
A list of files to be added to the context of the spark-submit
conf
Object (key-value pairs)
​
No
​
Mapping containing additional Spark configurations. See Spark documentation.
envVars
Object (key-value pairs)
​
No
​
Env variables when running the spark-submit command
sparkSubmitExec
String
It should be a path that matches an existing executable
No
The default for the Spark installation
The path of the spark-submit executable command
sparkHome
String
It should be a path that matches an existing directory
No
The default for the Spark installation
The path of the SPARK_HOME
proxyUser
String
​
No
​
The user to be used to execute Spark applications
verbose
Boolean
​
No
true
If additional debugging output should be displayed
component
String
It should match the name of an existing Component of the System under test
Yes
​
The name of the component whose properties can be used as arguments of the operator

Name	Type	Value Restrictions	Required	Default	Description
`file`	String	It should be a path to a valid java or python spark application file	Yes		Spark application to submit (jar or python file)
`args`	List of Strings, Numbers or Booleans		Yes		Additional application arguments
`master`	String	It should be a valid supported Master URL: local local[K] local[K,F] local[] local[,F] spark://HOST:PORT spark://HOST1:PORT1, HOST2:PORT2 yarn	Yes		The master URL for the Spark cluster
`deployMode`	`client` `cluster`		No	`cluster`	Whether to launch the driver locally (`client`) or in the cluster (`cluster`)
`className`	String		No		The entry point of the java application. Required for java applications.
`name`	String		No		Name of the task. When submitted the id of the study, experiment and trial will be appended.
`jars`	List of Strings	Each item of the list should be a path that matches an existing jar file	No		A list of jars to be added in the classpath.
`pyFiles`	List of Strings	Each item of the list should be a path that matches an existing python file	No		A list of python scripts to be added to the PYTHONPATH
`files`	List of Strings	Each item of the list should be a path that matches an existing file	No		A list of files to be added to the context of the spark-submit
`conf`	Object (key-value pairs)		No		Mapping containing additional Spark configurations. See Spark documentation.
`envVars`	Object (key-value pairs)		No		Env variables when running the spark-submit command
`sparkSubmitExec`	String	It should be a path that matches an existing executable	No	The default for the Spark installation	The path of the spark-submit executable command
`sparkHome`	String	It should be a path that matches an existing directory	No	The default for the Spark installation	The path of the SPARK_HOME
`proxyUser`	String		No		The user to be used to execute Spark applications
`verbose`	Boolean		No	true	If additional debugging output should be displayed
`component`	String	It should match the name of an existing Component of the System under test	Yes		The name of the component whose properties can be used as arguments of the operator

SparkSSHSubmit Operator

SparkLivy Operator

Last modified 4mo ago