The SSHSparkSubmit operator connects to a Spark instance invoking a spark-submit on a machine reachable via SSH.
Name
Type
Values restrictions
Required
Default
Description
file
String
It should be a path to a valid java or python spark application file
Yes
Spark application to submit (jar or python file)
args
List of Strings, Numbers or Booleans
Yes
Additional application arguments
master
String
t should be a valid supported Master URL:
local
local[K]
local[K,F]
local[]
local[,F]
spark://HOST:PORT
spark://HOST1:PORT1, HOST2:PORT2
yarn
Yes
The master URL for the Spark cluster
deployMode
client cluster
No
cluster
Whether to launch the driver locally (client) or in the cluster (cluster)
className
String
No
The entry point of the java application. Required for java applications.
name
String
No
Name of the task. When submitted the id of the study, experiment and trial will be appended.
jars
List of Strings
Each item of the list should be a path that matches an existing jar file
No
A list of jars to be added in the classpath.
pyFiles
List of Strings
Each item of the list should be a path that matches an existing python file
No
A list of python scripts to be added to the PYTHONPATH
files
List of Strings
Each item of the list should be a path that matches an existing file
No
A list of files to be added to the context of the spark-submit command
conf
Object (key-value pairs)
No
Mapping containing additional Spark configurations. See Spark documentation.
envVars
Object (key-value pairs)
No
Env variables when running the spark-submit command
verbose
Boolean
No
true
If additional debugging output should be output
sparkSubmitExec
String
It should be a path that matches an existing executable
No
The default for the Spark installation
The path of the spark-submit executable command
sparkHome
String
It should be a path that matches an existing directory
No
The default for the Spark installation
The path of the SPARK_HOME
proxyUser
String
No
The user to be used to execute Spark applications
hostname
String
It should be a valid SSH host address
No, if the Component whose name is defined in component has a property named hostname
SSH host address
username
String
No, if the Component whose name is defined in component has a property named username
SSH login username
sshPort
Number
1≤sshPort≤65532
No
22
SSH port
password
String
Cannot be set if key is already set
No, if the Component whose name is defined in component has a property named password
SSH login password
key
String
Cannot be set if password is already set
No, if the Component whose name is defined in component has a property named key
SSH login key, provided directly its value or the path of the file to import from. The operator supports RSA and DSA Keys.
component
String
It should match the name of an existing Component of the System under test
Yes
The name of the Component whose properties can be used as arguments of the operator
componentThis operator automatically maps some properties of its component to some arguments. In case the mapped arguments are already provided to the operator, the is no override.
hostname
hostname
username
username
sshPort
sshPort
password
password
key
key