Comment on page
SparkSubmit Operator
The SparkSubmit operator connects to a Spark instance and invokes a local spark-submit to schedule a job.
Name | Type | Value Restrictions | Required | Default | Description |
---|---|---|---|---|---|
file | String | It should be a path to a valid java or python spark application file | Yes | | Spark application to submit (jar or python file) |
args | List of Strings, Numbers or Booleans | | Yes | | Additional application arguments |
master | String | It should be a valid supported Master URL:
| Yes | | The master URL for the Spark cluster |
deployMode | client cluster | | No | cluster | Whether to launch the driver locally ( client ) or in the cluster (cluster ) |
className | String | | No | | The entry point of the java application. Required for java applications. |
name | String | | No | | Name of the task. When submitted the id of the study, experiment and trial will be appended. |
jars | List of Strings | Each item of the list should be a path that matches an existing jar file | No | | A list of jars to be added in the classpath. |
pyFiles | List of Strings | Each item of the list should be a path that matches an existing python file | No | | A list of python scripts to be added to the PYTHONPATH |
files | List of Strings | Each item of the list should be a path that matches an existing file | No | | A list of files to be added to the context of the spark-submit |
conf | Object (key-value pairs) | | No | | Mapping containing additional Spark configurations. See Spark documentation. |
envVars | Object (key-value pairs) | | No | | Env variables when running the spark-submit command |
sparkSubmitExec | String | It should be a path that matches an existing executable | No | The default for the Spark installation | The path of the spark-submit executable command |
sparkHome | String | It should be a path that matches an existing directory | No | The default for the Spark installation | The path of the SPARK_HOME |
proxyUser | String | | No | | The user to be used to execute Spark applications |
verbose | Boolean | | No | true | If additional debugging output should be displayed |
component | String | It should match the name of an existing Component of the System under test | Yes | | The name of the component whose properties can be used as arguments of the operator |
Last modified 4mo ago