# SparkSSHSubmit Operator

The **SSHSparkSubmit** operator connects to a Spark instance invoking a *spark-submit* on a machine reachable via SSH.

## Operator arguments <a href="#operator-arguments" id="operator-arguments"></a>

| **Name** | **Type** | **Values restrictions** | **Required** | **Default** | **Description** |
| -------- | -------- | ----------------------- | ------------ | ----------- | --------------- |

| Name              | Type                                 | Value Restrictions                                                                                                                                                                                                                | Required                                                                                  | Default                                | Description                                                                                                                |
| ----------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `file`            | String                               | It should be a path to a valid java or python spark application file                                                                                                                                                              | Yes                                                                                       |                                        | Spark application to submit (jar or python file)                                                                           |
| `args`            | List of Strings, Numbers or Booleans |                                                                                                                                                                                                                                   | Yes                                                                                       |                                        | Additional application arguments                                                                                           |
| `master`          | String                               | <p>t should be a valid supported Master URL:</p><ul><li>local</li><li>local\[K]</li><li>local\[K,F]</li><li>local\[]</li><li>local\[,F]</li><li>spark://HOST:PORT</li><li>spark://HOST1:PORT1, HOST2:PORT2</li><li>yarn</li></ul> | Yes                                                                                       |                                        | The master URL for the Spark cluster                                                                                       |
| `deployMode`      | `client` `cluster`                   |                                                                                                                                                                                                                                   | No                                                                                        | `cluster`                              | Whether to launch the driver locally (`client`) or in the cluster (`cluster`)                                              |
| `className`       | String                               |                                                                                                                                                                                                                                   | No                                                                                        |                                        | The entry point of the java application. Required for java applications.                                                   |
| `name`            | String                               |                                                                                                                                                                                                                                   | No                                                                                        |                                        | Name of the task. When submitted the id of the study, experiment and trial will be appended.                               |
| `jars`            | List of Strings                      | Each item of the list should be a path that matches an existing jar file                                                                                                                                                          | No                                                                                        |                                        | A list of jars to be added in the classpath.                                                                               |
| `pyFiles`         | List of Strings                      | Each item of the list should be a path that matches an existing python file                                                                                                                                                       | No                                                                                        |                                        | A list of python scripts to be added to the PYTHONPATH                                                                     |
| `files`           | List of Strings                      | Each item of the list should be a path that matches an existing file                                                                                                                                                              | No                                                                                        |                                        | A list of files to be added to the context of the *spark-submit* command                                                   |
| `conf`            | Object (key-value pairs)             |                                                                                                                                                                                                                                   | No                                                                                        |                                        | Mapping containing additional Spark configurations. See Spark documentation.                                               |
| `envVars`         | Object (key-value pairs)             |                                                                                                                                                                                                                                   | No                                                                                        |                                        | Env variables when running the *spark-submit* command                                                                      |
| `verbose`         | Boolean                              |                                                                                                                                                                                                                                   | No                                                                                        | true                                   | If additional debugging output should be output                                                                            |
| `sparkSubmitExec` | String                               | It should be a path that matches an existing executable                                                                                                                                                                           | No                                                                                        | The default for the Spark installation | The path of the *spark-submit* executable command                                                                          |
| `sparkHome`       | String                               | It should be a path that matches an existing directory                                                                                                                                                                            | No                                                                                        | The default for the Spark installation | The path of the SPARK\_HOME                                                                                                |
| `proxyUser`       | String                               |                                                                                                                                                                                                                                   | No                                                                                        |                                        | The user to be used to execute Spark applications                                                                          |
| `hostname`        | String                               | It should be a valid SSH host address                                                                                                                                                                                             | No, if the Component whose name is defined in `component` has a property named `hostname` |                                        | SSH host address                                                                                                           |
| `username`        | String                               |                                                                                                                                                                                                                                   | No, if the Component whose name is defined in `component` has a property named `username` |                                        | SSH login username                                                                                                         |
| `sshPort`         | Number                               | 1≤`sshPort`≤65532                                                                                                                                                                                                                 | No                                                                                        | 22                                     | SSH port                                                                                                                   |
| `password`        | String                               | Cannot be set if `key` is already set                                                                                                                                                                                             | No, if the Component whose name is defined in `component` has a property named `password` |                                        | SSH login password                                                                                                         |
| `key`             | String                               | Cannot be set if `password` is already set                                                                                                                                                                                        | No, if the Component whose name is defined in `component` has a property named `key`      |                                        | SSH login key, provided directly its value or the path of the file to import from. The operator supports RSA and DSA Keys. |
| `component`       | String                               | It should match the name of an existing Component of the System under test                                                                                                                                                        | Yes                                                                                       |                                        | The name of the Component whose properties can be used as arguments of the operator                                        |

### Get operator arguments from `component` <a href="#get-operator-arguments-from-component" id="get-operator-arguments-from-component"></a>

This operator automatically maps some properties of its `component` to some arguments. In case the mapped arguments are already provided to the operator, the is no override.

#### Component property to operator argument mapping <a href="#component-property-to-operator-argument-mapping" id="component-property-to-operator-argument-mapping"></a>

| `hostname` | `hostname` |
| ---------- | ---------- |
| `username` | `username` |
| `sshPort`  | `sshPort`  |
| `password` | `password` |
| `key`      | `key`      |