# SparkSSHSubmit Operator

The **SSHSparkSubmit** operator connects to a Spark instance invoking a *spark-submit* on a machine reachable via SSH.

## Operator arguments <a href="#operator-arguments" id="operator-arguments"></a>

| **Name** | **Type** | **Values restrictions** | **Required** | **Default** | **Description** |
| -------- | -------- | ----------------------- | ------------ | ----------- | --------------- |

| Name              | Type                                 | Value Restrictions                                                                                                                                                                                                                | Required                                                                                  | Default                                | Description                                                                                                                |
| ----------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `file`            | String                               | It should be a path to a valid java or python spark application file                                                                                                                                                              | Yes                                                                                       |                                        | Spark application to submit (jar or python file)                                                                           |
| `args`            | List of Strings, Numbers or Booleans |                                                                                                                                                                                                                                   | Yes                                                                                       |                                        | Additional application arguments                                                                                           |
| `master`          | String                               | <p>t should be a valid supported Master URL:</p><ul><li>local</li><li>local\[K]</li><li>local\[K,F]</li><li>local\[]</li><li>local\[,F]</li><li>spark://HOST:PORT</li><li>spark://HOST1:PORT1, HOST2:PORT2</li><li>yarn</li></ul> | Yes                                                                                       |                                        | The master URL for the Spark cluster                                                                                       |
| `deployMode`      | `client` `cluster`                   |                                                                                                                                                                                                                                   | No                                                                                        | `cluster`                              | Whether to launch the driver locally (`client`) or in the cluster (`cluster`)                                              |
| `className`       | String                               |                                                                                                                                                                                                                                   | No                                                                                        |                                        | The entry point of the java application. Required for java applications.                                                   |
| `name`            | String                               |                                                                                                                                                                                                                                   | No                                                                                        |                                        | Name of the task. When submitted the id of the study, experiment and trial will be appended.                               |
| `jars`            | List of Strings                      | Each item of the list should be a path that matches an existing jar file                                                                                                                                                          | No                                                                                        |                                        | A list of jars to be added in the classpath.                                                                               |
| `pyFiles`         | List of Strings                      | Each item of the list should be a path that matches an existing python file                                                                                                                                                       | No                                                                                        |                                        | A list of python scripts to be added to the PYTHONPATH                                                                     |
| `files`           | List of Strings                      | Each item of the list should be a path that matches an existing file                                                                                                                                                              | No                                                                                        |                                        | A list of files to be added to the context of the *spark-submit* command                                                   |
| `conf`            | Object (key-value pairs)             |                                                                                                                                                                                                                                   | No                                                                                        |                                        | Mapping containing additional Spark configurations. See Spark documentation.                                               |
| `envVars`         | Object (key-value pairs)             |                                                                                                                                                                                                                                   | No                                                                                        |                                        | Env variables when running the *spark-submit* command                                                                      |
| `verbose`         | Boolean                              |                                                                                                                                                                                                                                   | No                                                                                        | true                                   | If additional debugging output should be output                                                                            |
| `sparkSubmitExec` | String                               | It should be a path that matches an existing executable                                                                                                                                                                           | No                                                                                        | The default for the Spark installation | The path of the *spark-submit* executable command                                                                          |
| `sparkHome`       | String                               | It should be a path that matches an existing directory                                                                                                                                                                            | No                                                                                        | The default for the Spark installation | The path of the SPARK\_HOME                                                                                                |
| `proxyUser`       | String                               |                                                                                                                                                                                                                                   | No                                                                                        |                                        | The user to be used to execute Spark applications                                                                          |
| `hostname`        | String                               | It should be a valid SSH host address                                                                                                                                                                                             | No, if the Component whose name is defined in `component` has a property named `hostname` |                                        | SSH host address                                                                                                           |
| `username`        | String                               |                                                                                                                                                                                                                                   | No, if the Component whose name is defined in `component` has a property named `username` |                                        | SSH login username                                                                                                         |
| `sshPort`         | Number                               | 1≤`sshPort`≤65532                                                                                                                                                                                                                 | No                                                                                        | 22                                     | SSH port                                                                                                                   |
| `password`        | String                               | Cannot be set if `key` is already set                                                                                                                                                                                             | No, if the Component whose name is defined in `component` has a property named `password` |                                        | SSH login password                                                                                                         |
| `key`             | String                               | Cannot be set if `password` is already set                                                                                                                                                                                        | No, if the Component whose name is defined in `component` has a property named `key`      |                                        | SSH login key, provided directly its value or the path of the file to import from. The operator supports RSA and DSA Keys. |
| `component`       | String                               | It should match the name of an existing Component of the System under test                                                                                                                                                        | Yes                                                                                       |                                        | The name of the Component whose properties can be used as arguments of the operator                                        |

### Get operator arguments from `component` <a href="#get-operator-arguments-from-component" id="get-operator-arguments-from-component"></a>

This operator automatically maps some properties of its `component` to some arguments. In case the mapped arguments are already provided to the operator, the is no override.

#### Component property to operator argument mapping <a href="#component-property-to-operator-argument-mapping" id="component-property-to-operator-argument-mapping"></a>

| `hostname` | `hostname` |
| ---------- | ---------- |
| `username` | `username` |
| `sshPort`  | `sshPort`  |
| `password` | `password` |
| `key`      | `key`      |


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.akamas.io/akamas-docs/3.6/reference/workflow-operators/sparksshsubmit-operator.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.