# SparkLivy Operator

The **SparkLivy** operator uses Livy to run Spark applications on a Spark instance.

## Operator arguments <a href="#operator-arguments" id="operator-arguments"></a>

| Name              | type                                 | Value restrictions                                                          | Is required                         | Default | Description                                                                                  |
| ----------------- | ------------------------------------ | --------------------------------------------------------------------------- | ----------------------------------- | ------- | -------------------------------------------------------------------------------------------- |
| `file`            | String                               | It should be a path to a valid java or python spark application file        | Yes                                 |         | Spark application to submit (jar or python file)                                             |
| `args`            | List of Strings, Numbers or Booleans |                                                                             | Yes                                 |         | Additional application arguments                                                             |
| `className`       | String                               |                                                                             | No. Required for java applications. |         | The entry point of the java application.                                                     |
| `name`            | String                               |                                                                             | No                                  |         | Name of the task. When submitted the id of the study, experiment and trial will be appended. |
| `queue`           | String                               |                                                                             | No                                  |         | The name of the YARN queue to which submit a Spark application                               |
| `pyFiles`         | List of Strings                      | Each item of the list should be a path that matches an existing python file | No                                  |         | A list of python scripts to be added to the PYTHONPATH                                       |
| `proxyUser`       | String                               |                                                                             | No                                  |         | The user to be used to launch Spark applications                                             |
| `pollingInterval` | Number                               | `pollingInterval` > 0                                                       | No                                  | 10      | The number of seconds to wait before checking if a launched Spark application has finished   |
| `component`       | String                               | It should match the name of an existing Component of the System under test  | Yes                                 |         | The name of the component whose properties can be used as arguments of the operator          |

### Parameters applied from Experiments <a href="#parameters-applied-from-experiments" id="parameters-applied-from-experiments"></a>

The operator fetches the following parameters from the current Experiment to apply them to the System under test.

| Name                         | Description                         | Restrictions                    |
| ---------------------------- | ----------------------------------- | ------------------------------- |
| `spark_driver_memory`        | Memory for the driver               |                                 |
| `spark_executor_memory`      | Memory per executor                 |                                 |
| `spark_total_executor_cores` | Total cores used by the application | Spark standalone and Mesos only |
| `spark_executor_cores`       | Cores per executor                  | Spark standalone and YARN only  |
| `spark_num_executors`        | The number of executors             | YARN only                       |

### Examples <a href="#examples" id="examples"></a>

#### Execute with Livy <a href="#execute-spark-examples-with-livy" id="execute-spark-examples-with-livy"></a>

```yaml
- name: Run spark application
  operator: SparkLivy
  arguments:
    component: sparkemr
    file: /spark-examples.jar
```