# SparkLivy Operator

The **SparkLivy** operator uses Livy to run Spark applications on a Spark instance.

## Operator arguments <a href="#operator-arguments" id="operator-arguments"></a>

| Name              | type                                 | Value restrictions                                                          | Is required                         | Default | Description                                                                                  |
| ----------------- | ------------------------------------ | --------------------------------------------------------------------------- | ----------------------------------- | ------- | -------------------------------------------------------------------------------------------- |
| `file`            | String                               | It should be a path to a valid java or python spark application file        | Yes                                 |         | Spark application to submit (jar or python file)                                             |
| `args`            | List of Strings, Numbers or Booleans |                                                                             | Yes                                 |         | Additional application arguments                                                             |
| `className`       | String                               |                                                                             | No. Required for java applications. |         | The entry point of the java application.                                                     |
| `name`            | String                               |                                                                             | No                                  |         | Name of the task. When submitted the id of the study, experiment and trial will be appended. |
| `queue`           | String                               |                                                                             | No                                  |         | The name of the YARN queue to which submit a Spark application                               |
| `pyFiles`         | List of Strings                      | Each item of the list should be a path that matches an existing python file | No                                  |         | A list of python scripts to be added to the PYTHONPATH                                       |
| `proxyUser`       | String                               |                                                                             | No                                  |         | The user to be used to launch Spark applications                                             |
| `pollingInterval` | Number                               | `pollingInterval` > 0                                                       | No                                  | 10      | The number of seconds to wait before checking if a launched Spark application has finished   |
| `component`       | String                               | It should match the name of an existing Component of the System under test  | Yes                                 |         | The name of the component whose properties can be used as arguments of the operator          |

### Parameters applied from Experiments <a href="#parameters-applied-from-experiments" id="parameters-applied-from-experiments"></a>

The operator fetches the following parameters from the current Experiment to apply them to the System under test.

| Name                         | Description                         | Restrictions                    |
| ---------------------------- | ----------------------------------- | ------------------------------- |
| `spark_driver_memory`        | Memory for the driver               |                                 |
| `spark_executor_memory`      | Memory per executor                 |                                 |
| `spark_total_executor_cores` | Total cores used by the application | Spark standalone and Mesos only |
| `spark_executor_cores`       | Cores per executor                  | Spark standalone and YARN only  |
| `spark_num_executors`        | The number of executors             | YARN only                       |

### Examples <a href="#examples" id="examples"></a>

#### Execute with Livy <a href="#execute-spark-examples-with-livy" id="execute-spark-examples-with-livy"></a>

{% code lineNumbers="true" %}

```yaml
- name: "Run spark application"
  operator: "SparkLivy"
  arguments:
    component: "sparkemr"
    file: "/spark-examples.jar"
```

{% endcode %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.akamas.io/akamas-docs/3.1.2/akamas-reference/workflow-operators/sparklivy-operator.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.