> For the complete documentation index, see [llms.txt](https://docs.akamas.io/akamas-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.akamas.io/akamas-docs/3.5.0/reference/workflow-operators/sparklivy-operator.md).

# SparkLivy Operator

The **SparkLivy** operator uses Livy to run Spark applications on a Spark instance.

## Operator arguments <a href="#operator-arguments" id="operator-arguments"></a>

| Name              | type                                 | Value restrictions                                                          | Is required                         | Default | Description                                                                                  |
| ----------------- | ------------------------------------ | --------------------------------------------------------------------------- | ----------------------------------- | ------- | -------------------------------------------------------------------------------------------- |
| `file`            | String                               | It should be a path to a valid java or python spark application file        | Yes                                 |         | Spark application to submit (jar or python file)                                             |
| `args`            | List of Strings, Numbers or Booleans |                                                                             | Yes                                 |         | Additional application arguments                                                             |
| `className`       | String                               |                                                                             | No. Required for java applications. |         | The entry point of the java application.                                                     |
| `name`            | String                               |                                                                             | No                                  |         | Name of the task. When submitted the id of the study, experiment and trial will be appended. |
| `queue`           | String                               |                                                                             | No                                  |         | The name of the YARN queue to which submit a Spark application                               |
| `pyFiles`         | List of Strings                      | Each item of the list should be a path that matches an existing python file | No                                  |         | A list of python scripts to be added to the PYTHONPATH                                       |
| `proxyUser`       | String                               |                                                                             | No                                  |         | The user to be used to launch Spark applications                                             |
| `pollingInterval` | Number                               | `pollingInterval` > 0                                                       | No                                  | 10      | The number of seconds to wait before checking if a launched Spark application has finished   |
| `component`       | String                               | It should match the name of an existing Component of the System under test  | Yes                                 |         | The name of the component whose properties can be used as arguments of the operator          |

### Parameters applied from Experiments <a href="#parameters-applied-from-experiments" id="parameters-applied-from-experiments"></a>

The operator fetches the following parameters from the current Experiment to apply them to the System under test.

| Name                         | Description                         | Restrictions                    |
| ---------------------------- | ----------------------------------- | ------------------------------- |
| `spark_driver_memory`        | Memory for the driver               |                                 |
| `spark_executor_memory`      | Memory per executor                 |                                 |
| `spark_total_executor_cores` | Total cores used by the application | Spark standalone and Mesos only |
| `spark_executor_cores`       | Cores per executor                  | Spark standalone and YARN only  |
| `spark_num_executors`        | The number of executors             | YARN only                       |

### Examples <a href="#examples" id="examples"></a>

#### Execute with Livy <a href="#execute-spark-examples-with-livy" id="execute-spark-examples-with-livy"></a>

```yaml
- name: Run spark application
  operator: SparkLivy
  arguments:
    component: sparkemr
    file: /spark-examples.jar
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.akamas.io/akamas-docs/3.5.0/reference/workflow-operators/sparklivy-operator.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.