Optimizing Spark

When optimizing applications running on the Apache Spark framework, the goal is to find the configurations that best optimize the allocated resources or the execution time.

Please refer to the Spark optimization pack for the list of component types, parameters, metrics, and constraints.


Applying parameters

Akamas offers several operators that you can use to apply the parameters for the tuned Spark application. In particular, we suggest using the Spark SSH Submit operator, which connects to a target instance to submit the application using the configuration parameters to test.

Other solutions include:

A typical workflow

You can organize a typical workflow to optimize a Spark application in three parts:

  1. Setup the test environment

    1. prepare any required input data

    2. apply the Spark configuration parameters, if you are going for a file-based solution

  2. Execute the Spark application

  3. Perform cleanup

Here’s an example of a typical workflow where Akamas executes the Spark application using the Spark SSH Submit operator:

name: Spark workflow
   - name: cwspark
        master: yarn
        deployMode: cluster
        file: /home/hadoop/scripts/pi.py
        args: [ 100 ]L

Telemetry Providers

Akamas can access Spark History Server statistics using the Spark History Server Provider. This provider maps the metrics in this optimization pack to the statistics provided by the History Server endpoint.

Here’s a configuration example for a telemetry provider instance:

provider: SparkHistoryServer
  address: sparkmaster.akamas.io
  port: 18080


See this page for an example of a study leveraging the Spark pack.

Last updated