Spark optimization pack

The Spark optimization pack allows tuning applications running on the Apache Spark framework. Through this optimization pack, Akamas is able to explore the space of the Spark parameters in order to find the configurations that best optimize the allocated resources or the execution time.

To achieve these goals the optimization pack provides parameters that focus on the following areas:

• Driver and executors' resources allocation

• Parallelism

• Shuffling

• Spark SQL

Similarly, the bundled metrics provide visibility on the following statistics from the Spark History Server:

• Execution time

• Executors' resource usage

• Garbage collection time

Component Types

Component Type	Description
Spark Application 2.2.0	Spark Application 2.2.0
Spark Application 2.3.0	Spark Application 2.3.0
Spark Application 2.4.0	Spark Application 2.4.0

Installing

Here’s the command to install the Spark optimization pack using the Akamas CLI:

akamas install optimization-pack Spark