Spark Application 2.3.0
This page describes the Optimization Pack for Spark Application 2.3.0.
Metrics
Duration
Driver
Executors
Stages and Tasks
Parameters
Execution
CPU and Memory allocation
Shuffling
Dynamic allocation
SQL
Compression and Serialization
Constraints
The following tables show a list of constraints that may be required in the definition of the study, depending on the tuned parameters:
Cluster size
The overall resources allocated to the application should be constrained by a maximum and, sometimes, a minimum value:
the maximum value could be the sum of resources physically available in the cluster, or a lower limit to allow the concurrent execution of other applications
an optional minimum value could be useful to avoid configurations that allocate executors that are both small and scarce
Last updated