Create Spark History Server telemetry instances

Create a telemetry instance

To create an instance of the Spark History Server provider, build a YAML file (instance.yml in this example) with the definition of the instance:

provider: SparkHistoryServer
config:
  address: spark_master_node
  port: 18080
  importLevel: stage

Then you can create the instance for the system spark-system using the Akamas CLI:

akamas create telemetry-instance instance.yml spark-system

Configuration options

When you create an instance of the Spark History Server provider, you should specify some configuration information to allow the provider to correctly extract and process metrics from the Spark History server.

You can specify configuration information within the config part of the YAML of the instance definition.

Required properties

• address - hostname of the Spark History Server instance

Telemetry instance reference

The following YAML file describes the definition of a telemetry instance.

provider: SparkHistoryServer  # This is an instance of the Spark History Server provider
config:
  address: spark_master_node # The adress of Spark History Server
  port: 18080   # The port of Spark History Server
  importLevel: job  # The granularity of the imported metrics

The following table reports the reference for the config section within the definition of the Spark History Server provider instance:

Use cases

This section reports common use cases addressed by this provider.

Collect stage metrics of a Spark Application

Check Spark Application page for a list of all Spark application metrics available in Akamas

This example shows how to configure a Spark History Server provider in order to collect performance metrics about a Spark application submitted to the cluster using the Spark SSH Submit operator.

As a first step, you need to create a YAML file (spark_instance.yml) containing the configuration the provider needs to connect to the Spark History Server, plus the filter on the desired level of granularity for the imported metrics:

provider: SparkHistoryServer
config:
  address: spark_master_node
  port: 18080
  importLevel: stage

and then create the telemetry instance using the Akamas CLI:

akamas create telemetry-instance spark_instance.yml

Finally, you will need to define for your study a workflow that includes the submission of the Spark application to the cluster, in this case using the Spark SSH Submit operator:

name: spark_workflow
tasks:
  - name: Run Spark application
    operator: SSHSparkSubmit
    arguments:
      component: spark

Best practices

This section reports common best practices you can adopt to ease the use of this telemetry provider.

• configure metrics granularity: in order to reduce the collection time, configure the importLevel to import metrics with a granularity no finer than the study requires.

• wait for metrics publication: make sure in the workflow there is a few-minute interval between the end of the Spark application and the execution of the Spark telemetry instance, since the Spark History Server may take some time to complete the publication of the metrics.