Akamas Docs
3.1.2
3.1.2
  • How to use this documentation
  • Getting started with Akamas
    • Introduction to Akamas
    • Licensing
    • Deployment
      • Cloud Hosting
    • Security
    • Maintenance & Support (M&S) Services
      • Customer Support Services
      • Support levels for Customer Support Services
      • Support levels for software versions
      • Support levels with Akamas 3.1
  • Installing Akamas
    • Akamas Architecture
    • Prerequisites
      • Hardware Requirements
      • Software Requirements
      • Network requirements
    • Install Akamas dependencies
    • Install the Akamas Server
      • Online installation mode
        • Online installation behind a Proxy server
      • Offline installation mode
      • Changing UI Ports
      • Setup HTTPS configuration
    • Install the Akamas CLI
      • Setup the Akamas CLI
      • Verify the Akamas CLI
      • Initialize Akamas CLI
      • Change CLI configuration
    • Verify the Akamas Server
    • Install the Akamas license
    • Manage anonymous data collection
    • Install an Akamas Workstation
    • Troubleshoot install issues
    • Manage the Akamas Server
      • Akamas logs
      • Audit logs
      • Install upgrades and patches
      • Monitor the Akamas Server
      • Backup & Recover of the Akamas Server
  • Using Akamas
    • General optimization process and methodology
    • Preparing optimization studies
      • Modeling systems
      • Modeling components
        • Creating custom optimization packs
        • Managing optimization packs
      • Creating telemetry instances
      • Creating automation workflows
        • Creating workflows for offline studies
        • Performing load testing to support optimization activities
        • Creating workflows for live optimizations
      • Creating optimization studies
        • Defining optimization goal & constraints
        • Defining windowing policies
        • Defining KPIs
        • Defining parameters & metrics
        • Defining workloads
        • Defining optimization steps
        • Setting safety policies
    • Running optimization studies
      • Before running optimization studies
      • Analyzing results of offline optimization studies
        • Optimization Insights
      • Analyzing results of live optimization studies
      • Before applying optimization results
    • Guidelines for choosing optimization parameters
      • Guidelines for JVM (OpenJ9)
      • Guidelines for JVM layer (OpenJDK)
      • Guidelines for Oracle Database
      • Guidelines for PostgreSQL
    • Guidelines for defining optimization studies
      • Optimizing Linux
      • Optimizing Java OpenJDK
      • Optimizing OpenJ9
      • Optimizing Web Applications
      • Optimizing Kubernetes
      • Optimizing Spark
      • Optimizing Oracle Database
      • Optimizing MongoDB
      • Optimizing MySQL Database
      • Optimizing PostgreSQL
  • Integrating Akamas
    • Integrating Telemetry Providers
      • CSV provider
        • Install CSV provider
        • Create CSV provider instances
      • Dynatrace provider
        • Install Dynatrace provider
        • Create Dynatrace provider instances
      • Prometheus provider
        • Install Prometheus provider
        • Create Prometheus provider instances
        • CloudWatch Exporter
        • OracleDB Exporter
      • Spark History Server provider
        • Install Spark History Server provider
        • Create Spark History Server provider instances
      • NeoLoadWeb provider
        • Setup NeoLoadWeb telemetry provider
        • Create NeoLoadWeb provider instances
      • LoadRunner Professional provider
        • Install LoadRunner Professional provider
        • Create LoadRunner Professional provider instances
      • LoadRunner Enterprise provider
        • Install LoadRunner Enterprise provider
        • Create LoadRunner Enterprise provider instances
      • AWS provider
        • Install AWS provider
        • Create AWS provider instances
    • Integrating Configuration Management
    • Integrating Value Stream Delivery
    • Integrating Load Testing
      • Integrating NeoLoad
      • Integrating Load Runner Professional
      • Integrating LoadRunner Enterprise
  • Akamas Reference
    • Glossary
      • System
      • Component
      • Metric
      • Parameter
      • Component Type
      • Workflow
      • Telemetry Provider
      • Telemetry Instance
      • Optimization Pack
      • Goals & Constraints
      • KPI
      • Optimization Study
      • Offline Optimization Study
      • Live Optimization Study
      • Workspace
    • Construct templates
      • System template
      • Component template
      • Parameter template
      • Metric template
      • Component Types template
      • Telemetry Provider template
      • Telemetry Instance template
      • Workflows template
      • Study template
        • Goal & Constraints
        • Windowing policy
          • Trim windowing
          • Stability windowing
        • Parameter selection
        • Metric selection
        • Workload selection
        • KPIs
        • Steps
          • Baseline step
          • Bootstrap step
          • Preset step
          • Optimize step
        • Parameter rendering
    • Workflow Operators
      • General operator arguments
      • Executor Operator
      • FileConfigurator Operator
      • LinuxConfigurator Operator
      • WindowsExecutor Operator
      • WindowsFileConfigurator Operator
      • Sleep Operator
      • OracleExecutor Operator
      • OracleConfigurator Operator
      • SparkSSHSubmit Operator
      • SparkSubmit Operator
      • SparkLivy Operator
      • NeoLoadWeb Operator
      • LoadRunner Operator
      • LoadRunnerEnteprise Operator
    • Telemetry metric mapping
      • Dynatrace metrics mapping
      • Prometheus metrics mapping
      • NeoLoadWeb metrics mapping
      • Spark History Server metrics mapping
      • LoadRunner metrics mapping
    • Optimization Packs
      • Linux optimization pack
        • Amazon Linux
        • Amazon Linux 2
        • Amazon Linux 2022
        • CentOS 7
        • CentOS 8
        • RHEL 7
        • RHEL 8
        • Ubuntu 16.04
        • Ubuntu 18.04
        • Ubuntu 20.04
      • DotNet optimization pack
        • DotNet Core 3.1
      • Java-OpenJDK optimization pack
        • Java OpenJDK 8
        • Java OpenJDK 11
      • OpenJ9 optimization pack
        • IBM J9 VM 6
        • IBM J9 VM 8
        • Eclipse Open J9 11
      • NodeJS optimization pack
        • NodeJS
      • GO optimization pack
        • GO 1
      • Web Application optimization pack
        • Web Application
      • Docker optimization pack
        • Container
      • Kubernetes optimization pack
        • Kubernetes Pod
        • Kubernetes Container
        • Kubernetes Workload
        • Kubernetes Namespace
        • Kubernetes Cluster
      • WebSphere optimization pack
        • WebSphere 8.5
        • WebSphere Liberty ND
      • AWS optimization pack
        • EC2
        • Lambda
      • PostgreSQL optimization pack
        • PostgreSQL 11
        • PostgreSQL 12
      • Cassandra optimization pack
        • Cassandra
      • MySQL Database optimization pack
        • MySQL 8.0
      • Oracle Database optimization pack
        • Oracle Database 12c
        • Oracle Database 18c
        • Oracle Database 19c
        • RDS Oracle Database 11g
        • RDS Oracle Database 12c
      • MongoDB optimization pack
        • MongoDB 4
        • MongoDB 5
      • Elasticsearch optimization pack
        • Elasticsearch 6
      • Spark optimization pack
        • Spark Application 2.2.0
        • Spark Application 2.3.0
        • Spark Application 2.4.0
    • Command Line commands
      • Administration commands
      • User and Workspace management commands
      • Authentication commands
      • Resource management commands
      • Optimizer options commands
  • Knowledge Base
    • Setting up a Konakart environment for testing Akamas
    • Modeling a sample Java-based e-commerce application (Konakart)
    • Optimizing a web application
    • Optimizing a sample Java OpenJ9 application
    • Optimizing a sample Java OpenJDK application
    • Optimizing a sample Linux system
    • Optimizing a MongoDB server instance
    • Optimizing a Kubernetes application
    • Leveraging Ansible to automate AWS instance management
    • Guidelines for optimizing AWS EC2 instances
    • Optimizing a sample application running on AWS
    • Optimizing a Spark application
    • Optimizing an Oracle Database server instance
    • Optimizing an Oracle Database for an e-commerce service
    • Guidelines for optimizing Oracle RDS
    • Optimizing a MySQL server database running Sysbench
    • Optimizing a MySQL server database running OLTPBench
    • Optimizing a live K8s deployment
    • Optimizing a live full-stack deployment (K8s + JVM)
  • Akamas Free Trial
Powered by GitBook
On this page

Was this helpful?

Export as PDF
  1. Akamas Reference
  2. Telemetry metric mapping

Spark History Server metrics mapping

This page describes the mapping between metrics provided by Spark History Server to Akamas metrics for each supported component type

Component Type
Notes

Spark Application

Component metric
Granularity
Document Path
JSON query

spark_duration

job

/{appId}/1/jobs/{jobId}

.duration

spark_completed_tasks

job

/{appId}/1/jobs/{jobId}

.numCompletedTasks

spark_active_tasks

job

/{appId}/1/jobs/{jobId}

.numActiveTasks

spark_skipped_tasks

job

/{appId}/1/jobs/{jobId}

.numSkippedTasks

spark_failed_tasks

job

/{appId}/1/jobs/{jobId}

.numFailedTasks

spark_killed_tasks

job

/{appId}/1/jobs/{jobId}

.numKilledTasks

spark_completed_stages

job

/{appId}/1/jobs/{jobId}

.numCompletedStages

spark_failed_stages

job

/{appId}/1/jobs/{jobId}

.numFailedStages

spark_skipped_stages

job

/{appId}/1/jobs/{jobId}

.numSkippedStages

spark_active_stages

job

/{appId}/1/jobs/{jobId}

.numActiveStages

spark_duration

stage

/{appId}/1/stages/{stageId}

.getDuration

spark_task_stage_executor_run_time

stage

/{appId}/1/stages/{stageId}

.getExecutorRunTime

spark_task_stage_executor_cpu_time

stage

/{appId}/1/stages/{stageId}

.getExecutorCpuTime

spark_active_tasks

stage

/{appId}/1/stages/{stageId}

.getNumActiveTasks

spark_completed_tasks

stage

/{appId}/1/stages/{stageId}

.getNumCompleteTasks

spark_failed_tasks

stage

/{appId}/1/stages/{stageId}

.getNumFailedTasks

spark_killed_tasks

stage

/{appId}/1/stages/{stageId}

.getNumKilledTasks

spark_task_stage_input_bytes_read

stage

/{appId}/1/stages/{stageId}

.getInputBytes

spark_task_stage_input_records_read

stage

/{appId}/1/stages/{stageId}

.getInputRecords

spark_task_stage_output_bytes_written

stage

/{appId}/1/stages/{stageId}

.getOutputBytes

spark_task_stage_output_records_written

stage

/{appId}/1/stages/{stageId}

.getOutputRecords

spark_stage_shuffle_read_bytes

stage

/{appId}/1/stages/{stageId}

.getShuffleReadBytes

spark_task_stage_shuffle_read_records

stage

/{appId}/1/stages/{stageId}

.getShuffleReadRecords

spark_task_stage_shuffle_write_bytes

stage

/{appId}/1/stages/{stageId}

.getShuffleWriteBytes

spark_task_stage_shuffle_write_records

stage

/{appId}/1/stages/{stageId}

.getShuffleWriteRecords

spark_task_stage_memory_bytes_spilled

stage

/{appId}/1/stages/{stageId}

.getMemoryBytesSpilled

spark_task_stage_disk_bytes_spilled

stage

/{appId}/1/stages/{stageId}

.getDiskBytesSpilled

spark_duration

task

/{appId}/1/stages/{stageId}

.tasks[].duration

spark_task_executor_deserialize_time

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.executorDeserializeTime

spark_task_executor_deserialize_cpu_time

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.executorDeserializeCpuTime

spark_task_stage_executor_run_time

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.executorRunTime

spark_task_stage_executor_cpu_time

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.executorCpuTime

spark_task_result_size

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.resultSize

spark_task_jvm_gc_duration

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.jvmGcTime

spark_task_result_serialization_time

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.resultSerializationTime

spark_task_stage_memory_bytes_spilled

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.memoryBytesSpilled

spark_task_stage_disk_bytes_spilled

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.diskBytesSpilled

spark_task_peak_execution_memory

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.peakExecutionMemory

spark_task_stage_input_bytes_read

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.inputMetrics.bytesRead

spark_task_stage_input_records_read

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.inputMetrics.recordsRead

spark_task_stage_output_bytes_written

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.outputMetrics.bytesWritten

spark_task_stage_output_records_written

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.outputMetrics.recordsWritten

spark_task_shuffle_read_remote_blocks_fetched

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleReadMetrics.remoteBlocksFetched

spark_task_shuffle_read_local_blocks_fetched

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleReadMetrics.localBlocksFetched

spark_task_shuffle_read_fetch_wait_time

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleReadMetrics.fetchWaitTime

spark_task_shuffle_read_remote_bytes

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleReadMetrics.remoteBytesRead

spark_task_shuffle_read_remote_bytes_to_disk

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleReadMetrics.remoteBytesReadToDisk

spark_task_shuffle_read_local_bytes

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleReadMetrics.localBytesRead

spark_task_stage_shuffle_read_records

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleReadMetrics.recordsRead

spark_task_stage_shuffle_write_bytes

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleWriteMetrics.bytesWritten

spark_task_shuffle_write_time

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleWriteMetrics.writeTime

spark_task_stage_shuffle_write_records

task

/{appId}/1/stages/{stageId}

.tasks[].taskMetrics.shuffleWriteMetrics.recordsWritten

spark_executor_rdd_blocks

executor

/{appId}/1/allexecutors

select(.id!='driver) | .rddBlocks

spark_executor_mem_used

executor

/{appId}/1/allexecutors

select(.id!='driver) | .memoryUsed

spark_executor_disk_used

executor

/{appId}/1/allexecutors

select(.id!='driver) | .diskUsed

spark_executor_cores

executor

/{appId}/1/allexecutors

select(.id!='driver) | .totalCores

spark_active_tasks

executor

/{appId}/1/allexecutors

select(.id!='driver) | .activeTasks

spark_failed_tasks

executor

/{appId}/1/allexecutors

select(.id!='driver) | .failedTasks

spark_completed_tasks

executor

/{appId}/1/allexecutors

select(.id!='driver) | .completedTasks

spark_executor_total_tasks

executor

/{appId}/1/allexecutors

select(.id!='driver) | .totalTasks

spark_executor_total_duration

executor

/{appId}/1/allexecutors

select(.id!='driver) | .totalDuration

spark_executor_total_jvm_gc_duration

executor

/{appId}/1/allexecutors

select(.id!='driver) | .totalGCTime

spark_executor_total_input_bytes

executor

/{appId}/1/allexecutors

select(.id!='driver) | .totalInputBytes

spark_executor_total_shuffle_read

executor

/{appId}/1/allexecutors

select(.id!='driver) | .totalShuffleRead

spark_executor_total_shuffle_write

executor

/{appId}/1/allexecutors

select(.id!='driver) | .totalShuffleWrite

spark_executor_max_mem_used

executor

/{appId}/1/allexecutors

select(.id!='driver) | .maxMemory

spark_executor_used_on_heap_storage_memory

executor

/{appId}/1/allexecutors

select(.id!='driver) | .memoryMetrics.usedOnHeapStorageMemory

spark_executor_used_off_heap_storage_memory

executor

/{appId}/1/allexecutors

select(.id!='driver) | .memoryMetrics.usedOffHeapStorageMemory

spark_executor_total_on_heap_storage_memory

executor

/{appId}/1/allexecutors

select(.id!='driver) | .memoryMetrics.totalOnHeapStorageMemory

spark_executor_total_off_heap_storage_memory

executor

/{appId}/1/allexecutors

select(.id!='driver) | .memoryMetrics.totalOffHeapStorageMemory

spark_driver_rdd_blocks

driver

/{appId}/1/allexecutors

select(.id=='driver') | .rddBlocks

spark_driver_mem_used

driver

/{appId}/1/allexecutors

select(.id=='driver') | .memoryUsed

spark_driver_disk_used

driver

/{appId}/1/allexecutors

select(.id=='driver') | .diskUsed

spark_driver_cores

driver

/{appId}/1/allexecutors

select(.id=='driver') | .totalCores

spark_driver_total_duration

driver

/{appId}/1/allexecutors

select(.id=='driver') | .totalDuration

spark_driver_total_jvm_gc_duration

driver

/{appId}/1/allexecutors

select(.id=='driver') | .totalGCTime

spark_driver_total_input_bytes

driver

/{appId}/1/allexecutors

select(.id=='driver') | .totalInputBytes

spark_driver_total_shuffle_read

driver

/{appId}/1/allexecutors

select(.id=='driver') | .totalShuffleRead

spark_driver_total_shuffle_write

driver

/{appId}/1/allexecutors

select(.id=='driver') | .totalShuffleWrite

spark_driver_max_mem_used

driver

/{appId}/1/allexecutors

select(.id=='driver') | .maxMemory

spark_driver_used_on_heap_storage_memory

driver

/{appId}/1/allexecutors

select(.id=='driver') | .memoryMetrics.usedOnHeapStorageMemory

spark_driver_used_off_heap_storage_memory

driver

/{appId}/1/allexecutors

select(.id=='driver') | .memoryMetrics.usedOffHeapStorageMemory

spark_driver_total_on_heap_storage_memory

driver

/{appId}/1/allexecutors

select(.id=='driver') | .memoryMetrics.totalOnHeapStorageMemory

spark_driver_total_off_heap_storage_memory

driver

/{appId}/1/allexecutors

select(.id=='driver') | .memoryMetrics.totalOffHeapStorageMemory

Last updated 2 years ago

Was this helpful?

Spark Application