# Spark History Server metrics mapping This page describes the mapping between metrics provided by Spark History Server to Akamas metrics for each supported component type | Component Type | Notes | | ------------------------------------------------------- | ----- | | [Spark Application](#spark-application-component-types) | | ## Spark Application | Component metric | Granularity | Document Path | JSON query | | --------------------------------------------------- | ----------- | --------------------------- | ------------------------------------------------------------------- | | spark\_duration | job | /{appId}/1/jobs/{jobId} | `.duration` | | spark\_completed\_tasks | job | /{appId}/1/jobs/{jobId} | `.numCompletedTasks` | | spark\_active\_tasks | job | /{appId}/1/jobs/{jobId} | `.numActiveTasks` | | spark\_skipped\_tasks | job | /{appId}/1/jobs/{jobId} | `.numSkippedTasks` | | spark\_failed\_tasks | job | /{appId}/1/jobs/{jobId} | `.numFailedTasks` | | spark\_killed\_tasks | job | /{appId}/1/jobs/{jobId} | `.numKilledTasks` | | spark\_completed\_stages | job | /{appId}/1/jobs/{jobId} | `.numCompletedStages` | | spark\_failed\_stages | job | /{appId}/1/jobs/{jobId} | `.numFailedStages` | | spark\_skipped\_stages | job | /{appId}/1/jobs/{jobId} | `.numSkippedStages` | | spark\_active\_stages | job | /{appId}/1/jobs/{jobId} | `.numActiveStages` | | spark\_duration | stage | /{appId}/1/stages/{stageId} | `.getDuration` | | spark\_task\_stage\_executor\_run\_time | stage | /{appId}/1/stages/{stageId} | `.getExecutorRunTime` | | spark\_task\_stage\_executor\_cpu\_time | stage | /{appId}/1/stages/{stageId} | `.getExecutorCpuTime` | | spark\_active\_tasks | stage | /{appId}/1/stages/{stageId} | `.getNumActiveTasks` | | spark\_completed\_tasks | stage | /{appId}/1/stages/{stageId} | `.getNumCompleteTasks` | | spark\_failed\_tasks | stage | /{appId}/1/stages/{stageId} | `.getNumFailedTasks` | | spark\_killed\_tasks | stage | /{appId}/1/stages/{stageId} | `.getNumKilledTasks` | | spark\_task\_stage\_input\_bytes\_read | stage | /{appId}/1/stages/{stageId} | `.getInputBytes` | | spark\_task\_stage\_input\_records\_read | stage | /{appId}/1/stages/{stageId} | `.getInputRecords` | | spark\_task\_stage\_output\_bytes\_written | stage | /{appId}/1/stages/{stageId} | `.getOutputBytes` | | spark\_task\_stage\_output\_records\_written | stage | /{appId}/1/stages/{stageId} | `.getOutputRecords` | | spark\_stage\_shuffle\_read\_bytes | stage | /{appId}/1/stages/{stageId} | `.getShuffleReadBytes` | | spark\_task\_stage\_shuffle\_read\_records | stage | /{appId}/1/stages/{stageId} | `.getShuffleReadRecords` | | spark\_task\_stage\_shuffle\_write\_bytes | stage | /{appId}/1/stages/{stageId} | `.getShuffleWriteBytes` | | spark\_task\_stage\_shuffle\_write\_records | stage | /{appId}/1/stages/{stageId} | `.getShuffleWriteRecords` | | spark\_task\_stage\_memory\_bytes\_spilled | stage | /{appId}/1/stages/{stageId} | `.getMemoryBytesSpilled` | | spark\_task\_stage\_disk\_bytes\_spilled | stage | /{appId}/1/stages/{stageId} | `.getDiskBytesSpilled` | | spark\_duration | task | /{appId}/1/stages/{stageId} | `.tasks[].duration` | | spark\_task\_executor\_deserialize\_time | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.executorDeserializeTime` | | spark\_task\_executor\_deserialize\_cpu\_time | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.executorDeserializeCpuTime` | | spark\_task\_stage\_executor\_run\_time | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.executorRunTime` | | spark\_task\_stage\_executor\_cpu\_time | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.executorCpuTime` | | spark\_task\_result\_size | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.resultSize` | | spark\_task\_jvm\_gc\_duration | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.jvmGcTime` | | spark\_task\_result\_serialization\_time | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.resultSerializationTime` | | spark\_task\_stage\_memory\_bytes\_spilled | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.memoryBytesSpilled` | | spark\_task\_stage\_disk\_bytes\_spilled | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.diskBytesSpilled` | | spark\_task\_peak\_execution\_memory | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.peakExecutionMemory` | | spark\_task\_stage\_input\_bytes\_read | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.inputMetrics.bytesRead` | | spark\_task\_stage\_input\_records\_read | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.inputMetrics.recordsRead` | | spark\_task\_stage\_output\_bytes\_written | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.outputMetrics.bytesWritten` | | spark\_task\_stage\_output\_records\_written | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.outputMetrics.recordsWritten` | | spark\_task\_shuffle\_read\_remote\_blocks\_fetched | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleReadMetrics.remoteBlocksFetched` | | spark\_task\_shuffle\_read\_local\_blocks\_fetched | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleReadMetrics.localBlocksFetched` | | spark\_task\_shuffle\_read\_fetch\_wait\_time | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleReadMetrics.fetchWaitTime` | | spark\_task\_shuffle\_read\_remote\_bytes | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleReadMetrics.remoteBytesRead` | | spark\_task\_shuffle\_read\_remote\_bytes\_to\_disk | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleReadMetrics.remoteBytesReadToDisk` | | spark\_task\_shuffle\_read\_local\_bytes | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleReadMetrics.localBytesRead` | | spark\_task\_stage\_shuffle\_read\_records | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleReadMetrics.recordsRead` | | spark\_task\_stage\_shuffle\_write\_bytes | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleWriteMetrics.bytesWritten` | | spark\_task\_shuffle\_write\_time | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleWriteMetrics.writeTime` | | spark\_task\_stage\_shuffle\_write\_records | task | /{appId}/1/stages/{stageId} | `.tasks[].taskMetrics.shuffleWriteMetrics.recordsWritten` | | spark\_executor\_rdd\_blocks | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .rddBlocks` | | spark\_executor\_mem\_used | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .memoryUsed` | | spark\_executor\_disk\_used | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .diskUsed` | | spark\_executor\_cores | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .totalCores` | | spark\_active\_tasks | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .activeTasks` | | spark\_failed\_tasks | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .failedTasks` | | spark\_completed\_tasks | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .completedTasks` | | spark\_executor\_total\_tasks | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .totalTasks` | | spark\_executor\_total\_duration | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .totalDuration` | | spark\_executor\_total\_jvm\_gc\_duration | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .totalGCTime` | | spark\_executor\_total\_input\_bytes | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .totalInputBytes` | | spark\_executor\_total\_shuffle\_read | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .totalShuffleRead` | | spark\_executor\_total\_shuffle\_write | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .totalShuffleWrite` | | spark\_executor\_max\_mem\_used | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .maxMemory` | | spark\_executor\_used\_on\_heap\_storage\_memory | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .memoryMetrics.usedOnHeapStorageMemory` | | spark\_executor\_used\_off\_heap\_storage\_memory | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .memoryMetrics.usedOffHeapStorageMemory` | | spark\_executor\_total\_on\_heap\_storage\_memory | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .memoryMetrics.totalOnHeapStorageMemory` | | spark\_executor\_total\_off\_heap\_storage\_memory | executor | /{appId}/1/allexecutors | `select(.id!='driver) \| .memoryMetrics.totalOffHeapStorageMemory` | | spark\_driver\_rdd\_blocks | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .rddBlocks` | | spark\_driver\_mem\_used | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .memoryUsed` | | spark\_driver\_disk\_used | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .diskUsed` | | spark\_driver\_cores | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .totalCores` | | spark\_driver\_total\_duration | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .totalDuration` | | spark\_driver\_total\_jvm\_gc\_duration | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .totalGCTime` | | spark\_driver\_total\_input\_bytes | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .totalInputBytes` | | spark\_driver\_total\_shuffle\_read | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .totalShuffleRead` | | spark\_driver\_total\_shuffle\_write | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .totalShuffleWrite` | | spark\_driver\_max\_mem\_used | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .maxMemory` | | spark\_driver\_used\_on\_heap\_storage\_memory | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .memoryMetrics.usedOnHeapStorageMemory` | | spark\_driver\_used\_off\_heap\_storage\_memory | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .memoryMetrics.usedOffHeapStorageMemory` | | spark\_driver\_total\_on\_heap\_storage\_memory | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .memoryMetrics.totalOnHeapStorageMemory` | | spark\_driver\_total\_off\_heap\_storage\_memory | driver | /{appId}/1/allexecutors | `select(.id=='driver') \| .memoryMetrics.totalOffHeapStorageMemory` |