# Prometheus metrics mapping This page describes the mapping between metrics provided by Prometheus to Akamas metrics for each supported component type | Component Type | Notes | | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Linux](#linux) | | | [JVM](#jvm) | | | [Kubernetes Container](#kubernetes-container) | The default metrics in this table are based on the [cadvisor](https://github.com/google/cadvisor) and [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) | | [Kubernetes Pod](#kubernetes-component-types) | The default metrics in this table are based on the [cadvisor](https://github.com/google/cadvisor) and [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics) | | [EC2](#ec2) | The default metrics in this table are based on the [CloudWatch Exporter](https://docs.akamas.io/akamas-docs/3.6/integrating/integrating-telemetry-providers/prometheus-provider/cloudwatch-exporter), configured with the attached [custom configuration file](https://docs.akamas.io/akamas-docs/3.6/integrating/integrating-telemetry-providers/prometheus-provider/cloudwatch-exporter#custom-configuration-file) | | [Oracle Database](#oracle-database) | The default metrics in this table are based on the [OracleDB Exporter](https://docs.akamas.io/akamas-docs/3.6/integrating/integrating-telemetry-providers/prometheus-provider/oracledb-exporter), extending the default queries with the attached [custom configuration file](https://docs.akamas.io/akamas-docs/3.6/integrating/integrating-telemetry-providers/prometheus-provider/oracledb-exporter#custom-configuration-file) | | [**Web Application**](#web-application) | The default metrics in this table are based on the [Prometheus Listener for Jmeter](https://github.com/johrstrom/jmeter-prometheus-plugin) | ## Linux | Component metric | Prometheus query | | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | cpu\_load\_avg | node\_load1{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} | | cpu\_num | count(node\_cpu\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$", mode="system" %FILTERS%}) | | cpu\_used | sum by (job) (sum by (cpu, job) (rate(node\_cpu\_seconds\_total{instance=\~"$INSTANCE$", mode=\~"user\|system\|softirq\|irq\|nice", job=\~"$JOB$" %FILTERS%}\[$DURATION$]))) | | cpu\_util | avg by (job) (sum by (cpu, job) (rate(node\_cpu\_seconds\_total{instance=\~"$INSTANCE$", mode=\~"user\|system\|softirq\|irq\|nice", job=\~"$JOB$" %FILTERS%}\[$DURATION$]))) | | cpu\_util\_details | avg by (instance, cpu, mode, job) (sum by (instance, cpu, mode, job) (rate(node\_cpu\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]))) | | disk\_io\_inflight\_details | node\_disk\_io\_now{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} | | disk\_iops | sum by (instance, job) (rate(node\_disk\_writes\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) + sum by (instance, job) (rate(node\_disk\_reads\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_iops\_details | sum by (instance, device, job) (rate(node\_disk\_writes\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_iops\_details | sum by (instance, device, job) (rate(node\_disk\_reads\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_iops\_details | sum by (instance, device, job) (rate(node\_disk\_writes\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) + sum by (instance, device, job) (rate(node\_disk\_reads\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_iops\_reads | sum by (instance, job) (rate(node\_disk\_reads\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_iops\_writes | sum by (instance, job) (rate(node\_disk\_writes\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_read\_bytes | sum by (instance, device, job) (rate(node\_disk\_read\_bytes\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_read\_bytes\_details | sum by (instance, device, job) (rate(node\_disk\_read\_bytes\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_read\_write\_bytes | sum by (instance, device, job) (rate(node\_disk\_written\_bytes\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) + rate(node\_disk\_read\_bytes\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_response\_time | avg by (instance, job) ((rate(node\_disk\_read\_time\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) + rate(node\_disk\_write\_time\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) / (rate(node\_disk\_reads\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) + rate(node\_disk\_writes\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) > 0 )) | | disk\_response\_time\_details | avg by (instance, device, job) ((rate(node\_disk\_read\_time\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) + rate(node\_disk\_write\_time\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) / ((rate(node\_disk\_reads\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) + rate(node\_disk\_writes\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) > 0)) | | disk\_response\_time\_read | rate(node\_disk\_read\_time\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])/ rate(node\_disk\_reads\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | disk\_response\_time\_worst | max by (instance, job) ((rate(node\_disk\_read\_time\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) + rate(node\_disk\_write\_time\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) / (rate(node\_disk\_reads\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) + rate(node\_disk\_writes\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) > 0 )) | | disk\_response\_time\_write | rate(node\_disk\_write\_time\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])/ rate(node\_disk\_writes\_completed\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | disk\_swap\_used | node\_memory\_SwapTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_memory\_SwapFree\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} | | disk\_swap\_util | ((node\_memory\_SwapTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_memory\_SwapFree\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) / (node\_memory\_SwapTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} > 0)) or ((node\_memory\_SwapTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_memory\_SwapFree\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%})) | | disk\_util\_details | rate(node\_disk\_io\_time\_seconds\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | disk\_write\_bytes | sum by (instance, device, job) (rate(node\_disk\_written\_bytes\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | disk\_write\_bytes\_details | sum by (instance, device, job) (rate(node\_disk\_written\_bytes\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | filesystem\_size | node\_filesystem\_size\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} | | filesystem\_used | node\_filesystem\_size\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_filesystem\_free\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} | | filesystem\_util | ((node\_filesystem\_size\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_filesystem\_free\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) / node\_filesystem\_size\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) | | mem\_fault\_major | rate(node\_vmstat\_pgmajfault{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | mem\_fault\_minor | rate(node\_vmstat\_pgfault{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | mem\_swapins | rate(node\_vmstat\_pswpin{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | mem\_swapouts | rate(node\_vmstat\_pswpout{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | mem\_total | node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} | | mem\_used | (node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_memory\_MemFree\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) | | mem\_util | (node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_memory\_MemFree\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) / node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} | | mem\_util\_details | (node\_memory\_Active\_file\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} / node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) | | mem\_util\_details | (node\_memory\_Active\_anon\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} / node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) | | mem\_util\_details | (node\_memory\_Inactive\_file\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} / node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) | | mem\_util\_details | (node\_memory\_Inactive\_anon\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} / node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) | | mem\_util\_nocache | (node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_memory\_Buffers\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_memory\_Cached\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} - node\_memory\_MemFree\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}) / node\_memory\_MemTotal\_bytes{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} | | network\_in\_bytes\_details | rate(node\_network\_receive\_bytes\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | network\_out\_bytes\_details | rate(node\_network\_transmit\_bytes\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | network\_tcp\_retrans | rate(node\_netstat\_Tcp\_RetransSegs{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | os\_context\_switch | rate(node\_context\_switches\_total{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%}\[$DURATION$]) | | proc\_blocked | node\_procs\_blocked{instance=\~"$INSTANCE$", job=\~"$JOB$" %FILTERS%} | ## JVM | Component metric | Prometheus query | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | jvm\_heap\_size | avg(jvm\_memory\_bytes\_max{area="heap" %FILTERS%}) | | jvm\_heap\_committed | avg(jvm\_memory\_bytes\_committed{area="heap" %FILTERS%}) | | jvm\_heap\_used | avg(jvm\_memory\_bytes\_used{area="heap" %FILTERS%}) | | jvm\_off\_heap\_used | avg(jvm\_memory\_bytes\_used{area="nonheap" %FILTERS%}) | | jvm\_heap\_util | avg(jvm\_memory\_bytes\_used{area="heap" %FILTERS%} / jvm\_memory\_bytes\_max{area="heap" %FILTERS%}) | | jvm\_memory\_used | avg(sum by (instance) (jvm\_memory\_bytes\_used)) | | jvm\_heap\_young\_gen\_size | avg(sum by (instance) (jvm\_memory\_pool\_bytes\_max{pool=\~".\*Eden Space\|.\*Survivor Space" %FILTERS%})) | | jvm\_heap\_young\_gen\_used | avg(sum by (instance) (jvm\_memory\_pool\_bytes\_used{pool=\~".\*Eden Space\|.\*Survivor Space" %FILTERS%})) | | jvm\_heap\_old\_gen\_size | avg(sum by (instance) (jvm\_memory\_pool\_bytes\_max{pool=\~".\*Tenured Gen\|.\*Old Gen" %FILTERS%})) | | jvm\_heap\_old\_gen\_used | avg(sum by (instance) (jvm\_memory\_pool\_bytes\_used{pool=\~".\*Tenured Gen\|.\*Old Gen" %FILTERS%})) | | jvm\_memory\_buffer\_pool\_used | avg(sum by (instance) (jvm\_buffer\_pool\_used\_bytes)) | | jvm\_gc\_time | avg(sum by (instance) (rate(jvm\_gc\_collection\_seconds\_sum\[$DURATION$]))) | | jvm\_gc\_count | avg(sum by (instance) (rate(jvm\_gc\_collection\_seconds\_count\[$DURATION$]))) | | jvm\_gc\_duration | (sum(rate(jvm\_gc\_collection\_seconds\_sum\[$DURATION$])) / sum(rate(jvm\_gc\_collection\_seconds\_count\[$DURATION$])) > 0 ) or sum(rate(jvm\_gc\_collection\_seconds\_count\[$DURATION$])) | | jvm\_threads\_current | avg(jvm\_threads\_current) | | jvm\_threads\_deadlocked | avg(jvm\_threads\_deadlocked) | ## Kubernetes workload | Component metric | Prometheus query | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | | k8s\_workload\_desired\_pods | kube\_deployment\_spec\_replicas{namespace=\~"$NAMESPACE$", deployment=\~"$DEPLOYMENT$" %FILTERS%} | | k8s\_workload\_running\_pods | kube\_deployment\_status\_replicas\_available{namespace=\~"$NAMESPACE$", deployment=\~"$DEPLOYMENT$" %FILTERS%} | | k8s\_workload\_ready\_pods | kube\_deployment\_status\_replicas\_ready{namespace=\~"$NAMESPACE$", deployment=\~"$DEPLOYMENT$" %FILTERS%} | | k8s\_workload\_cpu\_used | 1e3 \* sum(rate(container\_cpu\_usage\_seconds\_total{container="", namespace=\~"$NAMESPACE$", pod=\~"$DEPLOYMENT$.\*" %FILTERS%}\[$DURATION$])) | | k8s\_workload\_memory\_used | sum(last\_over\_time(container\_memory\_usage\_bytes{container="", namespace=\~"$NAMESPACE$", pod=\~"$DEPLOYMENT$.\*" %FILTERS%}\[$DURATION$])) | | k8s\_workload\_cpu\_request | 1e3 \* sum(kube\_pod\_container\_resource\_requests{resource="cpu", namespace=\~"$NAMESPACE$", pod=\~"$DEPLOYMENT$.\*" %FILTERS%}) | | k8s\_workload\_cpu\_limit | 1e3 \* sum(kube\_pod\_container\_resource\_limits{resource="cpu", namespace=\~"$NAMESPACE$", pod=\~"$DEPLOYMENT$.\*" %FILTERS%}) | | k8s\_workload\_memory\_request | sum(kube\_pod\_container\_resource\_requests{resource="memory", namespace=\~"$NAMESPACE$", pod=\~"$DEPLOYMENT$.\*" %FILTERS%}) | | k8s\_workload\_memory\_limit | sum(kube\_pod\_container\_resource\_limits{resource="memory", namespace=\~"$NAMESPACE$", pod=\~"$DEPLOYMENT$.\*" %FILTERS%}) | ## Kubernetes Pod | Component metric | Prometheus metric | | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | | k8s\_pod\_cpu\_used | 1e3 \* avg(rate(container\_cpu\_usage\_seconds\_total{container="", namespace=\~"$NAMESPACE$", pod=\~"$POD$" %FILTERS%}\[$DURATION$])) | | k8s\_pod\_cpu\_request | 1e3 \* avg(sum by (pod) (kube\_pod\_container\_resource\_requests{resource="cpu", namespace=\~"$NAMESPACE$", pod=\~"$POD$" %FILTERS%})) | | k8s\_pod\_cpu\_limit | 1e3 \* avg(sum by (pod) (kube\_pod\_container\_resource\_limits{resource="cpu", namespace=\~"$NAMESPACE$", pod=\~"$POD$" %FILTERS%})) | | k8s\_pod\_memory\_used | avg(last\_over\_time(container\_memory\_usage\_bytes{container="", namespace=\~"$NAMESPACE$", pod=\~"$POD$" %FILTERS%}\[$DURATION$])) | | k8s\_pod\_memory\_working\_set | avg(container\_memory\_working\_set\_bytes{container="", namespace=\~"$NAMESPACE$", pod=\~"$POD$" %FILTERS%}) | | k8s\_pod\_memory\_request | avg(sum by (pod) (kube\_pod\_container\_resource\_requests{resource="memory", namespace=\~"$NAMESPACE$", pod=\~"$POD$" %FILTERS%})) | | k8s\_pod\_memory\_limit | avg(sum by (pod) (kube\_pod\_container\_resource\_limits{resource="memory", namespace=\~"$NAMESPACE$", pod=\~"$POD$" %FILTERS%})) | | k8s\_pod\_restarts | avg(sum by (pod) (increase(kube\_pod\_container\_status\_restarts\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$" %FILTERS%}\[$DURATION$]))) | ## Kubernetes Container The following metrics are configured to work for Kubernetes. | Component metric | Prometheus query | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | container\_cpu\_used | 1e3 \* avg(rate(container\_cpu\_usage\_seconds\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$])) | | container\_cpu\_used\_max | 1e3 \* max(rate(container\_cpu\_usage\_seconds\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$])) | | container\_cpu\_util | avg(rate(container\_cpu\_usage\_seconds\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$]) / on (pod) group\_left kube\_pod\_container\_resource\_limits{resource="cpu", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}) | | container\_cpu\_util\_max | max(rate(container\_cpu\_usage\_seconds\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$]) / on (pod) group\_left kube\_pod\_container\_resource\_limits{resource="cpu", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}) | | container\_cpu\_throttled\_millicores | 1e3 \* avg(rate(container\_cpu\_cfs\_throttled\_seconds\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$])) | | container\_cpu\_throttle\_time | avg(last\_over\_time(container\_cpu\_cfs\_throttled\_periods\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$]) / container\_cpu\_cfs\_periods\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}) | | container\_memory\_used | avg(last\_over\_time(container\_memory\_working\_set\_bytes{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$])) | | container\_memory\_used\_max | max(last\_over\_time(container\_memory\_working\_set\_bytes{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$])) | | container\_memory\_util | avg(last\_over\_time(container\_memory\_working\_set\_bytes{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$]) / on (pod) group\_left kube\_pod\_container\_resource\_limits{resource="memory", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}) | | container\_memory\_util\_max | max(last\_over\_time(container\_memory\_working\_set\_bytes{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$]) / on (pod) group\_left kube\_pod\_container\_resource\_limits{resource="memory", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}) | | container\_memory\_resident\_set | avg(last\_over\_time(container\_memory\_rss{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$])) | | container\_memory\_cache | avg(last\_over\_time(container\_memory\_cache{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$])) | | container\_cpu\_request | 1e3 \* avg(kube\_pod\_container\_resource\_requests{resource="cpu", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}) | | container\_cpu\_limit | 1e3 \* avg(kube\_pod\_container\_resource\_limits{resource="cpu", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}) | | container\_memory\_request | avg(kube\_pod\_container\_resource\_requests{resource="memory", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}) | | container\_memory\_limit | avg(kube\_pod\_container\_resource\_limits{resource="memory", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}) | | container\_restarts | avg(increase(kube\_pod\_container\_status\_restarts\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$])) | | container\_oom\_kills\_count | avg(increase(container\_oom\_events\_total{namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%}\[$DURATION$])) | | cost | sum(kube\_pod\_container\_resource\_requests{resource="cpu", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%})\*29 + sum(kube\_pod\_container\_resource\_requests{resource="memory", namespace=\~"$NAMESPACE$", pod=\~"$POD$", container=\~"$CONTAINER$" %FILTERS%})/1024/1024/1024\*8 | ## EC2 | Component metric | Prometheus query | | ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | cpu\_util | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() aws\_ec2\_cpuutilization\_average{job='$JOB$'}/100 | | network\_in\_bytes\_details | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() (aws\_ec2\_network\_in\_sum{job='$JOB$'} \* count\_over\_time(aws\_ec2\_network\_in\_sum{job='$JOB$'}\[300s]) / 300) | | network\_out\_bytes\_details | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() (aws\_ec2\_network\_out\_sum{job='$JOB$'} \* count\_over\_time(aws\_ec2\_network\_out\_sum{job='$JOB$'}\[300s]) / 300) | | aws\_ec2\_credits\_cpu\_available | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() aws\_ec2\_cpucredit\_balance\_average{job='$JOB$'} | | aws\_ec2\_credits\_cpu\_used | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() aws\_ec2\_cpucredit\_usage\_sum{job='$JOB$'} | | disk\_read\_bytes | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() (aws\_ec2\_ebsread\_bytes\_sum{job='$JOB$'} \* count\_over\_time(aws\_ec2\_ebsread\_bytes\_sum{job='$JOB$'}\[300s]) / 300) | | disk\_write\_bytes | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() (aws\_ec2\_ebswrite\_bytes\_sum{job='$JOB$'} \* count\_over\_time(aws\_ec2\_ebswrite\_bytes\_sum{job='$JOB$'}\[300s]) / 300) | | aws\_ec2\_disk\_iops | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() ((aws\_ec2\_ebsread\_ops\_sum{job='$JOB$'} + aws\_ec2\_ebswrite\_ops\_sum{job='$JOB$'}) \* count\_over\_time(aws\_ec2\_ebsread\_ops\_sum{job='$JOB$'}\[300s])/300) | | aws\_ec2\_disk\_iops\_reads | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() (aws\_ec2\_ebsread\_ops\_sum{job='$JOB$'} \* count\_over\_time(aws\_ec2\_ebsread\_ops\_sum{job='$JOB$'}\[300s]) / 300) | | aws\_ec2\_disk\_iops\_writes | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() (aws\_ec2\_ebswrite\_ops\_sum{job='$JOB$'} \* count\_over\_time(aws\_ec2\_ebswrite\_ops\_sum{job='$JOB$'}\[300s]) / 300) | | aws\_ec2\_ebs\_credits\_io\_util | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() aws\_ec2\_ebsiobalance\_\_average{job='$JOB$'} / 100 | | aws\_ec2\_ebs\_credits\_bytes\_util | aws\_resource\_info{instance='$INSTANCE$', job='$JOB$' %FILTERS%} \* on(instance\_id) group\_left() aws\_ec2\_ebsbyte\_balance\_\_average{job='$JOB$'} / 100 | ## Oracle Database | Component metric | Prometheus query | | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | oracle\_sga\_total\_size | oracledb\_memory\_size{component='SGA Target', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_sga\_free\_size | oracledb\_memory\_size{component='Free SGA Memory Available', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_sga\_max\_size | oracledb\_memory\_size{component='Maximum SGA Size', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_pga\_target\_size | oracledb\_memory\_size{component='PGA Target', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_redo\_buffers\_size | oracledb\_memory\_size{component='Redo Buffers', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_default\_buffer\_cache\_size | oracledb\_memory\_size{component='DEFAULT buffer cache', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_default\_2k\_buffer\_cache\_size | oracledb\_memory\_size{component='DEFAULT 2K buffer cache', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_default\_4k\_buffer\_cache\_size | oracledb\_memory\_size{component='DEFAULT 4K buffer cache', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_default\_8k\_buffer\_cache\_size | oracledb\_memory\_size{component='DEFULT 8K buffer cache', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_default\_16k\_buffer\_cache\_size | oracledb\_memory\_size{component='DEFAULT 16K buffer cache', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_default\_32k\_buffer\_cache\_size | oracledb\_memory\_size{component='DEFAULT 32K buffer cache', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_keep\_buffer\_cache\_size | oracledb\_memory\_size{component='KEEP buffer cache', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_recycle\_buffer\_cache\_size | oracledb\_memory\_size{component='RECYCLE buffer cache', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_asm\_buffer\_cache\_size | oracledb\_memory\_size{component='ASM Buffer Cache', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_shared\_io\_pool\_size | oracledb\_memory\_size{component='Shared IO Pool', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_java\_pool\_size | oracledb\_memory\_size{component='java pool', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_large\_pool\_size | oracledb\_memory\_size{component='large pool', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_shared\_pool\_size | oracledb\_memory\_size{component='shared pool', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_streams\_pool\_size | oracledb\_memory\_size{component='streams pool', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_sessions\_active\_user | oracledb\_sessions\_value{type='USER', status='ACTIVE', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_sessions\_inactive\_user | oracledb\_sessions\_value{type='USER', status='INACTIVE', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_sessions\_active\_background | oracledb\_sessions\_value{type='BACKGROUND', status='ACTIVE', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_sessions\_inactive\_background | oracledb\_sessions\_value{type='BACKGROUND', status='INACTIVE', instance='$INSTANCE$', job='$JOB$' %FILTERS%} | | oracle\_buffer\_cache\_hit\_ratio | ttps\://docs.oracle.com/database/121/TGDBA/tune\_buffer\_cache.htm#TGDBA533 | | oracle\_redo\_log\_space\_requests | rate(oracledb\_activity\_redo\_log\_space\_requests{instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]) | | oracle\_wait\_event\_log\_file\_sync | rate(oracledb\_system\_event\_time\_waited{event='log file sync', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_log\_file\_parallel\_write | rate(oracledb\_system\_event\_time\_waited{event='log file sequential read', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_log\_file\_sequential\_read | rate(oracledb\_system\_event\_time\_waited{event='log file parallel write', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_enq\_tx\_contention | rate(oracledb\_system\_event\_time\_waited{event='enq: TX - contention', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_enq\_tx\_row\_lock\_contention | rate(oracledb\_system\_event\_time\_waited{event='enq: TX - row lock contention', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_latch\_row\_cache\_objects | rate(oracledb\_system\_event\_time\_waited{event='latch: row cache objects', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_latch\_shared\_pool | rate(oracledb\_system\_event\_time\_waited{event='latch: shared pool', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_resmgr\_cpu\_quantum | rate(oracledb\_system\_event\_time\_waited{event='resmgr:cpu quantum', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_sql\_net\_message\_from\_client | rate(oracledb\_system\_event\_time\_waited{event='SQL\*Net message from client', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_rdbms\_ipc\_message | rate(oracledb\_system\_event\_time\_waited{event='rdbms ipc message', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_db\_file\_sequential\_read | rate(oracledb\_system\_event\_time\_waited{event='db file sequential read', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_log\_file\_switch\_checkpoint\_incomplete | rate(oracledb\_system\_event\_time\_waited{event='log file switch (checkpoint incomplete)', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_row\_cache\_lock | rate(oracledb\_system\_event\_time\_waited{event='row cache lock', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_buffer\_busy\_waits | rate(oracledb\_system\_event\_time\_waited{event='buffer busy waits', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_event\_db\_file\_async\_io\_submit | rate(oracledb\_system\_event\_time\_waited{event='db file async I/O submit', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$])/100 | | oracle\_wait\_class\_commit | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='Commit', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | | oracle\_wait\_class\_concurrency | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='Concurrency', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | | oracle\_wait\_class\_system\_io | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='System I/O', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | | oracle\_wait\_class\_user\_io | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='User I/O', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | | oracle\_wait\_class\_other | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='Other', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | | oracle\_wait\_class\_scheduler | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='Scheduler', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | | oracle\_wait\_class\_idle | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='Idle', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | | oracle\_wait\_class\_application | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='Application', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | | oracle\_wait\_class\_network | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='Network', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | | oracle\_wait\_class\_configuration | sum without(event) (rate(oracledb\_system\_event\_time\_waited{wait\_class='Configuration', instance='$INSTANCE$', job='$JOB$' %FILTERS%}\[$DURATION$]))/100 | ## Web Application | Component metric | Prometheus query | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | transactions\_response\_time | avg(rate(ResponseTime\_sum{code="200", job=\~"$JOB$" %FILTERS%}\[$DURATION$])/rate(ResponseTime\_count{code="200", job=\~"$JOB$" %FILTERS%}\[$DURATION$])>0) | | transactions\_response\_time\_max | max(rate(ResponseTime\_sum{code="200", job=\~"$JOB$" %FILTERS%}\[$DURATION$])/rate(ResponseTime\_count{code="200", job=\~"$JOB$" %FILTERS%}\[$DURATION$])>0) | | transactions\_response\_time\_min | min(rate(ResponseTime\_sum{code="200", job=\~"$JOB$" %FILTERS%}\[$DURATION$])/rate(ResponseTime\_count{code="200", job=\~"$JOB$" %FILTERS%}\[$DURATION$])>0) | | transactions\_response\_time\_p50 | ResponseTime{quantile="0.5", code="200", job=\~"$JOB$" %FILTERS%} | | transactions\_response\_time\_p85 | ResponseTime{quantile="0.85", code="200", job=\~"$JOB$" %FILTERS%} | | transactions\_response\_time\_p90 | ResponseTime{quantile="0.9", code="200", job=\~"$JOB$" %FILTERS%} | | transactions\_response\_time\_p99 | ResponseTime{quantile="0.99", code="200", job=\~"$JOB$" %FILTERS%} | | transactions\_throughput | sum(rate(Ratio\_success{job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | transactions\_error\_throughput | sum(rate(Ratio\_failure{job=\~"$JOB$" %FILTERS%}\[$DURATION$])) | | transactions\_error\_rate | (avg(rate(Ratio\_failure{job=\~"$JOB$" %FILTERS%}\[$DURATION$]))/avg(rate(Ratio\_total{job=\~"$JOB$" %FILTERS%}\[$DURATION$])))\*100 | | users | sum(jmeter\_threads{state="active", job=\~"$JOB$" %FILTERS%}) |