Create Prometheus telemetry instances

To create an instance of the Prometheus provider, edit a YAML file (instance.yml in this example) with the definition of the instance:

# Prometheus Telemetry Provider Instance
provider: Prometheus

config:
 address: host1  # URL or IP of the Prometheus from which extract metrics
 port: 9090      # Port of the Prometheus from which extract metrics

Then you can create the instance for the system using the Akamas CLI:

akamas create telemetry-instance instance.yml system

Configuration options

When you create an instance of the Prometheus provider, you should specify some configuration information to allow the provider to extract and process metrics from Prometheus correctly.

You can specify configuration information within the config part of the YAML of the instance definition.

Required properties

address, a URL or IP identifying the address of the host where Prometheus is installed
port, the port exposed by Prometheus

Optional properties

user, the username for the Prometheus service
password, the user password for the Prometheus service
job, a string to specify the scraping job name. The default is ".*" for all scraping jobs
logLevel, set this to "DETAILED" for some extra logs when searching for metrics (default value is "INFO")
headers, to specify additional custom headers e.g: headers: "custom_key": "custom_value"
namespace, a string to specify the namespace
duration, integer to determine the duration in seconds for data collection (use a number between 1 and 3600)
enableHttps, boolean to enable HTTPS in Prometheus (since 3.2.6)
ignoreCertificates, boolean to ignore SSL certificates
disableConnectionCheck, boolean to disable initial connection check to Prometheus

Custom queries

The Prometheus provider allows defining additional queries to populate custom metrics or redefine the default ones according to your use case. You can configure additional metrics using the metrics field as shown in the configuration below:

config:
  address: host1
  port: 9090

metrics:
- metric: cust_metric   # extra akamas metric to monitor
  datasourceMetric: 'http_requests_total{environment=~"staging|testing|development", method!="GET"}' # query to execute to extract the metric
  labels:
  - method   # The "method" label will be retained within akamas

In this example, the telemetry instance will populate cust_metric with the results of the query specified in datasource, maintaining the value of the labels listed under labels.

Please refer to Querying basics | Prometheus for a complete reference of PromQL

Akamas placeholders

Akamas pre-processes the queries before running them, replacing special-purpose placeholders with the fields provided in the components. For example, given the following component definition:

name: jvm1
description: jvm1 for payment services
properties:
  prometheus:
    instance: service01
    job: jmx

the query sum(jvm_memory_used_bytes{instance=~"$INSTANCE$", job=~"$JOB$"}) will be expanded for this component into sum(jvm_memory_used_bytes{instance=~"service01", job=~"jmx"}). This provides greater flexibility through the templatization of the queries, allowing the same query to select the correct data sources for different components.

The following is the list of available placeholders:

Placeholder

Usage example

Component definition example

Expanded query

Description

$INSTANCE$ , $JOB$

node_load1{instance=~"$INSTANCE$", job=~"$JOB$"}

See Example below

node_load1{instance=~"frontend", job=~"node"}

These placeholders are replaced respectively with the instance and job fields configured in the component’s prometheus configuration.

%FILTERS%

container_memory_usage_bytes{job=~"$JOB$" %FILTERS%}

See Example below

container_memory_usage_bytes{job=~"advisor", name=~"db-.*"}

This placeholder is replaced with a list containing any additional filter in the component’s definition (other than instance and job), where each field is expanded as field_name=~"field_value". This is useful to define additional label matches in the query without the need to hardcode them.

$DURATION$

rate(http_client_requests_seconds_count[$DURATION$])

rate(http_client_requests_seconds_count[30s])

If not set in the component properties, this placeholder is replaced with the duration field configured in the telemety-instance. You should use it with range vectors instead of hardcoding a fixed value.

$NAMESPACE$ , $POD$ , $CONTAINER$

1e3 * avg(kube_pod_container_resource_limits{resource="cpu", namespace=~"$NAMESPACE$", pod=~"$POD$", container=~"$CONTAINER$" %FILTERS%})

See Collect Kubernetes metrics

1e3 * avg(kube_pod_container_resource_limits{resource="cpu", namespace=~"boutique", pod=~"adservice.*", container=~"server"})

These placeholders are used within kubernetes environments

Example

prometheus:
  instance: frontend
  job: node

Use cases

This section reports common use cases addressed by this provider.

Collect Kubernetes metrics

To gather kubernetes metrics, the following exporters are required:

kube-state-metrics
cadvisor

As an example, you can define a component with type Kubernetes Container in this way:

name: adservice
description: The adservice of the online boutique by Google
componentType: Kubernetes Container
properties:
  prometheus:
    namespace: boutique
    pod: adservice.*
    container: server

Collect Java metrics

Check Java OpenJDK page for a list of all the Java metrics available in Akamas

You can leverage the Prometheus provider to collect Java metrics by using the JMX Exporter. The JMX Exporter is a collector of Java metrics for Prometheus that can be run as an agent for any Java application. Once downloaded, you execute it alongside a Java application with this command:

java -javaagent:the_downloaded_jmx_exporter_jar.jar=9100:config.yaml -jar yourJar.jar

The command will expose on localhost on port 9100 Java metrics of youJar.jar __ which can be scraped by Prometheus.

config.yaml is a configuration file useful for the activity of this exporter. It is suggested to use this configuration for an optimal experience with the Prometheus provider:

startDelaySeconds: 0
username:
password:
ssl: false
lowercaseOutputName: false
lowercaseOutputLabelNames: false
# using the property above we are telling the export to export only relevant java metrics
whitelistObjectNames:
- "java.lang:*"
- "jvm:*"

As a next step, add a new scraping target in the configuration of the Prometheus used by the provider:

...
scrape_configs:
# JMX Exporter
- job_name: "jmx"
  static_configs:
  - targets: ["jmx_exporter_host:9100"]

You can then create a YAML file with the definition of a telemetry instance (prom_instance.yml) of the Prometheus provider:

name: Prometheus
config:
  address: prometheus_host
  port: 9090

And you can create the telemetry instance using the Akamas CLI:

akamas create telemetry-instance prom_instance.yml

Finally, to bind the extracted metrics to the related component, you should add the following field to the properties of the component’s definition:

prometheus:
  job: jmx

Collect system metrics

Check the Linux page for a list of all the system metrics available in Akamas

You can leverage the Prometheus provider to collect system metrics (Linux) by using the Node exporter. The Node exporter is a collector of system metrics for Prometheus that can be run as a standalone executable or a service within a Linux machine to be monitored. Once downloaded, schedule it as a service using, for example, systemd:

systemctl start node_exporter

Here’s the manifest of the node_exporter service:

[Unit]
Description=Node Exporter

[Service]
ExecStart=/path/to/node_exporter/executable

[Install]
WantedBy=default.target

The service will expose on localhost on port 9100 system metrics __ which can be scraped by Prometheus.

As a final step, add a new scraping target in the configuration of the Prometheus used by the provider:

...
scrape_configs:
# Node Exporter
- job_name: "node"
  static_configs:
  - targets: ["node_exporter_host:9100"]
  relabel_configs:
  - source_labels: ["__address__"]
    regex: "(.*):.*"
    # here we put as "instance", the name of the component the metrics refer to
    target_label: "instance"
    replacement: "linux_component_name"

You can then create a YAML file with the definition of a telemetry instance (prom_instance.yml) of the Prometheus provider:

provider: Prometheus
config:
  address: prometheus_host
  port: 9090

And you can create the telemetry instance using the Akamas CLI:

akamas create telemetry-instance prom_instance.yml

Finally, to bind the extracted metrics to the related component, you should add the following field to the properties of the component’s definition:

prometheus:
  instance: linux_component_name
  job: node

Last updated 1 year ago