# Create Prometheus telemetry instances To create an instance of the Prometheus provider, edit a YAML file (`instance.yml` in this example) with the definition of the instance: ```yaml # Prometheus Telemetry Provider Instance provider: Prometheus config: address: host1 # URL or IP of the Prometheus from which extract metrics port: 9090 # Port of the Prometheus from which extract metrics ``` Then you can create the instance for the `system` using the Akamas CLI: ```bash akamas create telemetry-instance instance.yml system ``` ### Configuration options When you create an instance of the Prometheus provider, you should specify some configuration information to allow the provider to extract and process metrics from Prometheus correctly. You can specify configuration information within the `config` part of the YAML of the instance definition. #### Required properties * `address`, a URL or IP identifying the address of the host where Prometheus is installed * `port`, the port exposed by Prometheus #### Optional properties * `user`, the username for the Prometheus service * `password`, the user password for the Prometheus service * `job`, a string to specify the scraping job name. The default is ".\*" for all scraping jobs * `logLevel`, set this to "DETAILED" for some extra logs when searching for metrics (default value is "INFO") * `headers`, to specify additional custom headers (e.g.: `headers: {key: value}`) * `namespace`, a string to specify the namespace * `duration`, integer to determine the duration in seconds for data collection (use a number between 1 and 3600) * `enableHttps`, boolean to enable HTTPS in Prometheus (since 3.2.6) * `ignoreCertificates`, boolean to ignore SSL certificates * `disableConnectionCheck`, boolean to disable initial connection check to Prometheus * `scope`, value of the X-Scope-OrgID header * `urlSuffix`, a suffix to postpone to the prometheus url. Such as `http://prometeus:9090/` * `scope`, value of the X-Scope-OrgID header ### Custom queries The Prometheus provider allows defining additional queries to populate custom metrics or redefine the default ones according to your use case. You can configure additional metrics using the `metrics` field as shown in the configuration below: ```yaml config: address: host1 port: 9090 metrics: - metric: cust_metric # extra akamas metric to monitor datasourceMetric: 'http_requests_total{environment=~"staging|testing|development", method!="GET"}' # query to execute to extract the metric labels: - method # The "method" label will be retained within akamas ``` In this example, the telemetry instance will populate `cust_metric` with the results of the query specified in `datasource`, maintaining the value of the labels listed under `labels`. Please refer to [Querying basics | Prometheus](https://prometheus.io/docs/prometheus/latest/querying/basics/) for a complete reference of PromQL #### Akamas placeholders Akamas pre-processes the queries before running them, replacing special-purpose placeholders with the fields provided in the components. For example, given the following component definition: ```yaml name: jvm1 description: jvm1 for payment services properties: prometheus: instance: service01 job: jmx ``` the query `sum(jvm_memory_used_bytes{instance=~"$INSTANCE$", job=~"$JOB$"})` will be expanded for this component into `sum(jvm_memory_used_bytes{instance=~"service01", job=~"jmx"})`. This provides greater flexibility through the templatization of the queries, allowing the same query to select the correct data sources for different components. The following is the list of available placeholders: | Placeholder | Usage example | Component definition example | Expanded query | Description | | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `$INSTANCE$`, `$JOB$` | `node_load1{instance=~"$INSTANCE$", job=~"$JOB$"}` | See [Example](#example) below | `node_load1{instance=~"frontend", job=~"node"}` | These placeholders are replaced respectively with the `instance` and `job` fields configured in the component’s `prometheus` configuration. | | `%FILTERS%` | `container_memory_usage_bytes{job=~"$JOB$" %FILTERS%}` | See [Example](#example) below | `container_memory_usage_bytes{job=~"advisor", name=~"db-.*"}` | This placeholder is replaced with a list containing any additional filter in the component’s definition (other than `instance` and `job`), where each field is expanded as `field_name=~"field_value"`. This is useful to define additional label matches in the query without the need to hardcode them. | | `$DURATION$` | `rate(http_client_requests_seconds_count[$DURATION$])` | | `rate(http_client_requests_seconds_count[30s])` | If not set in the component properties, this placeholder is replaced with the *duration* field configured in the telemety-instance. You should use it with [range vectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#range-vector-selectors) instead of hardcoding a fixed value. | | `$NAMESPACE$`, `$POD$`, `$CONTAINER$` | `1e3 * avg(kube_pod_container_resource_limits{resource="cpu", namespace=~"$NAMESPACE$", pod=~"$POD$", container=~"$CONTAINER$" %FILTERS%})` | See [Collect Kubernetes metrics](#collect-kubernetes-metrics) | `1e3 * avg(kube_pod_container_resource_limits{resource="cpu", namespace=~"boutique", pod=~"adservice.*", container=~"server"})` | These placeholders are used within kubernetes environments | #### Example ```yaml prometheus: instance: frontend job: node ``` ## Use cases This section reports common use cases addressed by this provider. ### Collect Kubernetes metrics To gather kubernetes metrics, the following exporters are required: * `kube-state-metrics` * `cadvisor` As an example, you can define a component with type `Kubernetes Container` in this way: ```yaml name: adservice description: The adservice of the online boutique by Google componentType: Kubernetes Container properties: prometheus: namespace: boutique pod: adservice.* container: server ``` ### Collect Java metrics Check [Java OpenJDK page](https://docs.akamas.io/akamas-docs/3.6/reference/optimization-packs/java-openjdk-pack) for a list of all the Java metrics available in Akamas You can leverage the Prometheus provider to collect Java metrics by using the [JMX Exporter](https://github.com/prometheus/jmx_exporter). The JMX Exporter is a collector of Java metrics for Prometheus that can be run as an agent for any Java application. Once downloaded, you execute it alongside a Java application with this command: ```bash java -javaagent:the_downloaded_jmx_exporter_jar.jar=9100:config.yaml -jar yourJar.jar ``` The command will expose on localhost on port 9100 Java metrics of `youJar.jar` \_\_ which can be scraped by Prometheus. `config.yaml` is a configuration file useful for the activity of this exporter. It is suggested to use this configuration for an optimal experience with the Prometheus provider: ```yaml startDelaySeconds: 0 username: password: ssl: false lowercaseOutputName: false lowercaseOutputLabelNames: false # using the property above we are telling the export to export only relevant Java metrics whitelistObjectNames: - "java.lang:*" - "jvm:*" ``` As a next step, add a new scraping target in the configuration of the Prometheus used by the provider: ```yaml ... scrape_configs: # JMX Exporter - job_name: "jmx" static_configs: - targets: ["jmx_exporter_host:9100"] ``` You can then create a YAML file with the definition of a telemetry instance (`prom_instance.yml`) of the Prometheus provider: ```yaml name: Prometheus config: address: prometheus_host port: 9090 ``` And you can create the telemetry instance using the Akamas CLI: ```bash akamas create telemetry-instance prom_instance.yml ``` Finally, to bind the extracted metrics to the related component, you should add the following field to the `properties` of the component’s definition: ```yaml prometheus: job: jmx ``` ### Collect system metrics Check the [Linux page](https://docs.akamas.io/akamas-docs/3.6/reference/optimization-packs/linux-pack) for a list of all the system metrics available in Akamas You can leverage the Prometheus provider to collect system metrics (Linux) by using the [Node exporter](https://github.com/prometheus/node_exporter). The Node exporter is a collector of system metrics for Prometheus that can be run as a standalone executable or a service within a Linux machine to be monitored. Once downloaded, schedule it as a service using, for example, *systemd*: ```bash systemctl start node_exporter ``` Here’s the manifest of the `node_exporter` service: ```ini [Unit] Description=Node Exporter [Service] ExecStart=/path/to/node_exporter/executable [Install] WantedBy=default.target ``` The service will expose on localhost on port 9100 system metrics \_\_ which can be scraped by Prometheus. As a final step, add a new scraping target in the configuration of the Prometheus used by the provider: ```yaml scrape_configs: # Node Exporter - job_name: "node" static_configs: - targets: ["node_exporter_host:9100"] relabel_configs: - source_labels: ["__address__"] regex: "(.*):.*" # here we put as "instance", the name of the component the metrics refer to target_label: "instance" replacement: "linux_component_name" ``` You can then create a YAML file with the definition of a telemetry instance (`prom_instance.yml`) of the Prometheus provider: ```yaml provider: Prometheus config: address: prometheus_host port: 9090 ``` And you can create the telemetry instance using the Akamas CLI: ```bash akamas create telemetry-instance prom_instance.yml ``` Finally, to bind the extracted metrics to the related component, you should add the following field to the `properties` of the component’s definition: ```yaml prometheus: instance: linux_component_name job: node ```