Install the Telemetry Provider

Skip this part if the Telemetry Provider is already installed.

To install the Dynatrace provider, create a YAML file (called provider.yml in this example) with the definition of the provider:

Then you can install the provider using the Akamas CLI:

To install the CSV File provider, create a YAML file (called provider.yml in this example) with the specification of the provider:

# CSV File Telemetry Provider
name: CSV File
description: Telemetry Provider that enables to import of metrics from a remote CSV file
dockerImage: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/telemetry-providers/csv-file-provider:3.2.0

Then, you can then install the provider with the Akamas CLI:

akamas install telemetry-provider provider.yml

The CSV provider collects metrics from CSV files and makes them available to Akamas. It offers a very versatile way to integrate custom data sources.

Prerequisites

This section provides the minimum requirements that you should match before using the CSV File telemetry provider.

Network requirements

The following requirements should be met to enable the provider to gather CSV files from remote hosts:

• Port 22 (or a custom one) should be open from Akamas installation to the host where the files reside.

• The host where the files reside should support SCP or SFTP protocols.

Permissions

• Read access to the CSV files target of the integration

Akamas supported version

• Versions < 2.0.0 are compatibile with Akamas until version 1.8.0

• Versions >= 2.0.0 are compatible with Akamas from version 1.9.0

Supported component types

The CSV File provider is generic and allows integration with any data source, therefore it does not come with support for a specific component type.

Setup the data source

To operate properly, the CSV file provider expects the presence of four fields in each processed CSV file:

• A timestamp field used to identify the point in time a certain sample refers to.

• A component field used to identify the Akamas entity.

• A metric field used to identify the name of the metric.

• A value field used to store the actual value of the metric.

These fields can have custom names in the CSV file, you can specify them in the provider configuration.

The Install CSV provider page describes how to get this Telemetry Provider installed. Once installed, this provider is shared with all users of your Akamas installation and can be used to monitor many different systems, by configuring appropriate telemetry provider instances as described in the Create a CSV provider instance page.

The Dynatrace provider collects metrics from Dynatrace and makes them available to Akamas.

This provider includes support for several technologies. In any case, custom queries can be defined to gather the desired metrics.

Supported versions

Dynatrace SaaS/Managed version 1.187 or later

Supported component types:

• Kubernetes and Docker

• Web Application

• Ubuntu-16.04, Rhel-7.6

• java-openjdk-8, java-openjdk-11, java-openjdk-17

• java-ibm-j9vm-6, java-ibm-j9vm-8, java-eclipse-openj9-11

Refer to Dynatrace provider metrics mapping to see how component-types metrics are extracted by this provider.

Prerequisites

This section provides the minimum requirements that you should match before using the Prometheus provider.

• Dynatrace SaaS/Managed version 1.187 or later

• A valid Dynatrace license

• Dynatrace OneAgent installed on the servers where the Dynatrace entities to be monitored are running

• Connectivity between Akamas and the Dynatrace server on port 443

• A Dynatrace API token with the privileges described here.

Dynatrace Token

The Dynatrace provider needs a Dynatrace API token with the following privileges:

• metrics.read (Read metrics)

• entities.read (Read entities and tags)

• DataExport (Access problem and event feed, metrics, and topology)

• ReadSyntheticData (Read synthetic monitors, locations, and nodes)

• DataImport (Data ingest, e.g.: metrics and events). This permission is used to inform Dynatrace about configuration changes.

To generate an API Token for your Dynatrace installation you can follow these steps.

Component configuration

To instruct Akamas from which Dynatrace entities (e.g. Workloads, Services, Process Groups) metrics should be collected you can some specific properties on components.

Different strategies can be used to map Dynatrace entities to Akamas components:

• By id

• By name

• By tags

• By Kubernetes properties

By id

You can map a component to a Dynatrace entity by leveraging the unique id of the entity, which you should put under the id property in the component. This strategy is best used for long-lived instances whose ID does not change during the optimization such as Hosts, Process Groups, or Services.

Here is an example of how to setup host monitoring via id:

name: My Host
properties:
 dynatrace:
  id: HOST-12345YUAB1

You can find the id of a Dynatrace entity by looking at the URL of a Dynatrace dashboard relative to the entity. Watch out that the "host" key is valid only for Linux components, other components (e.g. the JVM) must drill down into the host entities to get the PROCESS_GROUP_INSTANCE or PROCESS_GROUP id.

By name

You can map a component to a Dynatrace entity by leveraging the entity’s display name. This strategy is similar to the map by id but provides a more friendly way to identify the mapped entity. Beware that id multiple entities in your Dynatrace installation share the same name they will all be mapped to the same component. The Dynatrace display name should be put under the name property in the component definition:

name: MyComponent
properties:
 dynatrace:
  name: host-1

By tags

You can map a component to a Dynatrace entity by leveraging Dynatrace tags that match the entity, tags which you should put under the tags property in the component definition.

If multiple tags are specified, instances matching any of the specified tags will be selected.

This sample configuration maps to the component all Dynatrace entities with tag environment: test or [AWS]dynatrace-monitored: true

name: MyComponent
properties:
 dynatrace:
  tags:
     environment: test
     [AWS]dynatrace-monitored: true

Dynatrace supports both key-value and key-only tags. Key-only tags can be specified as Key-value tags with an empty value as in the following example

name: MyComponent
properties:
 dynatrace:
  tags:
     myKeyOnlyTag: ""

By Kubernetes properties

By leveraging dedicated properties, you can map a component to a Dynatrace entity referring to a Kubernetes cluster (e.g., a Pod or a Container).

Container

To properly identify the set of containers to be mapped, you can specify the following properties. Any container matching all the properties will be mapped to the component.

You can retrieve all the information to setup the properties on the top of the Dynatrace container dashboard.

The following example shows how to map a component to a container running in Kubernetes:

dynatrace:
  type: CONTAINER_GROUP_INSTANCE
  kubernetes:
    namespace: boutique
    containerName: server
    basePodName: ak-frontend-*

Pod

To properly identify the set of pods to be mapped, you can specify the following properties. Any pod matching all the properties will be mapped to the component.

If you need to narrow your pod selection further you can also specify a set of tags as described in the by tags. Note that tags for Kubernetes resources are called Labels in the Dynatrace dashboard.

Labels are specified as key-value in the Akamas configuration. In Dynatrace’s dashboard key and value are separated by a column (:)

Example

The following example shows how to map a component to a pod running in Kubernetes:

dynatrace:
  type: CLOUD_APPLICATION_INSTANCE
  namePrefix: ak-frontend-
  kubernetes:
    labels:
      workload: ak-frontend
      product: hipstershop

Container, Pod, or Workload?

Please note, that when you are mapping components to Kubernetes entities the property type is required to instruct Akamas on which type of entity you want to map. Dynatrace maps Kubernetes entities to the following types:

Improve component mapping with type

You can improve the matching of components with Dynatrace by adding a type property in the component definition, this property will help the provider match only those Dynatrace entities of the given type.

name: MyComponent
properties:
 dynatrace:
  type: SERVICE     # the type helps the mapping by tags by filtering down entities that are only services
  tags:
     environment: test
     "[AWS]dynatrace-monitored": true

The type of an entity can be retrieved from the URL of the entity’s dashboard

Available entity types can be retrieved, from your Dynatrace instance, with the following command:

curl 'https://<Your Dynatrace host>/api/v2/entityTypes/?pageSize=500' \
  --header 'Authorization: Api-Token <API-TOKEN>'

Mapping multiple entities in one component

In some circumstances, you might want to map multiple Dyantrace entities (e.g. a set of hosts) to the same Akamas component and import aggregated metrics.

This can be easily done by using tags. If Akamas detects that multiple entities have been mapped to the same component it will try to aggregate metrics; some metrics, however, can not be automatically aggregated.

To force aggregation on all available metrics you can add the mergeable: trueproperty to the component under the Dynatrace element.

name: MyComponent
properties:
 dynatrace:
  mergeable: true
  tags:
     environment: test
     [AWS]dynatrace-monitored: true

To create an instance of the CSV provider, build a YAML file (instance.yml in this example) with the definition of the instance:

# CSV Telemetry Provider Instance
provider: CSV File
config:
  address: host1.example.com
  authType: password
  username: akamas
  auth: akamas
  remoteFilePattern: /monitoring/result-*.csv
  componentColumn: COMPONENT
  timestampColumn: TS
  timestampFormat: YYYY-MM-dd'T'HH:mm:ss
metrics:
  - metric: cpu_util
    datasourceMetric: user%

Then you can create the instance for the system using the Akamas CLI:

akamas create telemetry-instance instance.yml system

timestampFormat format

Notice that the week-year format YYYY is compliant with the ISO-8601 specification, but you should replace it with the year-of-era format yyyy if you are specifying a timestampFormat different from the ISO one. For example:

• Correct: yyyy-MM-dd HH:mm:ss

• Wrong: YYYY-MM-dd HH:mm:ss

You can find detailed information on timestamp patterns in the Patterns for Formatting and Parsing section on the DateTimeFormatter (Java Platform SE 8) page.

Configuration options

When you create an instance of the CSV provider, you should specify some configuration information to allow the provider to correctly extract and process metrics from your CSV files.

You can specify configuration information within the config part of the YAML of the instance definition.

Required properties

• address - a URL or IP identifying the address of the host where CSV files reside

• username - the username used when connecting to the host

• authType - the type of authentication to use when connecting to the file host; either password or key

• auth - the authentication credential; either a password or a key according to authType. When using keys, the value can either be the value of the key or the path of the file to import from

• remoteFilePattern - a list of remote files to be imported

Optional properties

• protocol - the protocol to use to retrieve files; either scp or sftp. Default is scp

• fieldSeparator - the character used as a field separator in the CSV files. Default is ,

• componentColumn - the header of the column containing the name of the component. Default is COMPONENT

• timestampColumn - the header of the column containing the timestamp. Default is TS

• timestampFormat - the format of the timestamp (e.g. yyyy-MM-dd HH:mm:ss zzz). Default is YYYY-MM-ddTHH:mm:ss

You should also specify the mapping between the metrics available in your CSV files and those provided by Akamas. This can be done in the metrics section of the telemetry instance configuration. To map a custom metric you should specify at least the following properties:

• metric - the name of a metric in Akamas

• datasourceMetric - the header of a column that contains the metric in the CSV file

The provider ignores any column not present as datasourceMetric in this section.

The sample configuration reported in this section would import the metric cpu_util from CSV files formatted as in the example below:

TS,                   COMPONENT,  user%
2020-04-17T09:46:30,  host,       20
2020-04-17T09:46:35,  host,       23
2020-04-17T09:46:40,  host,       32
2020-04-17T09:46:45,  host,       21

Telemetry instance reference

The following represents the complete configuration reference for the telemetry provider instance.

provider: CSV File             # this is an instance of the CSV provider
config:
  address: host1.example.com   # the address of the host with the CSV files
  port: 22                     # the port used to connect
  authType: password           # the authentication method
  username: akamas             # the username used to connect
  auth: akamas                 # the authentication credential
  protocol: scp                # the protocol used to retrieve the file
  fieldSeparator: ","          # the character used as field separator in the CSV files
  remoteFilePattern: /monitoring/result-*.csv    # the path of the CSV files to import
  componentColumn: COMPONENT                     # the header of the column with component names
  timestampColumn: TS                            # the header of the column with the time stamp
  timestampFormat: YYYY-mm-ddTHH:MM:ss           # the format of the timestamp
metrics:
  - metric: cpu_util                             # the name of the Akamas metric
    datasourceMetric: user%                      # the header of the column with the original metric
    staticLabels:
      mode: user                                 # (optional) additional labels to add to the metric

The following table reports the configuration reference for the config section

The following table reports the configuration reference for the metrics section

Use cases

Here you can find common use cases addressed by this provider.

Linux SAR

In this use case, you are going to import some metrics coming from SAR, a popular UNIX tool to monitor system resources. SAR can export CSV files in the following format.

hostname, interval,     timestamp, 		        %user,	%system,      %memory
machine1, 600,		2018-08-07 06:45:01 UTC,	30.01,	20.77,		96.21
machine1, 600,		2018-08-07 06:55:01 UTC,	40.07,	13.00,		84.55
machine1, 600,		2018-08-07 07:05:01 UTC,	5.00,	90.55,		89.23

Note that the metrics are percentages (between 1 and 100), while Akamas accepts percentages as values between 0 and 1, therefore each metric in this configuration has a scale factor of 0.001.

You can import the two CPU metrics and the memory metric from a SAR log using the following telemetry instance configuration.

provider: CSV File
config:
  remoteFilePattern: /csv/sar.csv
  address: 127.0.0.1
  port: 22
  username: user123
  auth: password123
  authType: password
  protocol: scp
  componentColumn: hostname
  timestampColumn: timestamp
  timestampFormat: yyyy-MM-dd HH:mm:ss zzz
metrics:
  - metric: cpu_util
    datasourceMetric: %user
    scale: 0.001
    staticLabels:
      mode: user
  - metric: cpu_util
    datasourceMetric: %system
    scale: 0.001
    staticLabels:
      mode: system
  - metric: mem_util
    scale: 0.001
    datasourceMetric: %memory

Using the configured instance, the CSV File provider will perform the following operations to import the metrics:

1. Retrieve the file "/csv/sar.csv" from the server "127.0.0.1" using the SCP protocol authenticating with the provided password.

2. Use the column hostname to lookup components by name.

3. Use the column timestamp to find the timestamps of the samples (that are expected to be in the format specified by timestampFormat).

4. Collect the metrics (two with the same name, but different labels, and one with a different name):

   • cpu_util: in the CSV file is in the column %user and attach to its samples the label "mode" with value "user".

   • cpu_util: in the CSV file is in the column %system and attach to its samples the label "mode" with value "system".

   • mem_util: in the CSV file is in the column %memory.

Akamas supports the integration with virtually any telemetry and observability tool.

Supported Telemetry Providers

The following table describes the supported Telemetry Providers, which are created automatically at installation time.

Notice that Telemetry Providers are shared across all the workspaces within the same Akamas installation, and only users with administrative privileges can manage them.

By default, only requests at the service level are imported by the telemetry provider.

To import specific key requests you can follow these steps.

Component Creation

Create a new component of type Web Application for each key request you want to import. This allows tracking response time, throughput, and error rates separately.

You can use the following yaml file as an example and customize it to suit your needs.

In order to instruct Akamas to import a specific key request you just need to change the id field of the yaml above to the one that matches your key request on Dynatarce.

To obtain that ID open the analysis page for the request as in the example below, take note of the URL of the page, and look for the SERVICE_METHOD keyword. The id is the one starting with SERVICE_METHOD and ending before the character %14

Considering the example below the id is SERVICE_METHOD-D4BCC949D5DD656A

Telemetry instance setup

Create a telemetry instance for your system using the yaml specified below as an example and modify it to provide your Dynatrace account and credentials. This will instruct Akamas to use key request metrics instead of service metrics.

The installed provider is shared with all users of your Akamas installation and can monitor many different systems, by configuring appropriate telemetry provider instances.

To create an instance of the Dynatrace provider, build a YAML file (instance.yml in this example) with the definition of the instance:

Then you can create the instance for the system using the Akamas CLI:

Configuration options

When you create an instance of the Dynatrace provider, you should specify some configuration information to allow the provider to correctly extract and process metrics from Dynatrace.

You can specify configuration information within the config part of the YAML of the instance definition.

Required properties

• url - URL of the Dynatrace installation API (see to retrieve the URL of your installation)

• token - A Dynatrace API Token with the

Collect additional metrics

You can collect additional metrics with the Dynatrace provider by using the metrics field:

Configure a proxy for Dynatrace

In the case in which Akamas cannot reach directly your Dynatrace installation, you can configure an HTTP proxy by using the proxy field:

Telemetry instance reference

This section reports the complete reference for the definition of a telemetry instance.

This table shows the reference for the config section within the definition of the Dynatrace provider instance:

Proxy options reference

This table reports the reference for the config → proxy section within the definition of the Dynatrace provider instance:

Metrics options reference

This table reports the reference for the metrics section within the definition of the Dynatrace provider instance. The section contains a collection of objects with the following properties:

Use cases

This section reports common use cases addressed by this provider.

Collect system metrics

Check the Linux optimization pack for a list of all the system metrics available in Akamas.

As a second step, choose a strategy to map your Linux component (MyLinuxComponent) with the corresponding Dyntrace entity.

Let’s assume you want to map by id your Dynatrace entity, you can find the id in the URL bar of a Dyntrace dashboard of the entity:

Grab the id and add it to the Linux component definition:

You can leverage the name of the entity as well:

As a third and final step, once the component is all set, you can create an instance of the Dynatrace provider and then build your first studies:

To install the Prometheus provider, create a YAML file (provider.yml in this example) with the definition of the provider:

Then you can install the provider using the Akamas CLI:

The installed provider is shared with all users of your Akamas installation and can monitor many different systems, by configuring appropriate telemetry provider instances.

The Prometheus provider collects metrics from a Prometheus instance and makes them available to Akamas.

This provider includes support for several technologies (Prometheus exporters). In any case, custom queries can be defined to gather the desired metrics.

Prerequisites

This section provides the minimum requirements that you should match before using the Prometheus provider.

Supported Prometheus versions:

Akamas supports Prometheus starting from version2.26.

Connectivity between the Akamas server and the Prometheus server is also required. By default, Prometheus is run on port 9090.

Supported Prometheus exporters

• (Linux system metrics)

• (Java metrics)

• (Docker container metrics)

• exporter (AWS resources metrics)

• (Web application metrics)

The Prometheus provider includes queries for most of the monitoring use cases these exporters cover. If you need to specify custom queries or make use of exporters not currently supported you can specify them as described in creating .

Supported Akamas component types

• Kubernetes (Pod, Container, Workload, Namespace)

• Web Application

• Java (java-ibm-j9vm-6, java-ibm-j9vm-8, java-eclipse-openj9-11, java-openjdk-8, java-openjdk-11, java-openjdk-17)

• Linux (Ubuntu-16.04, Rhel-7.6)

Component configuration

Akamas reasons in terms of a system to be optimized and in terms of parameters and metrics of components of that system. To understand which metrics collected from Prometheus should be mapped to a component, the Prometheus provider looks up some properties in the components of a system grouped under prometheus property. These properties depend on the exporter and the component type.

Nested under this property you can also include any additional field your use case may require to filter the imported metrics further. These fields will be appended in queries to the list of label matches in the form field_name=~'field_value', and can specify either exact values or patterns.

It is important that you add instance and, optionally, the job properties to the components of a system so that the Prometheus provider can gather metrics from them:

Prometheus configuration

The Prometheus provider does not usually require a specific configuration of the Prometheus instance it uses.

When gathering metrics for hosts it's usually convenient to set the value of the instance label so that it matches the value of the instance property in a component; in this way, the Prometheus provider knows which system component each data point refers to.

Here’s an example configuration for Prometheus that sets the instance label:

To create an instance of the Prometheus provider, edit a YAML file (instance.yml in this example) with the definition of the instance:

Then you can create the instance for the system using the Akamas CLI:

Configuration options

When you create an instance of the Prometheus provider, you should specify some configuration information to allow the provider to extract and process metrics from Prometheus correctly.

You can specify configuration information within the config part of the YAML of the instance definition.

Required properties

• address, a URL or IP identifying the address of the host where Prometheus is installed

• port, the port exposed by Prometheus

Optional properties

• user, the username for the Prometheus service

• password, the user password for the Prometheus service

• job, a string to specify the scraping job name. The default is ".*" for all scraping jobs

• logLevel, set this to "DETAILED" for some extra logs when searching for metrics (default value is "INFO")

• headers, to specify additional custom headers (e.g.: headers: {key: value})

• namespace, a string to specify the namespace

• duration, integer to determine the duration in seconds for data collection (use a number between 1 and 3600)

• enableHttps, boolean to enable HTTPS in Prometheus (since 3.2.6)

• ignoreCertificates, boolean to ignore SSL certificates

• disableConnectionCheck, boolean to disable initial connection check to Prometheus

Custom queries

The Prometheus provider allows defining additional queries to populate custom metrics or redefine the default ones according to your use case. You can configure additional metrics using the metrics field as shown in the configuration below:

In this example, the telemetry instance will populate cust_metric with the results of the query specified in datasource, maintaining the value of the labels listed under labels.

Akamas placeholders

Akamas pre-processes the queries before running them, replacing special-purpose placeholders with the fields provided in the components. For example, given the following component definition:

the query sum(jvm_memory_used_bytes{instance=~"$INSTANCE$", job=~"$JOB$"}) will be expanded for this component into sum(jvm_memory_used_bytes{instance=~"service01", job=~"jmx"}). This provides greater flexibility through the templatization of the queries, allowing the same query to select the correct data sources for different components.

The following is the list of available placeholders:

Example

Use cases

This section reports common use cases addressed by this provider.

Collect Kubernetes metrics

To gather kubernetes metrics, the following exporters are required:

• kube-state-metrics

• cadvisor

As an example, you can define a component with type Kubernetes Container in this way:

Collect Java metrics

The command will expose on localhost on port 9100 Java metrics of youJar.jar __ which can be scraped by Prometheus.

config.yaml is a configuration file useful for the activity of this exporter. It is suggested to use this configuration for an optimal experience with the Prometheus provider:

As a next step, add a new scraping target in the configuration of the Prometheus used by the provider:

You can then create a YAML file with the definition of a telemetry instance (prom_instance.yml) of the Prometheus provider:

And you can create the telemetry instance using the Akamas CLI:

Finally, to bind the extracted metrics to the related component, you should add the following field to the properties of the component’s definition:

Collect system metrics

Here’s the manifest of the node_exporter service:

The service will expose on localhost on port 9100 system metrics __ which can be scraped by Prometheus.

As a final step, add a new scraping target in the configuration of the Prometheus used by the provider:

You can then create a YAML file with the definition of a telemetry instance (prom_instance.yml) of the Prometheus provider:

And you can create the telemetry instance using the Akamas CLI:

Finally, to bind the extracted metrics to the related component, you should add the following field to the properties of the component’s definition:

To install the Spark History Server provider, create a YAML file (called provider.yml in this example) with the definition of the provider:

# Spark History Server Telemetry Provider
name: SparkHistoryServer
description: Telemetry Provider that enables to import of metrics from Spark History Server instances
dockerImage: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/telemetry-providers/spark-history-server-provider:3.2.0

Then you can install the provider using the Akamas CLI:

akamas install telemetry-provider provider.yml

The installed provider is shared with all users of your Akamas installation and can monitor many different systems, by configuring appropriate telemetry provider instances.

This page describes how to set up an OracleDB exporter in order to gather metrics regarding an Oracle Database instance through the Prometheus provider.

Installation

The OracleDB exporter repository is available on the official project page. The suggested deploy mode is through a Docker image, since the Prometheus instance can easily access the running container through the Akamas network.

Use the following command line to run the container, where cust-metrics.toml is your configuration file defining the queries for additional custom metrics (see paragraph below) and DATA_SOURCE_NAME an environment variable containing the Oracle EasyConnect string:

docker run -d --name oracledb_exporter --restart always \
  --network akamas -p 9161:9161 \
  -v ~/oracledb_exporter/cust-metrics.toml:/cust-metrics.toml \
  -e CUSTOM_METRICS=/cust-metrics.toml \
  -e DATA_SOURCE_NAME="username/password@//oracledb.mycompany.com/service" \
  iamseth/oracledb_exporter

You can refer to the official guide for more details or alternative deployment modes.

Custom queries

It is possible to define additional queries to expose custom metrics using any data in the database instance that is readable by the monitoring user (see the guide for more details about the syntax).

Custom Configuration file

The following is an example of exporting system metrics from the Dynamic Performance (V$) Views used by the Prometheus provider default queries for the Oracle Database optimization pack:

[[metric]]
context= "memory"
labels= [ "component" ]
metricsdesc= { size="Component memory extracted from v$memory_dynamic_components in Oracle." }
request = '''
SELECT component, current_size as "size"
FROM V$MEMORY_DYNAMIC_COMPONENTS
UNION
SELECT name, bytes as "size"
FROM V$SGAINFO
WHERE name in ('Free SGA Memory Available', 'Redo Buffers', 'Maximum SGA Size')
'''

[[metric]]
context = "activity"
metricsdesc = { value="Generic counter metric from v$sysstat view in Oracle." }
fieldtoappend = "name"
request = '''
SELECT name, value
FROM V$SYSSTAT WHERE name IN (
  'execute count',
  'user commits', 'user rollbacks',
  'db block gets from cache', 'consistent gets from cache', 'physical reads cache', /* CACHE */
  'redo log space requests'
 )
 '''

[[metric]]
context = "system_event"
labels = [ "event", "wait_class" ]
request = '''
SELECT
  event, wait_class,
  total_waits, time_waited
FROM V$SYSTEM_EVENT
'''
[metric.metricsdesc]
  total_waits= "Total number of waits for the event as per V$SYSTEM_EVENT in Oracle."
  time_waited= "Total time waited for the event (in hundredths of seconds) as per V$SYSTEM_EVENT in Oracle."

The Spark History Server provider collects metrics from a Spark History Server instance and makes them available to Akamas.

Prerequisites

This section provides the minimum requirements that you should match before using the Spark History Server telemetry provider.

Supported versions

• Apache Spark 2.3

Network requirements

• Spark History Server API must be reachable at the provided address and port (the default port is 18080).

Supported component types

• spark-application

You can check Spark History Server provider metrics mapping to see how component-types metrics are extracted by this provider.

Akamas supported version

• Versions < 2.0.0 are compatible with Akamas until version 1.8.0

• Versions >= 2.0.0 are compatible with Akamas from version 1.9.0

Workflow requirements

This section lists the workflow operators this provider depends on:

• SparkSubmit Operator

• SparkSSHSubmit operator

• SparkLivy Operator

Components configuration

Akamas uses components to identify specific elements of the system to be monitored and optimized. Your system might contain multiple components to model, for example, a Spark application and each host of the cluster. To point Akamas to the right component when extracting metrics you need to add a property called sparkApplication to your Spark Application component. The provider will only extract metrics for components for which this property has been specified.

name: My Application
properties:
 sparkApplication: "true"

The NeoLoad Web provider collects metrics from a NeoLoad Web instance and makes them available to Akamas.

Prerequisites

This section provides the minimum requirements that you should match before using the NeoLoad Web telemetry provider.

Supported versions

• NeoLoad Web SaaS or managed version 7.1 or later.

Network requirements

• The NeoLoad Web API must be reachable at a provided address and port (by default https://neoload-api.saas.neotys.com).

Permissions

• NeoLoad Web API access token.

Akamas supported version

• Versions < 2.0.0 are compatibile with Akamas untill version 1.8.0

• Versions >= 2.0.0 are compatible with Akamas from version 1.9.0

Supported component types

• Web Application

You can check NeoLoadWeb provider metrics mapping to see how component-types metrics are extracted by this provider.

Workflow requirements

This section lists the workflow operators this provider depends on.

• NeoLoadWeb operator

Components configuration

Akamas reasons in terms of a system to be optimized and in terms of parameters and metrics of components of that system. To understand which metrics collected from NeoloadWeb should refer to which component, the NeoloadWeb provider looks up the property neoloadweb in the components of a system:

name: MyComponent
properties:
 neoloadweb: "true" # The presence of this property helps akamas discriminate metrics imported using neoloadweb from the ones imported by other providers for the same component

Create a telemetry instance

To create an instance of the Spark History Server provider, build a YAML file (instance.yml in this example) with the definition of the instance:

provider: SparkHistoryServer
config:
  address: spark_master_node
  port: 18080
  importLevel: stage

Then you can create the instance for the system spark-system using the Akamas CLI:

akamas create telemetry-instance instance.yml spark-system

Configuration options

When you create an instance of the Spark History Server provider, you should specify some configuration information to allow the provider to correctly extract and process metrics from the Spark History server.

You can specify configuration information within the config part of the YAML of the instance definition.

Required properties

• address - hostname of the Spark History Server instance

Telemetry instance reference

The following YAML file describes the definition of a telemetry instance.

provider: SparkHistoryServer  # This is an instance of the Spark History Server provider
config:
  address: spark_master_node # The adress of Spark History Server
  port: 18080   # The port of Spark History Server
  importLevel: job  # The granularity of the imported metrics

The following table reports the reference for the config section within the definition of the Spark History Server provider instance:

Use cases

This section reports common use cases addressed by this provider.

Collect stage metrics of a Spark Application

Check Spark Application page for a list of all Spark application metrics available in Akamas

This example shows how to configure a Spark History Server provider in order to collect performance metrics about a Spark application submitted to the cluster using the Spark SSH Submit operator.

As a first step, you need to create a YAML file (spark_instance.yml) containing the configuration the provider needs to connect to the Spark History Server, plus the filter on the desired level of granularity for the imported metrics:

provider: SparkHistoryServer
config:
  address: spark_master_node
  port: 18080
  importLevel: stage

and then create the telemetry instance using the Akamas CLI:

akamas create telemetry-instance spark_instance.yml

Finally, you will need to define for your study a workflow that includes the submission of the Spark application to the cluster, in this case using the Spark SSH Submit operator:

name: spark_workflow
tasks:
  - name: Run Spark application
    operator: SSHSparkSubmit
    arguments:
      component: spark

Best practices

This section reports common best practices you can adopt to ease the use of this telemetry provider.

• configure metrics granularity: in order to reduce the collection time, configure the importLevel to import metrics with a granularity no finer than the study requires.

• wait for metrics publication: make sure in the workflow there is a few-minute interval between the end of the Spark application and the execution of the Spark telemetry instance, since the Spark History Server may take some time to complete the publication of the metrics.

To install the LoadRunner provider, create a YAML file (called provider.yml in this example) with the definition of the provider:

# LoadRunner Telemetry Provider
name: LoadRunner
description: Telemetry Provider that enables to import of metrics from LoadRunner installations
dockerImage: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/telemetry-providers/loadrunner-provider:3.2.0

Then you can install the provider using the Akamas CLI:

akamas install telemetry-provider provider.yml

The installed provider is shared with all users of your Akamas installation and can monitor many different systems, by configuring appropriate telemetry provider instances.

To install the NeoLoad Web provider, create a YAML file (called provider.yml in this example) with the definition of the provider:

# NeoLoad Web Telemetry Provider
name: NeoLoadWeb
description: Telemetry Provider that enables to import of metrics from NeoLoad Web instances
dockerImage: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/telemetry-providers/neoload-web-provider:3.2.0

Then you can install the provider using the Akamas CLI:

akamas install telemetry-provider provider.yml

The installed provider is shared with all users of your Akamas installation and can monitor many different systems, by configuring appropriate telemetry provider instances.

This page describes how to set up a CloudWatch exporter in order to gather AWS metrics through the Prometheus provider. This is especially useful to monitor system metrics when you don’t have direct SSH access to AWS resources like EC2 Instances or if you want to gather AWS-specific metrics not available in the guest OS.

AWS policies

In order to fetch metrics fromCloudWatch, the exporter requires an IAM user or role with the following privileges:

• cloudwatch:GetMetricData

• cloudwatch:GetMetricStatistics

• cloudwatch:ListMetrics

• tag:GetResources

You can assign AWS-managed policies CloudWatchReadOnlyAccess and ResourceGroupsandTagEditorReadOnlyAccess to the desired user to enable these permissions.

Exporter configuration

The CloudWatch exporter repository is available on the official project page. It requires a minimal configuration to fetch metrics from the desired AWS instances. Below is a short list of the parameters needed for a minimal configuration:

• region: AWS region of the monitored resource

• metrics: a list of objects containing filters for the exported metrics

  • aws_namespace: the namespace of the monitored resource

  • aws_metric_name: the name of the AWS metric to fetch

  • aws_dimensions: the dimension to expose as labels

  • aws_dimension_select: the dimension to filter over

  • aws_statistics: the list of metric statistics to expose

  • aws_tag_select: optional tags to filter on

    • tag_selections: map containing the list of values to select for each tag

    • resource_type_selection: resource type to fetch the tags from (see: Resource Types)

    • resource_id_dimension: dimension to use for the resource id (see: Resource Types)

For a complete list of possible values for namespaces, metrics, and dimensions please refer to the official AWS CloudWatch User Guide.

Notice: AWS bills CloudWatch usage in batches of 1 million requests, where every metric counts as a single request. To avoid unnecessary expenses configure only the metrics you need.

region: us-east-2
metrics:
  - aws_namespace: AWS/EC2
    aws_metric_name: CPUUtilization
    aws_statistics: [Average]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: NetworkIn
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: NetworkOut
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: NetworkPacketsIn
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: NetworkPacketsOut
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: CPUCreditUsage
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: CPUCreditBalance
    aws_statistics: [Average]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSReadOps
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSWriteOps
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSReadBytes
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSWriteBytes
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSIOBalance%
    aws_statistics: [Average]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSByteBalance%
    aws_statistics: [Average]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

The suggested deployment mode for the exporter is through a Docker image. The following snippet provides a command line example to run the container (remember to provide your AWS credentials if needed and the path of the configuration file):

docker run -d --name cloudwatch_exporter \
  -p 9106:9106 \
  -v $(pwd)/cloudwatch-exporter.yaml:/config/config.yml \
  -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
  prom/cloudwatch-exporter

You can refer to the official guide for more details or alternative deployment modes.

Prometheus configuration

In order to scrape the newly created exporter add a new job to the configuration file. You will also need to define some relabeling rules in order to add the instance label required by Akamas to properly filter the incoming metrics. In the example below the instance label is copied from the instance’s Name tag:

Notice: AWS bills CloudWatch usage in batches of 1 million requests, where every metric counts as a single request. To avoid unnecessary expenses configure an appropriate scraping interval.

scrape_configs:
  - job_name: cloudwatch_exporter
    scrape_interval: 60s
    scrape_timeout: 30s
    static_configs:
      - targets: [cloudwatch_exporter:9106]
    metric_relabel_configs:
      - source_labels: [tag_Name]
        regex: '(.+)'
        target_label: instance

Additional workflow task

Once you configured the exporter in the Prometheus configuration you can start to fetch metrics using the Prometheus provider. The following sections describe some scripts you can add as tasks in your workflow.

Wait for metrics

It’s worth noting that CloudWatch may require some minutes to aggregate the stats according to the configured granularity, causing the telemetry provider to fail while trying to fetch data points not available yet. To avoid such issues you can add at the end of your workflow a task using an Executor operator to wait for the CloudWatch metrics to be ready. The following script is an example of implementation:

METRIC=aws_rds_cpuutilization_sum   # metric to check for
DELAY_SEC=15
RETRIES=60

NOW=`date +'%FT%T.%3NZ'`

for i in `seq $RETRIES`; do
  sleep $DELAY_SEC
  curl -sS "http://prometheus_host/api/v1/query?query=${METRIC}&time=${NOW}" | jq -ce '.data.result[]' && exit 0
done

exit 255

Start/stop the exporter as needed

Since Amazon bills your CloudWatch queries is wise to run the exporter only when needed. The following script allows you to manage the exporter from the workflow by adding the following tasks:

• start the container right before the beginning of the load test (command: bash script.sh start)

• stop the container after the metrics publication, as described in the previous section (command: bash script.sh stop).

#!/bin/bash

set -e

CMD=$1
CONT_NAME=cloudwatch_exporter

stop_cont() {
  [ -z `docker ps -aq -f "name=${CONT_NAME}"` ] || (echo Removing ${CONT_NAME} && docker rm -f ${CONT_NAME})
}

case $CMD in
  stop|remove)
    stop_cont
    ;;

  start)
    stop_cont

    AWS_ACCESS_KEY_ID=`awk 'BEGIN { FS = "=" } /aws_access_key_id/ {print $2 }' ~/.aws/credentials | tr -d '[:space:]'`
    AWS_SECRET_ACCESS_KEY=`awk 'BEGIN { FS = "=" } /aws_secret_access_key/ {print $2 }' ~/.aws/credentials | tr -d '[:space:]'`

    echo Starting container $CONT_NAME
    docker run -d --name $CONT_NAME \
      -p 9106:9106 \
      -v ~/oracle-database/utils/cloudwatch-exporter.yaml:/config/config.yml \
      -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
      prom/cloudwatch-exporter
    ;;

    *)
    echo Unrecognized option $CMD
    exit 255
    ;;
esac

Custom Configuration file

The example below is the Akamas-supported configuration, fetching metrics of EC2 instances named server1 and server2.

region: us-east-2
metrics:
  - aws_namespace: AWS/EC2
    aws_metric_name: CPUUtilization
    aws_statistics: [Average]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: NetworkIn
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: NetworkOut
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: NetworkPacketsIn
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: NetworkPacketsOut
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: CPUCreditUsage
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: CPUCreditBalance
    aws_statistics: [Average]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSReadOps
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSWriteOps
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSReadBytes
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSWriteBytes
    aws_statistics: [Sum]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSIOBalance%
    aws_statistics: [Average]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId

  - aws_namespace: AWS/EC2
    aws_metric_name: EBSByteBalance%
    aws_statistics: [Average]
    aws_dimensions: [InstanceId]
    # aws_dimension_select:
    #   InstanceId: [i-XXXXXXXXXXXXXXXXX]
    aws_tag_select:
      tag_selections:
        Name: [server1, server2]
      resource_type_selection: ec2:instance
      resource_id_dimension: InstanceId