Akamas is based on a microservices architecture where each service is deployed as a container and communicates with other services via REST APIs. Akamas can be deployed on a dedicated machine (Akamas Server) or on a Kubernets cluster.

The following figure represents the high-level Akamas architecture.

Interact with Akamas

Users can interact with Akamas via either the Graphical User Interface (GUI), Command-Line Interface (CLI), or via Application Programmatic Interface (API).

Both the GUI and CLI leverage HTTP/S APIs which pass through an API gateway (based on Kong), which also takes care of authenticating users by interacting with Akamas access management and routing requests to the different services.

The Akamas CLI can be invoked on either the Akamas Server itself or on a different machine (e.g. a laptop or another server) where the Akamas CLI has been installed.

Repositories

Akamas data is securely stored in different databases:

• time series data gathered from telemetry providers are stored in Elasticsearch;

• application logs are also stored in Elasticsearch;

• data related to systems, studies, workflows, and other user-provided data are stored in a Postgres database.

Notice: both Postgres and Elasticsearch and any other service included within Akamas are provided by Akamas as part of the Akamas installation package.

Services

Core Services

The following Spring-based microservices represent Akamas core services:

• System Service: holds information about metrics, parameters, and systems that are being optimized

• Campaign Service: holds information about optimization studies, including configurations and experiments

• Metrics Service: stores raw performance metrics (in Elasticsearch)

• Analyzer Service: automates the analysis of load tests and provides related functionalities such as smart windowing

• Telemetry Service: takes care of integrating different data sources by supporting multiple Telemetry Providers

• Optimizer Service: combines different optimization engines to generate optimized configurations using ML techniques

• Orchestrator Service: manages the execution of user-defined workflows to drive load tests

• User Service: takes care of user management activities such as user creation or password changes

• License Service: takes care of license management activities, optimization pack, and study export.

Ancillary Services

Akamas also provides advanced management features like logging, self-monitoring, licensing, user management, and more.

Preliminary steps

Please make sure to read the Getting Started section before installing Akamas.

Before installing Akamas, please follow these steps:

1. Review hardware, software, and network prerequisites

2. Install all Akamas dependencies

Installation steps

Please follow these steps to install the Akamas Server:

1. Install the Akamas Server

2. Install the Akamas CLI

3. Verify the Akamas Server

4. Install an Akamas license

Please also read the section on how to troubleshoot the installation and how to manage the Akamas Server. Finally, read the relevant sections of Integrating Akamas to integrate Akamas into your specific ecosystem.

Before installing the Akamas Server please make sure to review all the following requirements:

• Hardware requirements

• Software requirements

• Network requirements

Running in your data center

The following table provides the minimal hardware requirements for the virtual or physical machine used to install the Akamas server in your data center.

Running on AWS EC2

To run Akamas on an AWS Instance you need to create a new virtual machine based on one of the supported operating systems. You can refer to AWS documentation for step-by-step instructions on creating the instance.

As shown in the following diagram, you can create the Akamas instance in the same AWS region, Virtual Private Cloud (VPC), and private subnet as your own already existing EC2 machines and by creating/configuring a new security group that allows communication between your application instances and Akamas instance. The inbound/outbound rules of this security group must be configured as explained in the Networking Requirements section of this page.

It is recommended to use an m6a.xlarge instance with at least 70GB of disks of type GP2 or GP3 and select the latest LTS version of Ubuntu.

Supported AWS Regions

Akamas can be run in any EC2 region.

You can find the latest version supported for your preferred region here.

AWS Service Limits

Before installing Akamas on an AWS Instance please make sure to meet your AWS service limits (please refer to the official AWS documentation here).

Operating System

The following table provides a list of the supported operating systems and their versions.

On RHEL systems Akamas containers might need to be run in privileged mode depending on how Docker was installed on the system.

Software packages

The following table provides a list of the required Software Packages (also referred to as Akamas dependencies) together with their versions.

The exact version of these prerequisites is listed in the following table:

Read more about how to set up Akamas dependencies.

Akamas user

To install and run Akamas it is recommended to create a dedicated user (usually "akamas"). The Akamas user is not required to be in the sudoers list but can be added to the docker (dockeroot) group so it can run docker and docker-compose commands.

Make sure that the Akamas user has the read, write, and execute permissions on /tmp. If your environment does not allow writing to the whole /tmp folder, please create a folder /tmp/build and assign read and write permission to the Akamas user on that folder.

Internet access

Internet access is required for Akamas online installation and updated procedures and allows retrieving the most updated Akamas container images from the Akamas private Amazon Elastic Container Registry (ECR).

If internet access is not available for policies or security reasons, Akamas installation and updates can be executed offline.

Internet access from the Akamas server is not mandatory but it’s strongly recommended.

Ports

The following table provides a list of the ports on the Akamas server that have to be reachable by Akamas administrators and users to properly operate the system.

In the specific case of AWS instance and customer instances sharing the same VPC/Subnet inside AWS, you should:

• open all of the ports listed in the table above for all inbound URLs (0.0.0.0/32) on your AWS security group

• open outbound rules to all traffic and then attach this AWS security group (which must reside inside a private subnet) to the Akamas machine and all customer application AWS machines

This page will guide you through the installation of software components that are required to get the Akamas Server installed on a machine. Please read the Akamas dependencies for a detailed list of these software components for each specific OS.

While some links to official documentation and installation resources are provided here, please make sure to refer to your internal system engineering department to ensure that your company deployment processes and best practices are correctly matched.

Dependencies Setup

As a preliminary step before installing any dependency, it is strongly suggested to create a user named akamas on your machine hosting Akamas Server.

Docker

Follow the reference documentation to install docker on your system.

Docker installation guide: https://docs.docker.com/engine/install

Docker compose is already installed since Docker 23+. To install it on previous versions of Docker follow this installation guide: https://docs.docker.com/compose/install/

AWS CLI v2: https://docs.aws.amazon.com/cli/latest/userguide

Verify dependencies

As a quick check to verify that all dependencies have been correctly installed, you can run the following commands

• Docker:

  Copy

  docker run hello-world

• Docker compose :

  Copy

  docker compose --version

• AWS CLI:

  Copy

  aws --version

Akamas is deployed as a set of containerized services running on Docker and managed via Docker Compose. The latest version of the Akamas Docker Compose file and all the images required by Docker can be downloaded from the AWS ECR repository.

Two installation modes are available:

• online installation mode, in case the Akamas Server has access to the Internet - installation behind a proxy server is also supported.

• offline installation mode, in case the Akamas Server does not have access to the Internet.

Akamas is deployed as a set of containerized services running on Docker and managed via Docker Compose. In the online installation mode, the latest version of the Akamas Docker Compose file and all the images required by Docker can be downloaded from the AWS ECR repository.

In case the Akamas Server is behind a proxy server please also read how to setup Akamas behind a Proxy.

Get Akamas Docker artifacts

It is suggested first to create a directory akamas in the home directory of your user, and then run the following command to get the latest compose file:

cd ~
mkdir akamas
cd akamas
curl -O https://s3.us-east-2.amazonaws.com/akamas/compose/$(curl https://s3.us-east-2.amazonaws.com/akamas/compose/stable.txt)/docker-compose.yml

Configure Akamas environment variables

To configure Akamas, you should set the following environment variables:

• AKAMAS_CUSTOMER: the customer name matching the one referenced in the Akamas license.

• AKAMAS_BASE_URL: the endpoint in the Akamas APIs that will be used to interact with the CLI, typically http://<akamas server dns address>:8000

You can export the variables using the following snippet:

# Add double quotes ("xx xx") if the name contains white spaces
export AKAMAS_CUSTOMER=<your name or your organization name>
export AKAMAS_BASE_URL=http://<akamas server DNS address>:8000

It is recommended to save these exported variables in your ~/.bashrc file for convenience.

Start Akamas

In order to login into AWS ECR and pull the most recent Akamas container images you also need to set the AWS authentication variables to the appropriate values provided by Akamas Customer Support Services by running the following command:

export AWS_ACCESS_KEY_ID=<your access key id>
export AWS_SECRET_ACCESS_KEY=<your secret access key>
export AWS_DEFAULT_REGION=us-east-2

At this point, you can start installing Akamas server by running the following AWS CLI commands:

aws ecr get-login-password --region us-east-2 | docker login -u AWS --password-stdin https://485790562880.dkr.ecr.us-east-2.amazonaws.com

docker compose up -d

This section describes how to setup an Akamas Server behind a proxy server and to allow Docker to connect to the Akamas repository on AWS ECR.

Configure Docker daemon

First, create the /etc/systemd/system/docker.service.d directory if it does not already exists. Then create or update the /etc/systemd/system/docker.service.d/http-proxy.conf file with the variables listed below, taking care of replacing <PROXY> with the address and port (and credentials if needed) of your target proxy server:

[Service]
Environment="HTTP_PROXY=<PROXY>"
Environment="HTTPS_PROXY=<PROXY>"

Once configured, flush the changes and restart Docker with the following commands:

sudo systemctl daemon-reload
sudo systemctl restart docker

For more details, refer to the official documentation page: Control Docker with systemd.

Configure the Akamas containers

To allow the Akamas services to connect to addresses outside your intranet, the Docker instance needs to be configured to forward the proxy configuration to the Akamas containers.

Update the ~/.docker/config.json file adding the following field to the JSON, taking care to replace <PROXY> with the address (and credentials if needed) of your target proxy server:

{
  ...
  "proxies": {
    "default": {
      "httpProxy": "<PROXY>",
      "httpsProxy": "<PROXY>",
      "ftpProxy": "<PROXY>",
      "noProxy": "localhost,127.0.0.1,/var/run/docker.sock,database,optimizer,campaign,analyzer,telemetry,log,elasticsearch,metrics,system,license,store,orchestrator,airflow-db,airflow-webserver,kong-database,kong,user-service,keycloak,logstash,kibana,akamas-ui,grafana,prometheus,node-exporter,cadvisor,konga,benchmark"
    }
  }
}

For more details, refer to the official documentation page: Configure Docker to use a proxy server.

Run Akamas

Set the following variables to configure your working environment, taking care to replace <PROXY> with the address (and credentials if needed) of your target proxy server:

export HTTP_PROXY='<PROXY>'
export HTTPS_PROXY='<PROXY>'

Once configured, you can log into the ECR repository through the AWS CLI and start the Akamas services manually.

Akamas is deployed as a set of containerized services running on Docker and managed via Docker Compose. In the offline installation mode, the latest version of the Akamas Docker Compose file and all the images required by Docker cannot be downloaded from the AWS ECR repository.

Get Akamas Docker artifacts

Get in contact with Akamas Customer Services to get the latest versions of the Akamas artifacts uploaded to a location of your choice on the dedicated Akamas Server.

Akamas installation artifacts will include:

• images.tar.gz: a tarball containing Akamas main images.

• docker-compose.yml: docker-compose file for Akamas.

• akamas: the binary file of the Akamas CLI that will be used to verify the installation.

Import Docker images

A preliminary step in the offline installation mode is to import the shipped Docker images by running the following commands in the same directory where the tar files have been stored:

cd <your bundle files location>
docker image load -i images.tar.gz

Notice that this import procedure could take quite some time!

Configure Akamas environment variables

To configure Akamas, the following environment variables are required to be set:

• AKAMAS_CUSTOMER: the customer name matching the one referenced in the Akamas license.

• AKAMAS_BASE_URL: the endpoint in the Akamas APIs that will be used to interact with the CLI, typically http://<akamas server dns address>:8000

Environment variables creation is performed by the snippet below:

# add double quotes ("xx xx") if the name contains white spaces
export AKAMAS_CUSTOMER=<your name or your organization name>
export AKAMAS_BASE_URL=http://<akamas server DNS address>:8000

It is recommended to save these exported variables in your ~/.bashrc file for convenience.

Run installation

To start Akamas you can now simply navigate into the akamas folder and run a docker-compose command as follows:

cd <your docker-compose file location>
docker compose up -d

You may get the following error:

Error saving credentials: error storing credentials - err: exit status 1, out: Cannot autolaunch D-Bus without X11 $DISPLAY

This is a documented docker bug (see this link) that can be solved by installing the "pass" package:

• Ubuntu

sudo apt-get install -y pass

• RHEL

yum install pass

By default, Akamas uses the following ports for its UI:

• 80 (HTTP)

• 443 (HTTPS)

Depending on the configuration of your environment, you may want to change the default settings: to do so, you’ll have to update the Akamas docker-compose file.

Inside the docker-compose.yml file, scroll down until you come across the akamas-ui service. There you will find a specification as follows:

  akamas-ui:
    ports:
      - "443:443"
      - "80:80"

Update the YAML file by remapping the UI ports to the desired ports of the host.

  akamas-ui:
    ports:
      - "<YOUR_HTTPS_PORT_OF_CHOICE>:443"
      - "<YOUR_HTTP_PORT_OF_CHOICE>:80"

In case you were running Akamas with host networking, you are allowed to bind different ports in the container itself. In order to do so you can expand the docker-compose service by adding a couple of environment variables like this:

  akamas-ui:
    environment:
      - HTTP_PORT=<HTTP_CONTAINER_PORT>
      - HTTPS_PORT=<HTTPS_CONTAINER_PORT>
    ports:
      - "<YOUR_HTTPS_PORT_OF_CHOICE>:<HTTP_CONTAINER_PORT>"
      - "<YOUR_HTTP_PORT_OF_CHOICE>:<HTTPS_CONTAINER_PORT>"

Akamas APIs and UI use plain HTTP when they are first installed. To enable the use of HTTPS you will need to:

1. Ask your security team to provide you with a valid certificate for your server. The certificate usually consists of two files with ".key" and ".pem" extensions. You will need to provide the Akamas server DNS name.

2. Create a folder named "certs" in the same directory as Akamas' docker-compose file;

3. Copy the ".key" and ".pem" files in the created "certs" folder and rename them to "akamas.key" and "akamas.pem" respectively. Make sure that the files belong to the same user and group you use to run Akamas.

4. Restart two Akamas services by running the following commands:

   Copy

   cd <Akamas docker-compose file folder>
   docker-compose restart akamas-ui kong

After the containers reboot is complete you will be able to access the UI over HTTPS from your browser:

https://<akamas server name here>

Setup CLI to use HTTPS

Now that your Akamas server is configured to use HTTPS you can update the Akamas CLI configuration in order to use the secure protocol.

If you have not yet installed the Akamas CLI follow the CLI installation guide in order to install it. If you already have the CLI available, you can run the following command:

akamas init config

You will be prompted to enter some input, please value it as follows:

Api address [http://localhost:8000]: https://<akamas server dns address>:443/akapi
Workspace [default]: default
Verify SSL: [True]: True

You can test the connection by running:

akamas status

It should return ‘OK’, meaning that Akamas has been properly configured to work over HTTPS.

Issues when installing Docker

Centos 7 and RHEL 7

Notice: this distro features a known issue since Docker default execution group is named dockerroot instead of docker . To make docker work edit (or create) /etc/docker/daemon.json to include the following fragment:

{
  "group": "dockerroot"
}

After editing or creating the file, please restart Docker and then check the group permission of the Docker socket (/var/run/docker.sock), which should show dockerroot as a group:

srw-rw----. 1 root dockerroot 0 Jul  4 09:57 /var/run/docker.sock

Then, add the newly created akamas user to the dockerroot group so that it can run docker containers:

sudo usermod -aG dockerroot <user_name>

and check the akamas user has been correctly added to dockerroot group by running:

lid -g dockerroot

Issues when running AWS CLI

In case of issues in logging in through AWS CLI, when executing the following command:

aws ecr get-login-password --region us-east-2

Please check that:

• Environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION are correctly set

• AWS CLI version is 2.0+

We recommend using the official AWS CLI installation guide for a smoother experience.

Issue when starting Akamas services

Akamas failed to start some services

Please notice that the very first time Akamas is started, up to 30 minutes might be required to initialize the environment.

In case the issue persists you can run the following command to identify which service is not able to start up correctly

akamas status -d

License service unable to access docker socket

In some systems, the Docker socket, usually located in /var/run/docker.sock can not be accessed within a container. This causes Akamas to signal this behavior by reporting the Access Denied error in the license service logs.

To overcome this limitation edit the docker-compose.yaml file adding the line privileged: true to the following services:

• License

• Optimizer

• Telemetry

• Airflow

The following is a sample configuration where this change is applied to the license service:

license:
  image: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/license_service:2.3.0
  container_name: license
  privileged: true

Finally, you can issue the following command to apply these changes

docker-compose up -d

Missing Akamas Customer variable

When installing Akamas it’s mandatory to provide the AKAMAS_CUSTOMER variable as illustrated in the installation guide. This variable must match the one provided by Akamas representatives when issuing a license. If the variable is not properly exported license installation will fail with an error message indicating that the name of the customer installation does not match the one provided in the license.

You can easily inspect which value of this variable has been used when starting Akamas by running the following command on the Akamas server:

docker inspect license | grep AKAMAS_CUSTOMER

If you find out that the value is not the one you expect you can update it by running the following command on the Akamas server:

AKAMAS_CUSTOMER=<your-new-value> docker-compose up -d license

Once Akamas is up and running you can re-install your license.

Other issues

For any other issues please contact Akamas Customer Support Services.

Preliminary steps

Before installing Akamas, please follow these steps:

1. Review the cluster requirements

2. Install the software requirements

Installation steps

Please follow these steps to install the Akamas application:

1. Install the application

2. Install the CLI

3. Verify the installation

4. Install the license

Please also read the section on how to manage Akamas. Finally, read the relevant sections of Integrating Akamas to integrate Akamas into your specific ecosystem.

Before installing the Akamas please make sure to review all the following requirements:

• Cluster requirements

• Software requirements

Kubernetes version

Running Akamas requires a cluster running Kubernetes version 1.23 or higher.

Resources requirements

Akamas can be deployed in three different sizes depending on the number of concurrent optimization studies that will be executed. If you are unsure about which size is appropriate for your environment we suggest you start with the small one and upgrade to bigger ones as you expand the optimization activity to more applications.

The tables below report the required resources both for requests and limits that should be available in the cluster to use Akamas.

Small

The small tier is suited for environments that needs to support up to 10 concurrent optimization studies

Storage requirements

The cluster must provide the definition of a Storage Class so that the application installation can leverage Persistent Volume Claims to dynamically provision the volumes required to persist data.

For more information on this topic refer to Kubernetes' official documentation.

Permissions

To work properly, Akamas needs to manage some resources inside the Namespace. For this reason, it is recommended to run Akamas in a dedicated Namespace.

To manage resources, Akamas uses a ServiceAccount bound to the application's pods, which must be created either manually by the cluster administrator or automatically by the provided Helm chart.

This snippet describes the namespaced required permissions for the service account:

- apiGroups: ["batch"]
  resources:
    - jobs
  verbs: ["get", "list", "create", "delete", "patch", "update"]
- apiGroups: [""]
  resources:
    - configmaps
  verbs: ["get", "list", "create", "delete", "patch", "update"]
- apiGroups: [""]
  resources:
    - pods
  verbs: ["get", "list", "patch", "update", "watch"]
- apiGroups: [""]
  resources:
    - pods/log
  verbs: ["get"]
- apiGroups: [""]
  resources:
    - secrets
  verbs:
    - get
    - create
    - patch

Networking

Networking requirements depend on how users interact with Akamas. Services can be exposed via Ingress, LoadBalancers, NodePorts, or using kubectl as a proxy. Refer to Accessing Akamas for a more detailed description of the available options.

This page describes the requirements that should be fulfilled by the user when installing or managing an Akamas installation on Kubernetes. The software listed below is usually installed on the user's workstation or laptop.

Kubectl

Kubectl must be installed and configured to interact with the desired cluster. Refer to the official kubectl documentation to set up the client.

To interact with the Kubernetes API server you will need kubectl, preferably with a version matching the cluster. To check both the client and cluster versions, run:

kubectl version --short

Helm

Installing Akamas requires Helm 3.0 or higher. To check the version run:

helm version --short

Privileged access

Akamas uses Elasticsearch to store logs and time-series. When running Akamas on Kubernetes, Elasticsearch is installed, automatically using the official Elasticsearch helm chart. This chart required running an init container with privileged access to set up a configuration on the host running the Elasticsearch pod. If running such a container is not permitted in your environment you can add the following snippet to the akamas.yaml file when installing Akamas to disable this feature. \

# Disable ES privileged initialization container. 
elasticsearch:
  sysctlInitContainer:
    enabled: false

Akamas is deployed on your Kubernetes cluster through a Helm chart, and all the required images can be downloaded from the AWS ECR repository.

Two installation modes are available:

• online installation, in case the Kubernetes cluster can access the Internet.

• offline installation, in case the Kubernetes cluster does not have access to the Internet or you need to use a private image registry.

Before starting the installation, make sure the requirements are met.

Create the configuration file

Akamas on Kubernetes is provided as a set of templates packaged in a chart archive managed by Helm.

To proceed with the installation, you need to create a file, called akamas.yaml in this guide, containing the mandatory configuration values required to customize your application. The following template contains the minimal set of values required to install Akamas:

# AWS credentials to fetch ECR images (required)
awsAccessKeyId: <AWS_ACCESS_KEY_ID>
awsSecretAccessKey: <AWS_SECRET_ACCESS_KEY>

# Akamas customer name. Must match the value in the license (required)
akamasCustomer: <CUSTOMER_NAME>

# Akamas administrator password. If not set a random password will be generated
akamasAdminPassword: <ADMIN_PASSWORD>

You can also download the template file running the following snippet:

curl -so akamas.yaml  http://helm.akamas.io/templates/1.0.2/akamas.yaml.template

This minimal configuration is enough to have Akamas up and running on your cluster, even though the endpoint will only be accessible through Kubectl's port forwarding.

The page Accessing Akamas provides some configuration examples using different types of services: edit the akamas.yaml file using the strategy that best suits your needs, or continue directly with the next sections and configure the endpoints at a later time.

Start the installation

Add the Akamas' repository to the Helm client with the following command:

helm repo add akamas http://helm.akamas.io/charts

If you wish to see the values that Helm will use to install Akamas and override some of them, you may execute the following command:

helm show values akamas/akamas

Now, with the configuration file you just created (and the new variables you added to override the defaults), you can start the installation with the following command:

helm upgrade --install \
  --create-namespace --namespace akamas \
  -f akamas.yaml \
  akamas akamas/akamas

This command will create the Akamas resources within the specified namespace. You can define a different namespace by changing the argument --namespace <your-namespace>

An example output of a successful installation is the following:

Release "akamas" does not exist. Installing it now.
NAME: akamas
LAST DEPLOYED: Wed Apr  5 11:40:19 2023
NAMESPACE: akamas
STATUS: deployed
REVISION: 1
NOTES:
Akamas has been installed

NOTES:
Akamas has been installed

To get the initial password use the following command:

kubectl get secret akamas-admin-credentials -o go-template='{{ .data.password | base64decode }}'

Check the installation

To monitor the application startup, run the command kubectl get pods. After a few minutes, the expected output should be similar to the following:

NAME                           READY   STATUS    RESTARTS   AGE
airflow-6ffbbf46d8-dqf8m       3/3     Running   0          5m
analyzer-67cf968b48-jhxvd      1/1     Running   0          5m
campaign-666c5db96-xvl2z       1/1     Running   0          5m
database-0                     1/1     Running   0          5m
elasticsearch-master-0         1/1     Running   0          5m
keycloak-66f748d54-7l6wb       1/1     Running   0          5m
kibana-6d86b8cbf5-6nz9v        1/1     Running   0          5m
kong-7d6fdd97cf-c2xc9          1/1     Running   0          5m
license-54ff5cc5d8-tr64l       1/1     Running   0          5m
log-5974b5c86b-4q7lj           1/1     Running   0          5m
logstash-8697dd69f8-9bkts      1/1     Running   0          5m
metrics-577fb6bf8d-j7cl2       1/1     Running   0          5m
optimizer-5b7576c6bb-96w8n     1/1     Running   0          5m
orchestrator-95c57fd45-lh4m6   1/1     Running   0          5m
store-5489dd65f4-lsk62         1/1     Running   0          5m
system-5877d4c89b-h8s6v        1/1     Running   0          5m
telemetry-8cf448bf4-x68tr      1/1     Running   0          5m
ui-7f7f4c4f44-55lv5            1/1     Running   0          5m
users-966f8f78-wv4zj           1/1     Running   0          5m

At this point, you should be able to access the Akamas UI on http://localhost:8000 and the Akamas CLI http://localhost:8000/akapi by running Kubectl's port forwarding command:

kubectl port-forward service/ui 8000:http

Mind that, before logging in, you need to configure the Akamas CLI and install a valid license.

If you haven't already, you can update your configuration file to use a different type of service to expose Akamas' endpoints. To do so, pick from the Accessing Akamas the configuration snippet for the service type of your choice, add it to the akamas.yaml file, and re-run the installation command to update your Helm release.

Before starting the installation, make sure the requirements are met.

Configure the registry

If your cluster is in an air-gapped network or is unable to reach the default repository, you need to mirror the required images on a private repository.

The procedure described here leverages your local environment to upload the images, so it requires that Docker is installed and configured to interact with the private registry.

Obtain the Docker images

Get in contact with Akamas Customer Services to get the latest versions of the Akamas artifacts. This will include:

• images.tar.gz: a tarball containing the Akamas images.

• akamas: the binary file of the Akamas CLI that will be used to verify the installation.

Upload the Docker images

The offline installation mode requires importing the shipped Docker images into your local environment. Run the following command in the same directory where the tar file is stored:

cd '<IMAGES_ARCHIVE_LOCATION>'
docker image load -i images.tar.gz

Once the import is complete, you need to re-tag and upload the images. Run the following snippet, replacing <REGISTRY_URL> with the actual URL of the private registry:

REGISTRY='<REGISTRY_URL>'
for image in `docker images | awk '/485790562880.dkr.ecr.us-east-2.amazonaws.com/ {print $1 ":" $2}'`; do
  newImage="${image/485790562880.dkr.ecr.us-east-2.amazonaws.com/${REGISTRY}}"
  docker tag ${image} ${newImage}
  docker push ${newImage}
done

Once the upload is complete, you can proceed with the next steps.

Create the configuration file

Akamas on Kubernetes is provided as a set of templates packaged in a chart archive managed by Helm.

To proceed with the installation, you need to create a file, called akamas.yaml in this guide, containing the mandatory configuration values required to customize your application. The following template contains the minimal set of values required to install Akamas:

# Akamas customer name. Must match the value in the license (required)
akamasCustomer: <CUSTOMER_NAME>

# Akamas administrator password. If not set a random password will be generated
akamasAdminPassword: <ADMIN_PASSWORD>

registry: <REGISTRY_URL>
postgresql:
  image:
    registry: <REGISTRY_URL>
elasticsearch:
  image: <REGISTRY_URL>/elasticsearch
kibana:
  image: <REGISTRY_URL>/kibana

This minimal configuration is enough to have Akamas up and running on your cluster, even though the endpoint will only be accessible through Kubectl's port forwarding.

The page Accessing Akamas provides some configuration examples using different types of services: edit the akamas.yaml file using the strategy that better suits your needs, or continue directly with the sections and configure the endpoints at a later time.

Configure the authentication

This section describes how to configure the authentication to your private registry. If your registry does not require any authentication, skip directly to the installation section.

To authenticate to your private registry, you must manually create the Secret required to pull the images. If the registry uses basic authentication, you can create the credentials in the namespace by running the following command:

kubectl create secret docker-registry registry-token \
  --namespace akamas \
  --docker-server=<REGISTRY_URL> \
  --docker-username=<USER> \
  --docker-password=<PASSWORD>

Otherwise, you can leverage any credential already configured on your machine by running the following command:

kubectl create secret docker-registry registry-token \
  --namespace akamas \
  --from-file=.dockerconfigjson=<PATH/TO/.docker/config.json>

Start the installation

From a machine that can reach the endpoint, add the Akamas' repository to the Helm client with the following command:

helm repo add akamas http://helm.akamas.io/charts

If you can not reach helm.akamas.io from the machine where the installation will be run, pull the chart by running:

helm pull akamas/akamas

The command downloads the latest version chart version as an archive named akamas-<version>.tgz. The file can be transferred to the machine where the installation will be run. Replace akamas/akamas with the download package in the following commands.

If you wish to see and override the values that Helm will use to install Akamas, you may execute the following command.

helm show values akamas/akamas

Now, with the configuration file you just created (and the new variables you added to override the defaults), you can start the installation with the following command:

helm upgrade --install \
  --create-namespace --namespace akamas \
  -f akamas.yaml \
  akamas akamas/akamas

This command will create the Akamas resources within the specified namespace. You can define a different namespace by changing the argument --namespace <your-namespace>

An example output of a successful installation is the following:

Release "akamas" does not exist. Installing it now.
NAME: akamas
LAST DEPLOYED: Wed Apr  5 11:40:19 2023
NAMESPACE: akamas
STATUS: deployed
REVISION: 1
NOTES:
Akamas has been installed

NOTES:
Akamas has been installed

To get the initial password use the following command:

kubectl get secret akamas-admin-credentials -o go-template='{{ .data.password | base64decode }}'

Check the installation

To monitor the application startup, run the command kubectl get pods. After a few minutes, the expected output should be similar to the following:

NAME                           READY   STATUS    RESTARTS   AGE
airflow-6ffbbf46d8-dqf8m       3/3     Running   0          5m
analyzer-67cf968b48-jhxvd      1/1     Running   0          5m
campaign-666c5db96-xvl2z       1/1     Running   0          5m
database-0                     1/1     Running   0          5m
elasticsearch-master-0         1/1     Running   0          5m
keycloak-66f748d54-7l6wb       1/1     Running   0          5m
kibana-6d86b8cbf5-6nz9v        1/1     Running   0          5m
kong-7d6fdd97cf-c2xc9          1/1     Running   0          5m
license-54ff5cc5d8-tr64l       1/1     Running   0          5m
log-5974b5c86b-4q7lj           1/1     Running   0          5m
logstash-8697dd69f8-9bkts      1/1     Running   0          5m
metrics-577fb6bf8d-j7cl2       1/1     Running   0          5m
optimizer-5b7576c6bb-96w8n     1/1     Running   0          5m
orchestrator-95c57fd45-lh4m6   1/1     Running   0          5m
store-5489dd65f4-lsk62         1/1     Running   0          5m
system-5877d4c89b-h8s6v        1/1     Running   0          5m
telemetry-8cf448bf4-x68tr      1/1     Running   0          5m
ui-7f7f4c4f44-55lv5            1/1     Running   0          5m
users-966f8f78-wv4zj           1/1     Running   0          5m

At this point, you should be able to access the Akamas UI on http://localhost:8000 and the Akamas CLI http://localhost:8000/akapi by running Kubectl's port forwarding command:

kubectl port-forward service/ui 8000:http

Mind that, before logging in, you need to configure the Akamas CLI and install a valid license.

If you haven't already, you can update your configuration file to use a different type of service to expose Akamas' endpoints. To do so, pick from the Accessing Akamas the configuration snippet for the service type of your choice, add it to the akamas.yaml file, and re-run the installation command to update your Helm release.

To interact with your Akamas instance you need the UI and API Gateway to be accessible from outside the cluster. This means you need to expose the ui and kong service respectively (although a minimal configuration only requires exposing the ui service, since it can forward requests to the API Gateway through the path /akapi).

Kubernetes offers different options to expose a service outside of the cluster. The following is a list of the supported ones, with examples of how to configure them to work in your chart release:

• Port Forwarding

• LoadBalancer

• Ingress

• NodePort

Port Forwarding

By default, Akams uses Cluster IPs for its services, which only allow communication inside the cluster. Still, you can leverage Kubectl's port-forward to create a private connection and expose any internal service on your local machine.

This solution is suggested to perform quick tests without the need of exposing the application, or in scenarios where cluster access to the public is not allowed.

To make the Akamas UI accessible on http://localhost:8000, run the following command:

kubectl port-forward service/ui 8000:http

To interact with the Akamas CLI you can use the URL http://localhost:8000/akapi, or expose the kong service in the same way.

Refer to the official kubernetes documentation for more details about port-forwarding.

Load Balancer

Load Balancers expose services outside the cluster. This solution is often used with clusters managed by cloud providers such as Amazon EKS or Google Kubernetes Engine (GKE).

You can expose the Akamas UI through a Load Balancer by adding the snippet below to the akamas.yaml file from the previous section, and re-running the install command to update the configuration.

ui:
  service:
    type: "LoadBalancer"

To get the address of the load balancer run the command:

kubectl get svc ui -o 'custom-columns=NAME:metadata.name,TYPE:spec.type,HOSTNAME:status.loadBalancer.ingress[].hostname'

For more details on Load Balancers refer to the official kubernetes documentation.

Ingress

An Ingress is a Kubernetes object that provides service access, load balancing, and SSL termination to kubernetes services.

You can expose the Akamas UI through an Ingress by adding the snippet below to the akamas.yaml file from the previous section. After adding to className one of the ingress controllers available on the cluster, re-run the install command to update the configuration.

ui:
  ingress:
    enabled: true
    className: "<class-name>"
    hosts:
      - host: "<dns-name>"
        paths:
          - path: /
            pathType: Prefix

You can also configure a certificate on the Ingress: refer to HTTPS configuration section for instructions.

Refer to the official kubernetes documentation for more details on Ingresses.

NodePort

Node Ports make services accessible on specific ports of any node of the cluster.

You can expose the Akamas UI through a NodePort by adding the snippet below to the akamas.yaml file from the previous section, and re-running the install command to update the configuration.

ui:
  service:
    type: "NodePort"
    http:
      nodePort: "30010"

The Akamas UI will be accessible on any cluster node at http://<cluster-node>:30010. You can also omit the http.nodePort field and let Kubernetes automatically select a random port.

Refer to the official kubernetes documentation for more information on Node Ports.

HTTPS configuration can be set in the Akamas services (UI and Kong) or in the Ingress definition.

Certificate in the Ingress

Declare the certificate secret by adding a tls section to the Ingress definition:

ui:
  ingress:
    enabled: true
    className: "<class-name>"    # ingress class name

    hosts:
      - host: "example.company.com"
        paths:
          - path: /
            pathType: Prefix

    tls:
      - secretName: "<secret name>"  # secret name containing the certificate and key data
        hosts:
          - "example.company.com"

You can apply the same configuration to the kong service to add a certificate to the API Gateway.

For more information regarding the TLS definition refer to the official documentation.

Certificate in the Akamas services

To add a certificate to both the UI and API Gateway you need to generate the akamas.key and akamas.pem files, and create a secret in Akamas' namespace with the following command:

kubectl create secret generic certificate --from-file=akamas.key --from-file=akamas.pem

To complete the update, restart the deployments:

kubectl rollout restart deployment ui kong

The Akamas CLI allows users to invoke commands against the Akamas dedicated machine (Akamas Server). The Akamas CLI can also be installed on a different system than the Akamas Server.

Prerequisites

Linux and Windows operating systems are supported for installing Akamas CLI.

Installation steps

The Akamas CLI can be installed and configured in two simple steps:

1. Setup the CLI

2. Initialize the CLI

Refer to the section Change CLI config to modify the CLI ports the Akamas Server is listening to. Section Use a proxy server provides instructions on how to interact with Akamas via a proxy server.

Linux

To get Akamas CLI installed on Linux, run the following commands:

curl -o akamas_cli -O https://akamas.s3.us-east-2.amazonaws.com/cli/$(curl https://akamas.s3.us-east-2.amazonaws.com/cli/stable.txt)/linux_64/akamas
sudo mv akamas_cli /usr/local/bin/akamas
chmod 755 /usr/local/bin/akamas

You can now run the Akamas CLI following by running the akamas command.

In some installations, the /usr/local/bin folder is not present in the PATH environment variable. This prevents you from using akamas without specifying the complete file location. To fix this issue you can add an entry to the PATH system environment variable or move the executable to another folder in your PATH.

Auto-completion

To enable auto-completion on Linux systems with a bash shell (requires bash 4.4+), run the following commands:

curl -O https://s3.us-east-2.amazonaws.com/akamas/cli/$(curl https://s3.us-east-2.amazonaws.com/akamas/cli/stable.txt)/linux_64/akamas_autocomplete.sh
mkdir -p ~/.akamas
mv akamas_autocomplete.sh ~/.akamas
echo '. ~/.akamas/akamas_autocomplete.sh' >> ~/.bashrc
source ~/.bashrc

Windows

To install the Akamas CLI on Windows run the following command from the Powershell:

Invoke-WebRequest "https://s3.us-east-2.amazonaws.com/akamas/cli/$($(Invoke-WebRequest https://s3.us-east-2.amazonaws.com/akamas/cli/stable.txt | Select-Object -Expand Content) -replace '\n', '')/win_64/akamas.exe" -OutFile akamas.exe

You can now run the Akamas CLI following by running .\akamas in the same folder.

To invoke the akamas CLI from any folder, create a akamas folder (such as C:\Program Files\akamas), and move there the akamas.exe file. Then, add an entry to the PATH system environment variable with the value C:\Program Files\akamas. Now, you can invoke the CLI from any folder, by simply running the akamas command.

The Akamas CLI can be accessed by simply running the akamascommand.

Verify the CLI

You can verify that the CLI was installed correctly by running this command:

akamas --version

which should show an output similar to this one

Akamas CLI, version 2.7.2

At any time, you can see available commands and options with:

akamas --help

For the full list of Akamas commands please refer to the section CLI reference.

The CLI is used to interact with an akamas server. To initialize the configuration of the Akamas CLI you can run the command:

akamas init config

and follow the wizard to provide the required information such as the server IP.

Here is a summary of the configuration wizard options.

Api address [http://localhost:8000]: https://<akamas-hostname>:<ui-port>/akapi
Workpace [default]: default
Verify SSL: [True]: True
Is external certificate CA required? [y/N]: N

This configuration can be changed at any time (see how to change the CLI config).

After this step, the Akamas CLI can be used to login to the Akamas server, by issuing the following command:

akamas login

and providing the credentials as requested.

The CLI configuration contains the information required to communicate with the akamas server. It can be easily created and updated with a configuration wizard. This page describes the main options of the Akamas CLI and how to modify them. If your Akamas instance is installed with Kubernetes, ensure the UI service is properly configured.

API Address

The CLI, as well as the UI, interacts with the akamas server via APIs. The apiAddress configuration contains the information required in order to communicate with the server.

Docker

The Akamas Server provides different listeners to interact with APIs:

• an HTTP listener on port 80 under the path /akapi

• an HTTP listener on port 8000

• an HTTPS listener on port 443 under the path /akapi

• an HTTPS listener on port 8443

Depending on your networking setup you can either use the listeners on ports 80 and 443 which are also used for the UI or directly interact with the API gateway on ports 8000 and 8443. If you are not sure about your network setup we suggest you start with the HTTPS listener on port 443.

For improved security, it is recommended to configure CLI communications with the Akamas Server over HTTPS. Notice that you need to have a valid certificate installed on your Akamas server (at least a self-signed one) in order to enable HTTPS communication between CLI and the Akamas Server.

Changing CLI protocol

The CLI can be configured either directly via the CLI itself or via the YAML configuration file akamasconf.

Using the CLI

Issue the following command to change the configuration of the Akamas CLI:

akamas init config

and then follow the wizard to provide the required CLI configuration:

• enable HTTPS communications:

Api address [http://localhost:8000]: https://<akamas server dns name>:443/akapi
Workpace [default]: Workspace1
Verify SSL: [True]: True
Is external certificate CA required? [y/N]: N

• enable HTTP communications:

Api address [http://localhost:8000]: http://<akamas server dns name>:80
Workspace [default]: Workspace1

Please notice that by default Akamas CLI expects a valid SSL certificate. If you are using a self-signed certificate or a not valid one you can set the Verify SSL variable to false. This will mimic the behavior of accepting a not valid HTTPS certificate on your favorite browser.

Using the akamasconf file

Create a file and name it akamasconf to be located at the following location:

• Linux: ~/.akamas/akamasconf

• Windows: C:\Users\<username>\.akamas (where C: is the drive where the OS is installed)

The file location can be customized by setting an $AKAMASCONF environment variable.

Here is an example akamasconf file provided as a sample:

apiAddress: http[s]://<akamas server dns name>:80[443]/akapi
verifySsl: [true|false]
organization: akamas
workspace: default

The Akamas CLI supports interacting with the API server through an HTTP/HTTPS proxy server.

To enable access via an HTTP proxy, set the environment variable HTTP_PROXY. From the following snippet, replace proxy_ip and proxy_port with the desired values.

export HTTP_PROXY="http://<proxy_ip>:<proxy_port>"

Then, run an akamas command to verify access.

akamas status --debug

Access through an HTTPS proxy can be set by using environment variable HTTPS_PROXY, instead of HTTP_PROXY.

Run the following command to verify the correct startup and initialization of Akamas:

akamas status

When all services have been started this command will return an "OK" message. Please notice that it might take a few minutes for Akamas to start all services.

To check that also UI is properly working please access the following URL:

http://<akamas server name here>

You will see the Akamas login form:

Please notice that it is impossible to log into Akamas before a license has been installed. Read here how to Install an Akamas license.

Akamas provides a Management Container (or called Management Pod, for Kubernetes deployments) that contains the Akamas CLI executable and other popular command line tools to develop custom scripts.

On docker, it runs in the same network of the Akamas' services or, when running on Kubernetes, in the Akamas' namespace. The purposes of this management container are:

• Allow technical troubleshooting/maintenance from inside the Kubernetes cluster.

• Allow customers to launch/control Akamas without the need to install Akamas CLI on their systems. The akamas executable is configured to connect to the correct endpoint.

• Provide an environment for the Akamas workflow to execute custom scripts.

The following is the list of the installed tools:

• akamas-cli

• curl, ping, wget

• docker, docker-cli, docker-compose

• git

• gzip, zip

• jq, yq, vim

• kubectl, Helm

• openjdk 11

• openssh-client, openssh-server, ssh-keygen

Docker compose installation

To run the management container on your docker installation, add the following code block to the list of services of your docker-compose file.

  management-container:
    image: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/management-container:1.0.1
    container_name: management-container
    environment:
      - BASH_ENV=/home/akamas/.bashrc
    expose:
      - 22
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - akamas
    restart: unless-stopped

and launch docker-compose up -d as explained in Start Akamas (online) or Run installation (offline).

Kubernetes installation

To run the management pod in the Akamas' namespace, update the following variable in the Values file of the Akamas' Helm chart:

managementPod:
  enabled: true

Then you can issue the helm upgrade --install ... command to launch the pods, as described in Start the installation (online) or Start the installation (offline).

Accessing Management Pod on Kubernetes

When it's deployed to Kubernetes, you may access this management pod in two ways:

• via kubectl exec -it management-pod

• via SSH command

NOTE: both methods require kubectl to be installed and configured for this cluster.

Kubectl access

Accessing is as simple as:

kubectl exec -it deployment/management-pod -- bash

SSH access

For this type of access, you need to retrieve the password for the akamas user. You should issue the following command to read it from management-pod logs:

kubectl logs service/management-pod

# example response is:
# Container started
# You can ssh into this container with user 'akamas' and password 'd48020ab71be6a07'

A similar result could be obtained by reading the file akamas_password in the work folder:

kubectl exec -it deployment/management-pod -- cat /work/akamas_password

# example response is:
# d48020ab71be6a07

At this point, you should launch this command to port-forward the management port to your local terminal (number 2300 can as well be any other number: it should be an unused port on your machine):

kubectl port-forward service/management-pod 2300:22 &

then, on another terminal, you may launch:

ssh akamas@localhost -p 2300

and answer yes to the question, then insert the akamas password to successfully SSH access the management pod (see example below):

my_user@my_machine:~$  ssh akamas@localhost -p 2300
The authenticity of host '[localhost]:2300 ([127.0.0.1]:2300)' can't be established.
ED25519 key fingerprint is SHA256:34GXnmRz1YjWr2TTpUpJmRoHYck0NzeAxni2L857Exs.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '[localhost]:2300' (ED25519) to the list of known hosts.
akamas@localhost's password:
Welcome to Ubuntu 20.04.6 LTS (GNU/Linux 5.10.178-162.673.amzn2.x86_64 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

akamas@management-pod-6dd8b7f898-8xwzf:~$

Work folder

If you need to store Akamas artifacts, scripts, or any other file that need persistence, you can use the /work directory, which persists across restarts. This is the default folder at login time. It contains the akamas_password file mentioned above, the Kubernetes and SSH configuration files, which will be symlinked to your home folder.

Logging into Akamas requires a valid Akamas license.

To install a license get in touch with Akamas Customer Service to receive:

• the Akamas license file

• your "customer name" to configure in the variable AKAMAS_CUSTOMER for Docker installations or akamasCustomer for Kubernetes installations

• the URL to configure in the AKAMAS_BASE_URL variable for Docker installations

• login credentials

Once you have this information, you can issue the following commands:

cd <your bundle files location>

akamas install license <license file you have been provided>

akamas login
# prompt for user and password

kubectl get secret -n <NAMESPACE> akamas-admin-credentials -o go-template='{{.data.password | base64decode}}'

Akamas might collect anonymized usage information on running optimizations. Collection and tracking are disabled by default and can be manually enabled.

Docker installation

External tracking is managed through the following environment variables:

• AKAMAS_TRACKER_URL: the target URL for all tracking info.

• AKAMAS_TRACKING_OPT_OUT: when set to 1, disables anonymous data collection.

Tracking for a running instance can be enabled by editing the AKAMAS_TRACKING_OPT_OUT variable in the docker-compose.yaml file.

To enable tracking set the variable to the following value:

AKAMAS_TRACKING_OPT_OUT=0

Then issue the command:

docker-compose up -d

Kubernetes installation

External tracking is managed through the field trackingOptOut in the Values file. To enable tracking set trackingOptOut to 0 as in the following example and upgrade the installation:

awsAccessKeyId: "YOUR_ACCESSKEY_ID"
awsSecretAccessKey: "YOUR_SECRET_ACCESS_KEY"

trackingOptOut: 0

This section covers some topics on how to manage the Akamas Server:

• Akamas logs

• Audit logs

• Install upgrades and patches

• Backup & Recovery of the Akamas Server

• Monitor the Akamas Server

Akamas allows dumping log entries from a specific service, workspace, workflow, study, trial, and experiment, for a specific timeframe and at different log levels.

Akamas CLI for logs

Akamas logs can be dumped via the following CLI command:

akamas log

This command provides many filters which can be retrieved with the following command:

akamas log --help

which should return

Usage: akamas log [OPTIONS] [MESSAGE]

  Show Akamas logs

Options:
  -d, --debug                     Show extended error messages if present.
  --page-size INTEGER             Number of log's lines to be retrieved NOTE:
                                  This argument is mutually exclusive with
                                  arguments: [dump, no_pagination].
  --no-pagination                 Disable pagination and print all logs NOTE:
                                  This argument is mutually exclusive with
                                  arguments: [dump, page_size].
  --dump                          Print the logs without pagination and
                                  formatting NOTE: This argument is mutually
                                  exclusive with arguments: [page_size,
                                  no_pagination].
  -f, --from [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S|%Y-%m-%dT%H:%M:%S.%f|%Y-%m-%d %H:%M:%S.%f|[-]nw|[-]nd|[-]nh|[-]nm|[-]ns]
                                  The start timestamp of the logs
  -t, --to [%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S|%Y-%m-%dT%H:%M:%S.%f|%Y-%m-%d %H:%M:%S.%f|[-]nw|[-]nd|[-]nh|[-]nm|[-]ns]
                                  The end timestamp of the logs
  -s, --study TEXT                UUID or name of the Study
  -e, --exp INTEGER               Number of the experiment
  --trial INTEGER                 Number of the trial
  -y, --system TEXT               UUID or name of the System
  -W, --workflow TEXT             UUID or name of the Workflow
  -l, --log-level TEXT            Log level
  -S, --service TEXT              Akamas service
  --without-metadata              Hide metadata
  --sorting [ASC|DESC]            Sorting order of the timestamps
  -ws, --workspace TEXT           UUID or name of the Workspace to visualize.
                                  When empty, system logs will be returned
                                  instead
  --help                          Show this message and exit.

For example, to get the list of the most recent Akamas errors:

akamas log -l ERROR

which should return something similar to:

       timestamp                         system                  provider    service                                                                                   message
==============================================================================================================================================================================================================================================================
2022-05-02T15:51:26.88    -                                      -          airflow     Task failed with exception
2022-05-02T15:51:26.899   -                                      -          airflow     Failed to execute job 2 for task Akamas_LogCurator_Task
2022-05-02T15:56:29.195   -                                      -          airflow     Task failed with exception
2022-05-02T15:56:29.215   -                                      -          airflow     Failed to execute job 3 for task Akamas_LogCurator_Task
2022-05-02T16:01:55.587   -                                      -          license     2022-05-02 16:01:47.426 ERROR 1 --- [           main] c.a.m.utils.rest.RestHandlers            :  has failed with returning a response:
                                                                                        {"httpStatus":400,"timestamp":"2022-05-02T16:01:47.413638","error":"Bad Request","message":"The following metrics: 'spark.spark_application_duration' were not found
                                                                                        in any of the components of the system 'analytics_cluster'","path":null}
2022-05-02T16:01:55.587   -                                      -          license     2022-05-02 16:01:47.434 ERROR 1 --- [           main] c.a.m.MigrationApplication               : Unable to complete operation. Mode: RESTORE. Cause: A request to a
                                                                                        downstream service CampaignService has failed: 400 : [{"httpStatus":400,"timestamp":"2022-05-02T16:01:47.413638","error":"Bad Request","message":"The following
                                                                                        metrics: 'spark.spark_application_duration' were not found in any of the components of the system 'analytics_cluster'","path":null}]
2022-05-02T16:01:55.678   -                                      -          license     2022-05-02 16:01:47.434 ERROR 1 --- [           main] c.a.m.MigrationApplication               : Unable to complete operation. Mode: RESTORE. Cause: A request to a
                                                                                        downstream service CampaignService has failed: 400 : [{"httpStatus":400,"timestamp":"2022-05-02T16:01:47.413638","error":"Bad Request","message":"The following
                                                                                        metrics: 'spark.spark_application_duration' were not found in any of the components of the system 'analytics_cluster'","path":null}]
2022-05-02T16:01:55.678   -                                      -          license     2022-05-02 16:01:47.426 ERROR 1 --- [           main] c.a.m.utils.rest.RestHandlers            :  has failed with returning a response:
                                                                                        {"httpStatus":400,"timestamp":"2022-05-02T16:01:47.413638","error":"Bad Request","message":"The following metrics: 'spark.spark_application_duration' were not found
                                                                                        in any of the components of the system 'analytics_cluster'","path":null}
2022-05-02T16:12:10.261   -                                      -          license     2022-05-02 16:05:53.209 ERROR 1 --- [           main] c.a.m.services.CampaignService           : de9f5ff9-418e-4e25-ae2c-12fc8e72cafc
2022-05-02T16:32:07.216   -                                      -          license     2022-05-02 16:31:37.330 ERROR 1 --- [           main] c.a.m.services.CampaignService           : 06c4b858-8353-429c-bacd-0cc56cc44634
2022-05-02T16:38:18.522   -                                      -          campaign    Internal Server Error: Object of class [com.akamas.campaign_service.entities.campaign.experiment.Experiment] with identifier
                                                                                        [ExperimentIdentifier(workspace=ac8481d3-d031-4b6a-8ae9-c7b366f027e8, study=de9f5ff9-418e-4e25-ae2c-12fc8e72cafc, id=2)]: optimistic locking failed; nested exception
                                                                                        is org.hibernate.StaleObjectStateException: Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) :
                                                                                        [com.akamas.campaign_service.entities.campaign.experiment.Experiment#ExperimentIdentifier(workspace=ac8481d3-d031-4b6a-8ae9-c7b366f027e8,
                                                                                        study=de9f5ff9-418e-4e25-ae2c-12fc8e72cafc, id=2)]

Akamas audit logs

Akamas stores all its logs into an internal Elasticsearch instance: some of these logs are reported to the user in the GUI in order to ease the monitoring of workflow executions, while other logs are only accessible via CLI and are mostly used to provide more context and information to support requests.

Audit access can be performed by using the CLI in order to extract logs related to UI or API access. For instance, to extract audit logs from the last hour use the following commands:

• UI Logs

akamas logs --no-pagination -S kong -f -1h

• API Logs

akamas logs --no-pagination -S kong -f -1h

Notice: to visualize the system logs unrelated to the execution of workflows bound to workspaces, you need an account with administrative privileges.

Storing audit logs into files

To ease the integration with external logging systems, Akamas can be configured to store access logs into files. To enable this feature you should:

1. Create a logs folder next to the Akamas docker-compose.yml file

2. Edit the docker-compose.yml file by modifying the line FILE_LOG: "false" to FILE_LOG: "true"

3. If Akamas is already running issue the following command

docker-compose up -d logstash

otherwise, start Akamas first.

When the user interacts with the UI or the API Akamas will report detailed access logs both on the internal database and in a file in the logs folder. To ease log rolling and management every day Akamas will create a new file named according to the pattern access-%{+YYYY-MM-dd}.log.

Akamas patches and upgrades need to be installed by following the specific instructions specified in the package provided. In case of new releases, it is recommended to read the related Release Notes. Under normal circumstances, this usually requires the user to update the docker-compose configuration, as described in the next section.

Docker compose Configuration

When using docker compose to install Akamas, there’s a folder usually named akamas in the user home folder that contains a docker-compose.yml file. This is a YAML text file that contains a list of docker services with the URLs/version pointing to the ECR repo hosting all docker images needed to launch Akamas.

Here’s an excerpt of such a docker-compose.yml file (this example contains 3 services only):

services:
  #####################
  # Database Service #
  #####################
  database:
    image: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/master-db:1.7.0
    container_name: database2
    restart: always
    command: postgres -c max_connections=200

  #####################
  # Optimizer Service #
  #####################
  optimizer:
    image: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/optimizer_service:2.3.0
    container_name: optimizer
    restart: always
    networks:
      - akamas2
    depends_on:
      - database
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /tmp/build/engine_input:/tmp/build/engine_input

  ####################
  # Campaign Service #
  ####################
  campaign:
    image: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/campaign_service:2.3.0
    container_name: campaign
    restart: always
    volumes:
      - config:/config
    networks:
      - akamas2
    depends_on:
      - database
      - optimizer
      - analyzer

The relevant lines that usually have to be patched during an upgrade are the lines with key "image" like:

image: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/master-db:1.7.0

In order to update to a new version you should replace the versions (1.7.0 or 2.3.0) after the colon with the new versions (ask your Akamas support for the correct service versions for a specific Akamas release) then you should restart Akamas with the following console commands: First login to Akamas CLI with:

akamas login

and type username and password as in the example below

ubuntu@ak_machine:~/akamas/ $ akamas login
User: akamas
Password:
User akamas logged in. Welcome.

Now make sure you have the following AWS variables with the proper value in your Linux user environment:

AWS_DEFAULT_REGION
AWS_SECRET_ACCESS_KEY
AWS_ACCESS_KEY_ID

Then log in to AWS with the following command:

aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 485790562880.dkr.ecr.us-east-2.amazonaws.com
Login Succeeded

Then pull all new ECR images for the new service versions you just changed (this should be done from when inside the same folder where file docker-compose.yml resides, usually $HOME/akamas/) with the following command:

docker-compose pull

It should return an output like the following:

Pulling database                ... done
Pulling optimizer               ... done
Pulling elasticsearch           ... done
Pulling log                     ... done
Pulling metrics                 ... done
Pulling telemetry               ... done
Pulling analyzer                ... done
Pulling campaign                ... done
Pulling system                  ... done
Pulling license                 ... done
Pulling store                   ... done
Pulling airflow-db              ... done
Pulling benchmark               ... done
Pulling kong-database           ... done
Pulling kong                    ... done
Pulling user-service            ... done
Pulling keycloak                ... done
Pulling logstash                ... done
Pulling kibana                  ... done
Pulling kong-consumer-init      ... done
Pulling kong-migration          ... done
Pulling keycloak-initializer    ... done
Pulling telemetry-init          ... done
Pulling curator-only-pull-image ... done
Pulling airflow                 ... done
Pulling orchestrator            ... done
Pulling akamas-init             ... done
Pulling akamas-ui               ... done
Pulling pg-admin                ... done
Pulling grafana                 ... done
Pulling prometheus              ... done
Pulling node-exporter           ... done
Pulling cadvisor                ... done
Pulling konga                   ... done

Finally, relaunch all services with:

docker-compose up -d

(usage example below)

ubuntu@ak_machine:~/akamas/ $ docker compose up -d
pgadmin4 is up-to-date
prometheus is up-to-date
benchmark is up-to-date
kibana is up-to-date
node-exporter is up-to-date
store is up-to-date
grafana is up-to-date
cadvisor is up-to-date
Starting telemetry-init ...
Starting curator-only-pull-image ...
Recreating database2             ...
Recreating airflow-db            ...
Starting kong-initializer        ...
akamas-ui is up-to-date
elasticsearch is up-to-date
Recreating kong-db               ...
Recreating metrics               ...
logstash is up-to-date
Recreating log                   ...
...(some logging follows)

Wait for a few minutes and check the Akamas services are back up by running the command:

akamas status -d

The expected output should be like the following (repeat the command after a minute or two if the last line is not "OK" as expected):

Checking Akamas services on http://localhost:8000 service status
analyzer UP
campaign UP
metrics UP
optimizer UP
orchestrator UP
system UP
telemetry UP
license UP
log UP
users UP
OK

External tools

You can use any monitoring tool to check the availability of the Akamas instance.

Checking Akamas services

To check the status of the Akams services please run akamas status -d to identify which service is not able to start up correctly

Here is an example of output:

Checking Akamas services on http://localhost:8000
 service	 status
=========================
analyzer       	UP
campaign       	UP
metrics        	UP
optimizer      	UP
orchestrator   	UP
system         	UP
telemetry      	UP
license        	UP
log            	UP
users          	UP
OK

Akamas server backup

The process of backing up an Akamas server can be divided in two parts, that is system backup and otherwise start Akamas. Backup can be performed in any way you see fit: they’re just regular files so you can use any backup tool.

System backup

System services are hosted on AWS ECR repo so the only thing that fully defines a working Akamas application is the docker-compose.yml file. Performing a backup of the Akamas application is as simple as copying this single file to your backup location. you may schedule any script that performs this weekly or at any frequency you see fit

User data backup

You may list all existing Akamas studies via the Akamas CLI command:

akamas list study

Then you can export all existing studies one by one via the CLI command

akamas export study <UUID>

where UUID is the UUID of a single study. This command exports into a single archive file (tar.gz). These archive files can be backed up to your favorite backup folder.

Akamas server recovery

Akamas server recovery involves recovering the system backup, restarting the Akamas service then re-importing the studies.

System Restore

To restore the system you must recover the original docker-compose.yml then launch the command

docker-compose up &

from the folder where you placed this YAML file and then wait for the system to come up, by checking it with the command

akamas status -d

User data restore

All studies can be re-imported singularly with the CLI command (referring to the correct pathname of the archive):

akamas import study archive.tgz