Optimize cost of a Kubernetes deployment subject to Horizontal Pod Autoscaler
In this guide, you optimize the cost (or resource footprint) of a Kubernetes deployment where the number of replicas is controlled by the HPA. The study tunes both pod resource settings (CPU and memory requests and limits) and HPA options (target CPU utilization) at the same time, while also taking into account your application performance and reliability requirements (SLOs). This optimization happens in production, leveraging Akamas live optimization capabilities.
Prerequisites
an Akamas instance
a Kubernetes cluster, with a deployment to be optimized
a Horizontal Pod Autoscaler working on the desired deployment
a supported telemetry data source configured to collect metrics from the target Kubernetes cluster (see here for the full list)
a way to apply configuration changes recommended by Akamas to the target deployment and HPA. In this guide, Akamas interacts directly with the Kubernetes APIs via
kubectl.
You need a service account with permissions to update your deployment (see below for other integration options).
Optimization setup
In this guide, we assume the following setup:
the Kubernetes deployment to be optimized is called frontend (in the hipster-shop namespace)
in the deployment, there is a container named server, where the app runs
the HPA is called frontend-hpa
both Dynatrace and Prometheus are used as observability tools
Let's set up the Akamas optimization for this use case.
System
For this optimization, you need the following components to model the frontend tech stack:
The Kubernetes Workload, Container and Pod components, containing metrics like CPU used for the different objects and parameters to be tuned like CPU limits at the container levels (from the Kubernetes optimization pack)
An HPA component, which contains HPA parameters like the target CPU utilization
A Web Application component, which contains service-level metrics like throughput and response time of the microservice (from the Web Applicationoptimization pack)
Let's start by creating the system, which represents the Kubernetes deployment to be optimized. To create it, write a system.yaml
manifest like this:
name: frontend
description: The frontend Kubernetes deployment
Then run:
akamas create system system.yaml
Now create the three Kubernetes components. Create a workload.yaml
manifest like the following:
name: workload_frontend
description: The frontend Kubernetes workload
componentType: Kubernetes Workload
properties:
prometheus:
namespace: hipster-shop
deployment: frontend
Then create a container.yaml
manifest like the following:
name: server
description: The server Kubernetes container
componentType: Kubernetes Container
properties:
prometheus:
namespace: hipster-shop
pod: frontend.*
container: server
And a pod.yaml
manifest like the following:
name: pod_frontend
description: The frontend Kubernetes pod
componentType: Kubernetes Pod
properties:
prometheus:
namespace: hipster-shop
pod: frontend.*
Now create the entities by running:
akamas create component workload.yaml frontend-2
akamas create component container.yaml frontend-2
akamas create component pod.yaml frontend-2
Now create an application.yaml
manifest like the following:
name: webapp
description: The web application of frontend deployment
componentType: Web Application
properties:
dynatrace:
id: SERVICE-80258F7AA97F2E4D
prometheus:
namespace: hipster-shop-2
pod: frontend.*
container: server
The run:
akamas create component application.yaml frontend-2
Finally, create anhpa.yaml
manifest like the following:
name: frontend_hpa
description: The HPA for the frontend
componentType: HPA
Then run:
akamas create component hpa.yaml frontend-2
Workflow
To optimize a Kubernetes microservice in production, you need to create a workflow that defines how the new configuration recommended by Akamas will be deployed in production.
Let's explore the high-level tasks required in this scenario and the options you have to adapt it to your environment:
Let's now create a workflow.yaml
manifest like the following:
name: frontend-11-delayedApproval-hpa-1hour-system2
tasks:
- name: configure frontend
operator: FileConfigurator
arguments:
source:
hostname: toolbox
username: akamas
key: /home/stefano/tmp_ak_key
path: /work/examples/hipstershop-hpa/hipstershop-2/ak-frontend.sh.templ
target:
hostname: toolbox
username: akamas
key: /home/stefano/tmp_ak_key
path: /work/ak-frontend-2.sh
- name: apply frontend
operator: Executor
arguments:
timeout: 5m
host:
hostname: toolbox
username: akamas
key: /home/stefano/tmp_ak_key
command: sh /work/ak-frontend-2.sh hipster-shop-2 frontend
- name: verify frontend
operator: Executor
arguments:
timeout: 5m
host:
hostname: toolbox
username: akamas
key: /home/stefano/tmp_ak_key
command: kubectl rollout status --timeout=5m deployment/frontend -n hipster-shop-2;
- name: configure hpa
operator: FileConfigurator
arguments:
source:
hostname: toolbox
username: akamas
key: /home/stefano/tmp_ak_key
path: /work/examples/hipstershop-hpa/hipstershop-2/frontend-hpa-v2.yaml.templ
target:
hostname: toolbox
username: akamas
key: /home/stefano/tmp_ak_key
path: /work/frontend-hpa-v2-2.yaml
- name: apply hpa
operator: Executor
arguments:
timeout: 5m
host:
hostname: toolbox
username: akamas
key: /home/stefano/tmp_ak_key
command: kubectl apply -f /work/frontend-hpa-v2-2.yaml -n hipster-shop-2
- name: check if we are in time or wait for start of next hour
operator: Executor
arguments:
host:
hostname: toolbox
username: akamas
key: /home/stefano/tmp_ak_key
command: if [ $(date +%M) -lt 55 ]; then sleep $((60*(60 - $(date +%M)))); else sleep 0; fi
- name: observe 55 minutes
operator: Sleep
arguments:
seconds: 3300
Then run:
akamas create workflow workflow.yaml
Telemetry
To collect metrics of your target Kubernetes deployment, you create a telemetry instance based on your observability setup.
Create a dynatrace.yaml
manifest like the following:
provider: Dynatrace
config:
url: <YOUR_DYNATRACE_URL>
token: <YOUR_DYNATRACE_TOKEN>
pushEvents: false
Then run:
akamas create telemetry-instance dynatrace.yaml frontend-2
Create a prometheus.yaml
manifest like the following:
provider: Prometheus
config:
address: prom-kube-prometheus-stack-prometheus.monitoring
port: 9090
duration: 60
logLevel: DETAILED
metrics:
- metric: cost
datasourceMetric: 'sum(kube_pod_container_resource_requests{resource="cpu" %FILTERS%})*29 + sum(kube_pod_container_resource_requests{resource="memory" %FILTERS%})/1024/1024/1024*3.2'
Then run:
akamas create telemetry-instance prometheus.yaml frontend-2
Study
It's now time to create the Akamas study to achieve your optimization objectives.
Let's explore how the study is designed by going through the main concepts. The complete study manifest is available at the bottom.
You can now create a study.yaml
manifest like the following:
name: ak-frontend - live - system 2
system: frontend-2
workflow: frontend-11-delayedApproval-hpa-1hour-system2
goal:
name: Cost
objective: minimize
function:
formula: web_application.cost
constraints:
absolute:
- name: Application response time degradation
formula: web_application.requests_response_time_p50:p90 <= 60
- name: Application error rate degradation
formula: web_application.requests_error_rate:p90 <= 0.02
- name: Container CPU saturation
formula: server.container_cpu_util_max:p90 < 0.8
- name: Container memory saturation
formula: server.container_memory_used:max / server.container_memory_limit < 0.7
windowing:
type: trim
trim: [1m, 1m]
task: observe 55 minutes
parametersSelection:
- name: server.cpu_request
domain: [10, 500]
- name: server.cpu_limit
domain: [10, 500]
- name: server.memory_limit
domain: [16, 640]
- name: frontend_hpa.metrics_resource_target_averageUtilization
domain: [10, 90]
parameterConstraints:
- name: CPU request less or equal to limits
formula: server.cpu_request <= server.cpu_limit
- name: CPU limit within a given factor of request
formula: server.cpu_limit <= server.cpu_request * 2
workloadsSelection:
- name: web_application.requests_throughput:max
- name: web_application.requests_throughput
numberOfTrials: 1
steps:
- name: baseline
type: baseline
numberOfTrials: 3
values:
server.cpu_request: 200
server.cpu_limit: 400
server.memory_limit: 128
frontend_hpa.metrics_resource_target_averageUtilization: 60
renderParameters: [frontend_hpa.metrics_resource_target_averageUtilization]
- name: optimize
type: optimize
numberOfExperiments: 300
Then run:
akamas create study study.yaml
You can now follow the live optimization progress and explore the results using the Akamas UI.
Last updated
Was this helpful?