Study template

Optimization studies are defined using a YAML manifest with the following structure:

system: 1
name: Optimizing the e-shop application
goal:
  objective: maximize
  function:
    formula: payments_per_sec
    variables:
      payments_per_sec:
        metric: eshop_payments
        labels:
          componentName: eshop

workflow: eshop_jmeter_test
steps:
  - name: baseline
    type: baseline
    values:
      tomcat.maxThreads: 1024
      jvm.maxHeap: 2048
      jvm.garbageCollectorType: G1GC
      postgres.shared_buffers: 4096

with the following mandatory properties:

Field
Type
Value restrictions
Is required
Default Value
Description

system

object reference

TRUE

The system the study refers to

name

string

TRUE

The name of the study

goal

object

TRUE

The goal and constraint description - see Goal & Constraints

kpis

list

FALSE

The KPIs description - see KPI

numberOfTrials

integer

FALSE

1

The number of trials for each experiment - see below

trialAggregation

string

MAX, MIN, AVG

FALSE

AVG

The aggregation used to calculate the score across multiple trials - see below

parametersSelection

list

FALSE

all

The list of parameters to be tuned - see Parameter selection

metricsSelection

list

FALSE

all

The list of metrics - see Metric selection

workloadsSelection

object array

FALSE

The list of defined workloads - this only applies to live optimization studies - see Workload Selection

windowing

string

FALSE

trim

The windowing strategy - this only applies to offline optimization studies - see Windowing strategy

workflow

object reference

TRUE

The workflow the study refers to

steps

list

TRUE

The description of the steps - see Steps

Some of these optional properties depend on whether the study is an offline or live optimization study.

Number of trials

It is possible to perform more than one trial per experiment in order to validate the score of a configuration under test, e.g. to take into account noisy environments.

The following fragment of the YAML definition of a study sets the number of trials to 3:

numberOfTrials: 3

Notice: This is a global property of the study which can be overwritten for each step.

Trial aggregation

The trial aggregation policy defines how trial scores are aggregated to form experiment scores.

There are three different types of strategies to aggregate trial scores:

  • AVG: the score of an experiment is the average of the scores of its trials - this is the default

  • MIN: the score of an experiment is the minimum among the scores of its trials

  • MAX: the score of an experiment is the maximum among the scores of its trial

The following fragment of the YAML definition of a study sets the trial aggregation to MAX:

trialAggregation: MAX # other possible values are AVG, MIN

Examples

The following system refers to an offline optimization study for a system modeling an e-commerce service, where a windowing strategy is specified:

system: "bde4f259-9a51-4c67-87aa-3c5bc599c6b9" # id of the system to optimize with the actions defined in this study
workflow: "eshop_jmeter_test" # name of the workflow to use to perform trials
name: Optimizing the e-shop application # name of the study
goal: # the performance goal to achieve
  objective: "maximize"
  function:
    formula: "eshop.payments_per_second"
windowing: # the temporal window in which to compute the score of a trial
  type: "trim"
  trim: ["10s", "0s"] # use the duration of the trial minus 0s from start and end to compute the score of the trial
parametersSelection: "all" # use all available configuration parameters
metricsSelection: "all" # gather all metrics
steps: # the steps to conduct to perform experiments and trials
  - name: "my_baseline" # do first a baseline with the provided configuration
    type: "baseline"
    values:
      jvm.maxHeap: 2048
      jvm.gcType: "-XX:+UseParallelGC"
  - name: my_optimization # then do 20 optimization experiments of 2 trials each
    type: optimize
    numberOfExperiments: 200
    numberOfTrials: 2

The following offline study refers to a tuning initiative for a Cassandra-based system (ID 2)

system: 2
name: Optimizing the cassandra - team 2
goal:
  objective: minimize
  function:
    formula: read_response_time_p90
    variables:
      read_response_time_p90:
        metric: read_response_time_p90
        labels:
          componentName: cassandra

windowing:
  type: trim
  trim: [5m, 1m]

workflow: cassandra_workflow
parametersSelection:
  - name: cassandra_jvm.jvm_maxHeapSize
  - name: cassandra.cassandra_concurrentReads
  - name: cassandra.cassandra_concurrentWrites
  - name: cassandra.cassandra_fileCacheSizeInMb
  - name: cassandra.cassandra_memtableCleanupThreshold
  - name: cassandra.cassandra_concurrentCompactors

steps:
  - name: baseline_step
    type: baseline
    values:
      cassandra_jvm.jvm_maxHeapSize: 1024
      cassandra.cassandra_concurrentReads: 32
      cassandra.cassandra_concurrentWrites: 32
      cassandra.cassandra_fileCacheSizeInMb: 512
      cassandra.cassandra_memtableCleanupThreshold: 0.11
      cassandra.cassandra_concurrentCompactors: 2

  - name: optimization_step
    type: optimize
    optimizer: CALABI
    numberOfExperiments: 50

The following offline study is for tuning another Cassandra-based system (ID 3) by acting only on JVM and Linux parameters

system: 3
name: Optimizing a Cassandra NoSQL database version 3 (jvm + os parameters)
goal:
  objective: minimize
  function:
    formula: (x1+x2)/2
    variables:
      x1:
        metric: write_response_time_p90
        labels:
          componentName: cassandra_team1
      x2:
        metric: read_response_time_p90
        labels:
          componentName: cassandra_team1

windowing:
  type: trim
  trim: [8m,2m]

numberOfTrials: 2
workflow: cassandra_workflow_jvm_os

parametersSelection:
  - name: JVM1.jvm_maxHeapSize
  - name: JVM1.jvm_newRatio
  - name: JVM1.jvm_survivorRatio
  - name: JVM1.jvm_maxTenuringThreshold
  - name: JVM1.jvm_gcType
  - name: JVM1.jvm_concurrentGCThreads
  - name: os1.os_cpuSchedMinGranularity
  - name: os1.os_cpuSchedWakeupGranularity
  - name: os1.os_CPUSchedMigrationCost
  - name: os1.os_CPUSchedChildRunsFirst
  - name: os1.os_CPUSchedLatency

steps:
  - name: baseline_step
    type: baseline
    values:
      JVM_team1.jvm_maxHeapSize: 1024
      JVM_team1.jvm_newRatio: 2
      JVM_team1.jvm_survivorRatio: 8
      JVM_team1.jvm_maxTenuringThreshold: 15
      JVM_team1.jvm_gcType: UseConcMarkSweepGC
      JVM_team1.jvm_concurrentGCThreads: 8
      os_team1.os_cpuSchedMinGranularity: 3000000
      os_team1.os_cpuSchedWakeupGranularity: 4000000
      os_team1.os_CPUSchedMigrationCost: 500000
      os_team1.os_CPUSchedChildRunsFirst: 0
      os_team1.os_CPUSchedLatency: 24000000

  - name: optimization_sobol
    type: optimize
    optimizer: SOBOL
    numberOfExperiments: 3

  - name: optimization_calabi
    type: optimize
    optimizer: CALABI
    numberOfExperiments: 50

Last updated