Study template
Optimization studies are defined using a YAML manifest with the following structure:
1
system: 1
2
name: Optimizing the e-shop application
3
goal:
4
objective: maximize
5
function:
6
formula: payments_per_sec
7
variables:
8
payments_per_sec:
9
metric: eshop_payments
10
labels:
11
componentName: eshop
12
13
workflow: eshop_jmeter_test
14
steps:
15
- name: baseline
16
type: baseline
17
values:
18
tomcat.maxThreads: 1024
19
jvm.maxHeap: 2048
20
jvm.garbageCollectorType: G1GC
21
postgres.shared_buffers: 4096
with the following mandatory properties:
Field | Type | Value restrictions | Is required | Default Value | Description |
|---|---|---|---|---|---|
system | object reference | | TRUE | | The system the study refers to |
name | string | | TRUE | | The name of the study |
goal | object | | TRUE | | |
kpis | list | | FALSE | | |
numberOfTrials | integer | | FALSE | 1 | The number of trials for each experiment - see below |
trialAggregation | string | MAX, MIN, AVG | FALSE | AVG | The aggregation used to calculate the score across multiple trials - see below |
parametersSelection | list | | FALSE | all | |
metricsSelection | list | | FALSE | all | |
workloadsSelection | object array | | FALSE | | The list of defined workloads - this only applies to live optimization studies - see Workload Selection |
windowing | string | | FALSE | trim | The windowing strategy - this only applies to offline optimization studies - see Windowing strategy |
workflow | object reference | | TRUE | | The workflow the study refers to |
steps | list | | TRUE | |
Some of these optional properties depend on whether the study is an offline or live optimization study.
It is possible to perform more than one trial per experiment in order to validate the score of a configuration under test, e.g. to take into account noisy environments.
The following fragment of the YAML definition of a study sets the number of trials to 3:
numberOfTrials: 3
Notice: This is a global property of the study which can be overwritten for each step.
The trial aggregation policy defines how trial scores are aggregated to form experiment scores.
There are three different types of strategies to aggregate trial scores:
- AVG: the score of an experiment is the average of the scores of its trials - this is the default
- MIN: the score of an experiment is the minimum among the scores of its trials
- MAX: the score of an experiment is the maximum among the scores of its trial
The following fragment of the YAML definition of a study sets the trial aggregation to MAX:
trialAggregation: MAX # other possible values are AVG, MIN
The following system refers to an offline optimization study for a system modeling an e-commerce service, where a windowing strategy is specified:
1
system: "bde4f259-9a51-4c67-87aa-3c5bc599c6b9" # id of the system to optimize with the actions defined in this study
2
workflow: "eshop_jmeter_test" # name of the workflow to use to perform trials
3
name: Optimizing the e-shop application # name of the study
4
goal: # the performance goal to achieve
5
objective: "maximize"
6
function:
7
formula: "eshop.payments_per_second"
8
windowing: # the temporal window in which to compute the score of a trial
9
type: "trim"
10
trim: ["10s", "0s"] # use the duration of the trial minus 0s from start and end to compute the score of the trial
11
parametersSelection: "all" # use all available configuration parameters
12
metricsSelection: "all" # gather all metrics
13
steps: # the steps to conduct to perform experiments and trials
14
- name: "my_baseline" # do first a baseline with the provided configuration
15
type: "baseline"
16
values:
17
jvm.maxHeap: 2048
18
jvm.gcType: "-XX:+UseParallelGC"
19
- name: my_optimization # then do 20 optimization experiments of 2 trials each
20
type: optimize
21
numberOfExperiments: 200
22
numberOfTrials: 2
The following offline study refers to a tuning initiative for a Cassandra-based system (ID 2)
1
system: 2
2
name: Optimizing the cassandra - team 2
3
goal:
4
objective: minimize
5
function:
6
formula: read_response_time_p90
7
variables:
8
read_response_time_p90:
9
metric: read_response_time_p90
10
labels:
11
componentName: cassandra
12
13
windowing:
14
type: trim
15
trim: [5m, 1m]
16
17
workflow: cassandra_workflow
18
parametersSelection:
19
- name: cassandra_jvm.jvm_maxHeapSize
20
- name: cassandra.cassandra_concurrentReads
21
- name: cassandra.cassandra_concurrentWrites
22
- name: cassandra.cassandra_fileCacheSizeInMb
23
- name: cassandra.cassandra_memtableCleanupThreshold
24
- name: cassandra.cassandra_concurrentCompactors
25
26
steps:
27
- name: baseline_step
28
type: baseline
29
values:
30
cassandra_jvm.jvm_maxHeapSize: 1024
31
cassandra.cassandra_concurrentReads: 32
32
cassandra.cassandra_concurrentWrites: 32
33
cassandra.cassandra_fileCacheSizeInMb: 512
34
cassandra.cassandra_memtableCleanupThreshold: 0.11
35
cassandra.cassandra_concurrentCompactors: 2
36
37
- name: optimization_step
38
type: optimize
39
optimizer: CALABI
40
numberOfExperiments: 50
The following offline study is for tuning another Cassandra-based system (ID 3) by acting only on JVM and Linux parameters
1
system: 3
2
name: Optimizing a Cassandra NoSQL database version 3 (jvm + os parameters)
3
goal:
4
objective: minimize
5
function:
6
formula: (x1+x2)/2
7
variables:
8
x1:
9
metric: write_response_time_p90
10
labels:
11
componentName: cassandra_team1
12
x2:
13
metric: read_response_time_p90
14
labels:
15
componentName: cassandra_team1
16
17
windowing:
18
type: trim
19
trim: [8m,2m]
20
21
numberOfTrials: 2
22
workflow: cassandra_workflow_jvm_os
23
24
parametersSelection:
25
- name: JVM1.jvm_maxHeapSize
26
- name: JVM1.jvm_newRatio
27
- name: JVM1.jvm_survivorRatio
28
- name: JVM1.jvm_maxTenuringThreshold
29
- name: JVM1.jvm_gcType
30
- name: JVM1.jvm_concurrentGCThreads
31
- name: os1.os_cpuSchedMinGranularity
32
- name: os1.os_cpuSchedWakeupGranularity
33
- name: os1.os_CPUSchedMigrationCost
34
- name: os1.os_CPUSchedChildRunsFirst
35
- name: os1.os_CPUSchedLatency
36
37
steps:
38
- name: baseline_step
39
type: baseline
40
values:
41
JVM_team1.jvm_maxHeapSize: 1024
42
JVM_team1.jvm_newRatio: 2
43
JVM_team1.jvm_survivorRatio: 8
44
JVM_team1.jvm_maxTenuringThreshold: 15
45
JVM_team1.jvm_gcType: UseConcMarkSweepGC
46
JVM_team1.jvm_concurrentGCThreads: 8
47
os_team1.os_cpuSchedMinGranularity: 3000000
48
os_team1.os_cpuSchedWakeupGranularity: 4000000
49
os_team1.os_CPUSchedMigrationCost: 500000
50
os_team1.os_CPUSchedChildRunsFirst: 0
51
os_team1.os_CPUSchedLatency: 24000000
52
53
- name: optimization_sobol
54
type: optimize
55
optimizer: SOBOL
56
numberOfExperiments: 3
57
58
- name: optimization_calabi
59
type: optimize
60
optimizer: CALABI
61
numberOfExperiments: 50
Last modified 2mo ago