Analyzing results of offline optimization studies

Since an offline optimization study lasts for at most the number of configured experiments and typically runs in a test or pre-production environment, results could be safely either analyzed after the study has completely finished.

However, it is a good practice to analyze partial results while the study is still running as this may provide useful insights about both the system being optimized (e.g. understanding of the system dynamics and sub-optimal configurations that could be immediately applied) and about the optimization study itself (e.g. how to re-design a workflow or change constraints), early-on.

The Akamas UI displays the results of an offline optimization study in different visual areas:

• the Best Configuration section provides the optimal configuration identified by Akamas, as a list of recommended values for the optimization parameters compared to the baseline and ranked according to their relevance;

Best Configuration section

• the Progress tab see the following figures) displays the progression of the study with respect to the study steps, the status of each experiment (and trial), its associated score, and the parameter values of the corresponding configurations; this area is mostly used for study monitoring (e.g. identifying failing workflows) and troubleshooting purposes;

Progress tab showing the study steps and experiments

Progress tab showing the configuration associted to an experiment as compared to the best and baseline

• the Analysis tab (see the following figures) displays how the baseline and experiments score with respect to the optimization goal, and the values of metrics and parameters for the corresponding configurations; this area supports the analysis of the different configurations;

Higher part of the Analysis tab showing scored experiments over time

Lower part of the Analysis tab showing values for each confguration metric and parameter

• the Metrics tab (see the following figure) displays the behavior of the metrics for all executed experiments (and trials); this area supports both study validation activities and deeper analysis of the system behavior;

Analysis tab with selected metrics and relative constraints (toggle on)

• the Insights section (see the following figure) displays any suboptimal configurations that have been identified for the study KPIs, and also allows making comparisons among them and the best configuration - the Optimization Insights page describes in further detail the Insight section and the insights tags displayed in other areas of the Akamas UI.

Best Score and Insight section of an offline optinmization study