Executor Operator

The Executor Operator can be used to execute a shell command on a target machine using SSH. It allows configuring systems tuned by Akamas by interpolating configuration parameters and study information into commands on remote machines.

Templates for configuration parameters

The Executor allows writing templates for configuration parameters in two ways:

• specify that a parameter should be interpolated directly:

${component_name.parameter_name}

• specify that all parameters of a component should be interpolated:

${component_name.*}

Suffix or prefix for interpolated parameters

It is possible to add a prefix or suffix to interpolated configuration parameters by acting at the component-type level:

Notice that any parameter that does not contain the FileConfigurator element in the operators' attribute is ignored and not written.

name: Component Type 1
description: My Component type
parameters:
- name: x1
  domain:
    type: real
    domain: [-5.0, 10.0]
  defaultValue: -5.0
  # Under this section, the operator to be used to configure the parameters is defined
  operators:
    FileConfigurator:
      # using this OPTIONAL confTemplate property is possible to interpolate the parameter value with a prefix and a suffix
      confTemplate: "PREFIX${value}SUFFIX"

In the example above, the parameter x1 will be interpolated with the prefix PREFIX and the suffix SUFFIX, ${value} will be replaced with the actual value of the parameter at each experiment.

Example

Let's assume we want to apply the following configuration:

component1.param1: 1024
component1.param2: Category1
component2.param3: 7
component2.param4: 35.4

where component1 is of type MyComponentType and MyComponentType is defined as follows:

name: MyComponentType
description: "MyComponentType"
parameters:
- name: param1
  domain:
    type: real
    domain: [-5.0, 10.0]
  defaultValue: -5.0
  # Under this section, the operator to be used to configure the parameters is defined
  operators:
    FileConfigurator:
      # using this OPTIONAL confTemplate property, it is possible to interpolate the parameter value with a prefix and a suffix
      confTemplate: "X1:${value}MB"
...

A template command to interpolate only parameter component1.param1 and all parameters from component2 would look like this:

myexecutable.sh -PARAM ${component1.param1} -PARAMS ${component2.*}

The command after the configuration parameters are interpolated would look like this:

myexecutable.sh -PARAM X1:1024MB -PARAMS 7 35.4

Note that the command in this example is a bash command whose arguments are constructed by interpolating configuration parameters.

Configuration Template Override

Some commands may require formatting parameter value using a template different from the one provided by default in the componentType. For this reason, it is possible to override the configuration template of each parameter at the workflow task level.

To handle these situations, it is possible to override the configuration template at the workflow level. The override can be applied to all components of a certain componentType.

Example

name: Configure Benchmark
operator: FileConfigurator
arguments:
  confTemplate:
    MyComponentType.param1: ${value}

In this example, the param1 parameter for all components of type MyComponentType will be configured using the ${value} template.

Templates for study information

The Executor Operator supports the interpolation of study-related information into commands executed on remote machines. This feature allows you to dynamically include metadata about the study, experiment, and trial within your commands or scripts.

You can access the following study information using the corresponding placeholders:

• Study Name: ${ak-study.name}

• Study ID: ${ak-study.id}

• Step Name: ${ak-step.name}

• Experiment ID: ${ak-experiment.id}

• Trial ID: ${ak-trial.id}

• Trial Start Time: ${ak-trial.start_time}

• Task Try Number: ${ak-task.retry}

• Task Start Time: ${ak-task.start_time}

By including these placeholders in your command templates, the Executor Operator will replace them with the actual values when the command is executed.

Example

Saving Study Information as Environment Variables

A common use case is to pass study information to scripts executed in subsequent tasks. This can be achieved by exporting, in a task, the study information as environment variables which can then be accessed by scripts in later tasks. In the example below, target-machine is a customer machine that contains the bash script named script.sh

name: example-workflow
tasks:
  - name: Launch Script with Study Info
    operator: Executor
    arguments:
      command: "export STUDY_NAME='${ak-study.name}' && bash script.sh"
      host:
        hostname: target-machine-hostname
        username: target-machine-username
        password: target-machine-password

Passing Study Information to Scripts

Or the same study information can, instead, be passed directly to scripts as arguments:

name: example-workflow
tasks:
  - name: Launch Script with Study Info
    operator: Executor
    arguments:
      command: "bash script.sh --study-name '${ak-study.name}'"
      host:
        hostname: target-machine-hostname
        username: target-machine-username
        password: target-machine-password

Operator arguments

Name	Type	Values restrictions	Required	Default	Description
command	String	If the template ${componentType.param} is present in the command, the component type and the specified parameter must exist.	yes		The shell command to be executed on the remote machine
host	Object	See structure documented below	no		Information relative to the target machine onto which the command has to be executed using SSH
component	String	It should match the name of an existing Component of the System under test	no		The name of the Component whose properties can be used as arguments of the operator
detach	Boolean		no	False	The execution mode of the shell command. Default (False) execution will be synchronous, detached (True) execution will be asynchronous and will return immediately
replaceTemplate	Boolean		no	True	Enable/Disable template replacement ${componentType.parameter}
confTemplate	Object	A map where the keys are in the format componentType.parameter, and the values are the corresponding new templates	no		The values of new parameter templates

Host structure and arguments

Here follows the structure of the host argument:

host:
  hostname: this_is_a_hostname
  username: this_is_a_username
  password: this_is_a_password
  sshPort: 22
  key: this_is_a_key

with its arguments:

Name	Type	Value Restrictions	Required	Default	Description
hostname	String	should be a valid SSH host address	no, if the Component whose name is defined in component has a property named hostname		SSH endpoint
username	String		no, if the Component whose name is defined in component has a property named username		SSH login username
password	String	cannot be set if key is already set	no, if the Component whose name is defined in component has a property named password		SSH login password
sshPort	Number	1≤sshPort≤65532	no	22	SSH port
key	String	cannot be set if password is already set	no, if the Component whose name is defined in component has a property named key		SSH login key. Either provide directly the key value or specify the path of the file (local to the cli executing the create command) to read the key from. The operator supports RSA and DSA Keys.

Get operator arguments from component

The component argument can refer to a component by name and use its properties as the arguments of the operator (see mapping here below). In case the mapped arguments are already provided to the operator, there is no override.

Component property to operator argument mapping

Component Property	Operator Argument
hostname	host->hostname
username	host->username
sshPort	host->sshPort
password	host->password
key	host->key

Workflow examples

Let's assume the user wants to run a script on a remote host and expects the script to be executed successfully within 30 seconds but might fail occasionally.

The code below launches a script, waits for its completion and, in failure/timeout cases, retries 3 times by waiting 10 seconds between retries:

name: Run Script
operator: Executor
arguments:
  timeout: 30s
  retries: 3
  retry_delay: 10s
  command: bash /tmp/myscript.sh
  host:
    hostname: frontend.akamas.io
    username: akamas
    key: secret.key

This snippet executes a uname command with explicit host information (explicit SSH key)

name: TestConnectivity
operator: Executor
arguments:
  command: bash uname -a
  host:
    hostname: frontend.akamas.io
    username: akamas
    key: |-
      -----BEGIN RSA PRIVATE KEY-----
      RSA KEY HERE
      -----END RSA PRIVATE KEY-----

The next piece of code executes a uname command with explicit host information (imported SSH key)

name: TestConnectivity
operator: Executor
arguments:
  command: bash uname -a
  host:
    hostname: frontend.akamas.io
    username: akamas
    key: path/to/key

This executes a uname command with host information taken from a Component

name: TestConnectivity
operator: Executor
arguments:
  command: bash uname -a
  component: frontend1

This starts a load-testing script and keeps it running in the background during the workflow

name: TestConnectivity
operator: Executor
arguments:
  command: bash start_load.sh
  component: tester
  detach: true

Logging with command scripts

When executing a time-consuming script on a remote SSH machine, it may be helpful to let it provide user feedback during its execution. Adding logging lines in the script allows you to inspect its current status while it runs. For example, in the case of an offline study, if you're using the Akamas UI, you can view these logging lines in a dedicated drawer that pops up in the offline study detail page. This drawer is accessible by expanding the trial detail and selecting the desired workflow task (it will be indicated by a blue dot if the study is currently running that script). See the example image below:

By clicking on the blue dot in the Tasks column, a drawer will popup showing the workflow task log details while it's running. See example image:

Everything between the line Start of script and the bottom line is the console log of the executed script. By default the Executor logs everything to the standard output (black text) or to the standard error stream (red text).

To display the output in real-time in the UI task log, ensure that every text line sent to the console ends with a line break. The implementation depends on the language you're using. For example:

• Use methods that automatically append a newline, such as println instead of print.

• or when the only print command available doesn't append newline, ensure strings explicitly end with a newline (e.g.: in Golang messages, add trailing \n).

In most languages like Bash, Node.js, or Golang, this approach is sufficient. However, Python scripts executed through the Executor may present the following undesired effects:

• All output appears in red (typically caused by logging to an unspecified stream, when the default is standard error).

• Logs are displayed only at the end of the script (due to unflushed text buffers).

Solving Python-Specific Issues

To address these issues in Python, you can use one of the following solutions:

1. Explicitly flush each print statement:

# Avoid this: it doesn't flush
print("Finished pass 1")
# Use this instead
print("Finished pass 1", flush=True)

2. Globally enable auto-flushing: Add these two lines at the beginning of your script:

import functools
print = functools.partial(print, flush=True)

This ensures all print statements automatically flush without modifying the rest of the script.

3. Use the logging library: Replace print with logging.info() and configure the logger to use the standard output stream:

# Preliminary config at the start of the script
import logging
import sys
# Use stdout to write in black color
# Set to logging.INFO or only warning/error messages will be printed
logging.basicConfig(stream=sys.stdout, level=logging.INFO, format='%(message)s')
# ...
# Do something
# ...
# Then log something to console
logging.info("Finished pass 1")

NOTE: This logging.basicConfig approach requires few configuration lines but can only log every message line (errors included) in black text.

Advanced Logging Example

Below is an advanced example demonstrating how to log messages with different levels (INFO, WARNING, DEBUG, ERROR) and display them in two different colors:

#!/usr/bin/python3

import logging
import time
import platform
import sys
from random import randint

# Filter to ensure each log level is displayed only once
class FilterExactlevel(logging.Filter):
    def __init__(self, passlevel):
        self.passlevel = passlevel

    def filter(self, record):
        return (record.levelno == self.passlevel)

# Define logging handlers for different levels
handler_err = logging.StreamHandler(sys.stderr)
handler_err.setLevel(logging.ERROR)
handler_err.addFilter(FilterExactlevel(logging.ERROR))
handler_err.setFormatter(logging.Formatter('%(message)s'))
logging.getLogger().addHandler(handler_err)

handler_debug = logging.StreamHandler(sys.stderr)
handler_debug.setLevel(logging.DEBUG)
handler_debug.addFilter(FilterExactlevel(logging.DEBUG))
handler_debug.setFormatter(logging.Formatter('%(levelname)s - %(message)s'))
logging.getLogger().addHandler(handler_debug)

handler_info = logging.StreamHandler(sys.stdout)
handler_info.setLevel(logging.INFO)
handler_info.addFilter(FilterExactlevel(logging.INFO))
handler_info.setFormatter(logging.Formatter('%(message)s'))
logging.getLogger().addHandler(handler_info)

handler_warn = logging.StreamHandler(sys.stdout)
handler_warn.setLevel(logging.WARNING)
handler_warn.addFilter(FilterExactlevel(logging.WARNING))
handler_warn.setFormatter(logging.Formatter('%(levelname)s - %(message)s'))
logging.getLogger().addHandler(handler_warn)

logging.getLogger().setLevel(logging.DEBUG)

# Main script logic
def prepare_job():
    logging.info('Preparing job:')
    seconds = randint(3,7)
    time.sleep(seconds) # Simulates your time-consuming task
    if seconds >= 4:
        logging.warning("Preparation task took longer than usual")
    logging.info('End of prepare_job')

def perform_job():
    logging.info('Starting job:')
    seconds = randint(14,30)
    time.sleep(seconds) # Simulates your time-consuming task
    if seconds >= 16:
        logging.error("Job failed with NoFileException")
    else:
        logging.warning("Job took longer than usual")
    logging.info('End of job')

def cleanup():
    logging.info('Starting cleanup:')
    seconds = randint(4,8)
    time.sleep(seconds) # Simulates your time-consuming task
    if seconds >= 5:
        logging.warning("Cleanup task took longer than usual")
    logging.info('End of cleanup')

# Execution starts here
logging.debug(f'OS: {platform.system()} {platform.release()}')
logging.debug('Python: ' + platform.python_version())
logging.info('Start of script')
prepare_job()
perform_job()
cleanup()
logging.info('End of script')

This script correctly handles real-time output and is compatible with the Executor's logging system.

Troubleshooting

Troubles in running sh scripts remotely

Due to the stderr configuration, it could happen that invoking a bash script on a server has a different result than running the same script from Akamas Executor Operator. This is quite common with Tomcat startup scripts like $HOME/tomcat/apache-tomcat_1299/bin/startup.sh.

To avoid this issue simply create a wrapper bash file on the target server adding the set -m instruction before the sh command, eg:

#!/bin/bash
set -m;
$HOME/tomcat/apache-tomcat_1299/bin/startup.sh

and then configure the Executor Operator to run the wrapper script like:

command: "bash $HOME/akamasScript/tomcatStart.sh"

You can run the following to emulate the same behavior of Akamas running scripts over SSH:

ssh -t <user>@<server> <your command here>

Troubles in keeping a script running in the background

There are cases in which you would like to keep a script running for the whole duration of the test. Some examples could be:

• A script applying load to your system for the duration of the workflow

• The manual start of an application to be tested

• The setup of a listener that gathers logs, metrics, or data

In all the instances where you need to keep a task running beyond the task that started it, you must use the detach: true property. Note that a detached executor task returns immediately, so you should run only the background task in detached mode.

Remember to keep all tasks requiring synchronous (standard) behavior out of the detached task.

Example:

switch on machine and wait for SSH
run application test in background → detached mode
execute test run

Library references

The library used to execute scripts remotely is Fabric, a high-level Python library designed to execute shell commands remotely over SSH, yielding useful Python objects in return.

The Fabric library uses a connection object to execute scripts remotely (see connection — Fabric documentation). The option of a dedicated detach mode comes from implementing the more robust disown property from the Invoke Runner underlying the Connection (see runners — Invoke documentation). This is the reason you should rely on detach whenever possible instead of running the background processes straight into the script.

In the Frequently Asked/Answered Questions (FAQ) — Fabric documentation you may find some further information about the typical problems and solutions due to hanging problems for background processes.