# Optimizing a MongoDB server instance

In this example study, we are going to optimize a MongoDB single server instance by setting the performance goal of maximizing the throughput of operations toward the database.

Concerning performance tests, we are going to employ [YCSB](https://github.com/brianfrankcooper/YCSB), a popular benchmark created by Yahoo for testing various NoSQL databases.

To extract MongoDB metrics, we are going to spin up a [Prometheus](https://prometheus.io/) instance and we are going to use the [MongoDB Prometheus exporter](https://github.com/dcu/mongodb_exporter).

## Environment setup <a href="#environment-setup" id="environment-setup"></a>

### Hosts and ports <a href="#hosts-and-ports" id="hosts-and-ports"></a>

You can use a single host for both MongoDB and YCSB but, in the following example, we replicate a common pattern in performance engineering by externalizing the load injection tool into a separate instance to avoid performance issues and measurement noise.

* [mongo.mycompany.com](http://mongo.mycompany.com/) for the MongoDB server instance (port 27017) and the MongoDB Prometheus exporter (port 9100)
* [ycsb.mycompany.com](http://ycsb.mycompany.com/) for YCSB and Prometheus (port 9090)

Notice: in the following, the assumption is to be working with Linux hosts.

## Prometheus and exporters <a href="#prometheus-and-exporters" id="prometheus-and-exporters"></a>

To correctly extract MongoDB metrics we can leverage a solution like Prometheus, paired with the MongoDB Prometheus exporter. To do so we would need to:

1. Install the MongoDB Prometheus exporter on mongo.mycompany.com
2. Install and configure Prometheus on ycsb.mycompany.com

### Install the MongoDB Prometheus exporter <a href="#install-the-mongodb-prometheus-exporter" id="install-the-mongodb-prometheus-exporter"></a>

You can check how to install the exporter [here](https://github.com/dcu/mongodb_exporter). On Ubuntu, you can use the system package manager:

```bash
apt-get install prometheus-mongodb-provider
```

By default, the exporter will expose MongoDB metrics on port 9100

### Install and configure Prometheus <a href="#install-and-configure-prometheus" id="install-and-configure-prometheus"></a>

You can check how to configure Prometheus [here](https://prometheus.io/docs/introduction/first_steps/); by default, it will run on port 9090.

The following YAML file`prometheus.yaml` is an example of the Prometheus configuration that you can use.

```yaml
global:
  scrape_interval:     15s  # Set the scrape interval to every 15 seconds. The default is every 1 minute.
  evaluation_interval: 15s  # Evaluate rules every 15 seconds. The default is every 1 minute.
  scrape_timeout: 15s

scrape_configs:
  # Mongo exporter
  - job_name: 'mongo_exporter'
    scrape_interval: 15s
    static_configs:
      - targets: ['mongo.mycompany.com:9001']
    relabel_configs:
      - source_labels: ["__address__"]
        regex: ".*"
        target_label: "instance"
        # this replacement should match the name of the akamas component of MongoDB
        replacement: "mongo"
```

## System <a href="#system" id="system"></a>

Since we are interested in tuning MongoDB by acting on its configuration parameters and by observing its throughput measured using YCSB, we need two components:

* A `mongo` component which represents the MongoDB instance all the configuration parameters and maps directly to [mongo.mycompany.com](http://mongo.mycompany.com/)
* A `ycsb` component which represents YCSB, in particular, it "houses" the metrics of the performance test, which can be used as parts of the goal of a study. This component maps directly to [ucsb.mycompany.com](http://ucsb.mycompany.com/)

Here’s the definition of our system (`system.yaml`):

```yaml
name: mongodb system
description: reference system
```

Here’s the definition of our `mongo` component (`mongo.yaml`):

```yaml
name: mongo
description: The MongoDB server
componentType: MongoDB-4 # MongoDB version 4
properties:
  hostname: mongo.mycompany.com
  prometheus:
    # we are telling akamas that this component should be monitored using Prometheus, and each data-point with a label instance=mongo should be mapped to this component
    instance: mongo
  sshPort: 22
  username: myusername
  key: ... RSA KEY ...
```

Here’s the definition of our `ycsb` component (`ycsb.yaml`):

```yaml
name: ycsb
description: The YCSB client
componentType: YCSB
properties:
  hostname: ycsb.mycompany.io
  instance: ycsb
  sshPort: 22
  username: myusername
  key: ... RSA KEY ...
```

We can create the system by running:

```bash
akamas create system system.yaml
```

We can then create the components by running:

```bash
akamas create component mongo.yaml "mongodb system"
akamas create component ycsb.yaml "mongodb system"
```

## Workflow <a href="#workflow" id="workflow"></a>

As described in the [MongoDB optimization pack](https://docs.akamas.io/akamas-docs/3.6/reference/optimization-packs/mongodb-pack) page, a workflow for optimizing MongoDB can be structured in three main steps:

1. Configure MongoDB with the configuration parameters decided by Akamas
2. Test the performance of the application
3. Prepare test results

Notice: here we have omitted the *Cleanup* step because it is not necessary for the context of this study.

### Configure MongoDB <a href="#configure-mongodb" id="configure-mongodb"></a>

We can define a workflow task that uses the FileConfigurator operator to interpolate Akamas parameters into a MongoDB configuration script:

```yaml
name: configure mongo
operator: FileConfigurator
arguments:
  sourcePath: /home/myusername/mongo/templates/mongo_launcher.sh.templ # MongoDB configuration script with placeholders for Akamas parameters
  targetPath: /home/myusername/mongo/launcher.sh # configuration script with interpolated Akamas parameters
  component: mongo # mongo should match the component of your system that represents your MongoDB instance
```

Here’s an example of a templated configuration script for MongoDB:

```bash
#!/bin/sh

cd "$(dirname "$0")" || exit

CACHESIZE=${mongo.mongodb_cache_size}
SYNCDELAY=${mongo.mongodb_syncdelay}
EVICTION_DIRTY_TRIGGER=${mongo.mongodb_eviction_dirty_trigger}
EVICTION_DIRTY_TARGET=${mongo.mongodb_eviction_dirty_target}
EVICTION_THREADS_MIN=${mongo.mongodb_eviction_threads_min}
EVICTION_THREADS_MAX=${mongo.mongodb_eviction_threads_max}
EVICTION_TRIGGER=${mongo.mongodb_eviction_trigger}
EVICTION_TARGET=${mongo.mongodb_eviction_target}
USE_NOATIME=${mongo.mongodb_datafs_use_noatime}

# Here we have to remount the disk mongodb uses for data, to take advantage of the USE_NOATIME parameter

sudo service mongod stop
sudo umount /mnt/mongodb
if [ "$USE_NOATIME" = true ]; then
        sudo mount /dev/nvme0n1 /mnt/mongodb -o noatime
else
        sudo mount /dev/nvme0n1 /mnt/mongodb
fi
sudo service mongod start

# flush logs
echo -n | sudo tee /mnt/mongodb/log/mongod.log
sudo service mongod restart

until grep -q "waiting for connections on port 27017" /mnt/mongodb/log/mongod.log
do
        echo "waiting MongoDB..."
        sleep 60
done

sleep 5
sudo service prometheus-mongodb-exporter restart
# set knobs
mongo --quiet --eval "db.adminCommand({setParameter:1, 'wiredTigerEngineRuntimeConfig': 'cache_size=${CACHESIZE}m, eviction=(threads_min=$EVICTION_THREADS_MIN,threads_max=$EVICTION_THREADS_MAX), eviction_dirty_trigger=$EVICTION_DIRTY_TRIGGER, eviction_dirty_target=$EVICTION_DIRTY_TARGET', eviction_trigger=$EVICTION_TRIGGER, eviction_target=$EVICTION_TARGET})"
mongo --quiet --eval "db = db.getSiblingDB('admin'); db.runCommand({ setParameter : 1, syncdelay: $SYNCDELAY})"

sleep 3Shell
```

We can add a workflow task that actually executes the MongoDB configuration script produced by the FileConfigurator:

```yaml
name: launch mongo
operator: Executor
arguments:
  command: bash /home/myusername/mongo/launcher.sh
  component: mongo  # we can take all the ssh connection parameters from the properties of the mongo component
```

In each task, we leveraged the reference to the "mongo" component to fetch from its properties all the authentication info to SSH into the right machine and let the FileConfigurator and Executor do their work.

### Test the performance of the application <a href="#test-the-performance-of-the-application" id="test-the-performance-of-the-application"></a>

We can define a workflow task that uses the Executor operator to launch the YCSB benchmark against MongoDB:

```yaml
name: launch ycsb
operator: Executor
arguments:
  command: bash /home/myusername/ycsb/launch_load.sh
  component: ycsb # we can take all the SSH connection parameters from the properties of the ycsb component
```

Here’s an example of a launch script for YCSB:

```bash
#!/bin/bash

MONGODB_SERVER_IP="mongo.mycompany.com"
RECORDCOUNT=30000000
RUN_THREADS=10
LOAD_THREADS=10
DURATION=1800 # 30 minutes
WORKLOAD="a"

cd "$(dirname "$0")" || exit

# here we use the db_records file to check if we have already loaded the db with data
# if not we run a load script
db_records=$(cat db_records)
if [ "$RECORDCOUNT" != "$db_records" ]; then
        bash scripts/create_db_mongo.sh ${MONGODB_SERVER_IP} "$RECORDCOUNT" "$LOAD_THREADS" "$WORKLOAD"
        echo "$RECORDCOUNT" > db_records
fi

cd /home/myuser/ycsb-0.15.0 || exit
# launch task in background
./bin/ycsb run mongodb-async -s -P workloads/workload"$WORKLOAD" -threads "$RUN_THREADS"  -p recordcount="$RECORDCOUNT" -p operationcount=0 -p maxexecutiontime="$DURATION" -p mongodb.url=mongodb://"$MONGODB_SERVER_IP":27017 &> /home/myuser/ycsb/outputRun.txt &
PID=$!

while kill -0 "$PID" >/dev/null 2>&1; do
        echo running

        if grep -q "java.net.ConnectException: Connection refused (Connection refused)" /home/myuser/ycsb/outputRun.txt; then
                echo "No connection, killing time!"
                echo -n > /home/myuser/ycsb/outputRun.txt
                ps -ef | grep -i com.yahoo.ycsb.Client | awk '{print $2}' | xargs -I{} kill -9 {}
                echo "Let's wait sometime... maybe Mongo is recovering data??"
                sleep 900
		            exit 1
        fi

        if grep -Fxq "Could not create a connection to the server" /home/myuser/ycsb/outputRun.txt; then
                echo "Unable to connect to server!"
                kill -9 ${PID}
                rm /home/myuser/ycsb/outputRun.txt
                rm /home/myuser/ycsb/db_records
        else
                sleep 10
        fi
done
```

### Prepare test results <a href="#prepare-test-results" id="prepare-test-results"></a>

We can define a workflow task that launches a script that parses the YCSB results into a CSV file (Akamas will process the CSV file and then extract performance test metrics):

```yaml
name: parse ycsb results
operator: Executor
arguments:
  command: python /home/myusername/ycsb/parser.py
  component: ycsb # we can take all the ssh connection parameters from the properties of the ycsb component
```

### Complete workflow <a href="#complete-workflow" id="complete-workflow"></a>

By putting together all the tasks defined above we come up with the following workflow definition (`workflow.yaml`):

```yaml
name: mongo workflow
tasks:
  - name: configure mongo
    operator: FileConfigurator
    arguments:
      sourcePath: /home/myusername/mongo/templates/mongo_launcher.sh.templ # MongoDB configuration script with placeholders for Akamas parameters
      targetPath: /home/myusername/mongo/launcher.sh # configuration script with interpolated Akamas parameters
      component: mongo # we can take all the ssh connection parameters from the properties of the mongo component

  - name: launch mongo
    operator: Executor
    arguments:
      command: bash /home/myusername/mongo/launcher.sh
      component: mongo # mongo should match the component of your system that represents your MongoDB instance

  - name: launch ycsb
    operator: Executor
    arguments:
      command: bash /home/myusername/ycsb/launch_load.sh
      component: ycsb # we can take all the ssh connection parameters from the properties of the ycsb component

  - name: parse ycsb results
    operator: Executor
    arguments:
      command: python /home/myuser/ycsb/parser.py
      component: ycsb # we can take all the ssh connection parameters from the properties of the ycsb component
```

We can create the workflow by running:

```bash
akamas create workflow workflow.yaml
```

## Telemetry <a href="#telemetry" id="telemetry"></a>

### Prometheus <a href="#prometheus" id="prometheus"></a>

Since we are employing Prometheus to extract MongoDB metrics, we can leverage the [Prometheus provider](https://docs.akamas.io/akamas-docs/3.6/integrating/integrating-telemetry-providers/prometheus-provider) to start ingesting data points into Akamas. To use the Prometheus provider we need to define a telemetry-instance (`prom.yaml`):

```yaml
provider: Prometheus # we are using Prometheus
config:
  address: ycsb.mycompany.com # address of Prometheus
  port: 9090

metrics:
  - metric: mongodb_connections_current
    datasourceMetric: mongodb_connections{instance="$INSTANCE$"}
    labels:
      - state
  - metric: mongodb_heap_used
    datasourceMetric: mongodb_extra_info_heap_usage_bytes{instance="$INSTANCE$"}
  - metric: mongodb_page_faults_total
    datasourceMetric: rate(mongodb_extra_info_page_faults_total{instance="$INSTANCE$"}[$DURATION$])
  - metric: mongodb_global_lock_current_queue
    datasourceMetric: mongodb_global_lock_current_queue{instance="$INSTANCE$"}
    labels:
      - type
  - metric: mongodb_mem_used
    datasourceMetric: mongodb_memory{instance="$INSTANCE$"}
    labels:
      - type
  - metric: mongodb_documents_inserted
    datasourceMetric: rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="inserted"}[$DURATION$])
  - metric: mongodb_documents_updated
    datasourceMetric: rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="updated"}[$DURATION$])
  - metric: mongodb_documents_deleted
    datasourceMetric: rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="deleted"}[$DURATION$])
  - metric: mongodb_documents_returned
    datasourceMetric: rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="returned"}[$DURATION$])
```

Notice: the fact that the instance definition contains the specification of Prometheus queries to map to Akamas metrics is temporary. In the next release, these queries will be embedded in Akamas.

By default, `$DURATION$` will be replaced with 30s. You can override it to your needs by setting a `duration` property under `prometheus` within your `mongo` component.

We can now create the telemetry instance and attach it to our system by running:

```bash
akamas create telemetry-instance prom.yaml "mongodb system"
```

### CSV <a href="#csv" id="csv"></a>

Beyond MongoDB metrics, it is important to ingest into Akamas metrics related to the performance tests run with YCSB, in particular the throughput of operations. To achieve this we can leverage the [CSV Provider](https://docs.akamas.io/akamas-docs/3.6/integrating/integrating-telemetry-providers/csv-provider) which parses a CSV file to extract relevant metrics. The CSV file we are going to parse with the help of the provider is the one produced in the last task of the workflow of the study.

To start using the provider, we need to define a telemetry instance (`csv.yaml`):

```yaml
provider: CSV
config:
  protocol: scp
  address: ycsb.mycompany.com
  username: myuser
  authType: key
  auth: ... RSA KEY ...
  remoteFilePattern: /home/ubuntu/ycsb/output.csv
  csvFormat: horizontal
  componentColumn: Component
  timestampColumn: timestamp
  timestampFormat: yyyy-MM-dd HH:mm:ss

metrics:
 # here we put which metric found in the csv provider should be mapped to which akamas metrics
 # we are only interested in the throughput, but you can add other metrics if you want
  - metric: throughput
    datasourceMetric: throughput
 ....
```

We can create the telemetry instance and attach it to our system by running:

```bash
akamas create telemetry-instance csv.yaml "mongodb system"
```

## Study <a href="#study" id="study"></a>

Our goal for optimizing MongoDB is to maximize its throughput, measured using a performance test executed with YCSB.

### Goal <a href="#goal" id="goal"></a>

Here’s the definition of the goal of our study, to maximize the throughput:

```yaml
goal:
  objective: maximize
  function:
    formula: ycsb.throughput
```

### Windowing <a href="#windowing" id="windowing"></a>

It is important that the throughput of our MongoDB instance should be considered valid only when it is stable, for this reason, we can use the *stability* windowing policy. This policy identifies a period of time with at least 100 samples with a standard deviation lower than 200 when the application throughput is maximum.

```yaml
windowing:
  type: stability
  stability:
    # measure the goal function where the throughput has stdDev <= 200 for 100 consecutive data points
    metric: throughput
    labels:
      componentName: ycsb
    width: 100
    maxStdDev: 200
    # take only the temporal window when the throughput is maximum
    when:
      metric: throughput
      is: max
      labels:
        componentName: ycsb
```

### Parameters to optimize <a href="#parameters-to-optimize" id="parameters-to-optimize"></a>

We are going to optimize every MongoDB parameter:

```yaml
parametersSelection:
  - name: mongo.mongodb_syncdelay
  - name: mongo.mongodb_eviction_dirty_trigger
  - name: mongo.mongodb_eviction_dirty_target
  - name: mongo.mongodb_eviction_target
  - name: mongo.mongodb_eviction_trigger
  - name: mongo.mongodb_eviction_threads_min
  - name: mongo.mongodb_eviction_threads_max
  - name: mongo.mongodb_cache_size
    # here we have changed the domain of the cache size since we suppose our mongo.mycompany.com host has 32gb of RAM, you should adapt to your own instance
    domain: [500, 32000]
```

### Steps <a href="#steps" id="steps"></a>

We are going to add to our study two steps:

* A baseline step, in which we set a cache size of 1GB and use the default values for all the other MongoDB parameters
* An optimize step, in which we perform 100 experiments to generate the best configuration for MongoDB

Here’s what these steps look like:

```yaml
steps:
- name: baseline
  type: baseline
  values:
    mongo.mongodb_cache_size: 1024
  renderParameters:
  # Use also all the other MongoDB parameters at their default value
  - mongo.*
- name: optimize mongo
  type: optimize
  numberOfExperiments: 100
```

### Complete study <a href="#complete-study" id="complete-study"></a>

Here’s the study definition (`study.yaml`) for optimizing MongoDB:

```yaml
name: study to tune MongoDB
description: study to tune MongoDB with YCSB perf test
system: mongodb system
workflow: mongo workflow
# Goal
goal:
  objective: maximize
  function:
    formula: ycsb.throughput
# Windowing
windowing:
  type: stability
  stability:
    metric: throughput
    labels:
      componentName: ycsb
    width: 100
    maxStdDev: 200
    when:
      metric: throughput
      is: max
      labels:
        componentName: ycsb
# parameters selection
parametersSelection:
  - name: mongo.mongodb_syncdelay
  - name: mongo.mongodb_eviction_dirty_trigger
  - name: mongo.mongodb_eviction_dirty_target
  - name: mongo.mongodb_eviction_target
  - name: mongo.mongodb_eviction_trigger
  - name: mongo.mongodb_eviction_threads_min
  - name: mongo.mongodb_eviction_threads_max
  - name: mongo.mongodb_cache_size
    # here we have changed the domain of the cache size since we suppose our mongo.mycompany.com host has 32gb of RAM
    domain: [500, 32000]
  - name: mongo.mongodb_datafs_use_noatime
parameterConstraints:
- name: c1
  formula: mongo.mongodb_eviction_threads_min <= mongo.mongodb_eviction_threads_max
- name: c2
  formula: mongodb_eviction_dirty_target <= mongodb_eviction_target
- name: c3
  formula: mongodb_eviction_dirty_trigger <= mongodb_eviction_trigger
# steps
steps:
- name: baseline
  type: baseline
  values:
    mongo.mongodb_cache_size: 1024
  renderParameters:
  # use also all the other MongoDB parameters at their default value
  - mongo.*
- name: optimize mongo
  type: optimize
  numberOfExperiments: 100
```

You can create the study by running:

```bash
akamas create study study.yaml
```

You can then start it by running:

```bash
akamas start study "study to tune MongoDB"
```