Optimizing MongoDB

When optimizing a MongoDB instance, typically the goal is one of the following:

• Throughput optimization - increasing the capacity of a MongoDB deployment to serve clients

• Cost optimization - decreasing the size of a MongoDB deployment while guaranteeing the same service level

To reach such goals, it is recommended to tune the parameters that manage the cache, which is of the elements that impact performances the most, in particular those parameters that control the lifecycle and the size of the MongoDB’s cache.

Even though it is possible to evaluate performance improvements of MongoDB by looking at the business application that uses it as its database, looking at the end-to-end throughput or response time, or using a performance test like YCSB, the optimization pack provides internal MongoDB metrics that can shed a light too on how MongoDB is performing, in particular in terms of throughput, for example:

• The number of documents inserted in the database per second

• The number of active connections

Please refer to the MongoDB optimization pack for the list of component types, parameters, metrics, and constraints.

Workflows

Applying parameters

Akamas offers many operators that you can use to apply freshly tuned configuration parameters to your MongoDB deployment. In particular, we suggest using the FileConfigurator operator to create a configuration script file and the ExecutorOperator to execute it and thus apply the parameters.

FileConfigurator and Executor operator

You can leverage the FileConfigurator by creating a template file on a remote host that contains some scripts to configure MongoDB with placeholders that will be replaced with the values of parameters tuned by Akamas.

Here’s an example of the aforementioned template file:

#!/bin/sh

cd "$(dirname "$0")" || exit

CACHESIZE=${mongo.mongodb_cache_size}
SYNCDELAY=${mongo.mongodb_syncdelay}
EVICTION_DIRTY_TRIGGER=${mongo.mongodb_eviction_dirty_trigger}
EVICTION_DIRTY_TARGET=${mongo.mongodb_eviction_dirty_target}
EVICTION_THREADS_MIN=${mongo.mongodb_eviction_threads_min}
EVICTION_THREADS_MAX=${mongo.mongodb_eviction_threads_max}
EVICTION_TRIGGER=${mongo.mongodb_eviction_trigger}
EVICTION_TARGET=${mongo.mongodb_eviction_target}
USE_NOATIME=${mongo.mongodb_datafs_use_noatime}

# Here we have to remount the disk mongodb uses for data, to take advantage of the USE_NOATIME parameter

sudo service mongod stop
sudo umount /mnt/mongodb
if [ "$USE_NOATIME" = true ]; then
        sudo mount /dev/nvme0n1 /mnt/mongodb -o noatime
else
        sudo mount /dev/nvme0n1 /mnt/mongodb
fi
sudo service mongod start

# flush logs
echo -n | sudo tee /mnt/mongodb/log/mongod.log
sudo service mongod restart

until grep -q "waiting for connections on port 27017" /mnt/mongodb/log/mongod.log
do
        echo "waiting MongoDB..."
        sleep 60
done

sleep 5
sudo service prometheus-mongodb-exporter restart
# set knobs
mongo --quiet --eval "db.adminCommand({setParameter:1, 'wiredTigerEngineRuntimeConfig': 'cache_size=${CACHESIZE}m, eviction=(threads_min=$EVICTION_THREADS_MIN,threads_max=$EVICTION_THREADS_MAX), eviction_dirty_trigger=$EVICTION_DIRTY_TRIGGER, eviction_dirty_target=$EVICTION_DIRTY_TARGET', eviction_trigger=$EVICTION_TRIGGER, eviction_target=$EVICTION_TARGET})"
mongo --quiet --eval "db = db.getSiblingDB('admin'); db.runCommand({ setParameter : 1, syncdelay: $SYNCDELAY})"

sleep 3

You can leverage the FileConfigurator by creating a template file on a remote host that contains some scripts to configure MongoDB with placeholders that will be replaced with the values of parameters tuned by Akamas. Once the FileConfigurator has replaced all the tokens, you can use the Executor operator to actually execute the script to configure MongoDB.

A typical workflow

A typical workflow to optimize a MongoDB deployment can be structured in three parts:

1. Configure MongoDB

   1. Use the FileConfigurator to specify an input and an output template file. The input template file is used to specify how to interpolate MongoDB parameters into a script, and the output file contains the actual configuration.
   2. Use the Executor operator to reconfigure MongoDB exploiting the output file produced in the previous step. You may need to restart MongoDB depending on the configuration parameters you want to optimize.
   3. Either use the Sleep operator or the Executor operator to verify that the application is up and running and has finished any initialization logic (this step may not be necessary)
2. Test the performance of the application

   1. Use available operators to execute a performance test against the application
3. Prepare test results (optional)

   1. If Akamas does not already automatically import performance test metrics, then you can use available operators to extract test results and make them available to Akamas (for example, you can use an Executor to launch a script that produces a CSV of the test results that Akamas can consume using the CSV provider)
4. Cleanup

   1. Use available operators to bring back MongoDB into a clean state to avoid impacting subsequent tests

Finally, when running performance experiments on a database, is common practice to execute some cleanup tasks at the end of the test to restore the database initial condition and avoid impacting subsequent tests.

Here’s an example of a typical workflow for a MongoDB deployment, which uses the YCSB benchmark to run performance tests:

name: "ycsb_mongo_workflow"
tasks:
  - name: "configure mongo"
    operator: "FileConfigurator"
    arguments:
      sourcePath: "/home/ubuntu/mongo/templates/mongo_launcher.sh.templ"
      targetPath: "/home/ubuntu/mongo/launcher.sh"
      component: "mongo"

  - name: "launch mongo"
    operator: "Executor"
    arguments:
      command: "bash /home/ubuntu/mongo/launcher.sh 2>&1 | tee -a /tmp/log"
      component: "mongo"

  - name: "launch ycsb"
    operator: "Executor"
    arguments:
      command: "bash /home/ubuntu/ycsb/launch_load.sh 2>&1 | tee -a /tmp/log"
      component: "mongo_ycsb"

  - name: "parse ycsb"
    operator: "Executor"
    arguments:
      command: "python /home/ubuntu/ycsb/parser.py"
      component: "mongo_ycsb"
  - name: "clean mongo"
    operator: "Executor"
    arguments:
      command: "bash /home/ubuntu/clean_mongodb.sh"
      component: "mongo"

Telemetry providers

Akamas offers many telemetry providers to extract MongoDB metrics; one of them is the Prometheus provider which we can use to query MongoDB metrics collected by a Prometheus instance via the MongoDB Prometheus exporter.

Here’s an example of a telemetry providers instance that uses Prometheus to extract all the MongoDB metrics defined in this optimization pack:

provider: "Prometheus"
config:
  address: "prometheus.mycompany.com"
  port: 9090

metrics:
  - metric: "mongodb_connections_current"
    datasourceMetric: "mongodb_connections{instance="$INSTANCE$"}"
    labels: ["state"]
  - metric: "mongodb_heap_used"
    datasourceMetric: "mongodb_extra_info_heap_usage_bytes{instance="$INSTANCE$"}"
  - metric: "mongodb_page_faults_total"
    datasourceMetric: "rate(mongodb_extra_info_page_faults_total{instance="$INSTANCE$"}[$DURATION$])"
  - metric: "mongodb_global_lock_current_queue"
    datasourceMetric: "mongodb_global_lock_current_queue{instance="$INSTANCE$"}"
    labels: ["type"]
  - metric: "mongodb_mem_used"
    datasourceMetric: "mongodb_memory{instance="$INSTANCE$"}"
    labels: ["type"]
  - metric: "mongodb_documents_inserted"
    datasourceMetric: "rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="inserted"}[$DURATION$])"
  - metric: "mongodb_documents_updated"
    datasourceMetric: "rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="updated"}[$DURATION$])"
  - metric: "mongodb_documents_deleted"
    datasourceMetric: "rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="deleted"}[$DURATION$])"
  - metric: "mongodb_documents_returned"
    datasourceMetric: "rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="returned"}[$DURATION$])"

Examples

See the page Optimizing a MongoDB server instance for an example of a study leveraging the MongoDB pack.