When optimizing a MongoDB instance, typically the goal is one of the following:
Throughput optimization - increasing the capacity of a MongoDB deployment to serve clients
Cost optimization - decreasing the size of a MongoDB deployment while guaranteeing the same service level
To reach such goals, it is recommended to tune the parameters that manage the cache, which is of the elements that impact performances the most, in particular those parameters that control the lifecycle and the size of the MongoDB’s cache.
Even though it is possible to evaluate performance improvements of MongoDB by looking at the business application that uses it as its database, looking at the end-to-end throughput or response time, or using a performance test like , the optimization pack provides internal MongoDB metrics that can shed a light too on how MongoDB is performing, in particular in terms of throughput, for example:
The number of documents inserted in the database per second
The number of active connections
Please refer to the for the list of component types, parameters, metrics, and constraints.
Akamas offers many operators that you can use to apply freshly tuned configuration parameters to your MongoDB deployment. In particular, we suggest using the to create a configuration script file and the ExecutorOperator to execute it and thus apply the parameters.
FileConfigurator and Executor operator
You can leverage the FileConfigurator by creating a template file on a remote host that contains some scripts to configure MongoDB with placeholders that will be replaced with the values of parameters tuned by Akamas.
Here’s an example of the aforementioned template file:
You can leverage the FileConfigurator by creating a template file on a remote host that contains some scripts to configure MongoDB with placeholders that will be replaced with the values of parameters tuned by Akamas. Once the FileConfigurator has replaced all the tokens, you can use the Executor operator to actually execute the script to configure MongoDB.
A typical workflow to optimize a MongoDB deployment can be structured in three parts:
Configure MongoDB
Use the to specify an input and an output template file. The input template file is used to specify how to interpolate MongoDB parameters into a script, and the output file contains the actual configuration.
Use the operator to reconfigure MongoDB exploiting the output file produced in the previous step. You may need to restart MongoDB depending on the configuration parameters you want to optimize.
Finally, when running performance experiments on a database, is common practice to execute some cleanup tasks at the end of the test to restore the database initial condition and avoid impacting subsequent tests.
Here’s an example of a typical workflow for a MongoDB deployment, which uses the YCSB benchmark to run performance tests:
Akamas offers many telemetry providers to extract MongoDB metrics; one of them is the which we can use to query MongoDB metrics collected by a Prometheus instance via the .
Here’s an example of a telemetry providers instance that uses Prometheus to extract all the MongoDB metrics defined in this optimization pack:
See the page for an example of a study leveraging the MongoDB pack.
Test the performance of the application
Use available operators to execute a performance test against the application
Prepare test results (optional)
If Akamas does not already automatically import performance test metrics, then you can use available operators to extract test results and make them available to Akamas (for example, you can use an Executor to launch a script that produces a CSV of the test results that Akamas can consume using the CSV provider)
Cleanup
Use available operators to bring back MongoDB into a clean state to avoid impacting subsequent tests
#!/bin/sh
cd "$(dirname "$0")" || exit
CACHESIZE=${mongo.mongodb_cache_size}
SYNCDELAY=${mongo.mongodb_syncdelay}
EVICTION_DIRTY_TRIGGER=${mongo.mongodb_eviction_dirty_trigger}
EVICTION_DIRTY_TARGET=${mongo.mongodb_eviction_dirty_target}
EVICTION_THREADS_MIN=${mongo.mongodb_eviction_threads_min}
EVICTION_THREADS_MAX=${mongo.mongodb_eviction_threads_max}
EVICTION_TRIGGER=${mongo.mongodb_eviction_trigger}
EVICTION_TARGET=${mongo.mongodb_eviction_target}
USE_NOATIME=${mongo.mongodb_datafs_use_noatime}
# Here we have to remount the disk mongodb uses for data, to take advantage of the USE_NOATIME parameter
sudo service mongod stop
sudo umount /mnt/mongodb
if [ "$USE_NOATIME" = true ]; then
sudo mount /dev/nvme0n1 /mnt/mongodb -o noatime
else
sudo mount /dev/nvme0n1 /mnt/mongodb
fi
sudo service mongod start
# flush logs
echo -n | sudo tee /mnt/mongodb/log/mongod.log
sudo service mongod restart
until grep -q "waiting for connections on port 27017" /mnt/mongodb/log/mongod.log
do
echo "waiting MongoDB..."
sleep 60
done
sleep 5
sudo service prometheus-mongodb-exporter restart
# set knobs
mongo --quiet --eval "db.adminCommand({setParameter:1, 'wiredTigerEngineRuntimeConfig': 'cache_size=${CACHESIZE}m, eviction=(threads_min=$EVICTION_THREADS_MIN,threads_max=$EVICTION_THREADS_MAX), eviction_dirty_trigger=$EVICTION_DIRTY_TRIGGER, eviction_dirty_target=$EVICTION_DIRTY_TARGET', eviction_trigger=$EVICTION_TRIGGER, eviction_target=$EVICTION_TARGET})"
mongo --quiet --eval "db = db.getSiblingDB('admin'); db.runCommand({ setParameter : 1, syncdelay: $SYNCDELAY})"
sleep 3name: "ycsb_mongo_workflow"
tasks:
- name: "configure mongo"
operator: "FileConfigurator"
arguments:
sourcePath: "/home/ubuntu/mongo/templates/mongo_launcher.sh.templ"
targetPath: "/home/ubuntu/mongo/launcher.sh"
component: "mongo"
- name: "launch mongo"
operator: "Executor"
arguments:
command: "bash /home/ubuntu/mongo/launcher.sh 2>&1 | tee -a /tmp/log"
component: "mongo"
- name: "launch ycsb"
operator: "Executor"
arguments:
command: "bash /home/ubuntu/ycsb/launch_load.sh 2>&1 | tee -a /tmp/log"
component: "mongo_ycsb"
- name: "parse ycsb"
operator: "Executor"
arguments:
command: "python /home/ubuntu/ycsb/parser.py"
component: "mongo_ycsb"
- name: "clean mongo"
operator: "Executor"
arguments:
command: "bash /home/ubuntu/clean_mongodb.sh"
component: "mongo"provider: "Prometheus"
config:
address: "prometheus.mycompany.com"
port: 9090
metrics:
- metric: "mongodb_connections_current"
datasourceMetric: "mongodb_connections{instance="$INSTANCE$"}"
labels: ["state"]
- metric: "mongodb_heap_used"
datasourceMetric: "mongodb_extra_info_heap_usage_bytes{instance="$INSTANCE$"}"
- metric: "mongodb_page_faults_total"
datasourceMetric: "rate(mongodb_extra_info_page_faults_total{instance="$INSTANCE$"}[$DURATION$])"
- metric: "mongodb_global_lock_current_queue"
datasourceMetric: "mongodb_global_lock_current_queue{instance="$INSTANCE$"}"
labels: ["type"]
- metric: "mongodb_mem_used"
datasourceMetric: "mongodb_memory{instance="$INSTANCE$"}"
labels: ["type"]
- metric: "mongodb_documents_inserted"
datasourceMetric: "rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="inserted"}[$DURATION$])"
- metric: "mongodb_documents_updated"
datasourceMetric: "rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="updated"}[$DURATION$])"
- metric: "mongodb_documents_deleted"
datasourceMetric: "rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="deleted"}[$DURATION$])"
- metric: "mongodb_documents_returned"
datasourceMetric: "rate(mongodb_metrics_document_total{instance="$INSTANCE$", state="returned"}[$DURATION$])"