Ctrlk
Akamas Insights
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Migration Procedures

From 3.6 to 4.0

The upgrade from Akamas 3.6 to 4.0 requires:

  • Upgrading the database to a new major version

  • Upgrading the CSV telemetry provider configuration

Those upgrades both involve some manual steps.

Pre-upgrade steps

Run these steps before updating the docker-compose or helm release

Database upgrade

Here's the guide for the database upgrade

DB upgrade - Docker version

Make sure you are logged in to the host running the Akamas instance before running the following commands.

Stop the services

Stop the Akamas services except the databases:

cd akamas
docker compose down
docker compose up -d database airflow-db kong-database

Export AWS credentials and login to AWS docker repo

export AWS_ACCESS_KEY_ID=AAAAAAAAAAA ## use your AWS access key ID
export AWS_SECRET_ACCESS_KEY=bbbbbbbbbbbbbbbbbb ## use your AWS secret access key id
aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 485790562880.dkr.ecr.us-east-2.amazonaws.com

Extract the database passwords

docker compose config | python3 -c 'import sys, yaml; s=yaml.safe_load(sys.stdin)["services"]; [print(k+"_DB_PASSWORD="+s[v]["environment"]["POSTGRES_PASSWORD"]) or print(k+"_DB_USER="+s[v]["environment"].get("POSTGRES_USER", "postgres")) for k, v in {"AKAMAS": "database", "AIRFLOW": "airflow-db"}.items()]' > pass.env

Dump the database

export PG_IMAGE='485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/master-db:1.13.0'

mkdir -p backup
source pass.env

echo " *** Dumping akamas services"
docker run --rm \
    --network akamas \
    -u "$(id -u):$(id -g)" \
    -v $(pwd)/backup:/backup \
    -e PGHOST=database \
    -e PGUSER=${AKAMAS_DB_USER} \
    -e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
    "$PG_IMAGE" \
    pg_dumpall --clean --if-exists --exclude-database=postgres -f /backup/akamas_dump.sql

echo " *** Dumping airflow"
docker run --rm \
    --network akamas \
    -u "$(id -u):$(id -g)" \
    -v $(pwd)/backup:/backup \
    -e PGHOST=airflow-db \
    -e PGUSER=${AIRFLOW_DB_USER} \
    -e PGPASSWORD=${AIRFLOW_DB_PASSWORD} \
    "$PG_IMAGE" \
    pg_dump --clean --if-exists -d airflow -f /backup/airflow_dump.sql

Clean up old database

echo " *** Removing old containers"
docker rm -fv database airflow-db kong-db
echo " *** Removing old volumes"
docker volume rm -f akamas_airflow-db-data akamas_database-data akamas_kong-data

Start the updated database

Update docker-compose.yml to use the new database image. Back up the old file and replace the image version using sed:

OLD_VERSION=$(python3 -c 'import sys, yaml; print(yaml.safe_load(sys.stdin)["services"]["database"]["image"].split(":")[1])' < docker-compose.yml)
echo "Current database image version: $OLD_VERSION"
cp docker-compose.yml docker-compose.yml.bak$(date +%s)
sed -i "s|master-db:${OLD_VERSION}|master-db:${PG_IMAGE##*:}|g" docker-compose.yml
diff -u docker-compose.yml.bak* docker-compose.yml

Verify the update and start the new database:

docker compose pull
docker compose up -d database
docker logs -f database

Then wait for the message database system is ready to accept connections to appear then press CTRL+C. NOTE: if the expected message doesn't pop up after a couple of minutes and you see the message database system was shut down, instead, some random bug occurred. In this case press CTRL+C then stop the database with docker compose down. Then relaunch the last 2 commands above (docker compose up -d database and docker logs -f database and wait again for the message database system is ready to accept connections to appear then press CTRL+C.

Restore the data

docker run --rm \
    --network akamas \
    -v $(pwd)/backup:/backup \
    -e PGHOST=database \
    -e PGUSER=${AKAMAS_DB_USER} \
    -e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
    "$PG_IMAGE" \
    psql -X -f /backup/akamas_dump.sql

docker run --rm \
    --network akamas \
    -v $(pwd)/backup:/backup \
    -e PGHOST=database \
    -e PGUSER=${AKAMAS_DB_USER} \
    -e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
    "$PG_IMAGE" \
    psql -X -f /backup/airflow_dump.sql

Restart Akamas

Replace docker-compose.yml with the latest Akamas 4.0 version, as described in Install the Akamas Server, and restart the remaining services:

docker compose pull
docker compose up -d
DB upgrade - Kubernetes version

Make sure you are using the correct namespace for the Akamas installation. To switch namespace, run kubectl config set-context --current --namespace <akamas>, replacing <akamas> with your namespace name.

Check preStop hooks

Ensure the current chart already supports preStop hooks for Postgres. Run:

kubectl get statefulset/database -o jsonpath='{.spec.template.spec.containers[].lifecycle}'

If the output contains preStop, like in the example below, your chart already supports graceful shutdown and no patching is needed.

{"preStop":{"exec":{"command":["/bin/sh","-c","PGUSER=postgres pg_ctl stop -m fast"]}}}

If the output is empty or missing the preStop entry, patch the statefulset with:

kubectl patch statefulset database -p '{"spec":{"template":{"spec":{"terminationGracePeriodSeconds":60,"containers":[{"name":"postgresql","lifecycle":{"preStop":{"exec":{"command":["/bin/sh","-c","PGUSER=postgres pg_ctl stop -m fast"]}}}}]}}}}'
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'

Stop the services

Stop the Akamas services:

kubectl scale deployment --all --replicas 0
kubectl scale statefulsets -l 'app.kubernetes.io/name notin (postgresql)' --replicas 0

Charts 1.6.4 and later stop all pods except the database automatically. For older charts, scale the database back up manually:

kubectl scale statefulset database --replicas 1
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'

Dump the database

Dump the database into a dedicated volume using the job defined in the attached pg16_dump.yaml file. It creates a 10Gi PersistentVolumeClaim to store the backup. If your database needs larger storage, update the PVC definition in the YAML file.

kubectl apply -f pg16_dump.yaml
kubectl wait job/pg-dump --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-dump -f --all-containers=true

Once the dump completes, scale down the database:

kubectl scale statefulset database --replicas 0

Clean up the datadir

Verify the database is scaled down before proceeding:

kubectl wait statefulset/database --for=jsonpath='{.status.replicas}=0' --timeout=5s && echo Ok || echo 'ERROR: database is still running'

Then clean up the datadir using the attached pg16_cleanup.yaml file:

kubectl apply -f pg16_cleanup.yaml
kubectl wait job/pg-cleanup --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-cleanup -f --all-containers=true

You can inspect the content of the backup volume using the pg-debug pod

kubectl apply -f pg16_debug.yaml

Upgrade the database

Once the cleanup completes, patch the statefulset with the new image:

export PG16_IMAGE='16.6.0-debian-12-r2'
kubectl patch statefulset database -p '{"spec":{"template":{"spec":{"containers":[{"name":"postgresql","image":"485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/bitnami/postgresql:'${PG16_IMAGE}'"}]}}}}'
kubectl scale statefulset/database --replicas=1
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'
kubectl logs statefulset/database --all-containers=true

Restore the database

Restore the database using the attached pg16_restore.yaml file:

kubectl apply -f pg16_restore.yaml
kubectl wait job/pg-restore --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-restore -f --all-containers=true

Restart the Akamas services

To complete the upgrade, restart the Akamas services:

kubectl scale deployment --all --replicas 1
kubectl scale statefulsets --all --replicas 1

Upgrade the Akamas release

Once verified that the new database is working correctly, upgrade Akamas using the chart associated with the latest 4.0 release, as described in Install Akamas:

helm upgrade --install \
    --create-namespace --namespace akamas \
    --repo http://helm.akamas.io/charts \
    --version '<1.8.1>' \
    -f akamas.yaml \
    akamas akamas

Final cleanup

Once you have verified that the instance was upgraded successfully and works correctly, you can delete the backup volume.

kubectl delete pvc pg-dump

Post-upgrade steps

Run these steps after updating the docker-compose or helm release

CSV telemetry provider configuration upgrade

This step upgrades the configuration (adding new config parameters: hostname, username, and password) of the CSV telemetry provider. If you have never used this provider in your studies (or systems) and have no CSV telemetry instance defined in any system, we recommend the faster way:

Recommended procedure when no CSV telemetry instances are present

An important prerequisite for this procedure is to start the services beforehand (see the final part of #from-3.6-to-3.7), since you need the telemetry service running.

Log in with Akamas CLI to your environment with:

Then issue the following command:

VERY IMPORTANT: If a message appears telling It will also delete the following telemetry-instances:

like in the following example:

Just STOP this procedure by pressing ENTER key and refer to alternate Docker or kubernetes procedures below. If there are no telemetry instances to delete, you may proceed with the deletion of the telemetry provider (press 'y' then ENTER), then you can recreate the new telemetry provider by creating a new file named csv-file-provider-official.yaml

with these exact contents:

then issuing the following command:

These, instead, are the alternate procedures for the Docker and kubernetes case, for use when you already have CSV telemetry instances defined in your systems. They have the advantage that you don't need Akamas services to be up and running (you only need a database, already running)

Alternate procedure for Docker

Log in to the machine where akamas is installed via SSH. Then issue the command:

Retrieve the telemetry DB password with this command:

Connect to the database container shell with:

Connect to postgres with:

Then type the DB password (retrieved above) when asked.

Then issue SQL command:

Alternate procedure for Kubernetes

Properly configure your kubectl configuration in order to connect to the correct cluster and namespace holding your Akamas installation.

Retrieve telemetry DB password with:

Connect to database pod shell with:

Connect to postgres with:

then type in DB password (retrieved above) when asked.

Then issue SQL command:

After using one of the two alternate procedures, you MUST upgrade the CSV Telemetry Provider to the latest version 3.3.0 (the 3.2.0 version that was shipped with 3.6.x will not work properly when using new features). For this, you need to start up services beforehand (refer to #from-3.6-to-3.7). Then you can upgrade the CSV telemetry provider by: Creating a new file named csv-file-provider-official.yaml with these contents:

and run:

PreviousKubernetesNextMonitor Akamas status

Last updated

Was this helpful?