Migration procedures

From 3.6 to 3.7

The upgrade from Akamas 3.6 to 3.7 requires upgrading the database to a new major version. This upgrade involves some manual steps.

Docker version

Make sure you are logged in to the host running the Akamas instance before running the following commands.

Stop the services

Stop the Akamas services except the databases:

docker compose down
docker compose up -d database airflow-db kong-database

Extract the database passwords

docker compose config | python3 -c 'import sys, yaml; s=yaml.safe_load(sys.stdin)["services"]; [print(k+"_DB_PASSWORD="+s[v]["environment"]["POSTGRES_PASSWORD"]) or print(k+"_DB_USER="+s[v]["environment"].get("POSTGRES_USER", "postgres")) for k, v in {"AKAMAS": "database", "AIRFLOW": "airflow-db"}.items()]' > pass.env

Dump the database

export PG_IMAGE='485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/master-db:1.13.0'

mkdir -p backup
source pass.env

echo " *** Dumping akamas services"
docker run --rm \
    --network akamas \
    -u "$(id -u):$(id -g)" \
    -v $(pwd)/backup:/backup \
    -e PGHOST=database \
    -e PGUSER=${AKAMAS_DB_USER} \
    -e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
    "$PG_IMAGE" \
    pg_dumpall --clean --if-exists --exclude-database=postgres -f /backup/akamas_dump.sql

echo " *** Dumping airflow"
docker run --rm \
    --network akamas \
    -u "$(id -u):$(id -g)" \
    -v $(pwd)/backup:/backup \
    -e PGHOST=airflow-db \
    -e PGUSER=${AIRFLOW_DB_USER} \
    -e PGPASSWORD=${AIRFLOW_DB_PASSWORD} \
    "$PG_IMAGE" \
    pg_dump --clean --if-exists -d airflow -f /backup/airflow_dump.sql

Clean up old database

echo " *** Removing old containers"
docker rm -fv database airflow-db kong-db
echo " *** Removing old volumes"
docker volume rm -f akamas_airflow-db-data akamas_database-data akamas_kong-data

Start the updated database

Update docker-compose.yml to use the new database image. Back up the old file and replace the image version using sed:

OLD_VERSION=$(python3 -c 'import sys, yaml; print(yaml.safe_load(sys.stdin)["services"]["database"]["image"].split(":")[1])' < docker-compose.yml)
echo "Current database image version: $OLD_VERSION"
cp docker-compose.yml docker-compose.yml.bak$(date +%s)
sed -i "s|master-db:${OLD_VERSION}|master-db:${PG_IMAGE##*:}|g" docker-compose.yml
diff -u docker-compose.yml.bak* docker-compose.yml

Verify the update and start the new database:

docker compose pull
docker compose up -d database
docker logs -f database

Restore the data

docker run --rm \
    --network akamas \
    -v $(pwd)/backup:/backup \
    -e PGHOST=database \
    -e PGUSER=${AKAMAS_DB_USER} \
    -e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
    "$PG_IMAGE" \
    psql -X -f /backup/akamas_dump.sql

docker run --rm \
    --network akamas \
    -v $(pwd)/backup:/backup \
    -e PGHOST=database \
    -e PGUSER=${AKAMAS_DB_USER} \
    -e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
    "$PG_IMAGE" \
    psql -X -f /backup/airflow_dump.sql

Restart Akamas

Replace docker-compose.yml with the latest Akamas 3.7.x version, as described in Install the Akamas Server, and restart the remaining services:

docker compose pull
docker compose up -d

Kubernetes version

Make sure you are using the correct namespace for the Akamas installation. To switch namespace, run kubectl config set-context --current --namespace <akamas>, replacing <akamas> with your namespace name.

Check preStop hooks

Ensure the current chart already supports preStop hooks for Postgres. Run:

kubectl get statefulset/database -o jsonpath='{.spec.template.spec.containers[].lifecycle}'

If the output contains preStop, like in the example below, your chart already supports graceful shutdown and no patching is needed.

{"preStop":{"exec":{"command":["/bin/sh","-c","PGUSER=postgres pg_ctl stop -m fast"]}}}

If the output is empty or missing the preStop entry, patch the statefulset with:

kubectl patch statefulset database -p '{"spec":{"template":{"spec":{"terminationGracePeriodSeconds":60,"containers":[{"name":"postgresql","lifecycle":{"preStop":{"exec":{"command":["/bin/sh","-c","PGUSER=postgres pg_ctl stop -m fast"]}}}}]}}}}'
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'

Stop the services

Stop the Akamas services:

kubectl scale deployment --all --replicas 0
kubectl scale statefulsets -l 'app.kubernetes.io/name notin (postgresql)' --replicas 0

Charts 1.6.4 and later stop all pods except the database automatically. For older charts, scale the database back up manually:

kubectl scale statefulset database --replicas 1
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'

Dump the database

Dump the database into a dedicated volume using the job defined in the attached pg16_dump.yaml file. It creates a 10Gi PersistentVolumeClaim to store the backup. If your database needs larger storage, update the PVC definition in the YAML file.

2KB

pg16_dump.yaml

Open

kubectl apply -f pg16_dump.yaml
kubectl wait job/pg-dump --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-dump -f --all-containers=true

Once the dump completes, scale down the database:

kubectl scale statefulset database --replicas 0

Clean up the datadir

Verify the database is scaled down before proceeding:

kubectl wait statefulset/database --for=jsonpath='{.status.replicas}=0' --timeout=5s && echo Ok || echo 'ERROR: database is still running'

Then clean up the datadir using the attached pg16_cleanup.yaml file:

1KB

pg16_cleanup.yaml

Open

kubectl apply -f pg16_cleanup.yaml
kubectl wait job/pg-cleanup --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-cleanup -f --all-containers=true

You can inspect the content of the backup volume using the pg-debug pod

kubectl apply -f pg16_debug.yaml

857B

pg16_debug.yaml

Open

Upgrade the database

Once the cleanup completes, patch the statefulset with the new image:

export PG16_IMAGE='16.6.0-debian-12-r2'
kubectl patch statefulset database -p '{"spec":{"template":{"spec":{"containers":[{"name":"postgresql","image":"485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/bitnami/postgresql:'${PG16_IMAGE}'"}]}}}}'
kubectl scale statefulset/database --replicas=1
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'
kubectl logs statefulset/database --all-containers=true

Restore the database

Restore the database using the attached pg16_restore.yaml file:

2KB

pg16_restore.yaml

Open

kubectl apply -f pg16_restore.yaml
kubectl wait job/pg-restore --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-restore -f --all-containers=true

Restart the Akamas services

To complete the upgrade, restart the Akamas services:

kubectl scale deployment --all --replicas 1
kubectl scale statefulsets --all --replicas 1

Upgrade the Akamas release

Once verified that the new database is working correctly, upgrade Akamas using the chart associated with the latest 3.7.x release, as described in Install Akamas:

helm upgrade --install \
    --create-namespace --namespace akamas \
    --repo http://helm.akamas.io/charts \
    --version '<1.7.x>' \
    -f akamas.yaml \
    akamas akamas

Final cleanup

Once you have verified that the instance was upgraded successfully and works correctly, you can delete the backup volume.

kubectl delete pvc pg-dump