# Migration procedures

## From 3.6 to 3.7

The upgrade from Akamas 3.6 to 3.7 requires upgrading the database to a new major version. This upgrade involves some manual steps.

<details>

<summary>Docker version</summary>

{% hint style="info" %}
Make sure you are logged in to the host running the Akamas instance before running the following commands.
{% endhint %}

**Stop the services**

Stop the Akamas services except the databases:

```bash
cd akamas
docker compose down
docker compose up -d database airflow-db kong-database
```

**Export AWS credentials and login to AWS docker repo**

```bash
export AWS_ACCESS_KEY_ID=AAAAAAAAAAA ## use your AWS access key ID
export AWS_SECRET_ACCESS_KEY=bbbbbbbbbbbbbbbbbb ## use your AWS secret access key id
aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 485790562880.dkr.ecr.us-east-2.amazonaws.com

```

**Extract the database passwords**

```bash
docker compose config | python3 -c 'import sys, yaml; s=yaml.safe_load(sys.stdin)["services"]; [print(k+"_DB_PASSWORD="+s[v]["environment"]["POSTGRES_PASSWORD"]) or print(k+"_DB_USER="+s[v]["environment"].get("POSTGRES_USER", "postgres")) for k, v in {"AKAMAS": "database", "AIRFLOW": "airflow-db"}.items()]' > pass.env
```

**Dump the database**

```bash
export PG_IMAGE='485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/master-db:1.13.0'

mkdir -p backup
source pass.env

echo " *** Dumping akamas services"
docker run --rm \
    --network akamas \
    -u "$(id -u):$(id -g)" \
    -v $(pwd)/backup:/backup \
    -e PGHOST=database \
    -e PGUSER=${AKAMAS_DB_USER} \
    -e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
    "$PG_IMAGE" \
    pg_dumpall --clean --if-exists --exclude-database=postgres -f /backup/akamas_dump.sql

echo " *** Dumping airflow"
docker run --rm \
    --network akamas \
    -u "$(id -u):$(id -g)" \
    -v $(pwd)/backup:/backup \
    -e PGHOST=airflow-db \
    -e PGUSER=${AIRFLOW_DB_USER} \
    -e PGPASSWORD=${AIRFLOW_DB_PASSWORD} \
    "$PG_IMAGE" \
    pg_dump --clean --if-exists -d airflow -f /backup/airflow_dump.sql
```

**Clean up old database**

```bash
echo " *** Removing old containers"
docker rm -fv database airflow-db kong-db
echo " *** Removing old volumes"
docker volume rm -f akamas_airflow-db-data akamas_database-data akamas_kong-data
```

**Start the updated database**

Update `docker-compose.yml` to use the new database image. Back up the old file and replace the image version using `sed`:

```bash
OLD_VERSION=$(python3 -c 'import sys, yaml; print(yaml.safe_load(sys.stdin)["services"]["database"]["image"].split(":")[1])' < docker-compose.yml)
echo "Current database image version: $OLD_VERSION"
cp docker-compose.yml docker-compose.yml.bak$(date +%s)
sed -i "s|master-db:${OLD_VERSION}|master-db:${PG_IMAGE##*:}|g" docker-compose.yml
diff -u docker-compose.yml.bak* docker-compose.yml
```

Verify the update and start the new database:

```bash
docker compose pull
docker compose up -d database
docker logs -f database
```

Then wait for the message `database system is ready to accept connections` to appear then press CTRL+C

**Restore the data**

```bash
docker run --rm \
    --network akamas \
    -v $(pwd)/backup:/backup \
    -e PGHOST=database \
    -e PGUSER=${AKAMAS_DB_USER} \
    -e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
    "$PG_IMAGE" \
    psql -X -f /backup/akamas_dump.sql

docker run --rm \
    --network akamas \
    -v $(pwd)/backup:/backup \
    -e PGHOST=database \
    -e PGUSER=${AKAMAS_DB_USER} \
    -e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
    "$PG_IMAGE" \
    psql -X -f /backup/airflow_dump.sql
```

**Restart Akamas**

Replace `docker-compose.yml` with the latest Akamas 3.7.x version, as described in [Install the Akamas Server](https://docs.akamas.io/akamas-docs/installing/docker/install-the-akamas-server), and restart the remaining services:

```bash
docker compose pull
docker compose up -d
```

</details>

<details>

<summary>Kubernetes version</summary>

{% hint style="info" %}
Make sure you are using the correct namespace for the Akamas installation. To switch namespace, run `kubectl config set-context --current --namespace <akamas>`, replacing `<akamas>` with your namespace name.
{% endhint %}

**Check `preStop` hooks**

Ensure the current chart already supports `preStop` hooks for Postgres. Run:

```bash
kubectl get statefulset/database -o jsonpath='{.spec.template.spec.containers[].lifecycle}'
```

If the output contains `preStop`, like in the example below, your chart already supports graceful shutdown and no patching is needed.

```json
{"preStop":{"exec":{"command":["/bin/sh","-c","PGUSER=postgres pg_ctl stop -m fast"]}}}
```

If the output is empty or missing the `preStop` entry, patch the statefulset with:

```bash
kubectl patch statefulset database -p '{"spec":{"template":{"spec":{"terminationGracePeriodSeconds":60,"containers":[{"name":"postgresql","lifecycle":{"preStop":{"exec":{"command":["/bin/sh","-c","PGUSER=postgres pg_ctl stop -m fast"]}}}}]}}}}'
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'
```

**Stop the services**

Stop the Akamas services:

```bash
kubectl scale deployment --all --replicas 0
kubectl scale statefulsets -l 'app.kubernetes.io/name notin (postgresql)' --replicas 0
```

Charts `1.6.4` and later stop all pods except the database automatically. For older charts, scale the database back up manually:

```bash
kubectl scale statefulset database --replicas 1
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'
```

**Dump the database**

Dump the database into a dedicated volume using the job defined in the attached `pg16_dump.yaml` file. It creates a 10Gi PersistentVolumeClaim to store the backup. If your database needs larger storage, update the PVC definition in the YAML file.

{% file src="<https://3902962638-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4eyIGD5kWiby93w0l2pW%2Fuploads%2Fgit-blob-be28974878d109688b23a9c6d1fc3492fd741a45%2Fpg16_dump.yaml?alt=media>" %}

```bash
kubectl apply -f pg16_dump.yaml
kubectl wait job/pg-dump --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-dump -f --all-containers=true
```

Once the dump completes, scale down the database:

```bash
kubectl scale statefulset database --replicas 0
```

**Clean up the datadir**

Verify the database is scaled down before proceeding:

```bash
kubectl wait statefulset/database --for=jsonpath='{.status.replicas}=0' --timeout=5s && echo Ok || echo 'ERROR: database is still running'
```

Then clean up the datadir using the attached `pg16_cleanup.yaml` file:

{% file src="<https://3902962638-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4eyIGD5kWiby93w0l2pW%2Fuploads%2Fgit-blob-cc51e30956d3a4985fe572d5e98c70affa696ece%2Fpg16_cleanup.yaml?alt=media>" %}

```bash
kubectl apply -f pg16_cleanup.yaml
kubectl wait job/pg-cleanup --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-cleanup -f --all-containers=true
```

{% hint style="info" %}
You can inspect the content of the backup volume using the `pg-debug` pod

```bash
kubectl apply -f pg16_debug.yaml
```

{% endhint %}

{% file src="<https://3902962638-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4eyIGD5kWiby93w0l2pW%2Fuploads%2Fgit-blob-e9dab6ff77f01b568f9079802c9c92cdf27b1d3b%2Fpg16_debug.yaml?alt=media>" %}

**Upgrade the database**

Once the cleanup completes, patch the statefulset with the new image:

```bash
export PG16_IMAGE='16.6.0-debian-12-r2'
kubectl patch statefulset database -p '{"spec":{"template":{"spec":{"containers":[{"name":"postgresql","image":"485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/bitnami/postgresql:'${PG16_IMAGE}'"}]}}}}'
kubectl scale statefulset/database --replicas=1
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'
kubectl logs statefulset/database --all-containers=true
```

**Restore the database**

Restore the database using the attached `pg16_restore.yaml` file:

{% file src="<https://3902962638-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F4eyIGD5kWiby93w0l2pW%2Fuploads%2Fgit-blob-1f2997ec5df5ddb5171bb0ac6503576c95e9a147%2Fpg16_restore.yaml?alt=media>" %}

```bash
kubectl apply -f pg16_restore.yaml
kubectl wait job/pg-restore --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-restore -f --all-containers=true
```

**Restart the Akamas services**

To complete the upgrade, restart the Akamas services:

```bash
kubectl scale deployment --all --replicas 1
kubectl scale statefulsets --all --replicas 1
```

**Upgrade the Akamas release**

Once verified that the new database is working correctly, upgrade Akamas using the chart associated with the latest 3.7.x release, as described in [Install Akamas](https://docs.akamas.io/akamas-docs/installing/kubernetes/install-akamas):

```bash
helm upgrade --install \
    --create-namespace --namespace akamas \
    --repo http://helm.akamas.io/charts \
    --version '<1.7.x>' \
    -f akamas.yaml \
    akamas akamas
```

**Final cleanup**

Once you have verified that the instance was upgraded successfully and works correctly, you can delete the backup volume.

```bash
kubectl delete pvc pg-dump
```

</details>