> For the complete documentation index, see [llms.txt](https://docs.akamas.io/akamas-docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.akamas.io/akamas-docs/managing-akamas/upgrade/migration-procedures.md).
# Migration Procedures
## From 3.6 to 4.0
The upgrade from Akamas 3.6 to 4.0 requires:
* Upgrading the database to a new major version
* Upgrading the CSV telemetry provider configuration
Those upgrades both involve some manual steps.
### Pre-upgrade steps
{% hint style="warning" %}
Run these steps **before** updating the docker-compose or helm release
{% endhint %}
#### Database upgrade
Here's the guide for the database upgrade
DB upgrade - Docker version
{% hint style="info" %}
Make sure you are logged in to the host running the Akamas instance before running the following commands.
{% endhint %}
**Stop the services**
Stop the Akamas services except the databases:
```bash
cd akamas
docker compose down
docker compose up -d database airflow-db kong-database
```
**Export AWS credentials and login to AWS docker repo**
```bash
export AWS_ACCESS_KEY_ID=AAAAAAAAAAA ## use your AWS access key ID
export AWS_SECRET_ACCESS_KEY=bbbbbbbbbbbbbbbbbb ## use your AWS secret access key id
aws ecr get-login-password --region us-east-2 | docker login --username AWS --password-stdin 485790562880.dkr.ecr.us-east-2.amazonaws.com
```
**Extract the database passwords**
```bash
docker compose config | python3 -c 'import sys, yaml; s=yaml.safe_load(sys.stdin)["services"]; [print(k+"_DB_PASSWORD="+s[v]["environment"]["POSTGRES_PASSWORD"]) or print(k+"_DB_USER="+s[v]["environment"].get("POSTGRES_USER", "postgres")) for k, v in {"AKAMAS": "database", "AIRFLOW": "airflow-db"}.items()]' > pass.env
```
**Dump the database**
```bash
export PG_IMAGE='485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/master-db:1.13.0'
mkdir -p backup
source pass.env
echo " *** Dumping akamas services"
docker run --rm \
--network akamas \
-u "$(id -u):$(id -g)" \
-v $(pwd)/backup:/backup \
-e PGHOST=database \
-e PGUSER=${AKAMAS_DB_USER} \
-e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
"$PG_IMAGE" \
pg_dumpall --clean --if-exists --exclude-database=postgres -f /backup/akamas_dump.sql
echo " *** Dumping airflow"
docker run --rm \
--network akamas \
-u "$(id -u):$(id -g)" \
-v $(pwd)/backup:/backup \
-e PGHOST=airflow-db \
-e PGUSER=${AIRFLOW_DB_USER} \
-e PGPASSWORD=${AIRFLOW_DB_PASSWORD} \
"$PG_IMAGE" \
pg_dump --clean --if-exists -d airflow -f /backup/airflow_dump.sql
```
**Clean up old database**
```bash
echo " *** Removing old containers"
docker rm -fv database airflow-db kong-db
echo " *** Removing old volumes"
docker volume rm -f akamas_airflow-db-data akamas_database-data akamas_kong-data
```
**Start the updated database**
Update `docker-compose.yml` to use the new database image. Back up the old file and replace the image version using `sed`:
```bash
OLD_VERSION=$(python3 -c 'import sys, yaml; print(yaml.safe_load(sys.stdin)["services"]["database"]["image"].split(":")[1])' < docker-compose.yml)
echo "Current database image version: $OLD_VERSION"
cp docker-compose.yml docker-compose.yml.bak$(date +%s)
sed -i "s|master-db:${OLD_VERSION}|master-db:${PG_IMAGE##*:}|g" docker-compose.yml
diff -u docker-compose.yml.bak* docker-compose.yml
```
Verify the update and start the new database:
```bash
docker compose pull
docker compose up -d database
docker logs -f database
```
Then wait for the message `database system is ready to accept connections` to appear then press CTRL+C.\
NOTE: if the expected message doesn't pop up after a couple of minutes and you see the message `database system was shut down`, instead, some random bug occurred. In this case press CTRL+C then stop the database with `docker compose down`. Then relaunch the last 2 commands above (`docker compose up -d database` and `docker logs -f database` and wait again for the message `database system is ready to accept connections` to appear then press CTRL+C.
**Restore the data**
```bash
docker run --rm \
--network akamas \
-v $(pwd)/backup:/backup \
-e PGHOST=database \
-e PGUSER=${AKAMAS_DB_USER} \
-e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
"$PG_IMAGE" \
psql -X -f /backup/akamas_dump.sql
docker run --rm \
--network akamas \
-v $(pwd)/backup:/backup \
-e PGHOST=database \
-e PGUSER=${AKAMAS_DB_USER} \
-e PGPASSWORD=${AKAMAS_DB_PASSWORD} \
"$PG_IMAGE" \
psql -X -f /backup/airflow_dump.sql
```
**Restart Akamas**
Replace `docker-compose.yml` with the latest Akamas 4.0 version, as described in Install the Akamas Server, and restart the remaining services:
```bash
docker compose pull
docker compose up -d
```
DB upgrade - Kubernetes version
{% hint style="info" %}
Make sure you are using the correct namespace for the Akamas installation. To switch namespace, run `kubectl config set-context --current --namespace `, replacing `` with your namespace name.
{% endhint %}
**Check `preStop` hooks**
Ensure the current chart already supports `preStop` hooks for Postgres. Run:
```bash
kubectl get statefulset/database -o jsonpath='{.spec.template.spec.containers[].lifecycle}'
```
If the output contains `preStop`, like in the example below, your chart already supports graceful shutdown and no patching is needed.
```json
{"preStop":{"exec":{"command":["/bin/sh","-c","PGUSER=postgres pg_ctl stop -m fast"]}}}
```
If the output is empty or missing the `preStop` entry, patch the statefulset with:
```bash
kubectl patch statefulset database -p '{"spec":{"template":{"spec":{"terminationGracePeriodSeconds":60,"containers":[{"name":"postgresql","lifecycle":{"preStop":{"exec":{"command":["/bin/sh","-c","PGUSER=postgres pg_ctl stop -m fast"]}}}}]}}}}'
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'
```
**Stop the services**
Stop the Akamas services:
```bash
kubectl scale deployment --all --replicas 0
kubectl scale statefulsets -l 'app.kubernetes.io/name notin (postgresql)' --replicas 0
```
Charts `1.6.4` and later stop all pods except the database automatically. For older charts, scale the database back up manually:
```bash
kubectl scale statefulset database --replicas 1
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'
```
**Dump the database**
Dump the database into a dedicated volume using the job defined in the attached `pg16_dump.yaml` file. It creates a 10Gi PersistentVolumeClaim to store the backup. If your database needs larger storage, update the PVC definition in the YAML file.
```bash
kubectl apply -f pg16_dump.yaml
kubectl wait job/pg-dump --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-dump -f --all-containers=true
```
Once the dump completes, scale down the database:
```bash
kubectl scale statefulset database --replicas 0
```
**Clean up the datadir**
Verify the database is scaled down before proceeding:
```bash
kubectl wait statefulset/database --for=jsonpath='{.status.replicas}=0' --timeout=5s && echo Ok || echo 'ERROR: database is still running'
```
Then clean up the datadir using the attached `pg16_cleanup.yaml` file:
```bash
kubectl apply -f pg16_cleanup.yaml
kubectl wait job/pg-cleanup --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-cleanup -f --all-containers=true
```
{% hint style="info" %}
You can inspect the content of the backup volume using the `pg-debug` pod
```bash
kubectl apply -f pg16_debug.yaml
```
{% endhint %}
**Upgrade the database**
Once the cleanup completes, patch the statefulset with the new image:
```bash
export PG16_IMAGE='16.6.0-debian-12-r2'
kubectl patch statefulset database -p '{"spec":{"template":{"spec":{"containers":[{"name":"postgresql","image":"485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/bitnami/postgresql:'${PG16_IMAGE}'"}]}}}}'
kubectl scale statefulset/database --replicas=1
kubectl wait statefulset/database --for=jsonpath='{.status.availableReplicas}=1'
kubectl logs statefulset/database --all-containers=true
```
**Restore the database**
Restore the database using the attached `pg16_restore.yaml` file:
```bash
kubectl apply -f pg16_restore.yaml
kubectl wait job/pg-restore --for=jsonpath='{.status.ready}=1' -o template='{{"initContainer complete\n"}}'
kubectl logs job/pg-restore -f --all-containers=true
```
**Restart the Akamas services**
To complete the upgrade, restart the Akamas services:
```bash
kubectl scale deployment --all --replicas 1
kubectl scale statefulsets --all --replicas 1
```
**Upgrade the Akamas release**
Once verified that the new database is working correctly, upgrade Akamas using the chart associated with the latest 4.0 release, as described in Install Akamas:
```bash
helm upgrade --install \
--create-namespace --namespace akamas \
--repo http://helm.akamas.io/charts \
--version '<1.8.1>' \
-f akamas.yaml \
akamas akamas
```
**Final cleanup**
Once you have verified that the instance was upgraded successfully and works correctly, you can delete the backup volume.
```bash
kubectl delete pvc pg-dump
```
### Post-upgrade steps
{% hint style="warning" %}
Run these steps **after** updating the docker-compose or helm release
{% endhint %}
#### CSV telemetry provider configuration upgrade
This step upgrades the configuration (adding new config parameters: hostname, username, and password) of the CSV telemetry provider. If you have never used this provider in your studies (or systems) and have no CSV telemetry instance defined in any system, we recommend the faster way:
Recommended procedure when no CSV telemetry instances are present
An important prerequisite for this procedure is to start the services beforehand (see the final part of #from-3.6-to-3.7), since you need the telemetry service running.
Log in with Akamas CLI to your environment with:
```
akamas login
```
Then issue the following command:
```
akamas uninstall tp "CSV File"
```
**VERY IMPORTANT:** If a message appears telling `It will also delete the following telemetry-instances:`
like in the following example:
```bash
$> akamas uninstall tp "CSV File"
This command will delete telemetry provider 'CSV File'.
It will also delete the following telemetry-instances:
id workspace system name
====================================================================================================
0f13e9b9-c110-46a8-9bb1-c53fe2e6c199 default renaissance CSV File telem instance
e86a3e01-11a9-4a35-bcf1-ead3e492afeb work1 customerpay-bff-service CSV File
b22a72e0-7013-4ac0-84a3-7cc337ba0d73 work2 Mongo CSV File 2
2c257de0-faca-445b-b7ac-81044c239efb work3 renaissance CSV File telem instance
Do you want to proceed? [y/N]:
```
Just STOP this procedure by pressing ENTER key and refer to alternate Docker or kubernetes procedures below.\
If there are no telemetry instances to delete, you may proceed with the deletion of the telemetry provider (press 'y' then ENTER), then you can recreate the new telemetry provider by creating a new file named `csv-file-provider-official.yaml`
with these exact contents:
```yaml
name: CSV File
description: Telemetry provider to collect metrics from CSV files
dockerImage: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/telemetry-providers/csv-file-provider:3.3.0
```
then issuing the following command:
```bash
akamas install tp csv-file-provider-official.yaml
```
These, instead, are the alternate procedures for the Docker and kubernetes case, for use when you already have CSV telemetry instances defined in your systems. They have the advantage that you don't need Akamas services to be up and running (you only need a database, already running)
Alternate procedure for Docker
Log in to the machine where akamas is installed via SSH. Then issue the command:
```bash
cd akamas
```
Retrieve the telemetry DB password with this command:
```bash
docker compose exec telemetry bash -c "echo \$SPRING_DATASOURCE_PASSWORD "
```
Connect to the database container shell with:
```bash
docker compose exec -it database bash
```
Connect to postgres with:
```bash
psql -U akamas_telemetry -p -d telemetry_service -p 5432
```
Then type the DB password (retrieved above) when asked.
Then issue SQL command:
```sql
update provider set config = '[{"required": false, "paramName": "logLevel", "paramType": "STRING_ENUM", "protected": false, "validationParams": {"validStrings": ["INFO", "DETAILED"]}}, {"required": true, "paramName": "remoteFilePattern", "paramType": "STRING", "protected": false}, {"required": true, "paramName": "hostname", "paramType": "IP_OR_FQDN", "protected": false, "requiredAlternates": ["address"]}, {"required": false, "paramName": "port", "paramType": "INTEGER", "protected": false, "validationParams": {"max": 65563, "min": 1}}, {"required": true, "paramName": "username", "paramType": "STRING", "protected": false}, {"required": true, "paramName": "password", "paramType": "STRING", "protected": true, "requiredAlternates": ["auth", "key"]}, {"required": true, "paramName": "key", "paramType": "STRING", "protected": true, "requiredAlternates": ["password", "auth"]}, {"required": true, "paramName": "authType", "paramType": "STRING_ENUM", "protected": false, "validationParams": {"validStrings": ["password", "key"]}, "requiredAlternates": ["password", "key"]}, {"required": false, "paramName": "protocol", "paramType": "STRING_ENUM", "protected": false, "validationParams": {"validStrings": ["scp", "sftp"]}}, {"required": false, "paramName": "componentColumn", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "componentName", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "timestampColumn", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "timestampFormat", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "filenamePattern", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "fieldSeparator", "paramType": "STRING", "protected": false, "validationParams": {"regex": "[,;\t]"}}, {"required": false, "paramName": "labelColumns", "paramType": "STRING_LIST", "protected": false}]' where name = 'CSV File';
```
Alternate procedure for Kubernetes
Properly configure your `kubectl` configuration in order to connect to the correct cluster and namespace holding your Akamas installation.
Retrieve telemetry DB password with:
```bash
kubectl get secret database-user-credentials -o go-template='{{ .data.telemetry | base64decode }} '
```
Connect to database pod shell with:
```bash
kubectl exec -it database-0 -- bash
```
Connect to postgres with:
```bash
psql -U akamas_telemetry -p -d telemetry_service -p 5432
```
then type in DB password (retrieved above) when asked.
Then issue SQL command:
```sql
update provider set config = '[{"required": false, "paramName": "logLevel", "paramType": "STRING_ENUM", "protected": false, "validationParams": {"validStrings": ["INFO", "DETAILED"]}}, {"required": true, "paramName": "remoteFilePattern", "paramType": "STRING", "protected": false}, {"required": true, "paramName": "hostname", "paramType": "IP_OR_FQDN", "protected": false, "requiredAlternates": ["address"]}, {"required": false, "paramName": "port", "paramType": "INTEGER", "protected": false, "validationParams": {"max": 65563, "min": 1}}, {"required": true, "paramName": "username", "paramType": "STRING", "protected": false}, {"required": true, "paramName": "password", "paramType": "STRING", "protected": true, "requiredAlternates": ["auth", "key"]}, {"required": true, "paramName": "key", "paramType": "STRING", "protected": true, "requiredAlternates": ["password", "auth"]}, {"required": true, "paramName": "authType", "paramType": "STRING_ENUM", "protected": false, "validationParams": {"validStrings": ["password", "key"]}, "requiredAlternates": ["password", "key"]}, {"required": false, "paramName": "protocol", "paramType": "STRING_ENUM", "protected": false, "validationParams": {"validStrings": ["scp", "sftp"]}}, {"required": false, "paramName": "componentColumn", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "componentName", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "timestampColumn", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "timestampFormat", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "filenamePattern", "paramType": "STRING", "protected": false}, {"required": false, "paramName": "fieldSeparator", "paramType": "STRING", "protected": false, "validationParams": {"regex": "[,;\t]"}}, {"required": false, "paramName": "labelColumns", "paramType": "STRING_LIST", "protected": false}]' where name = 'CSV File';
```
After using one of the two alternate procedures, you MUST upgrade the CSV Telemetry Provider to the latest version 3.3.0 (the 3.2.0 version that was shipped with 3.6.x will not work properly when using new features). For this, you need to start up services beforehand (refer to #from-3.6-to-3.7). Then you can upgrade the CSV telemetry provider by:\
Creating a new file named `csv-file-provider-official.yaml` with these contents:
```yaml
name: CSV File
description: Telemetry provider to collect metrics from CSV files
dockerImage: 485790562880.dkr.ecr.us-east-2.amazonaws.com/akamas/telemetry-providers/csv-file-provider:3.3.0
```
and run:
```bash
akamas install tp -u csv-file-provider-official.yaml
```
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.akamas.io/akamas-docs/managing-akamas/upgrade/migration-procedures.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.