Troubleshoot Kubernetes installation issues

This section describes some of the most common issues found during the Akamas installation on Kubernetes.

Issues with HTTP being redirected to HTTPS

This issue may potentially happen in the early stages of adoption when the customer still hasn't fully configured HTTPS and is temporarily using HTTP instead. If that's the case, there may be misconfigurations in your ingress controller that can cause automatic HTTP to HTTPS redirections. In fact, there's a problem with these redirections if HTTP code 301 is issued: this code automatically changes the HTTP method in every REST API call to a GET, even if it was a PUT, POST or DELETE. As a result Akamas will only receive GET calls which are informative only and no creation/update/deletion of any Akamas resource can be performed. The solution would be either to disable this redirection or to issue HTTP code 308 instead of 301. This because HTTP code 308 still means redirection but it ALSO retains the original HTTP method while redirecting, so HTTP POST calls will correctly become HTTPS POST calls.

NGINX Ingress controller case

Kubernetes allows different ingress types. We'll describe the case for NGINX ingress controller, which is one of the most popular choices. Other solution may vary technically but should be conceptually similar.

When using Kubernetes NGINX ingress, Akamas namespace ingress is automatically created and configured with a config file that starts with:

server {

listen 80;
listen [::]:80;

listen 443 ssl;
listen [::]:443 ssl;

ssl_certificate /etc/nginx/secrets/AKAMAS_NAMESPACE-tls;
ssl_certificate_key /etc/nginx/secrets/AKAMAS_NAMESPACE-tls;

if ($scheme = http) {
    return 301 https://$host:443$request_uri;
}

This file will be located in a NGINX dedicated pod (most probably named nginx-ingress-controller-NNNNNNNNNNN) in a dedicated namespace whose name should be nginx (or similar names). Let's refer to this namespace name as NGINX_NAMESPACE. Similarly, let's refer to the namespace where Akamas resides as AKAMAS_NAMESPACE. The name of such a file inside the pod will be /etc/nginx/conf.d/AKAMAS_NAMESPACE-AKAMAS_NAMESPACE.conf (where AKAMAS_NAMESPACE must be replaced with the real name of your Akamas namespace). As can be seen by the code above, the last IF forces, for every HTTP request, a HTTPS call with a 301 redirection which, as already anticipated, is not the proper redirect code since every PUT/POST/DELETE request is converted to a GET call and this causes Akamas to be unable to receive the correct API calls. This issue was recently caused by NGINX option SSLredirect being turned to true by default (it was false in the past). In fact the IF line above is constructed by the template nginx.ingress.tmpl which is located in the same pod. The relevant section is:

{{if $server.SSL}}
{{if not $server.GRPCOnly}}
{{- if $server.SSLRedirect}}
if ($scheme = http) {
    return 301 https://$host:{{index $server.SSLPorts 0}}$request_uri
}
{{- end}}
{{end}}
{{- end}}

and the best way to remove this line is to disable server SSLRedirect. Although the documentation states that this could be attained by adding the annotation nginx.ingress.kubernetes.io/ssl-redirect: "false" to the ingress yaml in your kubernetes chart for the AKAMAS_NAMESPACE, this doesn't seem to work. The only method I found to be working is to add to the ingress controller configmap (this too will be inside namespace NGINX_NAMESPACE) the following key/value: ssl-redirect: "false" So, in order to solve the issue you should edit the Kube deployment of the nginx-ingress-controller (inside NGINX_NAMESPACE) and add the above value. In fact its container will have some args similar to the following:

Args:
│ -nginx-plus=false
│ -nginx-reload-timeout=60000
│ -enable-app-protect=false
│ -enable-app-protect-dos=false
│ -nginx-configmaps=$(NGINX_NAMESPACE)/nginx-ingress

Take note of the nginx-configmaps value. Its name in this case is nginx-ingress in NGINX_NAMESPACE namespace. You must add the value: ssl-redirect: "false" to this exact config map. Assuming that this configmap is empty, you could simply create a file with this content:

apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-ingress
data:
ssl-redirect: "false"

and apply it with command (replace FILENAME and NGINX_NAMESPACE with correct names):

kubectl apply -f FILENAME.yaml - n NGINX_NAMESPACE

If configmap is not empty, you should use the following command to add the missing ssl-redirect key/value (then edit with your favorite editor such as vi):

kubectl configmap edit nginx-ingress -n NGINX_NAMESPACE

Either way, after this change all HTTP POST/PUT/DELETE calls will not be redirected to HTTPS GETs but will remain on HTTP (with same method type POST/PUT/DELETE)