Upgrade Kubernetes With Verrazzano Installed

After Verrazzano is installed in a cluster, you may want to upgrade the Kubernetes version of that cluster. For information on the Kubernetes versions that Verrazzano supports, see the Prerequisites.

The following lists documentation sources for Kubernetes cluster updates. If yours does not appear on this list, then consult your cluster provider’s upgrade documentation.

Upgrade a multinode cluster

For a typical multinode Kubernetes cluster, we recommend keeping one or more nodes present and available while upgrading nodes. This allows the cordoned nodes to distribute the pods to available nodes, which eliminates downtime during an upgrade. Your cluster provider can provide information on maintaining node availability for an in-place upgrade.

Upgrade a single-node cluster

For a single-node cluster upgrade, there will be downtime in the cluster to allow the node to cordon while it is upgraded. For this reason, there are a few manual workaround steps that you may need to perform to be able to fully drain the Kubernetes node.

Disable the MySQL pod disruption budget

The MySQL Operator deploys a Pod Disruption Budget for the MySQL database pods. This Pod Disruption Budget will prevent the node from being drained. To change this, you can patch the Pod Disruption budget to allow the MySQL replicas to be drained from the node.

$ kubectl patch poddisruptionbudgets.policy -n keycloak mysql-pdb -p '{"spec":{"minAvailable":0, "maxUnavailable":null}}' --type=merge

Remove the MySQL pod finalizers

It is possible that the MySQL pods will be stuck in the Terminating state while the node is being drained. If you find that the MySQL pod will not complete termination, then you can remove the finalizers to manually terminate these pods.

$ kubectl patch pod -n keycloak mysql-0 -p '{"metadata":{"finalizers":null}}' --type=merge

Delete the Rancher Helm pods

Rancher spins up Helm pods for cluster operations. Because these pods are not managed by any parent resources, they can prevent the node from being drained. If this is the case, then you can delete these pods with the following command.

$ kubectl get pods --no-headers=true -n cattle-system | awk '{print $1}' | grep helm | xargs kubectl delete pod --ignore-not-found -n cattle-system