Reading Time: 4 minutes

So, your workloads started to hit the ceiling with requests and the autoscaling is going through the roof. It’s that time, time to upgrade.

Here I will describe how to upgrade the whole node pool to a different instance or machine type with more CPU and RAM.

Upgrading is… inconvenient, it doesn’t matter if it’s a simple 5min fix to change an image or upgrade something else. There is still that teeny tiny negative thought that something may or can go wrong during the upgrade.

Nonetheless, working on the cloud has it’s perks and the chances are significantly lower of something going sideways. We still strive for better reliability, security, and uptime and so do the cloud providers with their services.

Let’s start.

Here we have our K8s cluster up and running with its three worker nodes, but doing a ‘top nodes’ shows us that they are near their memory limit, so let’s begin with the upgrade and swapping out those nodes.

NAME                                                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
gke-kubernetes-cluster-default-pool-0a3ad3e1-b93k   54m          5%     954Mi           82%
gke-kubernetes-cluster-default-pool-0a3ad3e1-hgrx   70m          7%     917Mi           79%
gke-kubernetes-cluster-default-pool-0a3ad3e1-tzv6   121m         12%    1136Mi          98%

Now before starting you’ll need to choose the new instance type, and how much nodes would you like.

(I’m doing this on GKE, but it all similar on Azure or AWS)

Go to Kubernetes Engine>Cluster>Your cluster and click add node pool from the top bar.

I’m gonna name the new pool “e2-pool”, as E2 are the newer instances that we are gonna upgrade to.

I’m choosing E2-medium which have 2 CPU’s and 4GB of RAM per node, that should be sufficent for my testing purposes.

After waiting a bit, the new pool is up and running:

Issuing ‘get nodes’ or ‘top nodes’ will show us that the new pool is registered:

NAME                                                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
gke-kubernetes-cluster-default-pool-0a3ad3e1-b93k   63m          6%     970Mi           83%
gke-kubernetes-cluster-default-pool-0a3ad3e1-hgrx   63m          6%     928Mi           80%
gke-kubernetes-cluster-default-pool-0a3ad3e1-tzv6   102m         10%    1156Mi          99%
gke-kubernetes-cluster-e2-pool-7fd36867-033p        56m          5%     615Mi           21%
gke-kubernetes-cluster-e2-pool-7fd36867-ktcb        57m          6%     584Mi           20%
gke-kubernetes-cluster-e2-pool-7fd36867-p9tv        57m          6%     589Mi           20%

Halfway there, now we will need to mark the current node pool as unscheduleable and evict the pods.

From the old pool, to mark the nodes as unchedulable or what is know as cordoning issue:

$ kubectl cordon node-name

In my case

$ kubectl cordon gke-kubernetes-cluster-default-pool-0a3ad3e1-b93k
node/gke-kubernetes-cluster-default-pool-0a3ad3e1-b93k cordoned

To check if the cordoning is successful you’ll see the node marked with “SchedulingDisabled” under status.

NAME                                                STATUS                     ROLES    AGE   VERSION
gke-kubernetes-cluster-default-pool-0a3ad3e1-b93k   Ready,SchedulingDisabled   <none>   66m   v1.16.13-gke.1
gke-kubernetes-cluster-default-pool-0a3ad3e1-hgrx   Ready                      <none>   66m   v1.16.13-gke.1
gke-kubernetes-cluster-default-pool-0a3ad3e1-tzv6   Ready                      <none>   66m   v1.16.13-gke.1
gke-kubernetes-cluster-e2-pool-7fd36867-033p        Ready                      <none>   29m   v1.16.13-gke.1
gke-kubernetes-cluster-e2-pool-7fd36867-ktcb        Ready                      <none>   29m   v1.16.13-gke.1
gke-kubernetes-cluster-e2-pool-7fd36867-p9tv        Ready                      <none>   29m   v1.16.13-gke.1

Also if you do ‘describe’ on the node, you’ll find that the node has been tainted and marked as unschedulable. Pay attention to the Taints and Unschedulable flags.

Continue on the other nodes. Also, make note that this is only tainting the nodes so no new pods will be scheduled to them, the existing running pods will stay! For that, we will need to drain the nodes.

gke-kubernetes-cluster-default-pool-0a3ad3e1-b93k Ready,SchedulingDisabled
gke-kubernetes-cluster-default-pool-0a3ad3e1-hgrx Ready,SchedulingDisabled
gke-kubernetes-cluster-default-pool-0a3ad3e1-tzv6 Ready,SchedulingDisabled
gke-kubernetes-cluster-e2-pool-7fd36867-033p Ready
gke-kubernetes-cluster-e2-pool-7fd36867-ktcb Ready
gke-kubernetes-cluster-e2-pool-7fd36867-p9tv Ready

After all the nodes have been cordoned, now we will start draining the nodes in order to evict the pods and force them to migrate to the new node pool.

Start with the first node, the command will be:

$ kubectl drain gke-kubernetes-cluster-default-pool-0a3ad3e1-b93k --ignore-daemonsets

Note: Since daemon set managed pods will be deployed to every node, you will get an error that the operator cannot evict those pods. The argument to add is in bold and self-explanatory. Those pods will be ignored and continue to run, to which they make no difference since the new node pool will be prepared, and the old one can be safely deleted.

Here is the output, note the evictions:

evicting pod default/api-gateway-0
evicting pod kube-system/event-exporter-gke-59b99fdd9c-dslhp
evicting pod kube-system/fluentd-gke-scaler-cd4d654d7-glk9d
evicting pod kube-system/kube-dns-7c976ddbdb-4xhqj
pod/api-gateway-0 evicted
pod/fluentd-gke-scaler-cd4d654d7-glk9d evicted
pod/kube-dns-7c976ddbdb-4xhqj evicted
pod/event-exporter-gke-59b99fdd9c-dslhp evicted
node/gke-kubernetes-cluster-default-pool-0a3ad3e1-b93k evicted

Now do describe on the same node and you will find that it doesn’t have any pods scheduled to it(apart from the monitoring ones, from the daemon set).

Now if we do describe on any of the new nodes, we can see that our pods have migrated:

$ kubectl describe node gke-kubernetes-cluster-e2-pool-7fd36867-ktcb

Continue draining all the old nodes, and check if the new ones have the pods from your namespace.

Running ‘get pods’ with wide output will show us also on which node they reside in:

Now we are left to delete the old pool.

$ kubectl get nodes
gke-kubernetes-cluster-e2-pool-7fd36867-033p Ready 76m v1.16.13-gke.1
gke-kubernetes-cluster-e2-pool-7fd36867-ktcb Ready 76m v1.16.13-gke.1
gke-kubernetes-cluster-e2-pool-7fd36867-p9tv Ready 76m v1.16.13-gke.1

That’s it. You’ve successfully managed to migrate the workloads to a new pool with zero downtime and no pods lost :)!