Draining of Nodes for Kubernetes

Cordon and Drain the Nodes before it gets terminated by ASG in AWS

Overview

When you do patching and/or updating Nodes in Kubernetes, it is required to stop scheduling to that node and drain the pods running on that node.

To drain a node e.g. worker (stop schedule to that node and evict pods from that node), Kubernetes provides the following commands -

Drain a single worker node:

kubectl drain node_name --ignore-daemonsets

Note: Replace node_name with your value.

To undo the isolation, run the following commands:

kubectl uncordon node_name

Note: Replace node_name with your value.

This works well for static set of Nodes. When you Autoscaling Group (ASG) for the management of WorkNodes for EKS or self managed Kubernetes, you will find that pods will be terminated abruptly when node is terminated and may corrupt the application.

AWS Node Termination handler, handles situations like spot instance termination and EC2 scheduled events. But it does not cater for ASG scale-in events.

aws/aws-node-termination-handler