When users requests On-Demand instances from a pool to the point that the pool is depleted, the system will select a set of spot instances from the pool to be terminated. A Spot instance pool is a set of unused EC2 instances with the same instance type (for example, m5.large), operating system, Availability Zone, and network platform. The Spot Instance is sent an interruption notice two minutes ahead to gracefully wrap up things.
We will deploy a pod on each spot instance to detect the instance termination notification signal so that we can both terminate gracefully any pod that was running on that node, drain from load balancers and redeploy applications elsewhere in the cluster.
To deploy Spot Interrupt Handler on each Spot Instance we will use a DaemonSet. This will monitor the EC2 metadata service on the instance for a interruption notice.
Within the Spot Interrupt Handler DaemonSet, the workflow can be summarized as:
We have provided an example K8s DaemonSet manifest. A DaemonSet runs one pod per node.
mkdir ~/environment/spot curl -o ~/environment/spot/spot-interrupt-handler-example.yml https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/content/using_ec2_spot_instances_with_eks/spotworkers/deployhandler.files/spot-interrupt-handler-example.yml
As written, the manifest will deploy pods to all nodes including On-Demand, which is a waste of resources. We want to edit our DaemonSet to only be deployed on Spot Instances. Let’s use the labels to identify the right nodes.
nodeSelector to constrain our deployment to spot instances. View this link for more details.
Configure our Spot Handler to use nodeSelector
Place this at the end of the DaemonSet manifest under spec.template.spec.nodeSelector
nodeSelector: lifecycle: Ec2Spot
If you need a reference of where those two lines should be inserted, check the highlighted text and line numbers in the section below. Note the syntax in the hierarchy maps with spec.template.spec.nodeSelector.
Once that you have added the nodeSelector section to your file, deploy the DaemonSet using the following line on the console:
kubectl apply -f ~/environment/spot/spot-interrupt-handler-example.yml
If you receive an error deploying the DaemonSet, there is likely a small error in the YAML file. We have provided a solution file at the bottom of this page that you can use to compare.
View the pods. There should be one for each spot node.
kubectl get daemonsets --all-namespaces
Use kube-ops-view to confirm the spot-interrupt-handler-example DaemonSet has been deployed only to EC2 Spot nodes.