If you are starting with self managed Spot workers chapter directly and planning to run only self managed node groups with Spot Instances, then first complete below chapters and then return back here: Start the workshop Launch using eksctl Install Kubernetes Tools Select Instance Types for Diversification
If you are have already completed EKS managed Spot workers chapters and still want to explore self managed node groups with Spot Instances, then continue with this chapter.
In this section we will create self managed node groups with Spot best practices. To adhere to the best practice of instance diversification we will include instance types we identified in Select Instance Types for Diversification chapter.
Let’s first create the configuration file:
cat <<EoF > ~/environment/add-ngs-spot.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: eksworkshop-eksctl
region: $AWS_REGION
nodeGroups:
- name: ng-spot-4vcpu-16gb
minSize: 0
maxSize: 4
desiredCapacity: 2
instancesDistribution:
instanceTypes: ["m4.xlarge", "m5.xlarge", "m5a.xlarge", "m5ad.xlarge", "m5d.xlarge", "t2.xlarge", "t3.xlarge", "t3a.xlarge"]
onDemandBaseCapacity: 0
onDemandPercentageAboveBaseCapacity: 0
spotAllocationStrategy: capacity-optimized
labels:
eks.amazonaws.com/capacityType: SPOT
intent: apps
type: self-managed-spot
taints:
spotInstance: "true:PreferNoSchedule"
tags:
k8s.io/cluster-autoscaler/node-template/label/eks.amazonaws.com/capacityType: SPOT
k8s.io/cluster-autoscaler/node-template/label/intent: apps
k8s.io/cluster-autoscaler/node-template/label/type: self-managed-spot
k8s.io/cluster-autoscaler/node-template/taint/spotInstance: "true:PreferNoSchedule"
iam:
withAddonPolicies:
autoScaler: true
cloudWatch: true
albIngress: true
- name: ng-spot-8vcpu-32gb
minSize: 0
maxSize: 2
desiredCapacity: 1
instancesDistribution:
instanceTypes: ["m4.2xlarge", "m5.2xlarge", "m5a.2xlarge", "m5ad.2xlarge", "m5d.2xlarge", "t2.2xlarge", "t3.2xlarge", "t3a.2xlarge"]
onDemandBaseCapacity: 0
onDemandPercentageAboveBaseCapacity: 0
spotAllocationStrategy: capacity-optimized
labels:
eks.amazonaws.com/capacityType: SPOT
intent: apps
type: self-managed-spot
taints:
spotInstance: "true:PreferNoSchedule"
tags:
k8s.io/cluster-autoscaler/node-template/label/eks.amazonaws.com/capacityType: SPOT
k8s.io/cluster-autoscaler/node-template/label/intent: apps
k8s.io/cluster-autoscaler/node-template/label/type: self-managed-spot
k8s.io/cluster-autoscaler/node-template/taint/spotInstance: "true:PreferNoSchedule"
iam:
withAddonPolicies:
autoScaler: true
cloudWatch: true
albIngress: true
EoF
This will create a add-ngs-spot.yaml
file that we will use to instruct eksctl to create two node groups.
eksctl create nodegroup -f add-ngs-spot.yaml
The creation of the workers will take about 3-4 minutes.
There are a few things to note in the configuration that we just used to create these nodegroups.
First node group has xlarge (4 vCPU and 16 GB) instance types with minSize 0, maxSize 4 and desiredCapacity 2.
Second node group has 2xlarge (8 vCPU and 32 GB) instance types with minSize 0, maxSize 2 and desiredCapacity 1.
onDemandBaseCapacity and onDemandPercentageAboveBaseCapacity both set to 0. which implies all nodes in the node group would be Spot instances.
spotAllocationStrategy is set as Capacity Optimized. This will ensure the capacity we provision in our node groups is procured from the pools that will have less chances of being interrupted.
We applied a Taint spotInstance: "true:PreferNoSchedule"
. PreferNoSchedule is used to indicate we prefer pods not be scheduled on Spot Instances. This is a “preference” or “soft” version of NoSchedule – the system will try to avoid placing a pod that does not tolerate the taint on the node, but it is not required.
Notice that the we added 3 node labels per node:
eks.amazonaws.com/capacityType: SPOT, to identify Spot nodes and use as affinities.
intent: apps, to allow you to deploy stateless applications on nodes that have been labeled with value apps
type: self-managed-spot, to identify self managed Spot nodes and use as nodeSelectors.
Notice that the we added 4 cluster autoscaler related tags to node groups. These tags are used by cluster autoscaler when node groups scale down to 0 (and scale up from 0). Cluster autoscaler acts on Auto Scaling groups belonging to node groups, therefore it requires same tags on ASG as well. EKSCTL adds these tags to ASG automatically while creating self managed node groups.
If you are wondering at this stage: Where is spot bidding price ? you are missing some of the changes EC2 Spot Instances had since 2017. Since November 2017 EC2 Spot price changes infrequently based on long term supply and demand of spare capacity in each pool independently. You can still set up a maxPrice in scenarios where you want to set maximum budget. By default maxPrice is set to the On-Demand price; Regardless of what the maxPrice value, Spot Instances will still be charged at the current spot market price.
Aside from familiarizing yourself with the kubectl commands below to obtain the cluster information, you should also explore your cluster using kube-ops-view and find out the nodes that were just created.
Confirm that the new nodes joined the cluster correctly. You should see the nodes added to the cluster.
kubectl get nodes
You can use the node-labels to identify the lifecycle of the nodes
kubectl get nodes --label-columns=eks.amazonaws.com/capacityType --selector=type=self-managed-spot | grep SPOT
ip-192-168-11-135.ap-southeast-1.compute.internal NotReady <none> 13s v1.21.4-eks-033ce7e SPOT
ip-192-168-78-145.ap-southeast-1.compute.internal NotReady <none> 5s v1.21.4-eks-033ce7e SPOT
ip-192-168-78-41.ap-southeast-1.compute.internal NotReady <none> 12s v1.21.4-eks-033ce7e SPOT
You can use the kubectl describe nodes
with one of the spot nodes to see the taints applied to the EC2 Spot Instances.
Explore your cluster using kube-ops-view and find out the nodes that have just been created.