So far we have seen some advanced use cases of Karpenter. In this section we will see how Karpenter can define different Provisioners. This allows to handle different configurations.
Each Provisioner CRD (Custom Resource Definition) provides a set of unique configurations, this that define the resources it supports as well as labels and taints that will also be applied to the newly resources created by that Provisioner. In large clusters with multiple applications, new applications may need to create nodes with specific Taints or specific labels. In these scenarios you can configure alternative Provisioners. For this workshop we have already defined a team1
Provisioner. You can list the available Provisioners by running the following command:
kubectl get provisioners
team1
Provisioner.Let’s create a new deployment this time, let’s force the deployment to use the team1
provisioner.
cat <<EOF > inflate-team1.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: inflate-team1
spec:
replicas: 0
selector:
matchLabels:
app: inflate-team1
template:
metadata:
labels:
app: inflate-team1
spec:
nodeSelector:
intent: apps
kubernetes.io/arch: amd64
karpenter.sh/provisioner-name: team1
containers:
- image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
name: inflate-team1
resources:
requests:
cpu: "1"
memory: 256M
tolerations:
- key: team1
operator: Exists
topologySpreadConstraints:
- labelSelector:
matchLabels:
app: inflate-team1
maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: ScheduleAnyway
EOF
kubectl apply -f inflate-team1.yaml
There seem to be quite a few new entries in this section! Let’s cover a few of those. The rest we will discuss in the Challenge section.
The deployment sets the NodeSelector intent: apps
so that this application does not overlap with the Managed Node group created with the cluster.
The Node Selector karpenter.sh/provisioner-name: team1
is also set. This NodeSelector is used by Karpenter to select which Provisioner configuration should be used when Provisioning capacity for this deployment. This allow different provisioner to for example define different Taint sections.
The NodeSelector kubernetes.io/arch
has been set to use x86_64 amd64
instances.
If you recall well the team1
provisioner had a section that defined a team1
taint. The deployment must add also that toleration so it’s allowed to be placed in the newly created instances with the Taint team1: NoSchedule
. Note the NoSchedule means that only applications that have the toleration for team1
will be allowed on it.
You can use Kube-ops-view or just plain kubectl cli to visualize the changes and answer the questions below. In the answers we will provide the CLI commands that will help you check the resposnes. Remember: to get the url of kube-ops-view you can run the following command kubectl get svc kube-ops-view | tail -n 1 | awk '{ print "Kube-ops-view URL = http://"$4 }'
Answer the following questions. You can expand each question to get a detailed answer and validate your understanding.
inflate-team1
deployment to 4 replicas ?To scale up the deployment run the command:
kubectl scale deployment inflate-team1 --replicas 4
We should be able to see the number of replicas deployed by running the command below and looking for the instances of inflate-team1
pods. There should be 4 of them.
kubectl get pods
Over this workshop we’ve used a few mechanisms. We can run the following command to check the nodes created and the properties label properties for those.
kubectl get node --selector=intent=apps,karpenter.sh/provisioner-name=team1 --show-labels
This should display an output similar to the one that follows. From this output below in our example we can see that the instance selected was on-demand
and of type t3a.xlarge
.
NAME STATUS ROLES AGE VERSION LABELS
xxxxxxxxxxxxxxxxxxxxxxxxxxxx.compute.internal Ready <none> 66s v1.21.5-eks-bc4871b beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=t3a.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=eu-west-1,failure-domain.beta.kubernetes.io/zone=eu-west-1b,intent=apps,karpenter.sh/provisioner-name=team1,kubernetes.io/arch=amd64,kubernetes.io/hostname=xxxxxxxxxxxxxxxxxxxxxxxxxxxx.compute.internal,kubernetes.io/os=linux,kubernetes.sh/capacity-type=on-demand,node.kubernetes.io/instance-type=t3a.xlarge,topology.kubernetes.io/region=eu-west-1,topology.kubernetes.io/zone=eu-west-1b
xxxxxxxxxxxxxxxxxxxxxxxxxxx.compute.internal Ready <none> 66s v1.21.5-eks-bc4871b beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=t3a.xlarge,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=eu-west-1,failure-domain.beta.kubernetes.io/zone=eu-west-1a,intent=apps,karpenter.sh/provisioner-name=team1,kubernetes.io/arch=amd64,kubernetes.io/hostname=xxxxxxxxxxxxxxxxxxxxxxxxxxx.compute.internal,kubernetes.io/os=linux,kubernetes.sh/capacity-type=on-demand,node.kubernetes.io/instance-type=t3a.xlarge,topology.kubernetes.io/region=eu-west-1,topology.kubernetes.io/zone=eu-west-1a
But there is something that does not match with what we have seen so far with Karpenter. In previous scenarios Karpenter will be bin-packing instances to fit the workload. In this case 2 instances have been created one on each AZ (in this case eu-west-1a and eu-west-1b) !
Well, let’s check first Karpenter log.
kubectl logs -f deployment/karpenter -c controller -n karpenter
The output of Karpenter should look similar to the one below
...
2022-05-12T04:33:28.625Z INFO controller Batched 4 pod(s) in 1.038819076s {"commit": "00661aa"}
2022-05-12T04:33:29.361Z DEBUG controller Discovered 401 EC2 instance types {"commit": "00661aa"}
2022-05-12T04:33:29.482Z DEBUG controller Discovered EC2 instance types zonal offerings {"commit": "00661aa"}
2022-05-12T04:33:29.638Z DEBUG controller Discovered subnets: [subnet-0204b1b3b885ca98d (eu-west-1a) subnet-037d1d97a6a473fd1 (eu-west-1b) subnet-04c2ca248972479e7 (eu-west-1b) subnet-063d5c7ba912986d5 (eu-west-1a)] {"commit": "00661aa"}
2022-05-12T04:33:29.728Z DEBUG controller Discovered security groups: [sg-03ab1d5d49b00b596 sg-06e7e2ca961ab3bed] {"commit": "00661aa", "provisioner": "team1"}
2022-05-12T04:33:29.732Z DEBUG controller Discovered kubernetes version 1.21 {"commit": "00661aa", "provisioner": "team1"}
2022-05-12T04:33:29.796Z DEBUG controller Discovered ami-0440c10a3f77514d8 for query "/aws/service/eks/optimized-ami/1.21/amazon-linux-2/recommended/image_id" {"commit": "00661aa", "provisioner": "team1"}
2022-05-12T04:33:29.971Z DEBUG controller Created launch template, Karpenter-eksworkshop-eksctl-2228580094236845875 {"commit": "00661aa", "provisioner": "team1"}
2022-05-12T04:33:31.885Z INFO controller Launched instance: i-02df8ea1e99895e78, hostname: ip-192- 68-14-13.eu-west-1.compute.internal, type: t3a.xlarge, zone: eu-west-1a, capacityType: on-demand {"commit": "00661aa", "provisioner": "team1"}
2022-05-12T04:33:31.896Z INFO controller Created node with 2 pods requesting {"cpu":"2125m","memory":"512M","pods":"4"} from types c4.xlarge, c6a.xlarge, c6i.xlarge, c5.xlarge, c5a.xlarge and 263 other(s) {"commit": "00661aa", "provisioner": "team1"}
2022-05-12T04:33:32.443Z INFO controller Launched instance: i-0b6984823f26d5b15, hostname: ip-192-168-39-218.eu-west-1.compute.internal, type: t3a.xlarge, zone: eu-west-1b, capacityType: on-demand {"commit": "00661aa", "provisioner": "team1"}
2022-05-12T04:33:32.453Z INFO controller Created node with 2 pods requesting {"cpu":"2125m","memory":"512M","pods":"4"} from types c4.xlarge, c6a.xlarge, c6i.xlarge, c5.xlarge, c5a.xlarge and 267 other(s) {"commit": "00661aa", "provisioner": "team1"}
2022-05-12T04:33:32.464Z INFO controller Waiting for unschedulable pods {"commit": "00661aa"}
...
It’s interesting to see how the nodes were selected in different availability zones.
As you are probably guessing by now, this may have to do with the section of the deployment section for topologySpreadConstraints:
. Latest versions of Karpenter (>0.4.1) support Kubernetes Topology Spread Constraints. You can define one or multiple topologySpreadConstraint to instruct the kube-scheduler how to place each incoming Pod in relation to the existing Pods across your cluster.
To all effect the bin-packing is still happening is just that the topologySpreadConstraint is applied and forces us to spread the workload across the available kubernetes.io/zone
This one should be really easy. Just run the following command
To scale up the deployment run the command:
kubectl scale deployment inflate-team1 --replicas 0
In this section we have learned:
Applications requiring specific labels or Taints can make use of alternative Provisioners that are customized for that set of applications. This is a common setup for large clusters.
Pods can select the Provisioner by setting a nodeSelector with the lable karpenter.sh/provisioner-name
pointing to the right Provisioner.
Karpenter supports topologySpreadConstraints. Topology Spread constraints instruct the kube-scheduler how to place each incoming Pod in relation to the existing Pods across your cluster. In this scenario we discover how to balance pods across Availability Zones.