In this section, you’re going to start creating the experiment to trigger the interruption of Amazon EC2 Spot Instances using AWS Fault Injection Simulator (FIS). When using Spot Instances, you need to be prepared to be interrupted. With FIS, you can test the resiliency of your workload and validate that your application is reacting to the interruption notices that EC2 sends before terminating your instances. You can target individual Spot Instances or a subset of instances in clusters managed by services that tag your instances such as ASG, Fleet and EMR.
You’re going to use the CLI, so launch your terminal to run the commands included in this section.
Before you start launching Spot interruptions with FIS, you need to create an experiment template. Here is where you define which resources you want to interrupt (targets), and when you want to interrupt the instance.
You’re going to use the following CloudFormation template which creates the IAM role (
FISSpotRole) with the minimum permissions FIS needs to interrupt an instance, and the experiment template (
FISExperimentTemplate) you’re going to use to trigger a Spot interruption:
AWSTemplateFormatVersion: 2010-09-09 Description: FIS for Spot Instances Parameters: InstancesToInterrupt: Description: Number of instances to interrupt Default: 3 Type: Number DurationBeforeInterruption: Description: Number of minutes before the interruption Default: 2 Type: Number Resources: FISSpotRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Statement: - Effect: Allow Principal: Service: [fis.amazonaws.com] Action: ["sts:AssumeRole"] Path: / Policies: - PolicyName: root PolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Action: 'ec2:DescribeInstances' Resource: '*' - Effect: Allow Action: 'ec2:SendSpotInstanceInterruptions' Resource: 'arn:aws:ec2:*:*:instance/*' FISExperimentTemplate: Type: AWS::FIS::ExperimentTemplate Properties: Description: "Interrupt multiple random instances" Targets: SpotIntances: ResourceTags: ResourceTagKey: ResourceTagValue Filters: - Path: State.Name Values: - running ResourceType: aws:ec2:spot-instance SelectionMode: !Join ["", ["COUNT(", !Ref InstancesToInterrupt, ")"]] Actions: interrupt: ActionId: "aws:ec2:send-spot-instance-interruptions" Description: "Interrupt multiple Spot instances" Parameters: durationBeforeInterruption: !Join ["", ["PT", !Ref DurationBeforeInterruption, "M"]] Targets: SpotInstances: SpotIntances StopConditions: - Source: none RoleArn: !GetAtt FISSpotRole.Arn Tags: Name: "FIS_EXP_NAME" Outputs: FISExperimentID: Value: !GetAtt FISExperimentTemplate.Id
Here are some important notes about the template:
InstancesToInterruptparameter. In the template it’s defined that it’s going to interrupt three instances.
DurationBeforeInterruptionparameter. By default, it’s going to take two minutes. This means that as soon as you launch the experiment, the instance is going to receive the two-minute notification Spot interruption warning.
Targetsfrom the experiment template. The template has two placeholders
ResourceTagValuewhich are basically the key/value for the tags to use when choosing the instances to interrupt. We’re going to run a
sedcommand to replace them with the proper values for this workshop.
Let’s continue by creating the Spot interruption experiment template using Cloudformation. You can view the CloudFormation template (fisspotinterruption.yaml) at GitHub here. To download it, you can run the following command:
Now, simply run the following commands to create the FIS experiment:
export FIS_EXP_NAME=fis-spot-interruption sed -i -e "s#ResourceTagKey#aws:elasticmapreduce:instance-group-role#g" fisspotinterruption.yaml sed -i -e "s#ResourceTagValue#TASK#g" fisspotinterruption.yaml sed -i -e "s#FIS_EXP_NAME#$FIS_EXP_NAME#g" fisspotinterruption.yaml aws cloudformation create-stack --stack-name $FIS_EXP_NAME --template-body file://fisspotinterruption.yaml --capabilities CAPABILITY_NAMED_IAM aws cloudformation wait stack-create-complete --stack-name $FIS_EXP_NAME