Lab 2: Deploy testing environments using Spot & Launch Templates

Now that you are carrying out your software builds on Spot instances, the next step is to build out a CI/CD pipeline to have your software deployed to a test environment also running on Spot instances. Pipelines within Jenkins are defined using another plugin, and you’ll define pipeline steps to both deploy and terminate your testing environment. The pipeline steps that you define in this lab will call a Lambda function, which in turn deploys a CloudFormation template to brings up your testing environment, and deletes the template to tear it down.

If you view the CloudFormation template that is used to deploy the test environment, pay close attention to the following lines:

  • Lines 183-218 contains the resource definition for the Launch Template used within the Spot Fleet. Of particular note, the User Data defined within the Launch Template updates the system with all current patches, installs the Java 8 OpenJDK and Apache Tomcat 8, deploys the build artifact generated by Jenkins and starts up the Tomcat service;
  • Lines 220-250 define the Spot Fleet that is used to build out the test environment. Specifically, we are following the best practices for stateless applications using Spot: the fleet uses a diversified allocation strategy to ensure that instances are distributed across multipe availablility zones and the fleet is set to replace any instances that are unhealthy;

INCREASE THE NUMBER OF BUILD EXECUTORS ON YOUR JENKINS BUILD AGENTS

When executing a pipeline job in Jenkins, the pipeline consumes one of the build execution slots. Given that the the software builds themselves are executed using a build agent on a Spot instance, you should configure the Jenkins master server to also have a build executor - for pipeline execution to run on.

Click to reveal detailed instructions

RECONFIGURE THE GAME OF LIFE PROJECT

The S3 Publisher Plugin should be pre-configured for use (it will use the IAM policy attached to the EC2 Instance Profile used by the Jenkins Master or the build agents to grant access to the S3 bucket where your deployment artifacts will be uploaded to). First up, you’ll need to reconfigure the Game of Life project to publish it’s artifacts to the S3 bucket that was created by the CloudFormation stack deployed at the beginning of this workshop.

  1. Back at the Jenkins home page, go into the Game of Life project by clicking on its Name link. Next, click on the Configure link in the left side menu;
  2. In the General section, click on the Execute concurrent builds if necessary checkbox and the Restrict where this project can be run checkbox. Next, enter spot-agents as the Label Expression;
  3. Scroll down to the Post-build Actions section and to the bottom of that section, click on the Add post-build action button, then select the Publish artifacts to S3 Bucket option. A new configuration block will appear;
  4. Within the new configuration block, select the Spot CICD Workshop Artifacts option from the S3 Profile dropdown. Click on the Add button next to the Files to upload label;
  5. In the Source field, enter **/*.war as the match condition for your build artifact - this will ensure that the WAR (Web Application Resource) file produce by our build process is uploaded;
  6. In the Destination bucket field, enter in the name of the S3 bucket that was created by the CloudFormation template deployed in Lab 1 (the S3 bucket name can be obtained from the Outputs tab of the SpotCICDWorkshop stack in the CloudFormation console or by clicking here - look for the value associated with the DeploymentArtifactsS3Bucket key);
  7. In the Bucket Region dropdown, select the eu-west-1 region;
  8. Finally, mark both the No upload on build failure and Publish from Slave checkboxes - this will ensure that uploads only occur when a build is successful, and to ensure that our build agents are aware that they will also have the permissions to upload (given that they are using the same IAM role that our master server is using);
  9. At the bottom of the page, click on the Save button.

INSPECT THE CI/CD PIPELINE AND LAMBDA FUNCTION THAT WILL MANIPULATE THE TEST ENVIRONMENT

Your Jenkins server has been preconfigured with a Game of Life Pipeline job that contains four stages as defined in the following Groovy script:

node {
	stage('Build') {
		// Build our project and upload artifact to S3
		build 'Game of Life'
	}
	stage('Deploy') {
		// Invoke a Lambda function that will create our test environment by launching a Spot Fleet
		invokeLambda([awsAccessKeyId: '', awsRegion: 'eu-west-1', awsSecretKey: '', functionName: 'SpotCICDWorkshop_ManageTestEnvironment', payload: '''{
		"action": "deploy",
			"stackName": "GameOfLife",
			"jobBaseName": "${JOB_BASE_NAME}",
			"buildId": "${BUILD_ID}"
		}''', synchronous: true, useInstanceCredentials: true])
	}
	stage('Test') {
		// A manual approval stage designed to halt the pipeline until someone indicates that the pipeline can proceed
		input 'Run your tests'
	}
	stage('Terminate') {
		// Invoke a Lambda function to tear down our test environment once testing has been completed
		invokeLambda([awsAccessKeyId: '', awsRegion: 'eu-west-1', awsSecretKey: '', functionName: 'SpotCICDWorkshop_ManageTestEnvironment', payload: '''{
			"action": "terminate",
			"stackName": "GameOfLife",
			"jobBaseName": "${JOB_BASE_NAME}",
			"buildId": "${BUILD_ID}"
		}''', synchronous: true, useInstanceCredentials: true])
	}
}
  • The Build stage should simply execute the Game of Life build job that you just reconfigured. As you’d expect from the configuration change that you made, the deployment artifact generated by this stage will be stored in an S3 bucket;
  • The Deploy stage will need to invoke a Lambda function, which in turn will deploy a CloudFormation template. The template will create a Spot Fleet consisting of two instances using configuration defined in an EC2 Launch Template; and within this Template will be EC2 UserData that will deploy your build artifact and all of its dependencies. The template will also deploy an Application Load Balancer to front the Game of Life application, through which you will test the application;
  • The Test stage will be a manual approval stage, designed to allow you the opportunity to test out the deployed artifact; and
  • The Terminate stage will invoke a Lambda function that will destroy the test environment by deleting the CloudFormation stack.

In the Groovy script, note that the same Lambda function (SpotCICDWorkshop_ManageTestEnvironment) is called for both the Deploy and Terminate stages, but a different “action” parameter value is passed on to the function.

Next, take a look at the Lambda function to observe what it’s going to do upon execution - you can view the function by going to the Lambda console and clicking on the SpotCICDWorkshop_ManageTestEnvironment function (or by clicking here) - or you can click on the code snippet link to the left to view a version of the code that has not been customised to access resources within your account by CloudFormation.

The deploy action within the function starts from line 4:

  • Lines 9-35 contain the CloudFormation parameters that are passed on to the CloudFormation template. These parameters were generated based on the outputs of the initial CloudFormation template that you deployed in the Workshop Preparation;
  • Line 36 defines the IAM role that is assumed when deploying the template - again, this role was deployed as a part of the initial CloudFormation template defined in the Workshop Preparation;
  • Line 37 contains the URL of the CloudFormation template that is going to be launched by the function: https://s3-us-west-2.amazonaws.com/amazon-ec2-spot-cicd-workshop/amazon-ec2-spot-cicd-workshop_game-of-life.yaml
  • Line 39 attempts to deploy the stack; and
  • If the request to deploy the stack was successful, lines 43-56 store some metadata (including the CloudFormation Stack ID) into a DynamoDB table that was created by the initial CloudFormation template.

The terminate action within the function starts from line 61:

  • Lines 63-70 queries the DynamoDB table to obtain the CloudFormation Stack ID added by the deploy action; and
  • If the stack ID was able to be obtained from DynamoDB, it is used to tell CloudFormation to delete the stack with that ID on lines 74-81.

Finally, lines 87-102 contain the boilerplate code used to load the relevant AWS SDK libraries and call the actions defined above.

DEPLOY, TEST, THEN TERMINATE A TEST ENVIRONMENT USING THE GAME OF LIFE PIPELINE

You should now be at the stage where your the Game of Life Pipeline can be tested. To commence this, go back to the Jenkins Home page and click on the Game of Life Pipeline project. Within the project, click on the Build Now link on the left. When the pipeline initiates, you’ll get to see the execution status of each stage as the workflow progresses through the pipeline.

Once the deploy stage has been completed successfully:

  1. Go to the CloudFormation console (or click here) and wait for the deployment of the GameOfLife stack to switch to a CREATE_COMPLETE status;
  2. When the stack has deployed, click on the checkbox next to the GameOfLife stack, then click on the Outputs tab at the bottom of the screen;
  3. Copy the value associated with the GameOfLifeDNSName key into the address bar of a new browser tab and append the /gameoflife suffix before hitting the Enter key.

Once you’ve verified that that the Game of Life web application loads, return to Jenkins and click on the Test pipeline stage and click on Proceed to get the pipeline to process the Terminate stage. After this final stage has executed, go back to the CloudFormation console to verify that the GameOfLife stack is being deleted.

PROCEED TO LAB 3

Once your test environment has been terminated, you may proceed with Lab 3.

Site Info