Lab 3: Externalise state data to add resiliency to Jenkins

You’re now using Spot instances for your code builds and for the environments that are built out for testing – but your Jenkins server is still using an on-demand instance. Jenkins itself does not natively support running in high-availability configurations because it persists all data on a local file system. If you can store this data durably somewhere else than on the local file system, you can move your Jenkins Master instance to a self-healing Spot instance. To provide persistence for this file system data, you’ll move your Jenkins data to an Elastic File System (EFS) volume and mount this volume on instance spawned by a Spot Fleet.

OBTAIN THE RELEVANT INFORMATION FOR CLOUDFORMATION FOR THIS LAB

As with the previous labs, the CloudFormation stack deployed during your Workshop Preparation has provisioned some of the resources required for this lab (in order to allow us to focus on the aspects of the workshop that directly apply to EC2 Spot). You will need to determine and make a note of what the EFS Filesystem ID is.

Click to reveal detailed instructions

COPY THE CONTENTS OF JENKINS_HOME TO YOUR EFS FILE SYSTEM

In order to copy the contents of the JENKINS_HOME directory to the EFS file system (for which the ID of which you determined in the previous step), you’ll need to first mount the file system on the EC2 instance currently running your Jenkins server.

Once the file system has been mounted, stop the Jenkins service and set the file system permission of the root of your filesystem so that the jenkins user and group are owners. Finally, copy the contents of /var/lib/jenkins (this is the JENKINS_HOME directory) across to the root of your EFS file system.

Click to reveal detailed instructions

PROVISION AN EC2 SPOT FLEET FOR YOUR NEW JENKINS HOST

The Spot Fleet that you’ll provision for your Jenkins server will be configured in a similar manner to what you did for the build agents - though this time you’ll be a bit more aggressive with the bid price to ensure that you see an overall saving over what you would have spent on an on-demand t3.medium instance. Additionally, you’ll configure this Spot Fleet so that the instances are associated with the Target Group used by the Application Load Balancer that you’ve been using to access Jenkins.

  1. Go to the EC2 console and click on the Spot Requests option from the left frame (or click here);
  2. Click on the Request Spot Instances button;
  3. At the first screen of the Spot instance launch wizard:
    1. Under the Tell us your application or task need heading, switch to the Load balancing workloads option;
    2. In the Configure your instances section, select the JenkinsMasterLaunchTemplate template from the Launch template dropdown. Change the Network to be the Spot CICD Workshop VPC. After making this selection, enable the check boxes for all three Availability Zones and then select the Amazon EC2 Spot CICD Workshop Public Subnet associated with each availability zone as the subnet to launch instances in;
    3. At the Tell us how much capacity you need section, keep the Total target capacity at 1 instance and the Optional On-Demand portion set to 0, and then tick the Maintain target capacity checkbox. Once selected, leave the Interruption behavior set to Terminate;
    4. Again, you’ll override the recommendations made by the console, so clear the tick from Apply recommendations checkbox. Click on the Remove links associated with the all of the instance types initially defined to remove them from the fleet configuration. Then click on the Select instance types button and add the m3.large, m4.large, t2.medium and t3.medium instance types to the fleet definition. Once the checkboxes for the required instance types have been ticked, click on the Select button. Once you have the four desired instance types listed in the fleet request, select the Lowest Price Fleet allocation strategy (since we’re interested in keeping cost to an absolute minimum for this use case, and it makes little sense to diversify a single instance across any number of instance pools);
    5. At the Additional request details section, remove the tick from the Apply defaults checkbox. As you want to keep the cost of running your Jenkins server below that for which you’re currently paying, select the Set your max price (per instance/hour) option, and set the price to be the on-demand price of a t3.medium instance in the Ireland region, which is $0.0456. In order to ensure that the server can receive HTTP requests from the Application Load Balancer you’ve been using, tick the checkbox labelled Receive traffic from one or more load balancers and from the Target groups dropdown, select JenkinsMasterEC2TargetGroup.
    6. Review the Your fleet request as a glance section - it should indicate that your Fleet strength is strong as a result of being able to draw instances from 12 instance pools, and your Estimated price should indicate that you’re expecting to make a 73% saving compared to the cost of equivalent on-demand resources;
    7. Lastly, click on the Launch button.

VERIFY THAT YOUR EC2 SPOT INSTANCE IS ATTACHED TO YOUR ALB TARGET GROUP

After a few moments, your Spot instance will start up and should attach itself the the Target Group being used by your Application Load Balancer. Determine if this registration has completed successfully and when it has done so, access Jenkins through your Load Balancer and fire off a build of Apache PDFBox to ensure that everything is still working as expected.

Click to reveal detailed instructions

TEST THE SELF-HEALING ARCHITECTURE BY TERMINATING THE RUNNING EC2 SPOT INSTANCE

What happens when the market price for the capacity pool that your EC2 Spot instance is running in goes over your $0.0456 per instance-hour bid price?

When this happens, your EC2 Spot instance will be provided with a two-minute notice of termination and after that time lapses, the instance will be terminated. At this point, Spot Fleet will observe that there are no running instances in the fleet and because the desired capacity is one, it will attempt to launch a replacement instance in a capacity pool that is still below the per instance-hour bid price that you set (this is why diversification across many capacity pool is a best practice). If such an instance is available, a replacement instance will be launched and bootstrapped in exactly the same manner as your original instance.

Test this out by terminating the Spot instance with the Jenkins Master (Spot) name tag and verifying that a replacement instance comes up to take its place.

Click to reveal detailed instructions

TERMINATE THE ON-DEMAND INSTANCE THAT WAS INITIALLY USED FOR YOUR JENKINS SERVER

Once you’ve verified that your Spot Fleet is self-healing, you no longer have any need for the On-demand instance. To prevent it from incurring unnecessary cost, it can be terminated.

Click to reveal detailed instructions

PROCEED TO LAB 4

Once your on-demand Jenkins instance has been terminated, you may proceed with Lab 4.