In this section we will look at the utilization of our EC2 Spot Instances while the application is running, and examine how many Spark executors are running.
To get started, let’s check that your EMR cluster and Spark application are running.
In this step, when you look at the utilization of the EMR cluster, do not expect to see full utilization of vCPUs and Memory on the EC2 instances, as the wordcount Spark application we are running is not very resource intensive and is just used for demo purposes.
To connect to the web interfaces running on our EMR cluster you need to use SSH tunneling. Click here to learn more about connecting to EMR interfaces.
First, we need to grant SSH access from the Cloud9 environment to the EMR cluster master node:
At this stage, we’ll be able to ssh into the EMR master node. First we will access the Ganglia web interface to look at cluster metrics:
Go to the EMR Management Console, click on your cluster, and open the Application user interfaces tab. You’ll see the list of on-cluster application interfaces.
Copy the master node DNS name from one of the interface urls, it will look like ec2.xx-xxx-xxx-xxx..compute.amazonaws.com
Establish an SSH tunnel to port 80, where Ganglia is bound, executing the below command on your Cloud9 environment (update the command with your master node DNS name):
ssh -i ~/environment/emr-workshop-key-pair.pem -N -L 8080:ec2-###-##-##-###.compute-1.amazonaws.com:80 hadoop@ec2-###-##-##-###.compute-1.amazonaws.com
You’ll get a message saying the authenticity of the host can’t be established. Type ‘yes’ and hit enter. The message will look similar to the following:
The authenticity of host 'ec2-54-195-131-148.eu-west-1.compute.amazonaws.com (172.31.36.62)' can't be established.
ECDSA key fingerprint is SHA256:Cmv0qkh+e4nm5qir6a9fPN5DlgTUEaCGBN42txhShoI.
ECDSA key fingerprint is MD5:ee:63:d0:4a:a2:29:8a:c9:41:1b:a1:f0:f6:8e:68:4a.
Are you sure you want to continue connecting (yes/no)?
Now, on your Cloud9 environment, click on the “Preview” menu on the top and then click on “Preview Running Application”. You’ll see a browser window opening on the environment with an Apache test page. on the URL, append /ganglia/ to access the Ganglia Interface. The url will look like https://xxxxxx.vfs.cloud9.eu-west-1.amazonaws.com/ganglia/.
Click on the button next to “Browser” (arrow inside a box) to open Ganglia in a dedicated browser page.Have a look around. Take notice of the heatmap (Server Load Distribution). Notable graphs are:
Now, let’s look at the Resource Manager application user interface.
Go to the Cloud9 terminal where you have established the ssh connection, and press ctrl+c to close it.
Create an SSH tunnel to the cluster master node on port 8088 by running this command (update the command with your master node DNS name):
ssh -i ~/environment/emr-workshop-key-pair.pem -N -L 8080:ec2-###-##-##-###.compute-1.amazonaws.com:8088 hadoop@ec2-###-##-##-###.compute-1.amazonaws.com
Now, on your browser, update the URL to “/cluster” i.e. https://xxxxxx.vfs.cloud9.eu-west-1.amazonaws.com/cluster
On the left pane, click Nodes, and in the node table, you should see the number of containers that each node ran.
Now, let’s look at Spark History Server application user interface:
Go to the Cloud9 terminal where you have established the ssh connection, and press ctrl+c to close it.
Create an SSH tunnel to the cluster master node on port 18080 by running this command (update the command with your master node DNS name):
ssh -i ~/environment/emr-workshop-key-pair.pem -N -L 8080:ec2-###-##-##-###.compute-1.amazonaws.com:18080 hadoop@ec2-###-##-##-###.compute-1.amazonaws.com
Now, on your browser, go to the base URL of your Cloud9 environment i.e. https://xxxxxx.vfs.cloud9.eu-west-1.amazonaws.com/
Click on the App ID in the table (where App Name = Amazon reviews word count) and go to the Executors tab
You can again see the number of executors that are running in your EMR cluster under the Executors table
EMR emits several useful metrics to CloudWatch metrics. You can use the AWS Management Console to look at the metrics in two ways:
The metrics will take a few minutes to populate.
Some notable metrics:
With 32 Spot Units in the Task Instance Fleet, EMR launched either 8 * xlarge (running one executor) or 4 * 2xlarge instances (running 2 executors) or 2 * 4xlarge instances (running 4 executors), so the Task Instance Fleet provides 8 executors / containers to the cluster.
The Core Instance Fleet launched one xlarge instance, able to run one executor.
Your Spark application was configured to run in Cluster mode, meaning that the Spark driver is running on the Core node. Since it is counted as a container, this adds a container to our count, but it is not an executor.