The first step to implement the rendering pipeline is to generate a Docker image with the script that will run Blender and FFmpeg. This container image will be used by AWS Batch when running jobs. You are going to host that image in Amazon Elastic Container Registry.
Amazon ECR is a fully managed container registry that makes it easy for developers to share and deploy container images and artifacts. Amazon ECR is integrated with Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and AWS Lambda, simplifying your development to production workflow. Amazon ECR eliminates the need to operate your own container repositories or worry about scaling the underlying infrastructure. Amazon ECR hosts your images in a highly available and scalable architecture, allowing you to deploy containers for your applications reliably.
To learn more about ECR, visit this web page.
If you want to learn more about containers, read this containers deep dive.
To create the Docker image you will need two files; the DockerFile, which is a text document that contains all the commands a user could call on the command line to assemble an image, and the bash script that will be executed when running the Docker container.
Download both files executing these commands:
wget "https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/content/rendering-with-batch/rendering-with-batch.files/docker-files/Dockerfile"
wget "https://raw.githubusercontent.com/awslabs/ec2-spot-workshops/master/content/rendering-with-batch/rendering-with-batch.files/docker-files/render.sh"
Retrieve the repository’s Uri and registry Id:
export REPOSITORY_DATA=$(aws ecr describe-repositories --repository-names "${RepositoryName}")
export REPOSITORY_URI=$((echo $REPOSITORY_DATA) | jq -r '.repositories[0].repositoryUri')
export REGISTRY_ID=$((echo $REPOSITORY_DATA) | jq -r '.repositories[0].registryId')
export IMAGE="${REPOSITORY_URI}:latest"
echo "Repository Uri: ${REPOSITORY_URI}"
echo "Registry Id: ${REGISTRY_ID}"
Retrieve an authentication token and authenticate your Docker client to your registry.
aws ecr get-login-password --region "${AWS_DEFAULT_REGION}" | docker login --username AWS --password-stdin "${REGISTRY_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
Build your Docker image using the following command. For information on building a Docker file from scratch see the instructions here. The execution if this step might take a couple of minutes.
docker build -t "${IMAGE}" .
Push the image to ECR .
docker push "${IMAGE}"
You are now done with the container part. Next, you will configure some environment variables needed to create resources in AWS Batch.
When we send a batch job, the container that we just created will be executed. The entry point of the container is the bash script render.sh. The script just takes
a few arguments that AWS Batch will pass to each task and does run either blender when an environment variable named ACTION is set to render or ffmpeg when is set to stitch.
The following section describes the render.sh script in more detail. You don’t need to go through this to run this workshop, but if you are interested in fully understanding how Blender and FFmpeg are called it will give you a clear description.
Reads the environment variable ACTION and decides from it what’s the type of job to run, either render or stitch. It also takes other arguments such as the the input, output
1#!/bin/bash
2
3parse_arguments() {
4 # Parses the command line arguments and stores the values in global variables.
5
6 ACTION=$1
7
8 if [ "${ACTION}" != "render" ] && [ "${ACTION}" != "stitch" ] ; then
9 echo "Unrecognised action"
10 exit 2
11 fi
12
13 while (( "$#" )); do
14...Downloads the blender file from S3.
53render() {
54 # Pipeline that is executed when this script is told to render.
55
56 # Download the blender file from S3
57 aws s3 cp "${INPUT_URI}" file.blend
58
59 # Calculate start frame and end frame
60 calculate_render_frame_range
61
62 # Start the rendering process
63 mkdir frames
64 echo "Rendering frames ${start_frame} to ${end_frame}"
65 blender -b file.blend -E CYCLES -o "frames/" -s "${start_frame}" -e "${end_frame}" -a
66
67 # Upload all the rendered frames to a folder in S3
68 aws s3 cp --recursive "frames" "${OUTPUT_URI}/frames"
69}Calculates the slice of frames that has to render (we will cover in more detail when we talk about AWS Batch).
38calculate_render_frame_range() {
39 # Calculates the start frame and end frame a job has to render
40 # using the value of the env var AWS_BATCH_JOB_ARRAY_INDEX
41
42 # If the env var AWS_BATCH_JOB_ARRAY_INDEX is empty, this is a single job. Render from start to end
43 if [[ -z "${AWS_BATCH_JOB_ARRAY_INDEX}" ]]; then
44 start_frame=1
45 end_frame="${F_PER_JOB}"
46 # Otherwise use the array index to calculate the corresponding frame slice
47 else
48 start_frame=$((AWS_BATCH_JOB_ARRAY_INDEX * F_PER_JOB + 1))
49 end_frame=$((AWS_BATCH_JOB_ARRAY_INDEX * F_PER_JOB + F_PER_JOB))
50 fi
51}Executes Blender.
53render() {
54 # Pipeline that is executed when this script is told to render.
55
56 # Download the blender file from S3
57 aws s3 cp "${INPUT_URI}" file.blend
58
59 # Calculate start frame and end frame
60 calculate_render_frame_range
61
62 # Start the rendering process
63 mkdir frames
64 echo "Rendering frames ${start_frame} to ${end_frame}"
65 blender -b file.blend -E CYCLES -o "frames/" -s "${start_frame}" -e "${end_frame}" -a
66
67 # Upload all the rendered frames to a folder in S3
68 aws s3 cp --recursive "frames" "${OUTPUT_URI}/frames"
69}Uploads all the frames to S3.
53render() {
54 # Pipeline that is executed when this script is told to render.
55
56 # Download the blender file from S3
57 aws s3 cp "${INPUT_URI}" file.blend
58
59 # Calculate start frame and end frame
60 calculate_render_frame_range
61
62 # Start the rendering process
63 mkdir frames
64 echo "Rendering frames ${start_frame} to ${end_frame}"
65 blender -b file.blend -E CYCLES -o "frames/" -s "${start_frame}" -e "${end_frame}" -a
66
67 # Upload all the rendered frames to a folder in S3
68 aws s3 cp --recursive "frames" "${OUTPUT_URI}/frames"
69}Downloads all the frames from S3.
71stitch() {
72 # Pipeline that is executed when this script is told to stitch.
73
74 # Download the frames from S3
75 mkdir frames
76 aws s3 cp --recursive "${INPUT_URI}/frames" frames/
77
78 # Start the stitching process
79 ffmpeg -i frames/%04d.png output.mp4
80
81 # Upload the output video to S3
82 aws s3 cp output.mp4 "${OUTPUT_URI}/output.mp4"
83}Executes FFmpeg.
71stitch() {
72 # Pipeline that is executed when this script is told to stitch.
73
74 # Download the frames from S3
75 mkdir frames
76 aws s3 cp --recursive "${INPUT_URI}/frames" frames/
77
78 # Start the stitching process
79 ffmpeg -i frames/%04d.png output.mp4
80
81 # Upload the output video to S3
82 aws s3 cp output.mp4 "${OUTPUT_URI}/output.mp4"
83}Uploads the video to S3.
71stitch() {
72 # Pipeline that is executed when this script is told to stitch.
73
74 # Download the frames from S3
75 mkdir frames
76 aws s3 cp --recursive "${INPUT_URI}/frames" frames/
77
78 # Start the stitching process
79 ffmpeg -i frames/%04d.png output.mp4
80
81 # Upload the output video to S3
82 aws s3 cp output.mp4 "${OUTPUT_URI}/output.mp4"
83}