How to create a custom EC2 image (AMI) using AWS EC2 Image Builder?
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
Today we will discuss how to create a custom EC2 image using AWS EC2 Image Builder with example.
What is AWS EC2 image builder?
EC2 Image Builder is a fully managed AWS service that makes it easier to automate the creation, management, and deployment of customized, secure, and up-to-date server images that are pre-installed and pre-configured with software and settings to meet specific IT standards. The images you build are created in your account and you can configure them for operating system patches on an ongoing basis.
Along with a final image, Image Builder creates an image recipe, which is a combination of the source image and components for building and testing. You can use the image recipe with existing source code version control systems and continuous integration/continuous deployment pipelines for repeatable automation.
What is an image pipeline?
Image Builder image pipelines provide an automation framework for creating and maintaining custom AMIs and container images.
Pipelines deliver the following functionality:
- Assemble the base image, components for building and testing, infrastructure configuration, and distribution settings.
- Facilitate scheduling for automated maintenance processes using the Schedule builder in the console wizard, or entering cron expressions for recurring updates to your images.
- Enable change detection for the base image and components, to automatically skip scheduled builds when there are no changes.
- Enable rule-based automation through Amazon EventBridge.
What is an image recipe?
An EC2 Image Builder recipe defines the base image to use as your starting point to create a new image, along with the set of components that you add to customize your image and verify that everything is working as expected. Automatic version choices are provided for each component. A maximum of 20 components, which include build and test, can be applied to a recipe.
After you create an image recipe, or a container recipe, you cannot modify or replace the recipe. To update components after a recipe is created, you must create a new recipe or recipe version. You can, however, always apply tags to your recipe.
What is a component?
A component defines the sequence of steps required to either customize an instance prior to image creation (a build component), or to test an instance that was launched from the created image (a test component).
A component is created from a declarative, plain-text YAML or JSON document that describes the runtime configuration for building and validating, or testing an instance that is produced by your pipeline. Components run on the instance using a component management application. The component management application parses the documents and runs the desired steps.
After they are created, one or more components are grouped together using an image recipe or container recipe to define the plan for building and testing a virtual machine or container image. You can use public components that are owned and managed by AWS, or you can create your own.
You define the component in an YAML or JSON file called component document that describes configuration for a customization you can apply to your image. The document is used to create a build or test component.
What is a distribution setting?
Once you build, validate and test an image using AWS Image builder pipeline, you need to tell AWS where to store the image using a distribution setting. You can share the final image to a different region within the same AWS account or to another AWS account.
What is infrastructure configuration?
Infrastructure configurations allow you to specify the infrastructure within which to build and test your EC2 Image Builder image.
Infrastructure settings include:
- Instance types for your build and test infrastructure. We recommend specifying more than one instance type because this allows Image Builder to launch an instance from a pool with sufficient capacity. This can reduce your transient build failures.
- An instance profile that provides your build and test instances with the permissions that are required to perform customization activities. For example, if you have a component that retrieves resources from Amazon S3, the instance profile requires permissions to access those files. The instance profile also requires a minimal set of permissions for EC2 Image Builder to successfully communicate with the instance. For more information, see Prerequisites.
- The VPC, subnet, and security groups for your pipeline’s build and test instances.
- The Amazon S3 location where Image Builder stores application logs from your build and testing. If you configure logging, the instance profile specified in your infrastructure configuration must have s3:PutObject permissions for the target bucket (arn:aws:s3:::BucketName/*).
- An Amazon EC2 key pair that allows you to log on to your instance to troubleshoot if your build fails and you set terminateInstanceOnFailure to false.
- An SNS topic to which Image Builder sends event notifications.
How to create a custom EC2 image (AMI) using AWS EC2 Image Builder
Prerequisites:
AWS CLI installed and configured.
Step 1: Create a new directory for this demo.
|
1 2 |
## Create a directory for the demo mkdir demo && cd demo |
Step 2: Create a component configuration file.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
## Create a component document cat << 'EOF' > component_config.yaml name: HelloWorld description: Hello World App schemaVersion: 1.0 phases: - name: build steps: - name: UpdateOS action: ExecuteBash inputs: commands: - sudo yum update -y - name: InstallWebServer action: ExecuteBash inputs: commands: - sudo yum install httpd -y - sudo systemctl start httpd - sudo systemctl is-enabled httpd - echo "hello world v1" > /var/www/html/index.html - curl -s localhost - name: validate steps: - name: ValidateWebServer action: ExecuteBash inputs: commands: - | CUR_STATE=$(sudo systemctl is-active httpd) if [[ $CUR_STATE == "active" ]]; then echo "Httpd service is active." exit 0 else echo "Httpd service is not active." exit 1 fi - name: TestWebServer action: ExecuteBash inputs: commands: - | OUTPUT=$(curl -s localhost) if [[ $OUTPUT == "hello world v1" ]]; then echo "Webserver is working fine" exit 0 else echo "Webserver not working fine" exit 0 fi EOF |
Step 3: Create a new IAM instance profile.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
## Create IAM Role trust policy configuration file cat << EOF > iam_trust_policy_config.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } EOF ## Create IAM role aws iam create-role \ --role-name HelloWorldIAMRole \ --assume-role-policy-document file://iam_trust_policy_config.json ## Attach IAM Policy to the role aws iam attach-role-policy \ --policy-arn arn:aws:iam::aws:policy/AdministratorAccess \ --role-name HelloWorldIAMRole ## Create an Instance Profile for EC2 aws iam create-instance-profile \ --instance-profile-name HelloWorldInstanceProfile ## Add IAM role to the instance profile aws iam add-role-to-instance-profile \ --role-name HelloWorldIAMRole \ --instance-profile-name HelloWorldInstanceProfile |
Step 4: Create an S3 bucket with a proper bucket policy.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
## Create an S3 bucket aws s3api create-bucket \ --bucket cloudaffaire-image-builder \ --create-bucket-configuration LocationConstraint=ap-south-1 ## Get S3 bucket ARN and AWS Account ID and ARN S3_BUCKET_ARN='arn:aws:s3:::cloudaffaire-image-builder' && ACCOUNT_ARN=$(aws sts get-caller-identity | jq -r .Arn) && ACCOUNT_ID=$(aws sts get-caller-identity | jq -r .Account) && IAM_ROLE_ARN=arn:aws:iam::$ACCOUNT_ID:role/HelloWorldIAMRole ## Create a s3 bucket policy definition file cat << EOF > bucket_policy_config.json { "Version": "2012-10-17", "Statement": [ { "Sid": "HelloWorldPolicy", "Effect": "Allow", "Principal": "*", "Action": "s3:*", "Resource": ["$S3_BUCKET_ARN/*"], "Condition": { "StringEquals": { "aws:SourceAccount": "$ACCOUNT_ID", "s3:x-amz-acl": "bucket-owner-full-control" } } } ] } EOF ## Create a s3 bucket policy aws s3api put-bucket-policy \ --bucket cloudaffaire-image-builder \ --policy file://bucket_policy_config.json |
Step 5: Create a new custom component.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
## Upload component document in S3 bucket aws s3 cp component_config.yaml \ s3://cloudaffaire-image-builder/component_config.yaml ## Create custom component config def cat << 'EOF' > image_component_config.json { "name": "HelloWorldComponent", "semanticVersion": "1.0.0", "description": "Hello World App", "changeDescription": "Initial version.", "platform": "Linux", "uri": "s3://cloudaffaire-image-builder/component_config.yaml", "tags": { "App": "Hello World" } } EOF ## Create the custom component aws imagebuilder create-component \ --cli-input-json file://image_component_config.json ## List all available components owned by You aws imagebuilder list-components \ --owner Self ## List component build version COMPONENT_VERSION_ARN=$(aws imagebuilder list-components \ --owner Self | jq -r .componentVersionList[].arn) && echo $COMPONENT_VERSION_ARN && COMPONENT_BUILD_VERSION_ARN=$(aws imagebuilder list-component-build-versions \ --component-version-arn $COMPONENT_VERSION_ARN | jq -r .componentSummaryList[].arn) && echo $COMPONENT_BUILD_VERSION_ARN && aws imagebuilder list-component-build-versions \ --component-version-arn $COMPONENT_VERSION_ARN ## Get custom component details aws imagebuilder get-component \ --component-build-version-arn $COMPONENT_BUILD_VERSION_ARN |
Step 6: Create a new Image recipe.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
## Get Amazon Linux 2 latest AMI ID AWS_AMI_ID=$(aws ec2 describe-images \ --owners 'amazon' \ --filters 'Name=name,Values=amzn2-ami-kernel-5.10-hvm-2.0.????????.0-x86_64-gp2' 'Name=state,Values=available' \ --query 'sort_by(Images, &CreationDate)[-1].[ImageId]' \ --output 'text') && echo $AWS_AMI_ID ## Create a image recipe document cat << EOF > image_recipe_config.json { "name": "HelloWorldRecipe", "description": "Hello World App", "semanticVersion": "2019.12.03", "components": [ { "componentArn": "$COMPONENT_VERSION_ARN" } ], "parentImage": "$AWS_AMI_ID" } EOF ## Create a custom image recipe aws imagebuilder create-image-recipe \ --cli-input-json file://image_recipe_config.json ## List all available image recipes owned by You aws imagebuilder list-image-recipes \ --owner Self ## Get image recipe details IMAGE_RECIPE_ARN=$(aws imagebuilder list-image-recipes \ --owner Self | jq -r .imageRecipeSummaryList[].arn ) && aws imagebuilder get-image-recipe \ --image-recipe-arn $IMAGE_RECIPE_ARN |
Step 7: Create a new infrastructure configuration.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 |
## Create a key-pair aws ec2 create-key-pair \ --key-name HelloWorldKP \ --query 'KeyMaterial' \ --output text > HelloWorldKP.pem ## Get default VPC, Subnet and SG id DEAFULT_VPC_ID=$(aws ec2 describe-vpcs \ --query 'Vpcs[?IsDefault == `true`].VpcId' \ --output text) && echo $DEAFULT_VPC_ID && SUBNET_ID=$(aws ec2 describe-subnets \ --query 'Subnets[?AvailabilityZone == `ap-south-1a`].SubnetId' \ --output text) && echo $SUBNET_ID && DEFAULT_SECURITY_GROUP_ID=$(aws ec2 describe-security-groups \ --filters "Name=vpc-id,Values=$DEAFULT_VPC_ID" \ --query 'SecurityGroups[?GroupName == `default`].GroupId' \ --output text) && echo $DEFAULT_SECURITY_GROUP_ID ## Create infrastructure configuration file cat << EOF > image_infra_config.json { "name": "HelloWorldInfrastructure", "description": "Hello World App", "instanceTypes": [ "t2.micro" ], "instanceProfileName": "HelloWorldInstanceProfile", "securityGroupIds": [ "$DEFAULT_SECURITY_GROUP_ID" ], "subnetId": "$SUBNET_ID", "logging": { "s3Logs": { "s3BucketName": "cloudaffaire-image-builder", "s3KeyPrefix": "Logs" } }, "keyPair": "HelloWorldKP", "terminateInstanceOnFailure": true } EOF ## Create infrastructure configuration aws imagebuilder create-infrastructure-configuration \ --cli-input-json file://image_infra_config.json ## List all infrastructure configurations aws imagebuilder list-infrastructure-configurations ## Get details on Infrastructure configuration INFRA_CONF_ARN=$(aws imagebuilder list-infrastructure-configurations | jq -r .infrastructureConfigurationSummaryList[].arn) && aws imagebuilder get-infrastructure-configuration \ --infrastructure-configuration-arn $INFRA_CONF_ARN |
Step 8: Create a new distribution configuration.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
## Create a distribution config file cat << EOF > image_distribution_config.json { "name": "HelloWorldDistribution", "description": "Hello World App", "distributions": [ { "region": "ap-south-1", "amiDistributionConfiguration": { "name": "Name {{imagebuilder:buildDate}}", "description": "Hello World AMI", "amiTags": { "Name": "HelloWorld" }, "launchPermission": { "userIds": [ "$ACCOUNT_ID" ] } } } ] } EOF ## Create an Image Distribution Configuration aws imagebuilder create-distribution-configuration \ --cli-input-json file://image_distribution_config.json ## List all distribution configurations aws imagebuilder list-distribution-configurations ## Get distribution configuration details DIST_CONFIG_ARN=$(aws imagebuilder list-distribution-configurations | jq -r .distributionConfigurationSummaryList[].arn) && aws imagebuilder get-distribution-configuration \ --distribution-configuration-arn $DIST_CONFIG_ARN |
Step 9: Create a new image pipeline.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
## Create image pipeline configuration cat << EOF > image_pipeline_config.json { "name": "HelloWorldPipeline", "description": "Hello World App", "enhancedImageMetadataEnabled": true, "imageRecipeArn": "$IMAGE_RECIPE_ARN", "infrastructureConfigurationArn": "$INFRA_CONF_ARN", "distributionConfigurationArn": "$DIST_CONFIG_ARN", "imageTestsConfiguration": { "imageTestsEnabled": true, "timeoutMinutes": 60 }, "schedule": { "scheduleExpression": "cron(0 8 1 * ? *)", "pipelineExecutionStartCondition": "EXPRESSION_MATCH_AND_DEPENDENCY_UPDATES_AVAILABLE" }, "status": "ENABLED" } EOF ## Create Image pipeline aws imagebuilder create-image-pipeline \ --cli-input-json file://image_pipeline_config.json ## List all image pipelines aws imagebuilder list-image-pipelines \ --filters '[{"name": "name", "values": ["HelloWorldPipeline"]}]' ## Get details on the image pipeline PIPELINE_ARN=$(aws imagebuilder list-image-pipelines \ --filters '[{"name": "name", "values": ["HelloWorldPipeline"]}]' \ --query 'imagePipelineList[].arn' \ --output text) && aws imagebuilder get-image-pipeline \ --image-pipeline-arn $PIPELINE_ARN |
Step 10: Start the image pipeline manually to build, validate and test your custom EC2 image (AMI).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
## Start image pipeline execution aws imagebuilder start-image-pipeline-execution \ --image-pipeline-arn $PIPELINE_ARN ## It may take upto 30 mins or more to build, validate and test the new image ## Check image pipeline image build status aws imagebuilder list-image-pipeline-images \ --image-pipeline-arn $PIPELINE_ARN | jq -r .imageSummaryList[].state.status ## List image pipeline images aws imagebuilder list-image-pipeline-images \ --image-pipeline-arn $PIPELINE_ARN ## Get AMI ID and Snapshot ID AMI_ID=$(aws imagebuilder list-image-pipeline-images \ --image-pipeline-arn $PIPELINE_ARN | jq -r .imageSummaryList[].outputResources.amis[].image) && echo $AMI_ID && SNAPSHOT_ID=$(aws ec2 describe-images \ --image-ids $AMI_ID | jq -r .Images[].BlockDeviceMappings[].Ebs.SnapshotId) && echo $SNAPSHOT_ID |
We have successfully created a custom image (AMI) using the AWS Image builder service.
If you are getting any errors during the image build or test phase, you can check the CloudWatch logs or S3 bucket logs to troubleshoot the issue with the AWS Image builder.
CloudWatch Logs:
S3 Logs:
Step 11: Clean up.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
## Delete image IAMGE_ARN=$(aws imagebuilder list-images | jq -r .imageVersionList[].arn) && IAMGE_BUILD_ARN=$(aws imagebuilder list-image-build-versions \ --image-version-arn $IAMGE_ARN | jq -r .imageSummaryList[].arn) && aws imagebuilder delete-image \ --image-build-version-arn $IAMGE_BUILD_ARN ## Delete AMI aws ec2 deregister-image \ --image-id $AMI_ID ## Delete Snapshot aws ec2 delete-snapshot \ --snapshot-id $SNAPSHOT_ID ## Delete the image pipeline aws imagebuilder delete-image-pipeline \ --image-pipeline-arn $PIPELINE_ARN ## Delete image recipe aws imagebuilder delete-image-recipe \ --image-recipe-arn $IMAGE_RECIPE_ARN ## Delete the distribution configuration aws imagebuilder delete-distribution-configuration \ --distribution-configuration-arn $DIST_CONFIG_ARN ## Delete the infrastructure configuration aws imagebuilder delete-infrastructure-configuration \ --infrastructure-configuration-arn $INFRA_CONF_ARN ## Delete the custom component aws imagebuilder delete-component \ --component-build-version-arn $COMPONENT_BUILD_VERSION_ARN ## Remove the IAM role from Instance profile aws iam remove-role-from-instance-profile \ --instance-profile-name HelloWorldInstanceProfile \ --role-name HelloWorldIAMRole ## Delete the IAM instance profile aws iam delete-instance-profile \ --instance-profile-name HelloWorldInstanceProfile ## Remove IAM policy from the IAM role aws iam detach-role-policy \ --role-name HelloWorldIAMRole \ --policy-arn arn:aws:iam::aws:policy/AdministratorAccess ## Delete the IAM role aws iam delete-role \ --role-name HelloWorldIAMRole ## Delete EC2 key pair aws ec2 delete-key-pair \ --key-name HelloWorldKP ## Delete the S3 bucket with objects aws s3 rb \ s3://cloudaffaire-image-builder --force ## Delete the cloudwatch log-group aws logs delete-log-group \ --log-group-name "/aws/imagebuilder/HelloWorldRecipe" ## Delete the demo directory cd .. && rm -rf demo |
Hope you have enjoyed this article. To get more details in AWS Image Builder, please refer the below documentation.
https://docs.aws.amazon.com/imagebuilder/index.html