How To Install And Update ECS Container Agent Using AWS CLI

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In the last AWS blog post, we have discussed how to create an ECS Container Instance with ECS-optimized AMI using AWS CLI..

https://cloudaffaire.com/how-to-create-an-ecs-container-instance-with-ecs-optimized-ami-using-aws-cli/

In this blog post, we will discuss ECS Container Agent. We will also learn how to install and update ECS container agent.

What is an ECS Container Agent?

The AWS ECS container agent allows container instances to connect to your cluster. The AWS ECS container agent is included in the AWS ECS-optimized AMIs, but you can also install it on any AWS EC2 instance that supports the AWS ECS specification.

Note: The AWS ECS container agent is only supported on AWS EC2 instances.

ECS Container Agent Configuration:

The AWS ECS container agent supports a number of configuration options, most of which should be set through environment variables. The following environment variables are available, and all of them are optional.

If your container instance was launched with a Linux variant of the AWS ECS-optimized AMI, you can set these environment variables in the /etc/ecs/ecs.config file and then restart the agent. You can also write these configuration variables to your container instances with AWS EC2 user data at launch time.

If you are manually starting the AWS ECS container agent (for non AWS ECS-optimized AMIs), you can use these environment variables in the docker run command that you use to start the agent. Use these variables with the syntax –env=VARIABLE_NAME=VARIABLE_VALUE. For sensitive information, such as authentication credentials for private repositories, you should store your agent environment variables in a file and pass them all at one time with the –env-file path_to_env_file option.

Available Container Agent Parameters:

ECS_CLUSTER: The cluster this agent should check into.
ECS_RESERVED_PORTS: An array of ports that should be marked as unavailable for scheduling on this container instance.
ECS_RESERVED_PORTS_UDP: An array of UDP ports that should be marked as unavailable for scheduling on this container instance.
ECS_ENGINE_AUTH_TYPE: The type of auth data that is stored in the ECS_ENGINE_AUTH_DATA key.
ECS_ENGINE_AUTH_DATA: Docker auth data formatted as defined by ECS_ENGINE_AUTH_TYPE.
AWS_DEFAULT_REGION: The region to be used in API requests as well as to infer the correct backend host.
AWS_ACCESS_KEY_ID: The access key used by the agent for all calls.
AWS_SECRET_ACCESS_KEY: The secret key used by the agent for all calls.
AWS_SESSION_TOKEN: The session token used for temporary credentials.
DOCKER_HOST: Used to create a connection to the Docker daemon; behaves similarly to this environment variable as used by the Docker client.
ECS_LOGLEVEL: The level of detail that should be logged.
ECS_LOGFILE: The location where logs should be written. Log level is controlled by ECS_LOGLEVEL.
ECS_CHECKPOINT: Whether to checkpoint state to the DATADIR specified below.
ECS_DATADIR: The container path where state is checkpointed for use across agent restarts.
ECS_UPDATES_ENABLED: Whether to exit for an updater to apply updates when requested.
ECS_UPDATE_DOWNLOAD_DIR: Where to place update tarballs within the container.
ECS_DISABLE_METRICS: Whether to disable metrics gathering for tasks.
ECS_POLL_METRICS: Whether to poll or stream when gathering metrics for tasks.
ECS_POLLING_METRICS_WAIT_DURATION: Time to wait to poll for new metrics for a task. Only used when ECS_POLL_METRICS is true
ECS_RESERVED_MEMORY: Memory, in MiB, to reserve for use by things other than containers managed by AWS ECS.
ECS_AVAILABLE_LOGGING_DRIVERS: Which logging drivers are available on the container instance.
ECS_DISABLE_PRIVILEGED: Whether launching privileged containers is disabled on the container instance.
ECS_SELINUX_CAPABLE: Whether SELinux is available on the container instance.
ECS_APPARMOR_CAPABLE: Whether AppArmor is available on the container instance.
ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION: Time to wait to delete containers for a stopped task. If set to less than 1 minute, the value is ignored.
ECS_CONTAINER_STOP_TIMEOUT: Instance scoped configuration for time to wait for the container to exit normally before being forcibly killed.
ECS_CONTAINER_START_TIMEOUT: Timeout before giving up on starting a container.
ECS_ENABLE_TASK_IAM_ROLE: Whether to enable IAM Roles for Tasks on the Container Instance
ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST: Whether to enable IAM Roles for Tasks when launched with host network mode on the Container Instance
ECS_DISABLE_IMAGE_CLEANUP: Whether to disable automated image cleanup for the ECS Agent.
ECS_IMAGE_CLEANUP_INTERVAL: The time interval between automated image cleanup cycles. If set to less than 10 minutes, the value is ignored.
ECS_IMAGE_MINIMUM_CLEANUP_AGE: The minimum time interval between when an image is pulled and when it can be considered for automated image cleanup.
NON_ECS_IMAGE_MINIMUM_CLEANUP_AGE: The minimum time interval between when a non ECS image is created and when it can be considered for automated image cleanup.
ECS_NUM_IMAGES_DELETE_PER_CYCLE: The maximum number of images to delete in a single automated image cleanup cycle. If set to less than 1, the value is ignored.
ECS_IMAGE_PULL_BEHAVIOR: The behavior used to customize the pull image process.
ECS_IMAGE_PULL_INACTIVITY_TIMEOUT: The time to wait after docker pulls complete waiting for extraction of a container. Useful for tuning large Windows containers.
ECS_INSTANCE_ATTRIBUTES: These attributes take effect only during initial registration.
ECS_ENABLE_TASK_ENI: Whether to enable task networking for task to be launched with its own network interface
ECS_ENABLE_HIGH_DENSITY_ENI: Whether to enable high density eni feature when using task networking
ECS_CNI_PLUGINS_PATH: The path where the cni binary file is located
ECS_AWSVPC_BLOCK_IMDS: Whether to block access to Instance Metadata for Tasks started with awsvpc network mode
ECS_AWSVPC_ADDITIONAL_LOCAL_ROUTES: In awsvpc network mode, traffic to these prefixes will be routed via the host bridge instead of the task ENI
ECS_ENABLE_CONTAINER_METADATA: When true, the agent will create a file describing the container’s metadata.
ECS_HOST_DATA_DIR: The source directory on the host from which ECS_DATADIR is mounted.
ECS_ENABLE_TASK_CPU_MEM_LIMIT: Whether to enable task-level cpu and memory limits
ECS_CGROUP_PATH: The root cgroup path that is expected by the ECS agent. This is the path that accessible from the agent mount.
ECS_CGROUP_CPU_PERIOD: CGroups CPU period for task level limits. This value should be between 8ms to 100ms
ECS_ENABLE_CPU_UNBOUNDED_WINDOWS_WORKAROUND: When true, ECS will allow CPU unbounded(CPU=0) tasks to run along with CPU bounded tasks in Windows.
ECS_ENABLE_MEMORY_UNBOUNDED_WINDOWS_WORKAROUND: When true, ECS will ignore the memory reservation parameter (soft limit) to run along with memory bounded tasks in Windows.
ECS_TASK_METADATA_RPS_LIMIT: Comma separated integer values for steady state and burst throttle limits for task metadata endpoint
ECS_SHARED_VOLUME_MATCH_FULL_CONFIG: When true, ECS Agent will compare name, driver options, and labels to make sure volumes are identical.
ECS_CONTAINER_INSTANCE_PROPAGATE_TAGS_FROM: If ec2_instance is specified, existing tags defined on the container instance will be registered to AWS ECS and will be discoverable using the ListTagsForResource API.
ECS_CONTAINER_INSTANCE_TAGS: The metadata that you apply to the container instance to help you categorize and organize them.
ECS_ENABLE_UNTRACKED_IMAGE_CLEANUP: Whether to allow the ECS agent to delete containers and images that are not part of ECS tasks.
ECS_EXCLUDE_UNTRACKED_IMAGE: Comma seperated list of imageName:tag of images that should not be deleted by the ECS agent if ECS_ENABLE_UNTRACKED_IMAGE_CLEANUP is enabled.
ECS_DISABLE_DOCKER_HEALTH_CHECK: Whether to disable the Docker Container health check for the ECS Agent.
ECS_NVIDIA_RUNTIME: The Nvidia Runtime to be used to pass Nvidia GPU devices to containers.
ECS_ENABLE_SPOT_INSTANCE_DRAINING: Whether to enable Spot Instance draining for the container instance.
ECS_LOG_ROLLOVER_TYPE: Determines whether the container agent logfile will be rotated based on size or hourly.
ECS_LOG_OUTPUT_FORMAT: Determines the log output format. When the json format is used, each line in the log would be a structured JSON map.
ECS_LOG_MAX_FILE_SIZE_MB: When the ECS_LOG_ROLLOVER_TYPE variable is set to size, this variable determines the maximum size (in MB) the log file before it is rotated.
ECS_LOG_MAX_ROLL_COUNT: Determines the number of rotated log files to keep. Older log files are deleted once this limit is reached.

Note: There are undocumented variables that the agent uses internally that may be visible but that are not intended for customer use.

Storing Container Instance Configuration in AWS S3:

AWS ECS container agent configuration is controlled with the environment variables described in the previous section. Linux variants of the AWS ECS-optimized AMI look for these variables in /etc/ecs/ecs.config when the container agent starts and configure the agent accordingly. Certain innocuous environment variables, such as ECS_CLUSTER, can be passed to the container instance at launch through AWS EC2 user data and written to this file without consequence. However, other sensitive information, such as your AWS credentials or the ECS_ENGINE_AUTH_DATA variable, should never be passed to an instance in user data or written to /etc/ecs/ecs.config in a way that would allow them to show up in a .bash_history file.

Storing configuration information in a private bucket in AWS S3 and granting read-only access to your container instance IAM role is a secure and convenient way to allow container instance configuration at launch. You can store a copy of your ecs.config file in a private bucket. You can then use AWS EC2 user data to install the AWS CLI and copy your configuration information to /etc/ecs/ecs.config when the instance launches.

Private Registry Authentication for Container Instances:

The AWS ECS container agent can authenticate with private registries, including Docker Hub, using basic authentication. When you enable private registry authentication, you can use private Docker images in your task definitions. This feature is only supported by tasks using the EC2 launch type.

Another method of enabling private registry authentication uses AWS Secrets Manager to store your private registry credentials securely and then reference them in your container definition. This allows your tasks to use images from private repositories. This method supports tasks using either the EC2 or Fargate launch types.

The AWS ECS container agent below two environment variables for authentication:

ECS_ENGINE_AUTH_TYPE, which specifies the type of authentication data that is being sent.
ECS_ENGINE_AUTH_DATA, which contains the actual authentication credentials.

Automated Task and Image Cleanup:

Each time a task is placed on a container instance, the AWS ECS container agent checks to see if the images referenced in the task are the most recent of the specified tag in the repository. If not, the default behavior allows the agent to pull the images from their respective repositories. If you frequently update the images in your tasks and services, your container instance storage can quickly fill up with Docker images that you are no longer using and may never use again. For example, you may use a continuous integration and continuous deployment (CI/CD) pipeline.

Likewise, containers that belong to stopped tasks can also consume container instance storage with log information, data volumes, and other artifacts. These artifacts are useful for debugging containers that have stopped unexpectedly, but most of this storage can be safely freed up after a period of time.

Note: By default, the AWS ECS container agent automatically cleans up stopped tasks and Docker images that are not being used by any tasks on your container instances.

AWS ECS Container Metadata File:

Beginning with version 1.15.0 of the AWS ECS container agent, various container metadata is available within your containers or the host container instance. By enabling this feature, you can query the information about a task, container, and container instance from within the container or the host container instance. The metadata file is created on the host instance and mounted in the container as a Docker volume.

This feature is disabled by default. You can enable container metadata at the container instance level by setting the ECS_ENABLE_CONTAINER_METADATA container agent variable to true. You can set this variable in the /etc/ecs/ecs.config configuration file and restart the agent. You can also set it as a Docker environment variable at runtime when the agent container is started.

By default, the container metadata file is written to the following host and container paths.

For Linux instances:
- Host path: /var/lib/ecs/data/metadata/cluster_name/task_id/container_name/ecs-container-metadata.json
- Container path: /opt/ecs/metadata/random_ID/ecs-container-metadata.json
For Windows instances:
- Host path: C:\ProgramData\AWS\ECS\data\metadata\task_id\container_name\ecs-container-metadata.json
- Container path: C:\ProgramData\AWS\ECS\metadata\random_ID\ecs-container-metadata.json

How To Install And Update ECS Container Agent Using AWS CLI:

Step 1: Create a custom VPC for your ECS container instance.

###############################################
## Install And Configure ECS Container Agent ##
###############################################
## Prerequisite: AWS CLI installed and configured with proper access
## https://cloudaffaire.com/category/aws/aws-cli/

##----------------------------------------------
## Create custom vpc for your container instance 
##----------------------------------------------

## Create a directory for this demo
mkdir ecs_container_agent_demo && cd ecs_container_agent_demo

## Create a VPC with DNS hostname enabled
AWS_VPC_ID=$(aws ec2 create-vpc \
--cidr-block 10.0.0.0/16 \
--query 'Vpc.{VpcId:VpcId}' \
--output text) &&
aws ec2 modify-vpc-attribute \
--vpc-id $AWS_VPC_ID \
--enable-dns-hostnames "{\"Value\":true}"

## Create one public subnet
AWS_SUBNET_PUBLIC_ID=$(aws ec2 create-subnet \
--vpc-id $AWS_VPC_ID --cidr-block 10.0.1.0/24 \
--availability-zone ap-south-1a --query 'Subnet.{SubnetId:SubnetId}' \
--output text) &&
aws ec2 modify-subnet-attribute \
  --subnet-id $AWS_SUBNET_PUBLIC_ID \
  --map-public-ip-on-launch

## Create an Internet Gateway and attach to the VPC
AWS_INTERNET_GATEWAY_ID=$(aws ec2 create-internet-gateway \
--query 'InternetGateway.{InternetGatewayId:InternetGatewayId}' \
--output text) &&
aws ec2 attach-internet-gateway \
--vpc-id $AWS_VPC_ID \
--internet-gateway-id $AWS_INTERNET_GATEWAY_ID

## Create a custom route table with route to internet gateway and explicite assosiation to public subnets
AWS_CUSTOM_ROUTE_TABLE_ID=$(aws ec2 create-route-table \
  --vpc-id $AWS_VPC_ID \
  --query 'RouteTable.{RouteTableId:RouteTableId}' \
  --output text ) &&
aws ec2 create-route \
--route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \
--destination-cidr-block 0.0.0.0/0 \
--gateway-id $AWS_INTERNET_GATEWAY_ID &&
AWS_ROUTE_TABLE_ASSOID_ONE=$(aws ec2 associate-route-table  \
--subnet-id $AWS_SUBNET_PUBLIC_ID \
--route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \
--query 'AssociationId' \
--output text) 

## Create a custom security group with ingress rules
AWS_CUSTOM_SECURITY_GROUP_ID=$(aws ec2 create-security-group \
--vpc-id $AWS_VPC_ID \
--group-name myvpc-security-group \
--description 'My VPC non default security group' \
--query 'GroupId' \
--output text) &&
aws ec2 authorize-security-group-ingress \
--group-id $AWS_CUSTOM_SECURITY_GROUP_ID \
--ip-permissions '[{"IpProtocol": "tcp", "FromPort": 22, "ToPort": 22, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow SSH"}]}]' &&
aws ec2 authorize-security-group-ingress \
--group-id $AWS_CUSTOM_SECURITY_GROUP_ID \
--ip-permissions '[{"IpProtocol": "tcp", "FromPort": 80, "ToPort": 80, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow HTTP"}]}]' &&
aws ec2 authorize-security-group-ingress \
--group-id $AWS_CUSTOM_SECURITY_GROUP_ID \
--ip-permissions '[{"IpProtocol": "tcp", "FromPort": 443, "ToPort": 443, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow HTTPS"}]}]'

###############################################

## Install And Configure ECS Container Agent ##

###############################################

## Prerequisite: AWS CLI installed and configured with proper access

## https://cloudaffaire.com/category/aws/aws-cli/

##----------------------------------------------

## Create custom vpc for your container instance

##----------------------------------------------

## Create a directory for this demo

mkdir ecs_container_agent_demo && cd ecs_container_agent_demo

## Create a VPC with DNS hostname enabled

AWS_VPC_ID=$(aws ec2 create-vpc \

--cidr-block 10.0.0.0/16 \

--query 'Vpc.{VpcId:VpcId}' \

--output text) &&

aws ec2 modify-vpc-attribute \

--vpc-id $AWS_VPC_ID \

--enable-dns-hostnames "{\"Value\":true}"

## Create one public subnet

AWS_SUBNET_PUBLIC_ID=$(aws ec2 create-subnet \

--vpc-id $AWS_VPC_ID --cidr-block 10.0.1.0/24 \

--availability-zone ap-south-1a --query 'Subnet.{SubnetId:SubnetId}' \

--output text) &&

aws ec2 modify-subnet-attribute \

--subnet-id $AWS_SUBNET_PUBLIC_ID \

--map-public-ip-on-launch

## Create an Internet Gateway and attach to the VPC

AWS_INTERNET_GATEWAY_ID=$(aws ec2 create-internet-gateway \

--query 'InternetGateway.{InternetGatewayId:InternetGatewayId}' \

--output text) &&

aws ec2 attach-internet-gateway \

--vpc-id $AWS_VPC_ID \

--internet-gateway-id $AWS_INTERNET_GATEWAY_ID

## Create a custom route table with route to internet gateway and explicite assosiation to public subnets

AWS_CUSTOM_ROUTE_TABLE_ID=$(aws ec2 create-route-table \

--vpc-id $AWS_VPC_ID \

--query 'RouteTable.{RouteTableId:RouteTableId}' \

--output text ) &&

aws ec2 create-route \

--route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \

--destination-cidr-block 0.0.0.0/0 \

--gateway-id $AWS_INTERNET_GATEWAY_ID &&

AWS_ROUTE_TABLE_ASSOID_ONE=$(aws ec2 associate-route-table \

--subnet-id $AWS_SUBNET_PUBLIC_ID \

--route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \

--query 'AssociationId' \

--output text)

## Create a custom security group with ingress rules

AWS_CUSTOM_SECURITY_GROUP_ID=$(aws ec2 create-security-group \

--vpc-id $AWS_VPC_ID \

--group-name myvpc-security-group \

--description 'My VPC non default security group' \

--query 'GroupId' \

--output text) &&

aws ec2 authorize-security-group-ingress \

--group-id $AWS_CUSTOM_SECURITY_GROUP_ID \

--ip-permissions '[{"IpProtocol": "tcp", "FromPort": 22, "ToPort": 22, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow SSH"}]}]' &&

aws ec2 authorize-security-group-ingress \

--group-id $AWS_CUSTOM_SECURITY_GROUP_ID \

--ip-permissions '[{"IpProtocol": "tcp", "FromPort": 80, "ToPort": 80, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow HTTP"}]}]' &&

aws ec2 authorize-security-group-ingress \

--group-id $AWS_CUSTOM_SECURITY_GROUP_ID \

--ip-permissions '[{"IpProtocol": "tcp", "FromPort": 443, "ToPort": 443, "IpRanges": [{"CidrIp": "0.0.0.0/0", "Description": "Allow HTTPS"}]}]'

Step 2: Create an IAM role for your ECS container instance.

##---------------------------------------------------------
## Create a custom IAM Role for your ECS Container Instance
##---------------------------------------------------------

## Create a policy for ecsInstanceRole IAM Role
vi ecsInstanceRolePolicy.json
-----------------
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.AWSaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
------------------
:wq

## Create an IAM EC2 service role and attach AWSEC2ContainerServiceforEC2Role policy
AWS_IAM_ROLE_ARN=$(aws iam create-role --role-name ecsInstanceRole \
--assume-role-policy-document file://ecsInstanceRolePolicy.json \
--query 'Role.Arn' \
--output text) &&
aws iam attach-role-policy \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSEC2ContainerServiceforEC2Role \
--role-name ecsInstanceRole 

## Create an IAM Instance Profile for Your ECS EC2 Instances using ecsInstanceRole role
aws iam create-instance-profile \
--instance-profile-name ecsInstanceProfileRole &&
aws iam add-role-to-instance-profile \
--instance-profile-name ecsInstanceProfileRole \
--role-name ecsInstanceRole &&
AWS_IAM_INSTANCE_PROFILE_ARN=$(aws iam list-instance-profiles-for-role \
--role-name ecsInstanceRole \
--query 'InstanceProfiles[0].Arn' \
--output text)

##---------------------------------------------------------

## Create a custom IAM Role for your ECS Container Instance

##---------------------------------------------------------

## Create a policy for ecsInstanceRole IAM Role

vi ecsInstanceRolePolicy.json

-----------------

{

"Version": "2008-10-17",

"Statement": [

{

"Effect": "Allow",

"Principal": {

"Service": "ec2.AWSaws.com"

"Action": "sts:AssumeRole"

}

]

}

------------------

:wq

## Create an IAM EC2 service role and attach AWSEC2ContainerServiceforEC2Role policy

AWS_IAM_ROLE_ARN=$(aws iam create-role --role-name ecsInstanceRole \

--assume-role-policy-document file://ecsInstanceRolePolicy.json \

--query 'Role.Arn' \

--output text) &&

aws iam attach-role-policy \

--policy-arn arn:aws:iam::aws:policy/service-role/AWSEC2ContainerServiceforEC2Role \

--role-name ecsInstanceRole

## Create an IAM Instance Profile for Your ECS EC2 Instances using ecsInstanceRole role

aws iam create-instance-profile \

--instance-profile-name ecsInstanceProfileRole &&

aws iam add-role-to-instance-profile \

--instance-profile-name ecsInstanceProfileRole \

--role-name ecsInstanceRole &&

AWS_IAM_INSTANCE_PROFILE_ARN=$(aws iam list-instance-profiles-for-role \

--role-name ecsInstanceRole \

--query 'InstanceProfiles[0].Arn' \

--output text)

Step 3: Create your ECS container instance.

##-----------------------
## Create an ec2 instance
##-----------------------

## Get AWS Linux 2 latest AMI ID
AWS_AMI_ID=$(aws ec2 describe-images \
--owners 'AWS' \
--filters 'Name=name,Values=amzn2-ami-hvm-2.0.????????-x86_64-gp2' 'Name=state,Values=available' \
--query 'sort_by(Images, &CreationDate)[-1].[ImageId]' \
--output 'text')

## Create a key-pair
aws ec2 create-key-pair \
--key-name myvpc-keypair \
--query 'KeyMaterial' \
--output text > myvpc-keypair.pem &&
chmod 400 myvpc-keypair.pem

## Create one EC2 instance in the public subnet
AWS_EC2_INSTANCE_ID=$(aws ec2 run-instances \
--image-id $AWS_AMI_ID \
--instance-type t2.micro \
--key-name myvpc-keypair \
--monitoring "Enabled=false" \
--security-group-ids $AWS_CUSTOM_SECURITY_GROUP_ID \
--subnet-id $AWS_SUBNET_PUBLIC_ID \
--iam-instance-profile Arn=$AWS_IAM_INSTANCE_PROFILE_ARN \
--private-ip-address 10.0.1.10 \
--query 'Instances[0].InstanceId' \
--output text)

## Check if the instance one is running
## It will take some time for the instance to get ready
aws ec2 describe-instance-status \
--instance-ids $AWS_EC2_INSTANCE_ID --output text

##-----------------------

## Create an ec2 instance

##-----------------------

## Get AWS Linux 2 latest AMI ID

AWS_AMI_ID=$(aws ec2 describe-images \

--owners 'AWS' \

--filters 'Name=name,Values=amzn2-ami-hvm-2.0.????????-x86_64-gp2' 'Name=state,Values=available' \

--query 'sort_by(Images, &CreationDate)[-1].[ImageId]' \

--output 'text')

## Create a key-pair

aws ec2 create-key-pair \

--key-name myvpc-keypair \

--query 'KeyMaterial' \

--output text > myvpc-keypair.pem &&

chmod 400 myvpc-keypair.pem

## Create one EC2 instance in the public subnet

AWS_EC2_INSTANCE_ID=$(aws ec2 run-instances \

--image-id $AWS_AMI_ID \

--instance-type t2.micro \

--key-name myvpc-keypair \

--monitoring "Enabled=false" \

--security-group-ids $AWS_CUSTOM_SECURITY_GROUP_ID \

--subnet-id $AWS_SUBNET_PUBLIC_ID \

--iam-instance-profile Arn=$AWS_IAM_INSTANCE_PROFILE_ARN \

--private-ip-address 10.0.1.10 \

--query 'Instances[0].InstanceId' \

--output text)

## Check if the instance one is running

## It will take some time for the instance to get ready

aws ec2 describe-instance-status \

--instance-ids $AWS_EC2_INSTANCE_ID --output text

Step 4: Install and configure ECS Container Agent.

##--------------------------------------------------------------
## Install and configure ecs container agent on the ec2 instance
##--------------------------------------------------------------

## Create your ECS cluster
aws ecs create-cluster \
--cluster-name myecscluster

## Get the public ip address of your ec2 instance
AWS_EC2_PUBLIC_IP=$(aws ec2 describe-instances \
--instance-ids $AWS_EC2_INSTANCE_ID \
--query 'Reservations[0].Instances[0].PublicIpAddress' \
--output text) && echo $AWS_EC2_PUBLIC_IP

## 13.235.103.30

## SSH into your ec2 instance (replace the public ip from above output)
ssh -i myvpc-keypair.pem ec2-user@13.235.103.30

## Disable the docker AWS Linux extra repository. The ecs AWS Linux 
## extra repository ships with its own version of Docker, so the docker 
## extra must be disabled to avoid any potential future conflicts.
sudo AWS-linux-extras disable docker

## Install and enable the ecs AWS Linux extra repository.
sudo AWS-linux-extras install -y ecs; sudo systemctl enable --now ecs

## Check to see if your agent uses the ECS_DATADIR environment variable to save its state.
sudo docker inspect ecs-agent | grep ECS_DATADIR

## "ECS_DATADIR=/data",

## Stop the AWS ECS container agent
sudo docker stop ecs-agent

## Create ecs config file
sudo mkdir -p /etc/ecs

sudo vi /etc/ecs/ecs.config
----------------------
ECS_DATADIR=/data
ECS_ENABLE_TASK_IAM_ROLE=true
ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true
ECS_LOGFILE=/log/ecs-agent.log
ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs"]
ECS_LOGLEVEL=info
ECS_CLUSTER=myecscluster
----------------------
:wq

## Start the AWS ECS container agent
sudo docker start ecs-agent

## You can verify that the agent is running and see some information about your new container instance
curl -s http://localhost:51678/v1/metadata | python -mjson.tool

# {
#    "Cluster": "myecscluster",
#    "ContainerInstanceArn": "<YOUR_CONTAINER_INSTANCE_ARN>",
#    "Version": "AWS ECS Agent - v1.33.0 (0d54fd61)"
# }

## Exit the container instance
exit

##--------------------------------------------------------------

## Install and configure ecs container agent on the ec2 instance

##--------------------------------------------------------------

## Create your ECS cluster

aws ecs create-cluster \

--cluster-name myecscluster

## Get the public ip address of your ec2 instance

AWS_EC2_PUBLIC_IP=$(aws ec2 describe-instances \

--instance-ids $AWS_EC2_INSTANCE_ID \

--query 'Reservations[0].Instances[0].PublicIpAddress' \

--output text) && echo $AWS_EC2_PUBLIC_IP

## 13.235.103.30

## SSH into your ec2 instance (replace the public ip from above output)

ssh -i myvpc-keypair.pem ec2-user@13.235.103.30

## Disable the docker AWS Linux extra repository. The ecs AWS Linux

## extra repository ships with its own version of Docker, so the docker

## extra must be disabled to avoid any potential future conflicts.

sudo AWS-linux-extras disable docker

## Install and enable the ecs AWS Linux extra repository.

sudo AWS-linux-extras install -y ecs; sudo systemctl enable --now ecs

## Check to see if your agent uses the ECS_DATADIR environment variable to save its state.

sudo docker inspect ecs-agent | grep ECS_DATADIR

## "ECS_DATADIR=/data",

## Stop the AWS ECS container agent

sudo docker stop ecs-agent

## Create ecs config file

sudo mkdir -p /etc/ecs

sudo vi /etc/ecs/ecs.config

----------------------

ECS_DATADIR=/data

ECS_ENABLE_TASK_IAM_ROLE=true

ECS_ENABLE_TASK_IAM_ROLE_NETWORK_HOST=true

ECS_LOGFILE=/log/ecs-agent.log

ECS_AVAILABLE_LOGGING_DRIVERS=["json-file","awslogs"]

ECS_LOGLEVEL=info

ECS_CLUSTER=myecscluster

----------------------

:wq

## Start the AWS ECS container agent

sudo docker start ecs-agent

## You can verify that the agent is running and see some information about your new container instance

curl -s http://localhost:51678/v1/metadata | python -mjson.tool

# {

# "Cluster": "myecscluster",

# "ContainerInstanceArn": "<YOUR_CONTAINER_INSTANCE_ARN>",

# "Version": "AWS ECS Agent - v1.33.0 (0d54fd61)"

# }

## Exit the container instance

exit

Step 5: Get details of your ECS container instance.

##----------------------------------------
## Get details for your container instance
##----------------------------------------

## List your ECS container instance
aws ecs list-container-instances \
--cluster myecscluster

## Get the details of ecs container instance
## If you have multiple ecs container instance, provide ARN of 
## the ecs container instance for which you want to get details
AWS_ECS_CONTAINER_INSTANCE_ARN=$(aws ecs list-container-instances \
--cluster myecscluster \
--query 'containerInstanceArns' \
--output text) &&
aws ecs describe-container-instances \
--cluster myecscluster \
--container-instances $AWS_ECS_CONTAINER_INSTANCE_ARN

##----------------------------------------

## Get details for your container instance

##----------------------------------------

## List your ECS container instance

aws ecs list-container-instances \

--cluster myecscluster

## Get the details of ecs container instance

## If you have multiple ecs container instance, provide ARN of

## the ecs container instance for which you want to get details

AWS_ECS_CONTAINER_INSTANCE_ARN=$(aws ecs list-container-instances \

--cluster myecscluster \

--query 'containerInstanceArns' \

--output text) &&

aws ecs describe-container-instances \

--cluster myecscluster \

--container-instances $AWS_ECS_CONTAINER_INSTANCE_ARN

Note: If you manually installing the ECS Container Agent, the container agent may not be of the latest version. Next, we are going to manually update our ECS Container Agent.

How To Install And Update ECS Container Agent Using AWS CLI

Step 6: Update your ECS Container Agent.

##--------------------------------
## Update your ECS container agent
##--------------------------------

## Get the public ip address of your ec2 instance
AWS_EC2_PUBLIC_IP=$(aws ec2 describe-instances \
--instance-ids $AWS_EC2_INSTANCE_ID \
--query 'Reservations[0].Instances[0].PublicIpAddress' \
--output text) && echo $AWS_EC2_PUBLIC_IP

## 13.235.103.30

## SSH into your ec2 instance (replace the public ip from above output)
ssh -i myvpc-keypair.pem ec2-user@13.235.103.30

## Pull the latest ECS container agent image from Docker Hub.
sudo docker pull AWS/AWS-ecs-agent:latest

## Stop the ECS container agent
sudo docker container stop ecs-agent

## Delete the ECS container agent
sudo docker container rm ecs-agent

## Check if new container with latest image spin up
sudo docker container ls

## Exit from the container instance
exit

##--------------------------------

## Update your ECS container agent

##--------------------------------

## Get the public ip address of your ec2 instance

AWS_EC2_PUBLIC_IP=$(aws ec2 describe-instances \

--instance-ids $AWS_EC2_INSTANCE_ID \

--query 'Reservations[0].Instances[0].PublicIpAddress' \

--output text) && echo $AWS_EC2_PUBLIC_IP

## 13.235.103.30

## SSH into your ec2 instance (replace the public ip from above output)

ssh -i myvpc-keypair.pem ec2-user@13.235.103.30

## Pull the latest ECS container agent image from Docker Hub.

sudo docker pull AWS/AWS-ecs-agent:latest

## Stop the ECS container agent

sudo docker container stop ecs-agent

## Delete the ECS container agent

sudo docker container rm ecs-agent

## Check if new container with latest image spin up

sudo docker container ls

## Exit from the container instance

exit

Our ECS Container Agent successfully updated to the latest version.

How To Install And Update ECS Container Agent Using AWS CLI

Step 7: Clenup.

##----------
## Clenup ##
##----------

## Deregister container instance
aws ecs deregister-container-instance \
--cluster myecscluster \
--container-instance $AWS_ECS_CONTAINER_INSTANCE_ARN \
--force 

## Delete your ECS cluster
aws ecs delete-cluster \
--cluster myecscluster

## Delete the ecs instance
aws ec2 terminate-instances \
--instance-ids $AWS_EC2_INSTANCE_ID

## Delete key pair
aws ec2 delete-key-pair \
--key-name myvpc-keypair

## Delete custom security group (once instances are terminated)
aws ec2 delete-security-group \
--group-id $AWS_CUSTOM_SECURITY_GROUP_ID

## Delete internet gateway
aws ec2 detach-internet-gateway \
--internet-gateway-id $AWS_INTERNET_GATEWAY_ID \
--vpc-id $AWS_VPC_ID &&
aws ec2 delete-internet-gateway \
--internet-gateway-id $AWS_INTERNET_GATEWAY_ID

## Disassociate the subnet from custom route table
aws ec2 disassociate-route-table \
--association-id $AWS_ROUTE_TABLE_ASSOID_ONE

## Delete custom route table
aws ec2 delete-route-table \
--route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID

## Delete the public subnet
aws ec2 delete-subnet \
--subnet-id $AWS_SUBNET_PUBLIC_ID

## Delete the vpc
aws ec2 delete-vpc \
--vpc-id $AWS_VPC_ID

## Remove instance profile & iam role
aws iam remove-role-from-instance-profile \
--instance-profile-name ecsInstanceProfileRole \
--role-name ecsInstanceRole &&
aws iam detach-role-policy \
--role-name ecsInstanceRole \
--policy-arn arn:aws:iam::aws:policy/service-role/AWSEC2ContainerServiceforEC2Role &&
aws iam delete-role \
--role-name ecsInstanceRole &&
aws iam delete-instance-profile \
--instance-profile-name ecsInstanceProfileRole

## Delete the directory used in this demo
cd .. && rm -rf ecs_container_agent_demo

##----------

## Clenup ##

##----------

## Deregister container instance

aws ecs deregister-container-instance \

--cluster myecscluster \

--container-instance $AWS_ECS_CONTAINER_INSTANCE_ARN \

--force

## Delete your ECS cluster

aws ecs delete-cluster \

--cluster myecscluster

## Delete the ecs instance

aws ec2 terminate-instances \

--instance-ids $AWS_EC2_INSTANCE_ID

## Delete key pair

aws ec2 delete-key-pair \

--key-name myvpc-keypair

## Delete custom security group (once instances are terminated)

aws ec2 delete-security-group \

--group-id $AWS_CUSTOM_SECURITY_GROUP_ID

## Delete internet gateway

aws ec2 detach-internet-gateway \

--internet-gateway-id $AWS_INTERNET_GATEWAY_ID \

--vpc-id $AWS_VPC_ID &&

aws ec2 delete-internet-gateway \

--internet-gateway-id $AWS_INTERNET_GATEWAY_ID

## Disassociate the subnet from custom route table

aws ec2 disassociate-route-table \

--association-id $AWS_ROUTE_TABLE_ASSOID_ONE

## Delete custom route table

aws ec2 delete-route-table \

--route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID

## Delete the public subnet

aws ec2 delete-subnet \

--subnet-id $AWS_SUBNET_PUBLIC_ID

## Delete the vpc

aws ec2 delete-vpc \

--vpc-id $AWS_VPC_ID

## Remove instance profile & iam role

aws iam remove-role-from-instance-profile \

--instance-profile-name ecsInstanceProfileRole \

--role-name ecsInstanceRole &&

aws iam detach-role-policy \

--role-name ecsInstanceRole \

--policy-arn arn:aws:iam::aws:policy/service-role/AWSEC2ContainerServiceforEC2Role &&

aws iam delete-role \

--role-name ecsInstanceRole &&

aws iam delete-instance-profile \

--instance-profile-name ecsInstanceProfileRole

## Delete the directory used in this demo

cd .. && rm -rf ecs_container_agent_demo

Hope you have enjoyed this article, In the next blog post, we will discuss ECS Task.

All the public cloud providers are changing the console user interface rapidly and due to this some of the screenshots used in our previous AWS blogs are no longer relevant. Hence, we have decided that from now onwards most of the demo will be done programmatically. Let us know your feedback on this in the comment section.

To get more details on AWS ECS, please refer below AWS documentation

https://docs.aws.amamzon.com/ecs/index.html

How To Install And Update ECS Container Agent Using AWS CLI

What is an ECS Container Agent?

ECS Container Agent Configuration:

Available Container Agent Parameters:

Storing Container Instance Configuration in AWS S3:

Private Registry Authentication for Container Instances:

Automated Task and Image Cleanup:

AWS ECS Container Metadata File:

How To Install And Update ECS Container Agent Using AWS CLI:

Step 1: Create a custom VPC for your ECS container instance.

Step 2: Create an IAM role for your ECS container instance.

Step 3: Create your ECS container instance.

Step 4: Install and configure ECS Container Agent.

Step 5: Get details of your ECS container instance.

Step 6: Update your ECS Container Agent.

Step 7: Clenup.

Leave a Reply Cancel reply