How to create and automatically mount AWS EFS storage in AWS EC2 instance?
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
Today we are going to discuss, how to create and automatically mount an Elastic File Share (EFS) to an EC2 instance using AWS CLI.
What is Amazon Elastic File System (EFS)?
Amazon Elastic File System (Amazon EFS) provides a simple, serverless, set-and-forget elastic file system for use with AWS Cloud services and on-premises resources. Amazon EFS supports the Network File System version 4 (NFSv4.1 and NFSv4.0) protocol, so the applications and tools that you use today work seamlessly with Amazon EFS. Multiple compute instances, including Amazon EC2, Amazon ECS, and AWS Lambda, can access an Amazon EFS file system at the same time, providing a common data source for workloads and applications running on more than one compute instance or server.
EFS storage classes:
Amazon EFS offers a range of storage classes that are designed for different use cases. These include EFS Standard, EFS Standard–Infrequent Access (Standard-IA), EFS One Zone, and EFS One Zone–Infrequent Access (EFS One Zone-IA).
EFS Standard and Standard-IA storage classes are regional storage classes that are designed to provide continuous availability to data, even when one or more Availability Zones in an AWS Region are unavailable. They offer the highest levels of availability and durability by storing file system data and metadata redundantly across multiple geographically separated Availability Zones within a Region.
EFS One Zone and One Zone–IA storage classes are designed to provide continuous availability to data within a single Availability Zone. The EFS One Zone storage classes store file system data and metadata redundantly within a single Availability Zone in an AWS Region. Because they store data in a single AWS Availability Zone, data that is stored in these storage classes might be lost in the event of a disaster or other fault that affects all copies of the data within the Availability Zone, or in the event of Availability Zone destruction.
EFS Throughput modes:
A file system’s throughput mode determines the throughput available to your file system. Amazon EFS offers two throughput modes, Bursting Throughput and Provisioned Throughput. Read throughput is discounted to allow you to drive higher read throughput than write throughput. Depending on the AWS Region, the discount for reads is between 1.66 and 3x.
Next, we will create a EFS file share and mount the EFS file share to an AWS EC2 instance.
Prerequisites:
AWS CLI installed and configured.
How to create and automatically mount AWS EFS storage in AWS EC2 instance?
Step 1: Create a EFS file share using AWS CLI.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
## Create a EFS fileshare aws efs create-file-system \ --performance-mode generalPurpose \ --throughput-mode bursting \ --no-encrypted \ --no-backup \ --availability-zone-name ap-south-1a \ --tags Key=Name,Value=myfilesystem ## Get details on the EFS file share aws efs describe-file-systems ## returns ## { ## "FileSystems": [ ## { ## "OwnerId": "<AWS_ACCOUNT_ID>", ## "CreationToken": "<SOME_TOKEN>", ## "FileSystemId": "<FILE_SYSTEM_ID>", ## "FileSystemArn": "<FILE_SYSTEM_ARN>", ## "CreationTime": "2022-03-14T19:14:51+05:30", ## "LifeCycleState": "available", ## "Name": "myfilesystem", ## "NumberOfMountTargets": 0, ## "SizeInBytes": { ## "Value": 6144, ## "ValueInIA": 0, ## "ValueInStandard": 6144 ## }, ## "PerformanceMode": "generalPurpose", ## "Encrypted": false, ## "ThroughputMode": "bursting", ## "AvailabilityZoneName": "ap-south-1a", ## "AvailabilityZoneId": "aps1-az1", ## "Tags": [ ## { ## "Key": "Name", ## "Value": "myfilesystem" ## } ## ] ## } ## ] ## } |
Step 2: Create a new VPC with public subnet and security group for EC2 and EFS.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 |
## Create a new VPC AWS_VPC_ID=$(aws ec2 create-vpc \ --cidr-block 10.0.0.0/16 \ --query 'Vpc.{VpcId:VpcId}' \ --output text) ## Create a public subnet AWS_SUBNET_PUBLIC_ID=$(aws ec2 create-subnet \ --vpc-id $AWS_VPC_ID --cidr-block 10.0.1.0/24 \ --availability-zone ap-south-1a --query 'Subnet.{SubnetId:SubnetId}' \ --output text) ## Enable Auto-assign Public IP on Public Subnet aws ec2 modify-subnet-attribute \ --subnet-id $AWS_SUBNET_PUBLIC_ID \ --map-public-ip-on-launch ## Create an Internet Gateway AWS_INTERNET_GATEWAY_ID=$(aws ec2 create-internet-gateway \ --query 'InternetGateway.{InternetGatewayId:InternetGatewayId}' \ --output text) ## Attach Internet gateway to your VPC aws ec2 attach-internet-gateway \ --vpc-id $AWS_VPC_ID \ --internet-gateway-id $AWS_INTERNET_GATEWAY_ID ## Create a route table AWS_CUSTOM_ROUTE_TABLE_ID=$(aws ec2 create-route-table \ --vpc-id $AWS_VPC_ID \ --query 'RouteTable.{RouteTableId:RouteTableId}' \ --output text ) ## Create route to Internet Gateway aws ec2 create-route \ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID \ --destination-cidr-block 0.0.0.0/0 \ --gateway-id $AWS_INTERNET_GATEWAY_ID ## Associate the public subnet with route table AWS_ROUTE_TABLE_ASSOID=$(aws ec2 associate-route-table \ --subnet-id $AWS_SUBNET_PUBLIC_ID \ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID | jq -r .AssociationId) ## Create two security groups aws ec2 create-security-group \ --group-name myvpc_ec2 \ --description "SG for EC2" \ --vpc-id $AWS_VPC_ID && aws ec2 create-security-group \ --group-name myvpc_efs \ --description "SG for EFS" \ --vpc-id $AWS_VPC_ID ## Get security group ID's EC2_SG_ID=$(aws ec2 describe-security-groups \ --filters "Name=vpc-id,Values=$AWS_VPC_ID" \ --query 'SecurityGroups[?GroupName == `myvpc_ec2`].GroupId' \ --output text) && EFS_SG_ID=$(aws ec2 describe-security-groups \ --filters "Name=vpc-id,Values=$AWS_VPC_ID" \ --query 'SecurityGroups[?GroupName == `myvpc_efs`].GroupId' \ --output text) ## Create security group ingress rules aws ec2 authorize-security-group-ingress \ --group-id $EC2_SG_ID \ --protocol tcp \ --port 22 \ --cidr 0.0.0.0/0 && aws ec2 authorize-security-group-ingress \ --group-id $EFS_SG_ID \ --protocol tcp \ --port 2049 \ --source-group $EC2_SG_ID |
Step 3: Create a EFS mount target using AWS CLI.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
## Create a mount target for your EFS FILE_SYSTEM_ID=$(aws efs describe-file-systems | jq -r .FileSystems[0].FileSystemId) && aws efs create-mount-target \ --file-system-id $FILE_SYSTEM_ID \ --subnet-id $AWS_SUBNET_PUBLIC_ID \ --security-groups $EFS_SG_ID ## Get details on EFS mount target aws efs describe-mount-targets \ --file-system-id $FILE_SYSTEM_ID ## returns ## { ## "MountTargets": [ ## { ## "OwnerId": "<AWS_ACCOUNT_ID>", ## "MountTargetId": "<MOUNT_TARGET_ID>", ## "FileSystemId": "<FILE_SYSTEM_ID>", ## "SubnetId": "<SUBNET_ID>", ## "LifeCycleState": "available", ## "IpAddress": "10.0.1.54", ## "NetworkInterfaceId": "<ENI_ID>", ## "AvailabilityZoneId": "aps1-az1", ## "AvailabilityZoneName": "ap-south-1a", ## "VpcId": "<VPC_ID>" ## } ## ] ## } ## Get details on EFS mount target security group MOUNT_TARGET_ID=$(aws efs describe-mount-targets \ --file-system-id $FILE_SYSTEM_ID | jq -r .MountTargets[0].MountTargetId) && aws efs describe-mount-target-security-groups \ --mount-target-id $MOUNT_TARGET_ID ## returns ## { ## "SecurityGroups": [ ## "<SECURITY_GROUP_ID>" ## ] ## } |
Step 4: Create a new EC2 instance and automatically mount the EFS file share to the EC2 instance using user data.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
## Get Amazon Linux 2 latest AMI ID AWS_AMI_ID=$(aws ec2 describe-images \ --owners 'amazon' \ --filters 'Name=name,Values=amzn2-ami-hvm-2.0.????????-x86_64-gp2' 'Name=state,Values=available' \ --query 'sort_by(Images, &CreationDate)[-1].[ImageId]' \ --output 'text') && echo $AWS_AMI_ID ## Create a key-pair aws ec2 create-key-pair \ --key-name myvpc-keypair \ --query 'KeyMaterial' \ --output text > myvpc-keypair.pem ## Change key pair permissions chmod 400 myvpc-keypair.pem ## Create user data auto mount efs on ec2 cat << 'EOF' > myuserdatatemp.txt #cloud-config package_update: true package_upgrade: true runcmd: - yum install -y amazon-efs-utils - apt-get -y install amazon-efs-utils - yum install -y nfs-utils - apt-get -y install nfs-common - file_system_id_1=__file_system_id__ - efs_mount_point_1=/mnt/efs/fs1 - mkdir -p "${efs_mount_point_1}" - test -f "/sbin/mount.efs" && printf "\n${file_system_id_1}:/ ${efs_mount_point_1} efs tls,_netdev\n" >> /etc/fstab || printf "\n${file_system_id_1}.efs.ap-south-1.amazonaws.com:/ ${efs_mount_point_1} nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,_netdev 0 0\n" >> /etc/fstab - test -f "/sbin/mount.efs" && grep -ozP 'client-info]\nsource' '/etc/amazon/efs/efs-utils.conf'; if [[ $? == 1 ]]; then printf "\n[client-info]\nsource=liw\n" >> /etc/amazon/efs/efs-utils.conf; fi; - retryCnt=15; waitTime=30; while true; do mount -a -t efs,nfs4 defaults; if [ $? = 0 ] || [ $retryCnt -lt 1 ]; then echo File system mounted successfully; break; fi; echo File system not available, retrying to mount.; ((retryCnt--)); sleep $waitTime; done; EOF ## replace the efs file system id sed "s/__file_system_id__/$FILE_SYSTEM_ID/g" myuserdatatemp.txt > myuserdata.txt ## Create an EC2 instance AWS_EC2_INSTANCE_ID=$(aws ec2 run-instances \ --image-id $AWS_AMI_ID \ --instance-type t2.micro \ --key-name myvpc-keypair \ --monitoring "Enabled=false" \ --security-group-ids $EC2_SG_ID \ --subnet-id $AWS_SUBNET_PUBLIC_ID \ --user-data file://myuserdata.txt \ --private-ip-address 10.0.1.10 \ --query 'Instances[0].InstanceId' \ --output text) |
Next, we will connect to the EC2 instance and create a new file in the EFS file share.
Step 5: Connect to the EC2 instance and create a file on EFS file share.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
## Check if EC2 instance is in running state aws ec2 describe-instance-status \ --instance-id $AWS_EC2_INSTANCE_ID | jq -r .InstanceStatuses[0].InstanceState.Name ## should be 'running' ## Connect to the EC2 instance EC2_PUBLIC_IP=$(aws ec2 describe-instances \ --instance-ids $AWS_EC2_INSTANCE_ID | jq -r .Reservations[0].Instances[0].PublicIpAddress) && ssh -i myvpc-keypair.pem -o "StrictHostKeyChecking=no" ec2-user@$EC2_PUBLIC_IP ## Create a file in your EFS storage cd /mnt/efs/fs1 sudo chmod go+rw . echo "hello world" > myfile.txt ls -l ## -rw-rw-r-- 1 ec2-user ec2-user 12 Mar 15 09:04 myfile.txt exit |
Step 6: Clean up.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
## Cleanup ## Terminate the ec2 instance aws ec2 terminate-instances \ --instance-ids $AWS_EC2_INSTANCE_ID && rm -f my* ## Delete key pair aws ec2 delete-key-pair \ --key-name myvpc-keypair ## Delete the efs mount target aws efs delete-mount-target \ --mount-target-id $MOUNT_TARGET_ID ## Delete the efs aws efs delete-file-system \ --file-system-id $FILE_SYSTEM_ID ## Delete custom security group aws ec2 delete-security-group \ --group-id $EFS_SG_ID && aws ec2 delete-security-group \ --group-id $EC2_SG_ID ## Delete internet gateway aws ec2 detach-internet-gateway \ --internet-gateway-id $AWS_INTERNET_GATEWAY_ID \ --vpc-id $AWS_VPC_ID && aws ec2 delete-internet-gateway \ --internet-gateway-id $AWS_INTERNET_GATEWAY_ID ## Delete the custom route table aws ec2 disassociate-route-table \ --association-id $AWS_ROUTE_TABLE_ASSOID && aws ec2 delete-route-table \ --route-table-id $AWS_CUSTOM_ROUTE_TABLE_ID ## Delete the public subnet aws ec2 delete-subnet \ --subnet-id $AWS_SUBNET_PUBLIC_ID ## Delete the vpc aws ec2 delete-vpc \ --vpc-id $AWS_VPC_ID |
Hope you have enjoyed this article. To get more details in AWS EFS, please refer the below documentation