Question:
I connected the Lambda to EFS, and I want to download small file from S3 to the EFS using Lambda.
I connected the Lambda function to file system, and added access point with permissions 777.
I have this small python function in Lambda to download to the mount:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
import json import boto3 import time import os s3 = boto3.client('s3') def lambda_handler(event, context): s3.download_file('my_bucket', 'img.jpg', '/mnt/my-efs/img.jpg') return{ "statusCode": 200 } |
And I get timeout after 1 min (this should take less than a second).
If I download for /tmp/img.jpg, this works. Even if I copy the file from /tmp/ to /mnt/my-efs/ this works.
I have this IAM for the Lambda user:
|
1 2 3 4 5 6 7 |
AmazonEC2FullAccess AmazonS3FullAccess AmazonVPCFullAccess AWSLambdaVPCAccessExecutionRole AmazonElasticFileSysstemClientReadWriteAccess AmazonElasticFileSysstemClientFullAccess |
Why can’t I download file from S3 to EFS using Lambda?
Answer:
This get’s a big tricky so let’s go slow and make sure we cover all the moving parts. I apologize if some of this is redundant, but I wanted to make sure a layman would have all the right parts. Please let me know if you think I’ve missed something.
Steps:
serverless.yaml- AWS Access point settings
- Adding EFS access to the Lambda
- Downloading items into the EFS drive
- Verifying your files are in EFS
1. serverless.yaml
I’m using serverless so this is what my serverless.yaml file looks like.
|
1 2 3 4 5 6 7 8 9 |
iamRoleStatements: - Effect: Allow Action: - elasticfilesystem:ClientMount - elasticfilesystem:ClientWrite - elasticfilesystem:ClientRootAccess Resource: - arn:aws:elasticfilesystem:us-west-2:123456789:file-system/fs-123456789 |
These were the minimum permissions i needed to download the file from S3 into the EFS drive location. Again, this project uses serverless so all this can be translated to a cloudFormation template if you’re not using serverless.
And under the handler function…
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
functions: myFunc: handler: handler.handle package: exclude: - env/** - node_modules/** description: Lambda to download S3 file to SFTP EFS folder. events: - s3: bucket: some-bucket event: s3:ObjectCreated:* rules: - prefix: some-folder/ - suffix: .jpg existing: true fileSystemConfig: localMountPath: /mnt/my-efs arn: arn:aws:elasticfilesystem:us-west-2:123456789:access-point/fsap-12345678910 |
NOTE: I left out a lot from my serverless.yaml file, and only included the parts pertaining to the EFS setup.
2. AWS EFS Access Point Settings
-
- go to EFS service.
-
- click Access points.
-
- Create new Access point
-
- Fill out the fields as needed. Below is how i configured mine
- Details
File System:Choose a dropdown selectionName - optionalPick a nameRoot directory path - optional:This will map to your underlying file system on the EC2, or SFTP server. I’m using an sftp server, so I simply input the address on the sftp server.sftp.companyX.com/images. This location will map to ourLocal Mount Pathset inside your Lambda settings within the EFS configuration window (we’ll look more at that in step 3).
- POSIX User
User ID:1054| Locate your/etc/passwd/file and verify this to be true for your server hosting the EFS drive. I think1054is the default setting for amazon linux but if you’re running a different OS on your EFS host, you should verify this value.Group ID:1054(see last line)
- Root directory creation permissions
Owner User ID:1054(see last line)Owner Group ID:1054(see last line)POSIX permissions to apply to the root directory path:777| understand these permissions translate to youriamRoleStatements.Actionvalues; if they’re different this777should be set appropriately.
3. Lambdas addition of the EFS
EFS File System:(choose from dropdown selection)Local Mount Path:| SUPER IMPORTANT e.g./mnt/my-efs. This part is very non-intuitive IMO. This location, whatever you want it to be, will map to theRoot directory path(step 2.4 above) you configured when you build your access point. Meaning, if you download an s3 file calledimg.jpgto/mnt/my-efs/.it will put that file intosftp.company.com/images/img.jpg.
4. Downloading items into the EFS drive
- Obviously, this can be done many ways. Your original code looked perfectly fine. I’ve simply extended it to extract the file location from an S3 event object as an example. If it’s not obvious, the S3 event is triggered whenever the S3 bucket get’s a new upload (a new object added to the bucket).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
def download_files_to_efs(self, event): """ Locates the S3 file name (i.e. S3 object "key" value) the initiated the Lambda call, then downloads the file into the locally attached EFS drive at the target location. :param: event | S3 event record :return: dict """ key = event.get('Records')[0].get('s3').get('object').get('key') # some-bucket/some-folder/img.jpg efs_loci = f"/mnt/my-efs/images/{key.split('/')[-1]}" # '/mnt/my-efs/images/img.jpg result = self.s3.download('some-bucket', key, efs_loci) if result: print('Download Success...') else: print('Ask Stack Overflow :/') return { 'status_code': 200 } |
5. Verifying Your Files are in EFS
- To verify the download works, you can just print the files from your EFS location from within your Lambda then check you built-in CloudWatch logs to see the results are as expected.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
def get_files_from_path(file_path): """ Returns the files found at the file_path :param file_path: string :return list: """ # NOTE: file_path = '/my-efs/ found_files = [] for _, _, f in os.walk(file_path): found_files.append(f) if found_files: print('efs files: ', found_files) return found_files |
Special attention here! Notice i didn’t read from /mnt/my-efs/ i read from /my-efs, this is because the EFS / directory maps to the /mnt directory in the Lambda environment.
Finished 🚀
NOTE: If i’ve left something out feel free to comment and i’ll add that as well if i can. Unfortunately i can’t share the repo as it’s a work related project (protected).