Question:
CloudFormation creates Lambda function. When the function is executed an ENI is provisioned automatically by lambda. The ENI seems to be left in existence after function execution for to speed up subsequent function execution. CloudFormation deletes the lambda function. TheEN remains behind. When attempting to delete the VPC CloudFormation stack, stack deletion fails as the ENI is using a security group and subnet.
in my lambda role the delete permission are there.
“Effect”: “Allow”, “Action”: [ “ec2:CreateNetworkInterface”, “ec2:DeleteNetworkInterface”, “ec2:DescribeNetworkInterfaces” ], “Resource”: “*”
I am using custom resource to run the lambda from CloudFormation template, so lambda will be called both stack creation and deletion. The ENI will be used in creation of stack and deletion of stack. Now how to handle the eni deletion?
Answer:
There is a known issue when using Lambda Functions in a VPC, as documented in Configuring a Lambda Function to Access Resources in an Amazon VPC:
There is a delay between the time your Lambda function executes and ENI deletion. If you do delete the role immediately after function execution, you are responsible for deleting the ENIs.
The documentation doesn’t specify exactly how long this “delay” will be, but a forum post by Richard@AWS suggests it can last up to 6 hours(!). (In my observations using AWS CloudTrail, the delay between Lambda execution and ENI deletion was around one hour.)
Until AWS addresses this issue further, you can workaround the issue by detaching and deleting the leftover ENIs in between deleting the Lambda function and deleting the associated Security Group(s) and Subnet(s). This is how Terraform currently handles this issue in its framework.
You can do this manually by separating the VPC/Subnet/SG layer and the Lambda-function layer into two different CloudFormation Stacks, or you can automate it by implementing a Custom Resource to delete the ENIs using the AWS SDK.
Here’s a complete working example that creates a VPC-Lambda Custom Resource, cleaning up its ENIs when deleted using the VPCDestroyENI Custom Resource:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 |
Description: Creates a VPC-Lambda Custom Resource, cleaning up ENIs when deleted. Parameters: VPCId: Description: VPC Id Type: AWS::EC2::VPC::Id SubnetId: Description: Private Subnet Id Type: AWS::EC2::Subnet::Id Resources: SecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Lambda VPC security group VpcId: !Ref VPCId LambdaExecutionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Principal: {Service: [lambda.amazonaws.com]} Action: ['sts:AssumeRole'] Path: "/" ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole Policies: - PolicyName: DetachNetworkInterface PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: ['ec2:DetachNetworkInterface'] Resource: '*' AppendTest: Type: Custom::Split DependsOn: VPCDestroyENI Properties: ServiceToken: !GetAtt AppendItemToListFunction.Arn List: [1, 2, 3] AppendedItem: 4 AppendItemToListFunction: Type: AWS::Lambda::Function Properties: Handler: index.handler Role: !GetAtt LambdaExecutionRole.Arn Code: ZipFile: !Sub | var response = require('cfn-response'); exports.handler = function(event, context) { var responseData = {Value: event.ResourceProperties.List}; responseData.Value.push(event.ResourceProperties.AppendedItem); response.send(event, context, response.SUCCESS, responseData); }; Timeout: 30 Runtime: nodejs4.3 VpcConfig: SecurityGroupIds: [!Ref SecurityGroup] SubnetIds: [!Ref SubnetId] VPCDestroyENIFunction: Type: AWS::Lambda::Function Properties: Handler: index.handler Role: !GetAtt LambdaExecutionRole.Arn Code: ZipFile: !Sub | var response = require('cfn-response'); var AWS = require('aws-sdk'); exports.handler = function(event, context) { console.log("REQUEST RECEIVED:\n", JSON.stringify(event)); if (event.RequestType != 'Delete') { response.send(event, context, response.SUCCESS, {}); return; } var ec2 = new AWS.EC2(); var params = { Filters: [ { Name: 'group-id', Values: event.ResourceProperties.SecurityGroups }, { Name: 'description', Values: ['AWS Lambda VPC ENI: *'] } ] }; console.log("Deleting attachments!"); // Detach all network-interface attachments ec2.describeNetworkInterfaces(params).promise().then(function(data) { console.log("Got Interfaces:\n", JSON.stringify(data)); return Promise.all(data.NetworkInterfaces.map(function(networkInterface) { var networkInterfaceId = networkInterface.NetworkInterfaceId; var attachmentId = networkInterface.Attachment.AttachmentId; return ec2.detachNetworkInterface({AttachmentId: attachmentId}).promise().then(function(data) { return ec2.waitFor('networkInterfaceAvailable', {NetworkInterfaceIds: [networkInterfaceId]}).promise(); }).then(function(data) { console.log("Detached Interface, deleting:\n", networkInterfaceId); return ec2.deleteNetworkInterface({NetworkInterfaceId: networkInterfaceId}).promise(); }); })); }).then(function(data) { console.log("Success!"); response.send(event, context, response.SUCCESS, {}); }).catch(function(err) { console.log("Failure:\n", JSON.stringify(err)); response.send(event, context, response.FAILED, {}); }); }; Timeout: 300 Runtime: nodejs4.3 VPCDestroyENI: Type: Custom::VPCDestroyENI Properties: ServiceToken: !GetAtt VPCDestroyENIFunction.Arn SecurityGroups: [!Ref SecurityGroup] Outputs: Output: Description: output Value: !Join [",", !GetAtt AppendTest.Value] |
Note: To create the VPC and Private Subnet required in the above example, you can use the AWS Quick Start Amazon VPC Architecture template.