Question:
I’m trying to create a stack with CloudFormation. The stack needs to take some data files from a central S3 bucket and copy them to it’s own “local” bucket.
I’ve written a lambda function to do this, and it works when I run it in the Lambda console with a test event (the test event uses the real central repository and successfully copies the file to a specified repo).
My current CloudFormation script does the following things:
- Creates the “local” S3 bucket
- Creates a role that the Lambda function can use to access the buckets
- Defines the Lambda function to move the specified file to the “local” bucket
- Defines some Custom resources to invoke the Lambda function.
It’s at step 4 where it starts to go wrong – the Cloudformation execution seems to freeze here (CREATE_IN_PROGESS). Also, when I try to delete the stack, it seems to just get stuck on DELETE_IN_PROGRESS instead.
Here’s how I’m invoking the Lambda function in the CloudFormation script:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
"DataSync": { "Type": "Custom::S3DataSync", "Properties": { "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] }, "InputFile": "data/provided-as-ip-v6.json", "OutputFile": "data/data.json" } }, "KeySync1": { "Type": "Custom::S3DataSync", "Properties": { "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] }, "InputFile": "keys/1/public_key.pem" } }, "KeySync2": { "Type": "Custom::S3DataSync", "Properties": { "ServiceToken": { "Fn::GetAtt" : [ "S3DataSync", "Arn" ] }, "InputFile": "keys/2/public_key.pem" } } |
And the Lambda function itself:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
exports.handler = function(event, context) { var buckets = {}; buckets.in = { "Bucket":"central-data-repository", "Key":"sandbox" + "/" + event.ResourceProperties.InputFile }; buckets.out = { "Bucket":"sandbox-data", "Key":event.ResourceProperties.OutputFile || event.ResourceProperties.InputFile }; var AWS = require('aws-sdk'); var S3 = new AWS.S3(); S3.getObject(buckets.in, function(err, data) { if (err) { console.log("Couldn't get file " + buckets.in.Key); context.fail("Error getting file: " + err) } else { buckets.out.Body = data.Body; S3.putObject(buckets.out, function(err, data) { if (err) { console.log("Couln't write to S3 bucket " + buckets.out.Bucket); context.fail("Error writing file: " + err); } else { console.log("Successfully copied " + buckets.in.Key + " to " + buckets.out.Bucket + " at " + buckets.out.Key); context.succeed(); } }); } }); } |
Answer:
Your Custom Resource function needs to send signals back to CloudFormation to indicate completion, status, and any returned values. You will see CREATE_IN_PROGRESS as the status in CloudFormation until you notify it that your function is complete.
The generic way of signaling CloudFormation is to post a response to a pre-signed S3 URL. But there is a cfn-response module to make this easier in Lambda functions. Interestingly, the two examples provided for Lambda-backed Custom Resources use different methods:
- Walkthrough: Refer to Resources in Another Stack – uses the cfn-response module
- Walkthrough: Looking Up Amazon Machine Image IDs – uses pre-signed URLs.