Question:
I have an AutoScaling group where I’m messing around with LifecycleHooks on instance start, some controlled by me and some controlled by other AWS services. I’ve been adding/removing LifecycleHooks and changing the size of the ASG, and instances are getting stuck in Pending:Wait. I assume it’s because it’s waiting for a signal from a LifecycleHook, but I can’t get it to budge.
I tried a number of things:
Terminate instances in EC2 console
I terminated the instances, but ASG waits for the LifecycleHook heartbeat to timeout before actually terminating the instance from the ASG, which is probably an hour.
Complete LifecycleHooks manually
I tried to complete the LifecycleHooks manually, illustrated by the following pseudocode:
|
1 2 3 4 5 6 7 8 9 10 11 |
describeAutoScalingGroups -> asg { instances = getPending(asg.instances) describeLifecycleHooks -> lifecycleHooks { lifecycleHooks.each { instances.each { completeLifecycleAction(instance, hook) } } } } |
This doesn’t do the trick. I’m guessing the LifecycleHook it’s waiting on was deleted from the ASG, so there’s no way to manually complete the LifecycleHook.
What next?
Obviously, I should be more careful about deleting these resources in the right order and all that and decreasing the heartbeat will help as well, but how can I force terminate instances from an ASG no matter what it’s waiting for?
Answer:
After terminating EC2 instance in Console, manually completing LifecycleHook worked for me:
|
1 2 |
aws autoscaling complete-lifecycle-action --lifecycle-hook-name $HOOK --auto-scaling-group-name $ASG --lifecycle-action-result ABANDON --instance-id $ID |