“docker-machine rm” failing on non-existent EC2 instance

Question:

I use docker-machine to manage Docker nodes on AWS. Today I tried to create a new node and failed due to an incompatibility between the instance type and specified AMI:

Whatever, I correct the error. My problem is that I seem to have created a ghost machine; if I run docker-machine ls, I get this:

I can’t remove it, kill it, restart it, or provision it; every command exits with a 400 error from the AWS driver:

How can I clear this dangling instance? I have many instances running under docker-machine, so I’d rather not do anything drastic, like reinstalling.

Edit:

Perhaps another way to frame this: where is docker-machine keeping track of managed hosts? Presumably there’s a file somewhere in my computer with a list of hosts, and when I run “docker-machine ls” it goes down that list and pings for status. Where is this file, and can I safely delete hosts that I know to be dead without leaving garbage elsewhere?

Answer:

Perhaps another way to frame this: where is docker-machine keeping track of managed hosts?

Check if you have an environment variable MACHINE_STORAGE_PATH.

If not, those VMS should be in ~/.docker.

You can see also the unresolved issue 3555:

Using docker-machine to provision a spot instance opens a spot request and waits for it to be fulfilled. If this fails, docker-machine exits but the spot request remains open and may be fulfilled later. This results in a dangling spot request with an active instance.

Spot instance support was discussed for docker-machine in issue 226.

From Spot Request, the main additional parameters seem to be Bid, Request valid From/To and Persistent request.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/images/spot_introduction.png

In the context of machine, I guess you could set request valid to = now + 2 minutes or so, at which point the command could fail.

The PR adding the support was PR 942 and PR 1000.
It is possible docker-machine rm does not adequately find/provide the extra parameters that a spot instance expects.

In addition to removing the corresponding directory from ~/.docker/machine/machines, you need to remove the corresponding EC2 key pair from the AWS EC2 console.

Leave a Reply