Kafka on AWS ECS, how to handle advertised.host without known instance?

Question 1

I’m trying to get Kafka running on an AWS ECS container. I have this setup already / working fine on my local docker environment, using the spotify/kafka image

To get this working locally, I needed to ensure the ADVERTISED_HOST environment variable was set. ADVERTISED_HOST needed to be set as the containers external IP, otherwise when I try to connect it was just giving me connection refused.

My local docker-compose.yaml has this for the kafka container:

  kafka:
    image: spotify/kafka
    hostname: kafka
    environment:
    - ADVERTISED_HOST=192.168.0.70
    - ADVERTISED_PORT=9092
    ports:
    - "9092:9092"
    - "2181:2181"
    restart: always

1

2

3

4

5

6

7

8

9

10

11

kafka:

image: spotify/kafka

hostname: kafka

environment:

- ADVERTISED_HOST=192.168.0.70

- ADVERTISED_PORT=9092

ports:

- "9092:9092"

- "2181:2181"

restart: always

Now the problem is, I don’t know what the IP is going to be, as I dont know which instance this will run on. So how do I set that environment variable?

Question 2

Your entrypoint script will need to call the EC2 Metadata Service on startup (in this case http://169.254.169.254/latest/meta-data/local-hostname) to get the external-to-docker hostname and set that variable.

Sample:

[ec2-user ~]$ curl http://169.254.169.254/latest/meta-data/local-hostname
ip-10-251-50-12.ec2.internal

1

2

3

[ec2-user ~]$ curl http://169.254.169.254/latest/meta-data/local-hostname

ip-10-251-50-12.ec2.internal

Kafka on AWS ECS, how to handle advertised.host without known instance?

Question:

Answer:

Leave a Reply Cancel reply