How to configure dead letter queue in SQS
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have discussed how to enable delay queue in SQS.
https://cloudaffaire.com/how-to-configure-delay-queue-in-sqs/
In this blog post, we will discuss how to configure a dead letter queue in SQS.
What is dead letter queue?
A dead letter queue is used to debug your messaging application. Dead letter queue is used with the source queue to debug messages in source queue that for some reason (for example network issue) cannot be processed by your application. When you configure a dead letter queue for a source queue, you need to provide a redrive policy defining your source queue, dead letter queue, and the conditions under which AWS SQS will move the message from your source queue to dead letter queue. By default, the dead letter queue is not created when you create an SQS queue.
The expiration of a message is always based on its original enqueue timestamp. When a message is moved to a dead-letter queue, the enqueue timestamp remains unchanged. For example, if a message spends 1 day in the original queue before being moved to a dead-letter queue, and the retention period of the dead-letter queue is set to 4 days, the message is deleted from the dead-letter queue after 3 days. Thus, it is a best practice to always set the retention period of a dead-letter queue to be longer than the retention period of the original queue.
Note: The dead-letter queue of a FIFO queue must also be a FIFO queue. Similarly, the dead-letter queue of a standard queue must also be a standard queue. Also, the dead letter queue needs to be created in the same region as your source queue.
Next, we are going to configure dead letter queue in SQS through AWS CLI.
How to configure dead letter queue in SQS:
Prerequisite for this demo:
- AWS SQS FIFO queue.
- AWS CLI configured with required access.
Step 1: Configure AWS CLI.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
## Create an EC2 instance with AWS Linux 2 AMI ## Create a role with admin access and attach to your instance ## Install aws cli and jq package if not installed already ## Create a script to configure AWS cli vi assume_role.sh ----------------------- ## Replace #!/bin/bash curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ export AWS_ACCESS_KEY_ID=$(cat cred.json| jq .AccessKeyId | xargs) export AWS_SECRET_ACCESS_KEY=$(cat cred.json| jq .SecretAccessKey| xargs) export AWS_SESSION_TOKEN=$(cat cred.json| jq .Token| xargs) export AWS_EXPIRATION=$(cat cred.json| jq .Credentials.Expiration| xargs) rm -f cred.json ------------------------ :wq ## Execute the script chmod +x assume_role.sh sh assume_role.sh ## Check if aws cli working aws sts get-caller-identity |
Step 2: Create a FIFO queue that will be used as a dead letter queue for your source FIFO queue.
|
1 2 3 4 5 |
## create a FIFO queue that will serve as a dead-letter-queue for your source FIFO queue aws sqs create-queue \ --queue-name mydeadletterqueue.fifo \ --attributes '{"FifoQueue": "true"}' \ --region ap-south-1 |
Step 3: Query the list of queues available and copy the queue URL.
|
1 |
aws sqs list-queues --region ap-south-1 |
Note: ‘myqueue’ will be our source queue and ‘mydeadletterqueue’ will serve as dead letter queue to myqueue.
Step 4: Get Queue ARN for your dead letter queue.
|
1 2 3 4 5 |
## Get dead letter queue arn aws sqs get-queue-attributes \ --queue-url --attribute-names QueueArn \ --region ap-south-1 |
Step 5: Configure your source queue to enable dead letter queue.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
## Configure dead letter queue for your source queue ## Create json file for dead letter queue configuration vi configure-dead-letter-queue.json ------------------------ { "DelaySeconds": "0", "RedrivePolicy": "{\"deadLetterTargetArn\":\" "VisibilityTimeout": "5", "ReceiveMessageWaitTimeSeconds": "20" } ------------------------ :wq ## Configure dead letter queue aws sqs set-queue-attributes \ --queue-url --region ap-south-1 \ --attributes file://configure-dead-letter-queue.json ## Get current queue attributes for your FIFO queue aws sqs get-queue-attributes \ --queue-url --region ap-south-1 \ --attribute-names All |
Note: Max Receive Count is set to 3, meaning if we consume a message from source queue more than 3 times without deleting it, the message will be moved from source queue to dead letter queue.
Step 6: Send a message to the queue and then try to consume the message more than 3 times.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
## Send a message to your queue and try to consume it more than 3 times aws sqs send-message \ --queue-url --message-body "message one" \ --message-group-id "mymsggrpid1" \ --message-deduplication-id "mymsgddupid1" \ --region ap-south-1 ## Try to receive message four times aws sqs receive-message \ --queue-url --region ap-south-1 aws sqs receive-message \ --queue-url --region ap-south-1 aws sqs receive-message \ --queue-url --region ap-south-1 aws sqs receive-message \ --queue-url --region ap-south-1 ## Observe last receive-message will return an empty response as the message is moved to dead-letter-queue aws sqs receive-message \ --queue-url --region ap-south-1 ## message is now available in dead letter queue |
Observation: Since ‘MaxReceiveCount’ is set to 3 for our dead letter queue ‘RedrivePolicy’, the message is sent to dead letter queue after 3 ‘ReceiveMessage’ request.
Hope you have enjoyed this article. In the next blog post, we will discuss how to enable encryption in SQS queue.
To get more details on AWS SQS, please refer below AWS documentation
https://docs.aws.amazon.com/sqs/index.html
That was a really good post on configuring dead letter queue.Step by step tutorials always helps in getting thing clear.Thank you so much for this.