SQS queue default visibility timeout
Hello Everyone
Welcome to CloudAffaire and this is Debjeet.
In the last blog post, we have discussed basic operations like sending, receiving and deleting a message to an SQS queue.
https://cloudaffaire.com/sqs-queue-operations/
In this blog post, we will discuss default visibility timeout for your SQS queue.
SQS queue default visibility timeout:
AWS SQS does not delete a message automatically, thus there is a chance that same message is consumed and processed by two different consumers. To prevent this, AWS SQS apply visibility timeout. Visibility timeout starts when a consumer consumes a message and during the visibility timeout no other consumer is allowed to consume the same message. This prevents duplicate processing of same message. Visibility timeout can be set to the entire queue or to individual messages. The default visibility timeout setting for a queue is 30 seconds.
You can change this setting for the entire queue. Typically, you should set the visibility timeout to the maximum time that it takes your application to process and delete a message from the queue. When receiving messages, you can also set a special visibility timeout (ChangeMessageVisibility) for the returned messages without changing the overall queue timeout.
Note: For standard queues, the visibility timeout isn’t a guarantee against receiving a message twice.
Next, we are going to explain visibility timeout through a demo.
Prerequisite for this demo:
- AWS SQS FIFO queue.
- AWS CLI configured with required access.
Step 1: Configure AWS CLI.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
## Create an EC2 instance with AWS Linux 2 AMI ## Create a role with admin access and attach to your instance ## Install aws cli and jq package if not already installed ## Create a script to configure AWS cli vi assume_role.sh ----------------------- ## Replace #!/bin/bash curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ export AWS_ACCESS_KEY_ID=$(cat cred.json| jq .AccessKeyId | xargs) export AWS_SECRET_ACCESS_KEY=$(cat cred.json| jq .SecretAccessKey| xargs) export AWS_SESSION_TOKEN=$(cat cred.json| jq .Token| xargs) export AWS_EXPIRATION=$(cat cred.json| jq .Credentials.Expiration| xargs) rm -f cred.json ------------------------ :wq ## Execute the script chmod +x assume_role.sh sh assume_role.sh ## Check if aws cli working aws sts get-caller-identity |
Step 2: Query the list of queues available and copy the queue URL.
|
1 |
aws sqs list-queues --region ap-south-1 |
Step 3: Get the current queue attributes.
|
1 2 3 4 5 |
## Get current queue attributes for your queue aws sqs get-queue-attributes \ --queue-url --region ap-south-1 \ --attribute-names All |
Note: Current queue visibility timeout is set to default 30 seconds.
Step 3: Send a message to the queue.
|
1 2 3 4 5 6 7 |
## Send a message to your queue aws sqs send-message \ --queue-url --message-body "message one" \ --message-group-id "mymsggrpid1" \ --message-deduplication-id "mymsgddupid1" \ --region ap-south-1 |
Step 4: Now try to receive the message. Execute below command couple of times within 30 seconds.
|
1 2 3 4 |
## receive a message from the queue aws sqs receive-message \ --queue-url --region ap-south-1 |
Observation: The 1st time a consumer receives the message, the visibility timeout starts and during the next 30 seconds, no other consumer can receive the same message (highlighted above). Only after 30 seconds is passed and provided no other consumer consumes your message, you can receive the message again.
Next, we are going to reduce the visibility timeout value to 1 second which in turn will reduce this timeout gap.
Step 5: Change the default visibility timeout value to 1 second for the entire queue and try to receive the message a couple of times.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
## Change default visibility timeout to 1 second for the entire queue aws sqs set-queue-attributes \ --queue-url --region ap-south-1 \ --attributes '{"VisibilityTimeout": "1"}' ## Try to receive a message from the queue couple of times aws sqs receive-message \ --queue-url --region ap-south-1 aws sqs receive-message \ --queue-url --region ap-south-1 |
Note: Since we have changed the visibility timeout to 1 second, the message is available and can be re-consume instantaneously once consumed.
Hope you have enjoyed this article. In the next blog post, we will discuss the SQS queue long polling option.
To get more details on AWS SQS, please refer below AWS documentation
https://docs.aws.amazon.com/sqs/index.html