SQS Standard Vs FIFO queue

Hello Everyone

Welcome to CloudAffaire and this is Debjeet.

In the last blog post, we have discussed different monitoring options available in SQS.

https://cloudaffaire.com/sqs-monitoring/

In this blog post, we will discuss Standard vs FIFO queue in SQS. AWS SQS supports two types of queue, standard and FIFO. A standard queue is used for application where the throughput of messages is more important than the ordering of messages. For example, an application for user registration where the order of users registering is not that important than the number of users that can register. Standard is the default SQS queue.

Standard Queue:

Support a nearly unlimited number of transactions per second (TPS) per API action.
Support “At-Least-Once” message delivery but occasionally more than one copy of a message might be delivered.
Support best effort basis ordering of messages.
Distributed in nature, stores copies of your messages on multiple servers for redundancy and high availability.
Does not support grouping of messages.
Does not support content-based deduplication.
Used when throughput is more important than ordering.

A FIFO queue, on the other hand, is used for application where the ordering of messages is more important than the throughput of the messages. For example, an application for online shopping checkout where the order (Payment needs to made first) is more important than the throughput of messages. FIFO is not the default queue of SQS and you need to provide additional configuration details to create a FIFO queue.

FIFO Queue:

Support high number (3,000 msg/ps with batching and 300 msg/ps without batching) of transactions per second (TPS) per API action.
Support “Exactly-Once” message processing in which a message is delivered once and remains available until a consumer processes and deletes it.
Support “First-In-First-Out” delivery where the order in which messages are sent and received is strictly preserved.
Not distributed in nature.
Support grouping of messages.
Support content-based deduplication.
FIFO queue name must end with the .fifo suffix.
Used when ordering is more important than throughput.

Next, we are going to explain the main difference between a Standard and FIFO queue through a demo.

Prerequisites for this demo:

An EC2 instance with proper access to SQS.

Step 1: Configure AWS EC2 instance for the demo.

## Create an EC2 instance with AWS Linux 2 AMI
## Create a role with admin access and attach to your instance
## Create a script to assume access
## Install AWS CLI, Python, PIP, Boto3 and JQ packages if not installed already
#sudo yum install jq -y
#sudo yum install python-pip -y
#sudo pip install boto3

vi assume_role.sh
-----------------------
## Replace <YOUR_IAM_ROLE> with the role that you have created and attached to the instance
#!/bin/bash
curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<YOUR_IAM_ROLE> > cred.json
export AWS_ACCESS_KEY_ID=$(cat cred.json| jq .AccessKeyId | xargs)
export AWS_SECRET_ACCESS_KEY=$(cat cred.json| jq .SecretAccessKey| xargs)
export AWS_SESSION_TOKEN=$(cat cred.json| jq .Token| xargs)
export AWS_EXPIRATION=$(cat cred.json| jq .Credentials.Expiration| xargs)
export AWS_DEFAULT_REGION=ap-south-1
rm -f cred.json
------------------------
:wq

## Assume access
chmod +x assume_role.sh
sh assume_role.sh

## Check if access is working
aws sts get-caller-identity

## Create an EC2 instance with AWS Linux 2 AMI

## Create a role with admin access and attach to your instance

## Create a script to assume access

## Install AWS CLI, Python, PIP, Boto3 and JQ packages if not installed already

#sudo yum install jq -y

#sudo yum install python-pip -y

#sudo pip install boto3

vi assume_role.sh

-----------------------

## Replace with the role that you have created and attached to the instance

#!/bin/bash

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/ > cred.json

export AWS_ACCESS_KEY_ID=$(cat cred.json| jq .AccessKeyId | xargs)

export AWS_SECRET_ACCESS_KEY=$(cat cred.json| jq .SecretAccessKey| xargs)

export AWS_SESSION_TOKEN=$(cat cred.json| jq .Token| xargs)

export AWS_EXPIRATION=$(cat cred.json| jq .Credentials.Expiration| xargs)

export AWS_DEFAULT_REGION=ap-south-1

rm -f cred.json

------------------------

:wq

## Assume access

chmod +x assume_role.sh

sh assume_role.sh

## Check if access is working

aws sts get-caller-identity

Step 2: Create a python script to perform queuing operations in a Standard queue.

## Standard Queue Example
## Create a python script to create queue, send and receive messages and delete queue
vi standard_queue.py
------------------
#!/usr/local/bin/python
import boto3
client = boto3.client('sqs')
print('Creating a standard queue')
client.create_queue(QueueName='my_standard_queue')
queues = client.list_queues(QueueNamePrefix='my_standard_queue')
queue_url = queues['QueueUrls'][0]
print('Sending 10 messages to the standard queue')
for i in range(0,10):
    print('message: '+ str(i))
    enqueue_response = client.send_message(QueueUrl=queue_url, MessageBody='message: '+str(i))
print('Receving 10 messages from the standard queue')
while True:
    messages = client.receive_message(QueueUrl=queue_url,MaxNumberOfMessages=5)
    if 'Messages' in messages:
        for message in messages['Messages']:
            print(message['Body'])
            client.delete_message(QueueUrl=queue_url,ReceiptHandle=message['ReceiptHandle'])
    else:
        print('Queue is now empty')
        break
client.delete_queue(QueueUrl=queue_url)
print('Queue is now deleted')
-------------------
:wq

## Standard Queue Example

## Create a python script to create queue, send and receive messages and delete queue

vi standard_queue.py

------------------

#!/usr/local/bin/python

import boto3

client = boto3.client('sqs')

print('Creating a standard queue')

client.create_queue(QueueName='my_standard_queue')

queues = client.list_queues(QueueNamePrefix='my_standard_queue')

queue_url = queues['QueueUrls'][0]

print('Sending 10 messages to the standard queue')

for i in range(0,10):

print('message: '+ str(i))

enqueue_response = client.send_message(QueueUrl=queue_url, MessageBody='message: '+str(i))

print('Receving 10 messages from the standard queue')

while True:

messages = client.receive_message(QueueUrl=queue_url,MaxNumberOfMessages=5)

if 'Messages' in messages:

for message in messages['Messages']:

print(message['Body'])

client.delete_message(QueueUrl=queue_url,ReceiptHandle=message['ReceiptHandle'])

else:

print('Queue is now empty')

break

client.delete_queue(QueueUrl=queue_url)

print('Queue is now deleted')

-------------------

:wq

Step 3: Execute the script.

## Execute the python script
python standard_queue.py

1 2	## Execute the python script python standard_queue.py

SQS Standard Vs FIFO Queue

Observation: In the case of the standard queue, while receiving messages, messages are not ordered as per ‘SendMessage’ request.

Step 4: Create a python script to perform queuing operations in a FIFO queue.

## FIFO Queue Example
## Create a python script to create queue, send and receive messages and delete queue
vi fifo_queue.py
--------------------
#!/usr/local/bin/python
import boto3
client = boto3.client('sqs')
print('Creating a fifo queue')
client.create_queue(QueueName='my_fifo_queue.fifo', Attributes={'FifoQueue': 'true','ContentBasedDeduplication': 'true'})
queues = client.list_queues(QueueNamePrefix='my_fifo_queue')
queue_url = queues['QueueUrls'][0]
print('Sending 10 messages to the fifo queue')
for i in range(0,10):
    print('message: '+ str(i))
    enqueue_response = client.send_message(QueueUrl=queue_url, MessageBody='message: '+str(i), MessageGroupId='msggpid1')
print('Receving 20 messages from the fifo queue')
while True:
    messages = client.receive_message(QueueUrl=queue_url,MaxNumberOfMessages=5)
    if 'Messages' in messages: 
        for message in messages['Messages']: 
            print(message['Body'])
            client.delete_message(QueueUrl=queue_url,ReceiptHandle=message['ReceiptHandle'])
    else:
        print('Queue is now empty')
        break
client.delete_queue(QueueUrl=queue_url)
print('Queue is now deleted')
-----------------------
:wq

## FIFO Queue Example

## Create a python script to create queue, send and receive messages and delete queue

vi fifo_queue.py

--------------------

#!/usr/local/bin/python

import boto3

client = boto3.client('sqs')

print('Creating a fifo queue')

client.create_queue(QueueName='my_fifo_queue.fifo', Attributes={'FifoQueue': 'true','ContentBasedDeduplication': 'true'})

queues = client.list_queues(QueueNamePrefix='my_fifo_queue')

queue_url = queues['QueueUrls'][0]

print('Sending 10 messages to the fifo queue')

for i in range(0,10):

print('message: '+ str(i))

enqueue_response = client.send_message(QueueUrl=queue_url, MessageBody='message: '+str(i), MessageGroupId='msggpid1')

print('Receving 20 messages from the fifo queue')