Amazon Simple Queue Service (SQS) is a fully managed, distributed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. While SQS is a powerful and reliable service, it does have some limitations that you should consider when using it. In this article, we will explore some of these limitations and discuss potential solutions that you can consider to overcome them.
🔁 Original post at 🔗 Dev Post
🔁 Reposted at 🔗 dev to @aravindvcyber
Check out lots more coming direct to your mailbox by joining my free newsletter
🔗 Exploring Serverless @ substack
Please find notable limitations in using SQS services below:
Message size limit,
Message retention period,
Visibility timeout,
Delivery guarantees,
Throughput limits,
Regional availability
In this article, we will discuss these limitations in more detail and provide potential solutions that you can consider to overcome them. By understanding these limitations and implementing appropriate solutions, you can ensure that SQS is an effective and reliable message-queuing service for your applications.
Message size limit: 🪂
Each message in an SQS queue has a maximum size limit of 256 KB. If you need to send larger messages, you may need to use a different service, such as Amazon SNS or Amazon Kinesis.
Solutions to use large message size 🐳
To overcome the message size limit of Amazon Simple Queue Service (SQS), you can use the following approaches:
Split the message into smaller chunks:
If your message is larger than the size limit, you can split it into smaller chunks and send multiple messages for each chunk. You can then have the consumer reassemble the chunks into the original message. This approach can be used for messages of any size, as long as you have the necessary logic to split and reassemble the message.
import * as AWS from 'aws-sdk';
// Set up an SQS client
const sqs = new AWS.SQS();
// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';
// Define the maximum chunk size
const chunkSize = 200000; // 200 KB
// Split the message into chunks
const chunks = [];
for (let i = 0; i < largeMessage.length; i += chunkSize) {
chunks.push(largeMessage.slice(i, i + chunkSize));
}
// Send a message for each chunk
for (const chunk of chunks) {
sqs.sendMessage({ QueueUrl: queueUrl, MessageBody: chunk }, (err) => {
if (err) {
console.error(err);
}
});
}
This example uses a loop to split the largeMessage string into chunks of chunkSize bytes and then sends a message for each chunk using the sendMessage method of the SQS client.
To reassemble the message on the consumer side, you can use a similar approach to receive and process the messages in a loop:
import * as AWS from 'aws-sdk';
// Set up an SQS client
const sqs = new AWS.SQS();
// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';
// Receive and process messages from the queue
while (true) {
sqs.receiveMessage({ QueueUrl: queueUrl, MaxNumberOfMessages: 10 }, (err, data) => {
if (err) {
console.error(err);
return;
}
for (const message of data.Messages) {
// Concatenate the message body with the reconstructed message
reconstructedMessage += message.Body;
// Delete the message from the queue
sqs.deleteMessage({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle }, (delErr) => {
if (delErr) {
console.error(delErr);
}
});
}
})
}
Using message attributes:
SQS allows you to attach up to 10 message attributes to each message, which can be used to store additional metadata or data. You can use message attributes to store parts of a larger message and then have the consumer retrieve and reassemble the message using the attributes. This approach is limited to a total of 1 KB of message attributes per message.
Amazon Simple Queue Service (SQS) allows you to attach up to 10 message attributes to each message, which can be used to store additional metadata or data. You can use message attributes to store parts of a larger message and then have the consumer retrieve and reassemble the message using the attributes. This approach is limited to a total of 1 KB of message attributes per message.
You can use message attributes to send a large message using the AWS SDK for JavaScript (AWS.SQS):
import * as AWS from 'aws-sdk';
// Set up an SQS client
const sqs = new AWS.SQS();
// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';
// Define the maximum chunk size
const chunkSize = 900; // 900 bytes
// Split the message into chunks
const chunks = [];
for (let i = 0; i < largeMessage.length; i += chunkSize) {
chunks.push(largeMessage.slice(i, i + chunkSize));
}
// Send a message for each chunk
for (let i = 0; i < chunks.length; i++) {
const messageAttributes = {
chunkNumber: { DataType: 'Number', StringValue: i.toString() }
};
sqs.sendMessage({ QueueUrl: queueUrl, MessageBody: chunks[i], MessageAttributes: messageAttributes }, (err) => {
if (err) {
console.error(err);
}
});
}
This example uses a loop to split the largeMessage string into chunks of chunkSize bytes and then sends a message for each chunk using the sendMessage method of the SQS client. The message attributes include a chunkNumber attribute that indicates the order of the chunk in the original message.
To receive a large message using Amazon Simple Queue Service (SQS), please follow the below example
import * as AWS from 'aws-sdk';
// Set up an SQS client
const sqs = new AWS.SQS();
// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';
// Receive and process messages from the queue
while (true) {
sqs.receiveMessage({ QueueUrl: queueUrl, MaxNumberOfMessages: 10 }, (err, data) => {
if (err) {
console.error(err);
return;
}
for (const message of data.Messages) {
// Get the chunk number from the message attributes
const chunkNumber = parseInt(message.MessageAttributes.chunkNumber.StringValue, 10);
// Store the chunk in the reconstructed message array
reconstructedMessage[chunkNumber] = message.Body;
// Delete the message from the queue
sqs.deleteMessage({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle }, (delErr) => {
if (delErr) {
console.error(delErr);
}
});
}
});
}
// Reassemble the message from the chunks
const reconstructedMessage = reconstructedMessage.join('');
This example uses a loop to receive and process messages from the queue using the receiveMessage method of the SQS client. For each message, it gets the chunkNumber attribute from the message attributes and stores the message body in the corresponding position in the reconstructedMessage array.
Once all the messages have been received and processed, the reconstructedMessage array can be joined to form the original message.
If you need to send larger messages, you can use a different service that supports larger message sizes, such as Amazon SNS or Amazon Kinesis. These services offer various options for sending large payloads, such as message batching, compression, and chunking.
Two popular options are Amazon Simple Notification Service (SNS) and Amazon Kinesis. These services offer various options for sending large payloads, such as message batching, compression, and chunking.
Using SNS service:
You can send a large message using Amazon SNS and with the AWS SDK for JavaScript (AWS.SNS):
import * as AWS from 'aws-sdk';
// Set up an SNS client
const sns = new AWS.SNS();
// Define the topic ARN
const topicArn = 'arn:aws:sns:ap-south-1:71717177117171:my-topic';
// Send the message
sns.publish({ TopicArn: topicArn, Message: largeMessage }, (err) => {
if (err) {
console.error(err);
}
});
This example uses the publish method of the SNS client to send the largeMessage string to the specified topic. SNS supports message sizes up to 2 MB.
To receive and process the message on the consumer side, you can set up a subscription to the SNS topic and specify a delivery mechanism, such as an SQS queue or an HTTP/S endpoint. The consumer can then receive and process the messages as usual.
You can set up an SQS queue as a subscription to an SNS topic:
import * as AWS from 'aws-sdk';
// Set up an SNS client
const sns = new AWS.SNS();
// Define the topic ARN and queue URL
const topicArn = 'arn:aws:sns:ap-south-1:71717177117171:my-topic';
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';
// Create the subscription
sns.subscribe({
Protocol: 'sqs',
TopicArn: topicArn,
Endpoint: queueUrl
}, (err) => {
if (err) {
console.error(err);
}
});
This example uses the subscribe method of the SNS client to create a subscription to the specified SNS topic, using the SQS queue as the delivery mechanism.
To receive and process the messages from the SQS queue, you can use the same approach as in the previous examples, using the receiveMessage and deleteMessage methods of the SQS client.
To set up an HTTP or HTTPS endpoint as a subscription for an SQS queue, you will need to create an Amazon Simple Notification Service (SNS) topic and then configure the topic to send notifications to the HTTP or HTTPS endpoint.
Here is an overview of the process:
Create an SNS topic and choose "HTTP/S" as the protocol for the endpoint.
Provide the URL for the HTTP or HTTPS endpoint as the endpoint address.
Create an SQS queue and choose "SNS" as the protocol for the subscription.
Choose the SNS topic that you created in step 1 as the topic for the subscription.
Once you have set up the subscription, any messages that are sent to the SQS queue will be forwarded to the HTTP or HTTPS endpoint as a POST request. The body of the request will contain a JSON object with details about the message, including the message body and any metadata.
It's important to note that the HTTP or HTTPS endpoint must be able to process the POST request and handle the message promptly. If the endpoint is unable to process the message, it may be retried multiple times before the message is considered to have failed delivery.
import * as cdk from 'aws-cdk-lib';
import * as sqs from 'aws-cdk-lib-sqs';
export class MyStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Create an SQS queue
const queue = new sqs.SqsQueue(this, 'MyQueue', {
visibilityTimeout: cdk.Duration.seconds(300)
});
// Create an HTTP subscription for the queue
const subscription = new sqs.HttpSubscription(this, 'MySubscription', {
queue,
endpoint: 'https://example.com/endpoint'
});
}
}
Using Kinesis service:
Amazon Web Services (AWS) Kinesis is a fully managed streaming data platform that enables real-time processing of streaming data at scale. It is designed to handle high volume and velocity data streams, such as clickstreams, log files, and social media feeds.
In this case, the SQS queue is being used to decouple Kinesis. This means that the system that is producing the data stream (e.g. a log generator or social media platform) can send the data to the Kinesis stream without needing to directly communicate with the consumer of the data (e.g. an analytics system or machine learning model). Instead, the producer can simply add a message to the SQS queue, which will trigger further processing to send the data to the Kinesis stream. This decoupling can make it easier to scale and maintain the system, as the producer and consumer can operate independently without needing to directly communicate with each other.
import * as cdk from 'aws-cdk-lib';
import * as kinesis from '@aws-cdk/aws-kinesis';
import * as sqs from '@aws-cdk/aws-sqs';
export class MyStack extends cdk.Stack {
constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Create a Kinesis stream
const stream = new kinesis.Stream(this, 'MyStream', {
shardCount: 1
});
// Create an SQS queue
const queue = new sqs.Queue(this, 'MyQueue', {
visibilityTimeout: cdk.Duration.seconds(300)
});
// Send data to the Kinesis stream and post a message to the SQS queue once it has been sent
stream.grantWrite(queue);
}
}
This code creates a new Kinesis stream with a single shard and an SQS queue with a visibility timeout of 300 seconds. It then grants the queue write permissions to the stream, which allows it to send data to the stream. When data is added to the stream, it will trigger further processing to send the data to any consumers that are subscribed to the stream. The message that is posted to the SQS queue will contain information about the data that was sent to the stream, such as the partition key and the sequence number. This can be used to track the progress of the data through the system.
import { Kinesis } from 'aws-sdk';
const kinesis = new Kinesis();
export async function handler(event: any) {
console.log(`Received event: ${JSON.stringify(event, null, 2)}`);
const message = event.Records[0].body;
const data = JSON.parse(message);
// Get the shard iterator for the specified shard and sequence number
const result = await kinesis.getShardIterator({
StreamName: process.env.STREAM_NAME,
ShardId: data.ShardId,
ShardIteratorType: 'AT_SEQUENCE_NUMBER',
StartingSequenceNumber: data.SequenceNumber
}).promise();
const shardIterator = result.ShardIterator;
// Read data from the stream using the shard iterator
let records = await kinesis.getRecords({
ShardIterator: shardIterator,
Limit: 1
}).promise();
while (records.Records.length > 0) {
// Process the records that were read from the stream
for (const record of records.Records) {
if (record.PartitionKey === data.PartitionKey) {
// This record has the desired partition key, process it
// ...
}
}
// Get the next shard iterator
shardIterator = records.NextShardIterator;
// Read the next batch of records
records = await kinesis.getRecords({
ShardIterator: shardIterator,
Limit: 1
}).promise();
}
// Delete the message from the queue once it has been processed
const deleteResult = await sqs.deleteMessage({
QueueUrl: process.env.QUEUE_URL,
ReceiptHandle: event.Records[0].receiptHandle
}).promise();
console.log(`Deleted message: ${JSON.stringify(deleteResult, null, 2)}`);
}
This code reads a single record from the stream using the ShardIterator from the message and the getRecords method of the Kinesis client. It then processes the data that was read from the stream. Finally, it deletes the message from the queue using the deleteMessage method of the SQS client. This will mark the message as processed and remove it from the queue.
Message retention period: 🪂
SQS queues can retain messages for a maximum of 14 days. If you need to retain messages for a longer period, you may need to use a different service, such as
Amazon S3orAmazon Glacier.
The message retention period of an Amazon Simple Queue Service (SQS) queue determines how long a message is retained in the queue before it is automatically deleted. The default retention period is 4 days, and the maximum retention period is 14 days.
You can set the message retention period when you create a queue or update an existing queue using the SetQueueAttributes action of the SQS API or the corresponding method of the AWS SDKs.
import * as AWS from 'aws-sdk';
// Set up an SQS client
const sqs = new AWS.SQS();
// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';
// Set the message retention period to 7 days
sqs.setQueueAttributes({
QueueUrl: queueUrl,
Attributes: {
MessageRetentionPeriod: '604800' // 7 days in seconds
}
}, (err) => {
if (err) {
console.error(err);
}
});
If you need to retain messages for a longer period, you can use a different service, such as Amazon S3 or Amazon Glacier. These services offer long-term storage options that can store messages indefinitely or for a specified period.
You can use Amazon S3 to store messages from an SQS queue for long-term retention:
import * as AWS from 'aws-sdk';
// Set up an SQS client
const sqs = new AWS.SQS();
// Set up an S3 client
const s3 = new AWS.S3();
// Define the queue URL and bucket name
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';
const bucketName = 'my-s3-bucket';
// Receive and process messages from the queue
while (true) {
sqs.receiveMessage({ QueueUrl: queueUrl, MaxNumberOfMessages: 10 }, (err, data) => {
if (err) {
console.error(err);
return;
}
for (const message of data.Messages) {
// Store the message in S3
s3.putObject({
Bucket: bucketName,
Key: message.MessageId,
Body: message.Body
}, (putErr) => {
if (putErr) {
console.error(putErr);
return;
}
// Delete the message from the queue
sqs.deleteMessage({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle }, (delErr) => {
if (delErr) {
console.error(delErr);
}
});
}
}
})
}
This example uses a loop to receive and process messages from the queue using the receiveMessage method of the SQS client. For each message, it stores the message body in an S3 bucket using the putObject method of the S3 client and then deletes the message from the queue using the deleteMessage method of the SQS client.
To retrieve the messages from S3, you can use the getObject method of the S3 client. You can also set up lifecycle policies on the S3 bucket to automatically delete or transition the messages to Amazon Glacier after a specified period.
Visibility timeout: 🪂
When a message is received from an SQS queue, it becomes hidden from other consumers for a specified period, known as the visibility timeout. If the consumer fails to process the message within this timeout, the message will become visible again and may be processed by another consumer. This can lead to duplicate processing if not handled properly.
The visibility timeout of an Amazon Simple Queue Service (SQS) queue is the amount of time that a message is hidden from other consumers when it is being processed. This is to prevent other consumers from processing the same message simultaneously, which can lead to duplicate processing of the message.
The visibility timeout is specified in seconds and has a maximum value of 12 hours (43200 seconds). The default value is 30 seconds.
The visibility timeout of an Amazon Simple Queue Service (SQS) queue is designed to prevent multiple consumers from processing the same message simultaneously, which can lead to duplicate processing of the message. However, there may be situations where you need to extend the visibility timeout of a message to allow more time for processing.
To extend the visibility timeout of a message in an SQS queue, you can use the ChangeMessageVisibility action of the SQS API or the corresponding method of the AWS SDKs.
You can extend the visibility timeout of a message by 5 minutes (300 seconds) using the AWS SDK for JavaScript (AWS.SQS):
import * as AWS from 'aws-sdk';
// Set up an SQS client
const sqs = new AWS.SQS();
// Define the queue URL
const queueUrl = 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue';
// Receive and process messages from the queue
while (true) {
sqs.receiveMessage({ QueueUrl: queueUrl, MaxNumberOfMessages: 10 }, (err, data) => {
if (err) {
console.error(err);
return;
}
for (const message of data.Messages) {
// Process the message
processMessage(message, (processErr) => {
if (processErr) {
console.error(processErr);
// Extend the visibility timeout by 5 minutes
sqs.changeMessageVisibility({
QueueUrl: queueUrl,
ReceiptHandle: message.ReceiptHandle,
VisibilityTimeout: 300
}, (visErr) => {
if (visErr) {
console.error(visErr);
}
});
} else {
// Delete the message from the queue
sqs.deleteMessage({ QueueUrl: queueUrl, ReceiptHandle: message.ReceiptHandle }, (delErr) => {
if (delErr) {
console.error(delErr);
}
});
}
});
}
});
}
This example uses a loop to receive and process messages from the queue using the receiveMessage method of the SQS client. For each message, it calls the processMessage function to process the message. If the processMessage function returns an error, it means that the message processing failed, and the visibility timeout of the message is extended by 5 minutes using the changeMessageVisibility method of the SQS client. If the processMessage function succeeds, the message is deleted from the queue using the deleteMessage method of the SQS client.
Delivery guarantees: 🪂
SQS is a best-effort delivery service, which means that it does not guarantee the order of messages or that every message will be delivered. If you need stronger delivery guarantees, you may need to use a different service, such as Amazon SNS or Amazon Kinesis.
Amazon Simple Queue Service (SQS) provides different delivery guarantees to ensure the reliability and availability of message delivery.
Here are the main delivery guarantees provided by SQS:
At least once delivery: This guarantee ensures that each message is delivered at least once, but it may be delivered more than once in certain circumstances, such as when the consumer fails to process the message or when the visibility timeout is extended. This is the default delivery guarantee for SQS queues.
At least once delivery is a delivery guarantee provided by Amazon Simple Queue Service (SQS) that ensures that each message is delivered at least once, but it may be delivered more than once in certain circumstances. This is the default delivery guarantee for SQS queues.
At least once delivery: 🐳
Here is how the "at least once delivery" guarantee works in SQS:
When you send a message to an SQS queue, the message is stored in the queue and made available to consumers.
When a consumer retrieves a message from the queue using the receiveMessage the action of the SQS API or the corresponding method of the AWS SDKs, the message is hidden from other consumers for a specified period known as the visibility timeout.
During the visibility timeout, the consumer processes the message and then sends a request to delete the message from the queue using the deleteMessage the action of the SQS API or the corresponding method of the AWS SDKs.
If the consumer successfully deletes the message from the queue, the message is removed from the queue and is not delivered again.
If the consumer fails to delete the message from the queue, the message becomes visible to other consumers again after the visibility timeout expires, and the process is repeated until the message is successfully deleted from the queue.
This means that a message may be delivered more than once in the following circumstances:
The consumer fails to process the message and does not delete it from the queue before the visibility timeout expires. The consumer extends the visibility timeout of the message using the changeMessageVisibility the action of the SQS API or the corresponding method of the AWS SDKs. To avoid these issues, it is important to design your consumer application to process the message and delete it from the queue promptly, and to handle duplicate messages if they occur.
At most once delivery: 🐳
This guarantee ensures that each message is delivered at most once, but it may not be delivered at all in certain circumstances, such as when the consumer is unable to process the message. To use this delivery guarantee, you can use the SQS FIFO (First-In-First-Out) queue, which is designed to prevent duplicates and guarantee the order of delivery.
A FIFO (First-In, First-Out) queue is a type of Amazon Simple Queue Service (SQS) queue that preserves the order of messages and ensure that each message is processed and deleted exactly once.
At most once delivery is a delivery guarantee provided by Amazon Simple Queue Service (SQS) that ensures that each message is delivered at most once, but it may not be delivered at all in certain circumstances. To use this delivery guarantee, you can use the SQS FIFO (First-In-First-Out) queue, which is designed to prevent duplicates and guarantee the order of delivery.
Here is how the "at most once delivery" guarantee works in an SQS FIFO queue:
When you send a message to an SQS FIFO queue, the message is stored in the queue and made available to consumers.
When a consumer retrieves a message from the queue using the receiveMessage the action of the SQS API or the corresponding method of the AWS SDKs, the message is hidden from other consumers for a specified period known as the visibility timeout.
During the visibility timeout, the consumer processes the message and then sends a request to delete the message from the queue using the deleteMessage the action of the SQS API or the corresponding method of the AWS SDKs.
If the consumer successfully deletes the message from the queue, the message is removed from the queue and is not delivered again.
If the consumer fails to delete the message from the queue, the message becomes visible to other consumers again after the visibility timeout expires, and the process is repeated until the message is successfully deleted from the queue.
This means that a message may not be delivered at all in the following circumstances:
The consumer fails to process the message and does not delete it from the queue before the visibility timeout expires. The consumer extends the visibility timeout of the message using the changeMessageVisibility the action of the SQS API or the corresponding method of the AWS SDKs. To avoid these issues, it is important to design your consumer application to process the message and delete it from the queue promptly, and to handle the failure to process the message if it occurs.
Exactly once delivery: 🐳
This guarantee ensures that each message is delivered exactly once, but it requires additional effort to implement and may not be suitable for all use cases. To use this delivery guarantee, you can use the SQS FIFO (First-In-First-Out) queue in combination with a distributed transaction system, such as Amazon DynamoDB or Amazon Aurora, to store the message processing state and coordinate the processing of the message.
Exactly once delivery is a delivery guarantee that ensures that each message is delivered exactly once. This guarantee requires additional effort to implement and may not be suitable for all use cases. To use this delivery guarantee, you can use the SQS FIFO (First-In-First-Out) queue in combination with a distributed transaction system, such as Amazon DynamoDB or Amazon Aurora, to store the message processing state and coordinate the processing of the message.
You can implement "exactly once delivery" using an SQS FIFO queue and Amazon DynamoDB:
When you send a message to an SQS FIFO queue, the message is stored in the queue and made available to consumers.
When a consumer retrieves a message from the queue using the receiveMessage the action of the SQS API or the corresponding method of the AWS SDKs, the consumer stores the message processing state in a DynamoDB table using the putItem action of the DynamoDB API or the corresponding method of the AWS SDKs.
The consumer processes the message and then sends a request to delete the message from the queue using the deleteMessage the action of the SQS API or the corresponding method of the AWS SDKs.
If the deleteMessage request succeeds, the consumer removes the message processing state from the DynamoDB table using the deleteItem action of the DynamoDB API or the corresponding method of the AWS SDKs.
If the deleteMessage the request fails, the consumer retrieves the message processing state from the DynamoDB table using the getItem action of the DynamoDB API or the corresponding method of the AWS SDKs, and checks whether the message has already been processed. If the message has already been processed, the consumer removes the message processing state from the DynamoDB table and does not process the message again. If the message has not been processed, the consumer processes the message and repeats the deleteMessage request until it succeeds.
This example uses DynamoDB to store the message processing state and coordinate the processing of the message, which ensures that the message is delivered exactly once.
Throughput limits: 🪂
SQS has limits on the number of requests per second that can be made to a queue, as well as the maximum number of inflight messages (messages that have been received by a consumer but have not yet been deleted). If you exceed these limits, you can design your consumers to handle the workload more efficiently or consider using a higher-throughput queue type (such as an Amazon SQS FIFO queue).
Workarounds 🐳
Optimize your consumer design: 🪴
One way to avoid exceeding throughput limits is to design your consumers to handle the workload efficiently. This may involve using techniques such as batch processing, multithreading, or asynchronous processing to reduce the number of requests made to the queue.
Use a higher-throughput queue type: SQS offers two types of queues: 🪴
standard and FIFO (First-In, First-Out). Standard queues have higher throughput limits but do not guarantee message order, while FIFO queues have lower throughput limits but guarantee message order. If message order is not critical for your use case, you may be able to increase your throughput by using a standard queue.
Monitor your usage and adjust as needed: 🪴
You can use Amazon CloudWatch to monitor your queue usage and adjust your workload as needed to stay within the limits. For example, if you are consistently approaching the inflight message limit, you may need to increase the number of consumers processing messages from the queue.
Use Auto scaling:
If you are using Amazon Elastic Container Service (ECS) or Amazon EC2 to run your consumers, you can use Auto scaling to adjust the number of consumers based on the workload. This can help you to optimize your usage of SQS and avoid exceeding throughput limits.
Regional availability: 🪂
SQS is available in multiple regions, but a queue can only be accessed from within the same region it was created in. If you need to access a queue from another region, you can set up a cross-region replication configuration or use a service such as AWS Global Accelerator to access the queue.
Workarounds: 🐳
Choose the right region: 🪴
When creating a queue, you should choose the region that is closest to the consumers that will be accessing the queue. This will help to minimize latencies and improve performance.
Use cross-region replication: 🪴
If you need to access a queue from a different region, you can set up a cross-region replication configuration to replicate messages to a secondary queue in the destination region. This allows you to access the queue from multiple regions while maintaining a consistent copy of the messages.
import * as cdk from 'aws-cdk-lib';
import * as sqs from '@aws-cdk/aws-sqs';
export class MyStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Create the source queue
const sourceQueue = new sqs.Queue(this, 'SourceQueue', {
queueName: 'my-queue',
redrivePolicy: {
deadLetterQueue: {
queueUrl: 'https://sqs.ap-south-1.amazonaws.com/71717177117171/my-queue-replica'
},
maxReceiveCount: 10
}
});
// Create the target queue in a different region
const targetQueue = new sqs.Queue(this, 'TargetQueue', {
queueName: 'my-queue-replica',
region: 'ap-south-1'
});
}
}
This will set up the source queue to send messages that it can't process to the target queue in the specified region. The deadLetterQueue property of the redrivePolicy specifies the target queue, and the maxReceiveCount the property specifies the maximum number of times a message can be delivered to the source queue before it is sent to the dead-letter queue.
Keep in mind that cross-region replication for SQS is based on the DLQ feature. When a message is sent to the DLQ, it means that the message couldn't be processed by the source queue. You can configure the number of times a message can be delivered to the source queue before it is sent to the DLQ using the maxReceiveCount parameter in the RedrivePolicy.
Use AWS Global Accelerator: 🪴
AWS Global Accelerator allows you to access resources in different regions using static IP addresses. This can be used to access an SQS queue from a different region without the need for cross-region replication.
AWS Global Accelerator is a network service that routes traffic to the optimal Amazon Web Services (AWS) Region for lower latency and higher performance. It uses static anycast IP addresses to route traffic to the optimal AWS Region, and it includes health checking to ensure that traffic is routed to healthy endpoints.
import * as cdk from 'aws-cdk-lib';
import * as globalaccelerator from '@aws-cdk/aws-globalaccelerator';
import * as sqs from '@aws-cdk/aws-sqs';
export class MyStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// Create a new Global Accelerator
const accelerator = new globalaccelerator.GlobalAccelerator(this, 'Accelerator', {
acceleratorType: globalaccelerator.AcceleratorType.STATIC,
enabled: true,
regions: [globalaccelerator.Regions.US_EAST_1, globalaccelerator.Regions.US_WEST_2]
});
// Create an SQS queue
const queue = new sqs.Queue(this, 'Queue', {
visibilityTimeout: cdk.Duration.seconds(300)
});
// Create an accelerator endpoint for the queue
const endpoint = new globalaccelerator.AcceleratorEndpoint(this, 'Endpoint', {
accelerator,
port: 80,
protocol: globalaccelerator.Protocol.TCP,
resource: queue
});
// Create a listener for the endpoint
new globalaccelerator.Listener(this, 'Listener', {
accelerator,
port: 80,
protocol: globalaccelerator.Protocol.TCP,
endpointGroups: [{
endpoints: [endpoint]
}]
});
}
}
Cost: 🪂
Using SQS can incur costs, including charges for the number of requests made, the number of messages sent and received, and the amount of data transferred. You will need to consider these costs and optimize your usage as needed.
SQS is charged based on the number of requests and the volume of data transferred.
Here are the main factors that affect the cost of using SQS:
Number of requests:
SQS charges a request fee for each action performed on a queue, such as sending a message, receiving a message, or deleting a message. The request fees vary depending on the type of queue and the region in which the queue is located. For example, the request fees for an SQS Standard queue in the US East (N. Virginia) region are $0.00000040 per request for sending a message and $0.0000004 per request for receiving and deleting a message.
Data transfer:
SQS charges for data transfer based on the amount of data transferred in and out of the service. The data transfer fees vary depending on the region in which the queue is located. For example, the data transfer fees for an SQS Standard queue in the US East (N. Virginia) region are $0.09 per GB for data transferred in and $0.01 per GB for data transferred out.
Number of messages:
SQS charges a fee for the number of messages stored in a queue, which is based on the volume of data stored in the queue. The number of messages fee varies depending on the type of queue and the region in which the queue is located. For example, the number of messages fee for an SQS Standard queue in the US East (N. Virginia) region is $0.00000040 per 10,000 messages.
Visibility timeout:
SQS charges a fee for the number of seconds that a message is in the queue and not visible to consumers. The visibility timeout fee varies depending on the type of queue and the region in which the queue is located. For example, the visibility timeout fee for an SQS Standard queue in the US East (N. Virginia) region is $0.00000160 per 10,000 seconds.
Optimization on Cost: 🐳
There are several ways you can reduce the cost of using Amazon Simple Queue Service (SQS) by optimizing the usage of the service:
Use the appropriate queue type: 🪴
SQS offers different queue types that have different characteristics and pricing models. For example, the SQS Standard queue provides best-effort delivery at the lowest cost, while the SQS FIFO (First-In-First-Out) queue provides guaranteed delivery and order at a higher cost. Choose the queue type that best fits your requirements and optimize your usage accordingly.
Optimize the number of requests: 🪴
SQS charges a request fee for each action performed on a queue, such as sending a message, receiving a message, or deleting a message. To reduce the cost of requests, you can batch requests together using the sendMessageBatch and deleteMessageBatch actions of the SQS API or the corresponding methods of the AWS SDKs, which allows you to send or delete multiple messages in a single request.
Optimize the data transfer: 🪴
SQS charges for data transfer based on the amount of data transferred in and out of the service. To reduce the cost of data transfer, you can minimize the size of the messages you send and receive, and compress the data if possible. You can also use Amazon CloudWatch to monitor the data transfer volume and identify opportunities to optimize your usage.
Optimize the number of messages: 🪴
SQS charges a fee for the number of messages stored in a queue, which is based on the volume of data stored in the queue. To reduce the cost of storing messages, you can delete messages from the queue as soon as they are processed, and use the SQS Dead Letter Queue feature to store and analyze failed messages.
Optimize the visibility timeout: 🪴
SQS charges a fee for the number of seconds that a message is in the queue and not visible to consumers. To reduce the cost of the visibility timeout, you can set a shorter visibility timeout for messages that can be processed quickly, and a longer visibility timeout for messages that require more time to process. You can also use the changeMessageVisibility the action of the SQS API or the corresponding method of the AWS SDKs to extend the visibility timeout of a message if necessary.
These are some of the main limitations of SQS to consider. It is important to carefully evaluate your specific needs and requirements to determine if SQS is the right solution for your use case.
🎉 Thanks for supporting! 🙏
Would be great if you like to ☕ Buy Me a Coffee, to help boost my efforts 😍.
Also free to comment/review if you find something which could be wrong or explained in a better way, I am open to your thoughts and also ready to clarify so that our readers are getting the best content.
🔁 Original post at 🔗 Dev Post
🔁 Reposted at 🔗 dev to @aravindvcyber
Check out lots more coming direct to your mailbox by joining my free newsletter