How should you implement such a system?
Use a DynamoDB table with an attribute defining the priority level. Transformation instances will scan the table for tasks, sorting the results by priority level.
Use Route 53 latency based-routing to send high priority tasks to the closest transformation instances.
Use two SQS queues, one for high priority messages, the other for default priority. Transformation instances first poll the high priority queue; if there is no message, they poll the default priority queue.
Use a single SQS queue. Each message contains the priority level. Transformation instances poll high-priority messages first.
Explanations:
While using a DynamoDB table to store tasks and sorting by priority level could work, it introduces complexity and latency as transformation instances would need to perform additional read operations to check for tasks, which isn’t as efficient as using SQS for message queuing.
Route 53 latency-based routing is designed for directing traffic based on latency to different endpoints but does not apply to prioritizing tasks for data transformations. It does not provide a mechanism for managing task priority directly.
Using two SQS queues allows for clear separation of high and default priority tasks. Transformation instances can first poll the high-priority queue, ensuring premium customer tasks are handled first, leading to efficient processing of high-priority requests.
A single SQS queue with messages containing priority levels requires transformation instances to sort through messages to identify high-priority ones, adding unnecessary complexity and potential delay in processing high-priority tasks compared to using two separate queues.
To the best of my knowledge, the answer is:
Use two SQS queues, one for high priority messages, the other for default priority. Transformation instances first poll the high priority queue; if there is no message, they poll the default priority queue.