Which solution will meet these requirements?
Configure AWS Lambda to poll the SOS queue and invoke a Lambda function to check whether the queue messages are valid. If validation fails, send a copy of the data that is not valid to an Amazon S3 bucket so that the scientists can review and correct the data. When the data is corrected, amend the message in the SOS queue by using a replay Lambda function with the corrected data
Convert the SQS standard queue to an SQS FIFO queue. Configure AWS Lambda to poll the SQS queue every 10 minutes by using an Amazon EventBridge schedule. Invoke the Lambda function to identify any messages with a SentTimestamp value that is older than 5 minutes, push the data to the same location as the application’s output location, and remove the messages from the queue.
Create an SOS dead-letter queue. Modify the existing queue by including a redrive policy that sets the Maximum Receives setting to 1 and sets the dead-letter queue ARN to the ARN of the newly created queue. Instruct the scientists to use the dead-letter queue to review the data that is not valid. Reprocess this data at a later time.
Configure API Gateway to send messages to different SOS virtual queues that are named for each of the satellites. Update the application to use a new virtual queue for any data that it cannot transform, and send the message to the new virtual queue. Instruct the scientists to use the virtual queue to review the data that is not valid. Reprocess this data at a later time.
Explanations:
While this option suggests sending invalid data to an S3 bucket for review, it does not address how to handle the messages remaining in the SQS queue. It also introduces complexity with the need for a replay mechanism and may not ensure that the invalid data is properly tracked or processed in a straightforward manner.
This option proposes converting the queue to FIFO and using an EventBridge schedule to poll for old messages. However, it does not specifically address the requirement to retain failed messages for review by scientists. Instead, it focuses on moving older messages, which may lead to loss of information regarding the failed transformations.
Creating a dead-letter queue (DLQ) allows for the retention of failed messages after a specified number of processing attempts. This meets the requirement for retaining and reviewing inconsistent data. The scientists can review the messages in the DLQ and decide on reprocessing, thus providing a clear mechanism for handling data transformation failures.
While sending messages to different virtual queues based on satellites might segregate the data, it does not provide a direct mechanism for retaining failed messages. The scientists would have to manage multiple queues and it complicates the overall architecture without providing a clear way to track and reprocess failed transformations.
I categorize that the answer is:
Create an SOS dead-letter queue. Modify the existing queue by including a redrive policy that sets the Maximum Receives setting to 1 and sets the dead-letter queue ARN to the ARN of the newly created queue. Instruct the scientists to use the dead-letter queue to review the data that is not valid. Reprocess this data at a later time.