Which solution meets these requirements and is the MOST operationally efficient?
Set up an Amazon EC2 instance running a Redis database. Configure both applications to use the instance. Store, process, and delete the messages, respectively.
Use an Amazon Kinesis data stream to receive the messages from the sender application. Integrate the processing application with the Kinesis Client Library (KCL).
Integrate the sender and processor applications with an Amazon Simple Queue Service (Amazon SQS) queue. Configure a dead-letter queue to collect the messages that failed to process.
Subscribe the processing application to an Amazon Simple Notification Service (Amazon SNS) topic to receive notifications to process. Integrate the sender application to write to the SNS topic.
Explanations:
Using an Amazon EC2 instance with Redis for message handling introduces operational overhead, such as managing the instance, scaling, and ensuring high availability. It does not inherently provide message retention or dead-letter capabilities, making it less efficient for handling message failures.
Amazon Kinesis is designed for real-time data processing and is not optimal for message queuing where message retention for long durations is required (up to 2 days). Additionally, it adds complexity by requiring integration with Kinesis Client Library (KCL), which may not be necessary for this use case.
Amazon SQS is a fully managed message queuing service that is ideal for decoupling applications. It provides built-in support for message retention and the ability to configure dead-letter queues to handle messages that fail processing. This approach minimizes operational overhead and meets the requirements effectively.
Amazon SNS is primarily a pub/sub messaging service, not designed for guaranteed message delivery and retention like SQS. While it can notify subscribers of new messages, it does not retain messages for long periods or handle failed message processing effectively, making it unsuitable for this scenario.
I plan that the answer is:
Integrate the sender and processor applications with an Amazon Simple Queue Service (Amazon SQS) queue. Configure a dead-letter queue to collect the messages that failed to process.