What service could be used to allow multiple consumers to process the data concurrently and MOST cost-effectively?
Amazon SNS with fanout to an SQS queue for each application
Amazon SNS with fanout to an SQS FIFO (first-in, first-out) queue for each application
Amazon Kinesis Firehose
Amazon Kinesis Streams
Explanations:
While Amazon SNS can deliver messages to multiple SQS queues, this setup can lead to higher costs due to the need to manage multiple queues and potential message duplication. Additionally, it may introduce latency as messages are sent to each queue, making it less efficient for real-time processing of millions of events.
Using SNS with SQS FIFO queues ensures that messages are processed in order, but this can increase costs and complexity due to the FIFO limitation and the need for message deduplication. The throughput of FIFO queues is also limited compared to standard queues, making this option less suitable for high-volume real-time processing.
Amazon Kinesis Firehose is primarily designed for batch processing and data delivery to storage destinations rather than real-time event processing with multiple consumers. It is not optimized for scenarios requiring low latency and concurrent processing by multiple applications.
Amazon Kinesis Streams is designed for real-time processing of streaming data with support for multiple consumers, allowing them to process events concurrently. It offers high throughput, low latency, and cost-effectiveness for handling millions of events, making it the best choice for this use case.