What changes should make the bid processing more reliable?
Refactor the web application to use the Amazon Kinesis Producer Library (KPL) when posting bids to Kinesis Data Streams. Refactor the bid processor to flag each record in Kinesis Data Streams as being unread, processing, and processed. At the start of each bid processing run, scan Kinesis Data Streams for unprocessed records.
Refactor the web application to post each incoming bid to an Amazon SNS topic in place of Kinesis Data Streams. Configure the SNS topic to trigger an AWS Lambda function that processes each bid as soon as a user submits it.
Refactor the web application to post each incoming bid to an Amazon SQS FIFO queue in place of Kinesis Data Streams. Refactor the bid processor to continuously the SQS queue. Place the bid processing EC2 instance in an Auto Scaling group with a minimum and a maximum size of 1.
Switch the EC2 instance type from t2.large to a larger general compute instance type. Put the bid processor EC2 instances in an Auto Scaling group that scales out the number of EC2 instances running the bid processor, based on the IncomingRecords metric in Kinesis Data Streams.
Explanations:
While using the Kinesis Producer Library (KPL) can enhance bid posting, flagging records and scanning for unprocessed records does not address the core issues of bid processing speed and reliability. Kinesis Data Streams are not designed to guarantee exactly-once processing, and the bid processor may still face performance bottlenecks under high load.
Switching to Amazon SNS and AWS Lambda for bid processing could introduce additional latency and does not guarantee the order of bid processing. SNS is designed for fan-out scenarios rather than strict message ordering, which is crucial for auction bids. This could lead to bids being processed out of order or potentially missed altogether.
Using an Amazon SQS FIFO queue ensures that bids are processed in the exact order they are received. Continuous processing of the SQS queue provides better reliability, and placing the bid processor in an Auto Scaling group allows for scaling based on demand without downtime, improving overall responsiveness and robustness of bid processing.
Although switching to a larger EC2 instance type might improve performance, it does not address the underlying issues of processing order and potential loss of bid records. Simply increasing instance size without addressing scaling or the processing mechanism may not provide the needed reliability and may still result in failures during peak loads.
From what I’ve seen, the answer is:
Refactor the web application to post each incoming bid to an Amazon SQS FIFO queue in place of Kinesis Data Streams. Refactor the bid processor to continuously the SQS queue. Place the bid processing EC2 instance in an Auto Scaling group with a minimum and a maximum size of 1.