Which approach is the MOST resilient way to achieve this goal, which also smooths out temporary volume spikes for the backend service?
Develop an AWS Lambda function to check the upload folder in the S3 bucket. If new uploaded pictures are detected, the Lambda function will scan and parse them.
Once a picture is uploaded to Amazon S3, publish the event to an Amazon SQS queue. Use the queue as an event source to trigger an AWS Lambda function. In the Lambda function, scan and parse the picture.
When the user uploads a picture. invoke an API hosted in Amazon API Gateway. The API will invoke an AWS Lambda function to scan and parse the picture.
Create a state machine in AWS Step Functions to check the upload folder in the S3 bucket. If a new picture is detected, invoke an AWS Lambda function to scan and parse it.
Explanations:
AWS Lambda can be triggered by events in S3, but polling the S3 bucket to check for new uploads is inefficient. It may lead to delays in processing and doesn’t handle temporary volume spikes well.
Using Amazon SQS allows for event-driven processing, which smooths out temporary volume spikes. The SQS queue acts as a buffer, enabling the Lambda function to process messages at its own pace, preventing overloading.
Invoking an API Gateway for each upload introduces a single point of failure and doesn’t scale efficiently under high traffic. This approach may not handle large spikes in load effectively.
AWS Step Functions are better suited for orchestrating complex workflows and states, not for simple event-driven tasks like scanning images. This option adds unnecessary complexity and doesn’t handle volume spikes as efficiently as SQS.
I plot that the answer is:
Once a picture is uploaded to Amazon S3, publish the event to an Amazon SQS queue. Use the queue as an event source to trigger an AWS Lambda function. In the Lambda function, scan and parse the picture.