What is the MOST operationally efficient solution that meets these requirements?
Create a web application to write records to Amazon S3. Use S3 Event Notifications to publish to an Amazon Simple Notification Service (Amazon SNS) topic. Use an EC2 instance to poll Amazon SNS and start processing. Save intermediate results to Amazon S3 to pass on to the next step.
Perform the processing steps by using logic in the application. Convert the application code to run in a container. Use AWS Fargate to manage the container instances. Configure the container to invoke itself to pass the state from one step to the next.
Create a web application to pass records to an Amazon Kinesis data stream. Decouple the processing by using the Kinesis data stream and AWS Lambda functions.
Create a web application to pass records to AWS Step Functions. Decouple the processing into Step Functions tasks and AWS Lambda functions.
Explanations:
While using S3 Event Notifications and SNS could decouple the processing, the need to save intermediate results and manage retries manually for failed steps can lead to complexity and operational overhead. S3 is not optimized for managing state across multiple processing steps.
Converting the application to run in containers and using AWS Fargate allows for better resource management, but it does not inherently manage state between processing steps. If a step fails, it would still require handling reprocessing from the start, which is not aligned with the requirement to only reprocess failed steps.
Utilizing Kinesis data streams for decoupled processing is beneficial for real-time data processing, but it does not provide a built-in mechanism to manage state across multiple steps in a straightforward manner. Each Lambda function would need to manage its own state and any failure would still necessitate a reprocess from the beginning.
AWS Step Functions is designed for orchestrating complex workflows and allows for managing state between steps effectively. It can track the progress of each step, handle retries for specific steps, and only require reprocessing of failed tasks, thereby aligning perfectly with the requirement of not starting over from the beginning.
I rate that the answer is:
Create a web application to pass records to AWS Step Functions. Decouple the processing into Step Functions tasks and AWS Lambda functions.