What is the MOST operationally efficient solution that meets these requirements?
Create a web application to write records to Amazon S3. Use S3 Event Notifications to publish to an Amazon Simple Notification Service (Amazon SNS) topic. Use an EC2 instance to poll Amazon SNS and start processing. Save intermediate results to Amazon S3 to pass on to the next step.
Perform the processing steps by using logic in the application. Convert the application code to run in a container. Use AWS Fargate to manage the container instances. Configure the container to invoke itself to pass the state from one step to the next.
Create a web application to pass records to an Amazon Kinesis data stream. Decouple the processing by using the Kinesis data stream and AWS Lambda functions.
Create a web application to pass records to AWS Step Functions. Decouple the processing into Step Functions tasks and AWS Lambda functions.
Explanations:
While this option saves intermediate results to S3, using S3 Event Notifications and SNS introduces latency and complexity in managing state across steps. It also requires polling and may not efficiently handle reprocessing only failed steps without additional logic.
Running the application in containers on AWS Fargate allows for easier management, but this approach does not inherently provide a mechanism for tracking and resuming failed steps. Additional state management would need to be implemented to handle partial processing, which adds complexity.
Although using Amazon Kinesis and Lambda decouples processing, it does not provide an easy way to manage and resume failed processing steps. The stateless nature of Lambda functions means additional logic would be needed to track the progress of each record’s processing across multiple steps.
AWS Step Functions is designed for coordinating complex workflows with multiple steps and provides built-in error handling and state management. It allows for easy reprocessing of only the failed steps, making it the most operationally efficient solution to meet the requirements.
As far as I can tell, the answer is:
Create a web application to pass records to AWS Step Functions. Decouple the processing into Step Functions tasks and AWS Lambda functions.