Which solution will meet these requirements in the MOST operationally efficient way?
Use an Amazon EC2 instance that runs an FTP server to store incoming files as objects in Amazon S3 Glacier Flexible Retrieval. Configure a job queue in AWS Batch. Use Amazon EventBridge rules to invoke the job to process the objects nightly from S3 Glacier Flexible Retrieval. Delete the objects after the job has processed the objects.
Use an Amazon EC2 instance that runs an FTP server to store incoming files on an Amazon Elastic Block Store (Amazon EBS) volume. Configure a job queue in AWS Batch. Use Amazon EventBridge rules to invoke the job to process the files nightly from the EBS volume. Delete the files after the job has processed the files.
Use AWS Transfer Family to create an FTP server to store incoming files on an Amazon Elastic Block Store (Amazon EBS) volume. Configure a job queue in AWS Batch. Use an Amazon S3 event notification when each file arrives to invoke the job in AWS Batch. Delete the files after the job has processed the files.
Use AWS Transfer Family to create an FTP server to store incoming files in Amazon S3 Standard. Create an AWS Lambda function to process the files and to delete the files after they are processed. Use an S3 event notification to invoke the Lambda function when the files arrive.
Explanations:
S3 Glacier Flexible Retrieval is not suitable for this use case because it is designed for infrequently accessed data and has retrieval times that are much longer than the required processing time of 3-8 minutes. This would not allow for timely processing of incoming files. Additionally, using EventBridge to invoke a nightly job does not align with the requirement for immediate processing.
Storing files on an EBS volume requires an EC2 instance to manage the files. This adds unnecessary operational overhead and complexity, as it requires management of the EC2 instance. Also, processing files only nightly does not meet the requirement of processing files as soon as they are received.
Although using AWS Transfer Family and EBS allows for an FTP solution, EBS still requires managing an EC2 instance, which increases operational complexity. Additionally, invoking the job nightly does not align with the requirement for processing files as soon as they arrive.
This option utilizes AWS Transfer Family for FTP, storing files directly in S3 Standard, which allows for immediate access and processing. The use of an S3 event notification to invoke a Lambda function enables processing of files as soon as they are uploaded. The Lambda function can process and delete the files efficiently, meeting the requirement for operational efficiency and timely processing.
It appears that the answer is:
Use AWS Transfer Family to create an FTP server to store incoming files in Amazon S3 Standard. Create an AWS Lambda function to process the files and to delete the files after they are processed. Use an S3 event notification to invoke the Lambda function when the files arrive.