Which solution will provide the LARGEST overall cost reduction while meeting these requirements?
Migrate the data from the existing shared file system to an Amazon S3 bucket that uses the S3 Intelligent-Tiering storage class. Before the job runs each month, use Amazon FSx for Lustre to create a new file system with the data from Amazon S3 by using lazy loading. Use the new file system as the shared storage for the duration of the job. Delete the file system when the job is complete.
Migrate the data from the existing shared file system to a large Amazon Elastic Block Store (Amazon EBS) volume with Multi-Attach enabled. Attach the EBS volume to each of the instances by using a user data script in the Auto Scaling group launch template. Use the EBS volume as the shared storage for the duration of the job. Detach the EBS volume when the job is complete
Migrate the data from the existing shared file system to an Amazon S3 bucket that uses the S3 Standard storage class. Before the job runs each month, use Amazon FSx for Lustre to create a new file system with the data from Amazon S3 by using batch loading. Use the new file system as the shared storage for the duration of the job. Delete the file system when the job is complete.
Migrate the data from the existing shared file system to an Amazon S3 bucket. Before the job runs each month, use AWS Storage Gateway to create a file gateway with the data from Amazon S3. Use the file gateway as the shared storage for the job. Delete the file gateway when the job is complete.
Explanations:
Amazon FSx for Lustre integrates well with S3 and provides high-performance, low-latency file system access. Using lazy loading from Amazon S3 ensures that only the needed data is retrieved, minimizing storage costs when not in use. Deleting the FSx for Lustre file system after the job is complete saves costs, making this solution the most cost-effective.
Amazon EBS is not designed for large-scale shared storage access across multiple EC2 instances. Multi-Attach can only be used with specific EBS volume types (io1/io2), but it doesn’t scale well for high-performance file system access in a data-intensive job. Additionally, this solution would require manual management of the volume and could incur unnecessary costs for continuous use.
Amazon FSx for Lustre can be used with Amazon S3, but the batch loading option is less efficient compared to lazy loading for this use case. Lazy loading allows the data to be accessed as needed without loading everything at once, whereas batch loading can result in higher storage and access costs.
AWS Storage Gateway with the file gateway option allows access to S3 data as if it were a file system, but it is not optimized for high-performance, data-intensive workloads. File Gateway is primarily designed for backup, archiving, and hybrid cloud file access rather than providing high-performance shared storage for compute-intensive applications.
I conclude that the answer is:
Migrate the data from the existing shared file system to an Amazon S3 bucket that uses the S3 Intelligent-Tiering storage class. Before the job runs each month, use Amazon FSx for Lustre to create a new file system with the data from Amazon S3 by using lazy loading. Use the new file system as the shared storage for the duration of the job. Delete the file system when the job is complete.