Which solution will meet these requirements?
Use Amazon Elastic File System (Amazon EFS) as a shared file system. Access the dataset from Amazon EFS.
Mount an Amazon S3 bucket to serve as the shared file system. Perform postprocessing directly from the S3 bucket.
Use Amazon FSx for Lustre as a shared file system. Link the file system to an Amazon S3 bucket for postprocessing.
Configure AWS Resource Access Manager to share an Amazon S3 bucket so that it can be mounted to all instances for processing and postprocessing.
Explanations:
Amazon EFS provides a managed NFS service but does not consistently deliver access latencies within 1 ms for HPC workloads that require high throughput and performance, especially at scale.
Mounting an Amazon S3 bucket directly is not supported in the same way as a file system. S3 is an object storage service and may introduce higher latencies than the required 1 ms for HPC workloads.
Amazon FSx for Lustre is designed for high-performance workloads and supports low-latency access, making it suitable for HPC applications. It can also be linked to Amazon S3 for postprocessing, providing a robust solution for both processing and accessing datasets.
AWS Resource Access Manager does not allow S3 buckets to be mounted as file systems. S3 is not designed for low-latency access and cannot meet the performance requirements of HPC workloads.
My best guess is:
Use Amazon FSx for Lustre as a shared file system. Link the file system to an Amazon S3 bucket for postprocessing.