Which solution will meet these requirements with the LEAST development effort?
Use AWS Lambda to run a predefined SageMaker pipeline to perform the transformations on each new dataset that arrives in the S3 bucket.
Run an AWS Step Functions step and a predefined SageMaker pipeline to perform the transformations on each new dataset that arrives in the S3 bucket.
Use Apache Airflow to orchestrate a set of predefined transformations on each new dataset that arrives in the S3 bucket.
Configure Amazon EventBridge to run a predefined SageMaker pipeline to perform the transformations when a new data is detected in the S3 bucket.
Explanations:
AWS Lambda can be used to trigger actions on S3 events, but it would require significant custom code to integrate with SageMaker, making it more complex than necessary.
AWS Step Functions is useful for orchestration but introduces additional complexity and management overhead for a simple data transformation task.
Apache Airflow is useful for complex workflows but is overkill for this task, requiring additional setup and management of infrastructure.
Amazon EventBridge can trigger predefined SageMaker pipelines based on S3 events, offering a native, low-development effort solution that meets the requirements.
I sort that the answer is:
Configure Amazon EventBridge to run a predefined SageMaker pipeline to perform the transformations when a new data is detected in the S3 bucket.