Which approach will address all of these requirements with the LEAST development effort?

By:
In: MLS-C01
Tagged:
With: 2 Comments
A company is building a new version of a recommendation engine.Machine learning (ML) specialists need to keep adding new data from users to improve personalized recommendations.The ML specialists gather data from the users’ interactions on the platform and from sources such as external websites and social media.The pipeline cleans, transforms, enriches, and compresses terabytes of data daily, and this data is stored in Amazon S3.A set of Python scripts was coded to do the job and is stored in a large Amazon EC2 instance.The whole process takes more than 20 hours to finish, with each script taking at least an hour.The company wants to move the scripts out of Amazon EC2 into a more managed solution that will eliminate the need to maintain servers.

Which approach will address all of these requirements with the LEAST development effort?

Load the data into an Amazon Redshift cluster. Execute the pipeline by using SQL. Store the results in Amazon S3.

Load the data into Amazon DynamoDB. Convert the scripts to an AWS Lambda function. Execute the pipeline by triggering Lambda executions. Store the results in Amazon S3.

Create an AWS Glue job. Convert the scripts to PySpark. Execute the pipeline. Store the results in Amazon S3.

Create a set of individual AWS Lambda functions to execute each of the scripts. Build a step function by using the AWS Step Functions Data Science SDK. Store the results in Amazon S3.

Explanations:

Amazon Redshift is primarily used for data warehousing and analytics, not for data processing pipelines. SQL is not ideal for complex data transformations and the required scripts.

Amazon DynamoDB is a NoSQL database, not suitable for large-scale data transformation tasks. AWS Lambda has resource limitations and may not be the best fit for processing terabytes of data.

AWS Glue is a managed ETL service designed for transforming and processing large-scale data. It supports PySpark, making it a good fit for the required pipeline, and can easily store the results in S3.

While AWS Lambda and Step Functions can be used for orchestrating tasks, managing multiple Lambda functions for such a large-scale pipeline introduces complexity, and Lambda is not ideal for heavy data processing tasks like those in this scenario.

2026-04-02

2 Comments

  1. Sean
    Author

    As far as I’m aware, the answer is:
    Create an AWS Glue job. Convert the scripts to PySpark. Execute the pipeline. Store the results in Amazon S3.

  2. Lucas
    Author

    From what I’ve heard, the answer is:
    Create an AWS Glue job. Convert the scripts to PySpark. Execute the pipeline. Store the results in Amazon S3.

Leave a Reply

Your email address will not be published. Required fields are marked *

five × three =