Which solution will meet these requirements with the MOST operational efficiency?
Configure the devices to send streaming data to an Amazon Kinesis data stream. Configure an Amazon Kinesis Data Firehose delivery stream to automatically consume the Kinesis data stream, transform the data with an AWS Lambda function, and save the output into an Amazon S3 bucket.
Configure the devices to send streaming data to an Amazon S3 bucket. Configure an AWS Lambda function that is invoked by S3 event notifications to transform the data and load the data into an Amazon Kinesis data stream. Configure an Amazon Kinesis Data Firehose delivery stream to automatically consume the Kinesis data stream and load the output back into the S3 bucket.
Configure the devices to send streaming data to an Amazon S3 bucket. Configure an AWS Glue job that is invoked by S3 event notifications to read the data, transform the data, and load the output into a new S3 bucket.
Configure the devices to send streaming data to an Amazon Kinesis Data Firehose delivery stream. Configure an AWS Glue job that connects to the delivery stream to transform the data and load the output into an Amazon S3 bucket.
Explanations:
This option uses Amazon Kinesis Data Streams for high-throughput data ingestion and Kinesis Data Firehose for automated delivery to S3. AWS Lambda transforms data during the ingestion process, ensuring high throughput, durability, scalability, and low latency, all while meeting the 5-minute tolerance.
This solution introduces unnecessary complexity. Using S3 as an ingestion point followed by Lambda and Kinesis adds more latency and is inefficient compared to directly using Kinesis Data Streams. Also, there’s no reason to send data back to S3 after Kinesis processes it.
This solution is inefficient because it starts with S3 as the data ingestion point, adding unnecessary complexity. Using S3 for event-driven Lambda execution and then loading the data into another S3 bucket is not optimal for high-throughput streaming ingestion.
Using AWS Glue with Kinesis Data Firehose is inefficient. Kinesis Firehose can automatically perform basic transformations and deliver data to S3, making the use of Glue unnecessary in this case. Glue jobs are better suited for more complex ETL processes, not simple streaming transformations.
I map out that the answer is:
Configure the devices to send streaming data to an Amazon Kinesis data stream. Configure an Amazon Kinesis Data Firehose delivery stream to automatically consume the Kinesis data stream, transform the data with an AWS Lambda function, and save the output into an Amazon S3 bucket.