How can this requirement be met at the LOWEST cost?
Have the application send its logs to an Amazon EMR cluster and normalize the logs before sending them to Amazon S3
Have the application send its logs to Amazon QuickSight, then use the Amazon QuickSight SPICE engine to normalize the logs. Do the analysis directly from Amazon QuickSight
Keep the logs in Amazon S3 and use Amazon Redshift Spectrum to normalize the logs in place
Use Amazon Kinesis Agent on each server to upload the logs and have Amazon Kinesis Data Firehose use an AWS Lambda function to normalize the logs before writing them to Amazon S3
Explanations:
Using an Amazon EMR cluster incurs higher costs due to resource allocation and management for processing logs, making it less economical compared to other options.
Amazon QuickSight is primarily a BI tool for visualization and analysis, not for log normalization. Additionally, using the SPICE engine involves costs related to data storage and computation.
Amazon Redshift Spectrum allows querying data in S3 but does not provide a mechanism for normalizing logs directly in S3, and utilizing Redshift incurs costs that may not be justified for simple log normalization.
Using Amazon Kinesis Agent with Kinesis Data Firehose and AWS Lambda allows for real-time log processing and normalization at a lower cost. This serverless architecture minimizes costs since you only pay for the resources consumed during log processing.
I figure that the answer is:
Use Amazon Kinesis Agent on each server to upload the logs and have Amazon Kinesis Data Firehose use an AWS Lambda function to normalize the logs before writing them to Amazon S3