What is the best approach to meet your customer’s requirements?
Send all the log events to Amazon SQS, setup an Auto Scaling group of EC2 servers to consume the logs and apply the heuristics.
Send all the log events to Amazon Kinesis, develop a client process to apply heuristics on the logs
Configure Amazon CloudTrail to receive custom logs, use EMR to apply heuristics the logs
Setup an Auto Scaling group of EC2 syslogd servers, store the logs on S3, use EMR to apply heuristics on the logs
Explanations:
Using Amazon SQS for log events is not ideal for real-time processing. SQS is designed for decoupling components rather than for real-time log analysis. The latency involved in pulling logs from SQS could hinder the real-time requirement.
Amazon Kinesis is designed for real-time data streaming and processing. It allows the customer to send log events, apply heuristics immediately, and retain data for 12 hours, meeting the requirements for real-time analysis and validation.
Amazon CloudTrail is primarily used for tracking API calls and actions in AWS services, not for general log streaming. Using EMR for log analysis does not address the real-time aspect and may not support immediate heuristics validation effectively.
While using an Auto Scaling group and storing logs in S3 can provide scalability, it does not enable real-time analysis since S3 is a storage service and requires batch processing for analysis, which is not suitable for the immediate heuristic validation needed.
I rate that the answer is:
Send all the log events to Amazon Kinesis, develop a client process to apply heuristics on the logs
From what I’ve heard, the answer is:
Send all the log events to Amazon Kinesis, develop a client process to apply heuristics on the logs