Which solution meets the company’s requirements?

By:
On:
In: SAP-C01
Tagged:
With: 0 Comments
A company that provides wireless services needs a solution to store and analyze log files about user activities.Currently, log files are delivered daily to AmazonLinux on an Amazon EC2 instance.A batch script is run once a day to aggregate data used for analysis by a third-party tool.The data pushed to the third-party tool is used to generate a visualization for end users.The batch script is cumbersome to maintain, and it takes several hours to deliver the ever-increasing data volumes to the third-party tool.The company wants to lower costs, and is open to considering a new tool that minimizes development effort and lowers administrative overhead.The company wants to build a more agile solution that can store and perform the analysis in near-real time, with minimal overhead.The solution needs to be cost effective and scalable to meet the company’s end-user base growth.

Which solution meets the company’s requirements?

Develop a Python script to capture the data from Amazon EC2 in real time and store the data in Amazon S3. Use a copy command to copy data from Amazon S3 to Amazon Redshift. Connect a business intelligence tool running on Amazon EC2 to Amazon Redshift and create the visualizations.

Use an Amazon Kinesis agent running on an EC2 instance in an Auto Scaling group to collect and send the data to an Amazon Kinesis Data Firehose delivery stream. The Kinesis Data Firehose delivery stream will deliver the data directly to Amazon ES. Use Kibana to visualize the data.

Use an in-memory caching application running on an Amazon EBS-optimized EC2 instance to capture the log data in near real-time. Install an Amazon ES cluster on the same EC2 instance to store the log files as they are delivered to Amazon EC2 in near real-time. Install a Kibana plugin to create the visualizations.

Use an Amazon Kinesis agent running on an EC2 instance to collect and send the data to an Amazon Kinesis Data Firehose delivery stream. The Kinesis Data Firehose delivery stream will deliver the data to Amazon S3. Use an AWS Lambda function to deliver the data from Amazon S3 to Amazon ES. Use Kibana to visualize the data.

Explanations:

This option relies on a Python script and a batch process to move data from S3 to Redshift, which does not address the need for near-real-time analysis and requires significant maintenance. Additionally, it involves more administrative overhead compared to streaming solutions.

This option utilizes Amazon Kinesis to stream data in real time to Amazon Elasticsearch Service (ES) for storage and visualization via Kibana. It provides scalability, minimizes administrative overhead, and allows for near-real-time analysis, which aligns well with the company’s requirements.

This option proposes an in-memory caching application and an ES cluster on the same EC2 instance, which could lead to resource contention and scalability issues. It does not provide the streaming capabilities required for near-real-time analysis and adds complexity in management and maintenance.

Although this option uses Kinesis Data Firehose to send data to S3 and involves AWS Lambda to deliver data to Amazon ES, it does not achieve the desired near-real-time analysis as effectively as option B. The use of S3 introduces latency compared to direct streaming to ES.

Leave a Reply

Your email address will not be published. Required fields are marked *

1 × 3 =