How should the solutions architect meet these requirements?
Enable VPC flows logs, and send them to CloudWatch. Create an AWS Lambda function to periodically export the CloudWatch logs to an Amazon S3 bucket by using the pre-defined export function. Generate ACCESS_KEY and SECRET_KEY AWS credentials. Configure Splunk to pull the logs from the S3 bucket by using those credentials.
Create an Amazon Kinesis Data Firehose delivery stream with Splunk as the destination. Configure a pre-processing AWS Lambda function with a Kinesis Data Firehose stream processor that extracts individual log events from records sent by CloudWatch Logs subscription filters. Enable VPC flows logs, and send them to CloudWatch. Create a CloudWatch Logs subscription that sends log events to the Kinesis Data Firehose delivery stream.
Ask the company to log every request that is made to the databases along with the EC2 instance IP address. Export the CloudWatch logs to an Amazon S3 bucket. Use Amazon Athena to query the logs grouped by database name. Export Athena results to another S3 bucket. Invoke an AWS Lambda function to automatically send any new file that is put in the S3 bucket to Splunk.
Send the CloudWatch logs to an Amazon Kinesis data stream with Amazon Kinesis Data Analytics for SQL Applications. Configure a 1-minute sliding window to collect the events. Create a SQL query that uses the anomaly detection template to monitor any networking traffic anomalies in near-real time. Send the result to an Amazon Kinesis Data Firehose delivery stream with Splunk as the destination.
Explanations:
While enabling VPC flow logs and sending them to CloudWatch is a good practice, the approach of exporting logs to S3 and configuring Splunk to pull logs from S3 introduces latency and does not meet the near-real-time requirement. Moreover, it requires additional setup for AWS Lambda and S3 management, which complicates the solution unnecessarily.
This option effectively addresses the requirement for near-real-time monitoring by using Amazon Kinesis Data Firehose to stream logs directly to Splunk. The integration of CloudWatch Logs subscription filters and a pre-processing Lambda function allows for the immediate extraction of relevant log events, ensuring timely data transfer to Splunk.
This option involves multiple steps of logging requests, exporting logs to S3, and querying with Amazon Athena. While it allows for analysis of logs, it does not provide near-real-time monitoring as it relies on periodic querying and exporting of results, leading to delays in data availability for Splunk.
Although using Kinesis Data Analytics can help in monitoring network traffic, this option adds complexity by involving SQL queries and anomaly detection which are unnecessary for the goal of simply monitoring which EC2 instances are connected to the databases. Additionally, it doesn’t specify a direct way to send data to Splunk, making it less efficient for the requirement at hand.
I surmise that the answer is:
Create an Amazon Kinesis Data Firehose delivery stream with Splunk as the destination. Configure a pre-processing AWS Lambda function with a Kinesis Data Firehose stream processor that extracts individual log events from records sent by CloudWatch Logs subscription filters. Enable VPC flows logs, and send them to CloudWatch. Create a CloudWatch Logs subscription that sends log events to the Kinesis Data Firehose delivery stream.