Which setup win meet the requirements?

By:
In: SAP-C01
Tagged:
With: 1 Comment
You have recently joined a startup company building sensors to measure street noise and air quality in urban areas.The company has been running a pilot deployment of around 100 sensors for 3 months each sensor uploads 1KB of sensor data every minute to a backend hosted on AWS.During the pilot, you measured a peak or 10 IOPS on the database, and you stored an average of 3GB of sensor data per month in the database.The current deployment consists of a load-balanced auto scaled Ingestion layer using EC2 instances and a PostgreSQL RDS database with 500GB standard storage.The pilot is considered a success and your CEO has managed to get the attention or some potential investors.The business plan requires a deployment of at least 100K sensors which needs to be supported by the backend.You also need to store sensor data for at least two years to be able to compare year over yearImprovements.To secure funding, you have to make sure that the platform meets these requirements and leaves room for further scaling.

Which setup win meet the requirements?

Add an SQS queue to the ingestion layer to buffer writes to the RDS instance

Ingest data into a DynamoDB table and move old data to a Redshift cluster

Replace the RDS instance with a 6 node Redshift cluster with 96TB of storage

Keep the current architecture but upgrade RDS storage to 3TB and 10K provisioned IOPS

Explanations:

While adding an SQS queue can help manage bursts in traffic and buffer writes, it does not directly address the long-term storage needs and scalability for 100K sensors. It could lead to increased complexity without ensuring sufficient database capacity for the expected data volume.

Ingesting data into DynamoDB allows for high scalability and can handle the required IOPS for 100K sensors. Additionally, moving older data to a Redshift cluster is suitable for long-term storage and analytics, fulfilling the requirement of storing data for at least two years.

Replacing RDS with a Redshift cluster may not be ideal for real-time ingestion of sensor data, as Redshift is optimized for analytics rather than transactional workloads. This could lead to challenges in processing incoming data from 100K sensors effectively.

Upgrading RDS storage to 3TB and 10K provisioned IOPS might improve the existing setup but is unlikely to support the volume of data generated by 100K sensors over two years. The total data storage requirement would exceed RDS capabilities, necessitating a more scalable solution.

2025-10-15

1 Comment

  1. Zachary
    Author

    I would say the answer is:
    Ingest data into a DynamoDB table and move old data to a Redshift cluster

Leave a Reply

Your email address will not be published. Required fields are marked *

3 × 5 =