How can the ML team solve this issue?
Decrease the cooldown period for the scale-in activity. Increase the configured maximum capacity of instances.
Replace the current endpoint with a multi-model endpoint using SageMaker.
Set up Amazon API Gateway and AWS Lambda to trigger the SageMaker inference endpoint.
Increase the cooldown period for the scale-out activity.
Explanations:
Decreasing the cooldown period for scale-in activity does not address the problem of new instances being launched before previous ones are ready. Increasing the maximum capacity also does not solve the issue of scaling too quickly.
A multi-model endpoint is used for serving multiple models from a single endpoint, which is not related to the issue of scaling behavior. This does not address the timing of instance scaling.
Using Amazon API Gateway and AWS Lambda to trigger the inference endpoint would introduce additional complexity and is not relevant to solving the scaling issue with SageMaker instances.
Increasing the cooldown period for scale-out activity ensures that the system waits for new instances to be ready before launching more, which addresses the problem of premature scaling.
I conclude that the answer is:
Increase the cooldown period for the scale-out activity.