What best practice should first be applied to address this issue?
Contact AWS Support for a limit increase.
Use the AWS CLI to get the metrics.
Analyze the applications and remove the API call.
Retry the call with exponential backoff.
Explanations:
Contacting AWS Support for a limit increase may be necessary if the application’s API usage consistently exceeds the allocated limits, but it is not the immediate best practice to address intermittent throttling exceptions. This option does not handle the current issue of failed calls due to throttling.
Using the AWS CLI to retrieve metrics does not address the underlying throttling issue causing the HTTP 400: ThrottlingException errors. This approach may also face the same throttling limits as the API calls from the application, leading to similar failures.
Analyzing the applications and removing the API call may be a long-term solution but does not directly resolve the intermittent throttling issue. This option could lead to loss of necessary functionality and does not address the problem of handling errors when they occur.
Retrying the call with exponential backoff is a recommended best practice for handling throttling exceptions. This method allows the application to pause and retry requests after increasing intervals, which helps avoid overwhelming the API and adheres to the rate limits imposed by the CloudWatch API, leading to better success rates in subsequent attempts.
I would say the answer is:
Retry the call with exponential backoff.