What is MOST likely causing this issue?
The EC2 instances are in the same Availability Zone, causing contention between the two.
The route tables are not updated to allow traffic to flow between the ALB and the EC2 instances.
The ALB health check has failed, and the ALB has taken EC2 instances out of service.
The Amazon Route 53 health check has failed, and the ALB has taken EC2 instances out of service.
Explanations:
The issue is not due to contention between EC2 instances in the same Availability Zone. The ALB does not have contention issues from instances in the same zone.
Route tables are typically not the cause for this issue. As long as the ALB and EC2 instances are within the same VPC, routing should not be the primary cause of the issue.
This is the most likely cause. The drop in HealthyHostCount suggests that the ALB health check failed, causing the instances to be marked as unhealthy and taken out of service.
The issue is related to the ALB’s internal health check, not the Route 53 health check. Route 53 health checks would affect DNS routing, not the ALB’s HealthyHostCount.
I surmise that the answer is:
The ALB health check has failed, and the ALB has taken EC2 instances out of service.