What is the root cause of this issue?

Explanations:

The IAM role that is attached to the EKS cluster typically has the necessary permissions to interact with AWS services, including Secrets Manager. If the EKS cluster IAM role itself did not have access, other applications or services running on the cluster would encounter similar issues.

The error indicates that the service account is unable to access Secrets Manager, which suggests that the key policy for the customer managed key may not grant the necessary permissions for the Kubernetes service account IAM role to decrypt the secrets. Without access to the KMS key, the service account cannot decrypt the secrets even if it can access Secrets Manager.

The key policy should primarily control access for the IAM role used by the Kubernetes service account. If the EKS cluster IAM role itself were the issue, it would have more broad implications on the cluster’s ability to access AWS services, not just Secrets Manager. The root cause is related to the permissions for the service account, not the cluster role.

The IAM role assumed by the Kubernetes service account specifically grants permissions needed to access Secrets Manager. An Access Denied error would not stem from a lack of permissions to access the EKS cluster itself but rather from issues related to accessing the secrets or the associated KMS key.