IAM Roles for Service Accounts configuration

IAM Role for Service Account on EKS Anywhere clusters with self-hosted signing keys

IAM Roles for Service Account (IRSA) enables applications running in clusters to authenticate with AWS services using IAM roles. The current solution for leveraging this in EKS Anywhere involves creating your own OIDC provider for the cluster, and hosting your cluster’s public service account signing key. The public keys along with the OIDC discovery document should be hosted somewhere that AWS STS can discover it. The steps below assume the keys will be hosted on a publicly accessible S3 bucket. Refer this doc to ensure that the s3 bucket is publicly accessible.

The steps below are based on the guide for configuring IRSA for DIY Kubernetes, with modifications specific to EKS Anywhere’s cluster provisioning workflow. The main modification is the process of generating the keys.json document. As per the original guide, the user has to create the service account signing keys, and then use that to create the keys.json document prior to cluster creation. This order is reversed for EKS Anywhere clusters, so you will create the cluster first, and then retrieve the service account signing key generated by the cluster, and use it to create the keys.json document. The sections below show how to do this in detail.

Create an OIDC provider and make its discovery document publicly accessible

1. Create an s3 bucket to host the public signing keys and OIDC discovery document for your cluster as per this section. Ensure you follow all the steps and save the $HOSTNAME and $ISSUER_HOSTPATH.

2. Create the OIDC discovery document as follows:

   cat <<EOF > discovery.json
   {
       "issuer": "https://$ISSUER_HOSTPATH",
       "jwks_uri": "https://$ISSUER_HOSTPATH/keys.json",
       "authorization_endpoint": "urn:kubernetes:programmatic_authorization",
       "response_types_supported": [
           "id_token"
       ],
       "subject_types_supported": [
           "public"
       ],
       "id_token_signing_alg_values_supported": [
           "RS256"
       ],
       "claims_supported": [
           "sub",
           "iss"
       ]
   }
   EOF
   

3. Upload it to the publicly accessible S3 bucket:

   aws s3 cp --acl public-read ./discovery.json s3://$S3_BUCKET/.well-known/openid-configuration
   

4. Create an OIDC provider for your cluster. Set the Provider URL to https://$ISSUER_HOSTPATH, and audience to sts.amazonaws.com.

5. Note down the Provider field of OIDC provider after it is created.

6. Assign an IAM role to this OIDC provider.

   1. From the IAM console, select and click on the OIDC provider created from above, and click on Assign role at the top right.

   2. Select Create a new role.

   3. In the Select type of trusted entity section, choose Web identity.

   4. In the Choose a web identity provider section:

      • For Identity provider, choose the auto selected Identity Provider URL for your cluster.
      • For Audience, choose sts.amazonaws.com.

   5. Choose Next: Permissions.

   6. In the Attach Policy section, select the IAM policy that has the permissions that you want your applications running in the pods to use.

   7. Continue with the next sections of adding tags if desired and a suitable name for this role and create the role.

   8. Below is a sample trust policy of IAM role for your pods. Remember to replace Account ID and ISSUER_HOSTPATH with required values.

      {
       "Version": "2012-10-17",
       "Statement": [
        {
         "Effect": "Allow",
         "Principal": {
          "Federated": "arn:aws:iam::111122223333:oidc-provider/ISSUER_HOSTPATH"
         },
         "Action": "sts:AssumeRoleWithWebIdentity",
         "Condition": {
          "__doc_comment": "scope the role to the service account (optional)",
          "StringEquals": {
           "ISSUER_HOSTPATH:sub": "system:serviceaccount:default:my-serviceaccount"
          },
          "__doc_comment": "OR scope the role to a namespace (optional)",
          "StringLike": {
           "ISSUER_HOSTPATH/CLUSTER_ID:sub": ["system:serviceaccount:default:*","system:serviceaccount:observability:*"]
          }
         }
        }
       ]
      }
      

   9. After the role is created, note down the name of this IAM Role as OIDC_IAM_ROLE.

   10. Once the cluster is created, you can create service accounts and grant them this role by editing the trust relationship of this role. You can use StringLike condition to add required service accounts, as mentioned in the above sample. Please also refer to section Configure the trust relationship for the OIDC provider’s IAM Role .

Create the EKS Anywhere cluster

1. When creating the EKS Anywhere cluster, you need to configure the kube-apiserver’s service-account-issuer flag so it can issue and mount projected service account tokens in pods. For this, use the value obtained in the first section for $ISSUER_HOSTPATH as the service-account-issuer. Configure the kube-apiserver by setting this value through the EKS Anywhere cluster spec as follows:

   apiVersion: anywhere.eks.amazonaws.com/v1alpha1
   kind: Cluster
   metadata:
     name: my-cluster-name
   spec:
     podIamConfig:
       serviceAccountIssuer: https://$ISSUER_HOSTPATH
   

Set the remaining fields in cluster spec as required and create the cluster using the eksctl anywhere create cluster command.

Generate keys.json and make it publicly accessible

1. The cluster provisioning workflow generates a pair of service account signing keys. Retrieve the public signing key generated and used by the cluster, and create a keys.json document containing the public signing key.

   git clone https://github.com/aws/amazon-eks-pod-identity-webhook
   cd amazon-eks-pod-identity-webhook
   kubectl get secret ${CLUSTER_NAME}-sa -n eksa-system -o jsonpath={.data.tls\\.crt} | base64 --decode > ${CLUSTER_NAME}-sa.pub    
   go run ./hack/self-hosted/main.go -key ${CLUSTER_NAME}-sa.pub | jq '.keys += [.keys[0]] | .keys[1].kid = ""' > keys.json
   

2. Upload the keys.json document to the s3 bucket.

   aws s3 cp --acl public-read ./keys.json s3://$S3_BUCKET/keys.json
   

Deploy pod identity webhook

1. After hosting the service account public signing key and OIDC discovery documents, the applications running in pods can start accessing the desired AWS resources, as long as the pod is mounted with the right service account tokens. This part of configuring the pods with the right service account tokens and env vars is automated by the amazon pod identity webhook . Once the webhook is deployed, it mutates any pods launched using service accounts annotated with eks.amazonaws.com/role-arn

2. Clone amazon-eks-pod-identity-webhook if not done already.

3. Set the $KUBECONFIG env var to the path of the EKS Anywhere cluster.

4. Create my-service-account.yaml with OIDC_IAM_ROLE and other annotations as mentioned in sample below.

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: my-serviceaccount
     namespace: default
     annotations:
       # set this with value of OIDC_IAM_ROLE      
       eks.amazonaws.com/role-arn: "arn:aws:iam::111122223333:role/s3-reader"
       # optional: Defaults to "sts.amazonaws.com" if not set
       eks.amazonaws.com/audience: "sts.amazonaws.com"
       # optional: When set to "true", adds AWS_STS_REGIONAL_ENDPOINTS env var
       #   to containers
       eks.amazonaws.com/sts-regional-endpoints: "true"
       # optional: Defaults to 86400 for expirationSeconds if not set
       #   Note: This value can be overwritten if specified in the pod 
       #         annotation as shown in the next step.
       eks.amazonaws.com/token-expiration: "86400"
   

5. Run the following command to apply the manifests for the amazon-eks-pod-identity-webhook. The image used here will be pulled from docker.io. Optionally, the image can be imported into (or proxied through) your private registry. Change the IMAGE= argument here to your private registry if needed.

   make cluster-up IMAGE=amazon/amazon-eks-pod-identity-webhook:latest
   

6. Finally, apply the my-service-account.yaml file to create your service account.

   kubectl apply -f my-service-account.yaml
   

7. You can validate IRSA by using test steps mentioned here . Ensure awscli pod is deployed in same namespace of ServiceAccount pod-identity-webhook.

Configure the trust relationship for the OIDC provider’s IAM Role

In order to grant certain service accounts access to the desired AWS resources, edit the trust relationship for the OIDC provider’s IAM Role (OIDC_IAM_ROLE) created in the first section, and add in the desired service accounts.

1. Choose the role in the console to open it for editing.

2. Choose the Trust relationships tab, and then choose Edit trust relationship.

3. Find the line that looks similar to the following:

   "$ISSUER_HOSTPATH:aud": "sts.amazonaws.com"
   

4. Change the line to look like the following line. Replace aud with sub and replace KUBERNETES_SERVICE_ACCOUNT_NAMESPACE and KUBERNETES_SERVICE_ACCOUNT_NAME with the name of your Kubernetes service account and the Kubernetes namespace that the account exists in.

   "$ISSUER_HOSTPATH:sub": "system:serviceaccount:KUBERNETES_SERVICE_ACCOUNT_NAMESPACE:KUBERNETES_SERVICE_ACCOUNT_NAME"
   

5. Refer this doc for different ways of configuring one or multiple service accounts through the condition operators in the trust relationship.

6. Choose Update Trust Policy to finish.