Question:
I have created the EKS cluster using AWS console, while creating a cluster I used my pre-created VPCs and subnets, I have created one role eks-role which has AmazonEKSClusterPolicy and AmazonEKSServicePolicy attached to it.
I have added the kubeconfig file using:
|
1 2 |
aws eks update-kubeconfig --name eks-cluster --role-arn "arn:aws:iam::############:role/eks-role" |
When I use kubectl get svc command I get the error as:
An error occurred (AccessDenied) when calling the AssumeRole operation: Access denied
I don’t know what might be wrong with this.
In My user, I have added a policy as:
|
1 2 3 4 5 6 7 8 9 10 11 |
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam::############:role/eks-role" } ] } |
And In the role I have added the trust relationship:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
{ "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::############:user/test" }, "Action": "sts:AssumeRole" } ] } |
My ~/.aws/credentials file looks like this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
**[default]** aws_access_key_id = ############## aws_secret_access_key = ############################# region=us-west-1 **[test]** aws_access_key_id = ########### aws_secret_access_key = ############################# region=ap-southeast-1 **[eks]** role_arn = arn:aws:iam::##########:role/eks-role source_profile = test |
Answer:
I am sure issue is resolved but I will be putting more information here so if any other people are still facing the issue then they might not waste time like me and use the steps.
When we create the EKS cluster by any method via CloudFormation/CLI/EKSCTL the IAM role/user who created the cluster will automatically binded to the default kubernetes RBAC API group system:masters (https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) and in this way creator of the cluster will get the admin access to the cluster. Although we can always give the access to other IAM user/role using the aws-auth file but for that we must have to use the IAM user/role who created the cluster.
To verify the role/user for the EKS cluster we can search for the CreateCluster" Api call on cloudtrail and it will tell us the creator of the cluster in the sessionIssuer section for field arn (https://docs.aws.amazon.com/awscloudtrail/latest/userguide/view-cloudtrail-events.html).
When we create the cluster using the IAM role or IAM user, setting up the access for the EKS cluster will become little tricky when we created the cluster using the role compare to user.
I will put the steps we can follow for each different method while setting up the access to EKS cluster.
Scenario-1: Cluster was Created using the IAM user (For example “eks-user”)
Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity
|
1 2 3 4 5 6 7 |
$ aws sts get-caller-identity { "Account": "xxxxxxxxxxxx", "UserId": "xxxxxxxxxxxxxxxxxxxxx", "Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user" } |
After that update the kubeconfig file using the below command
|
1 2 |
aws eks --region region-code update-kubeconfig --name cluster_name |
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
$ cat ~/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: CERT server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster contexts: - context: cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster kind: Config preferences: {} users: - name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: exec: apiVersion: client.authentication.k8s.io/v1alpha1 args: - --region - us-east-1 - eks - get-token - --cluster-name - eks-cluster command: aws |
Once above setup is done you should be able to run the kubectl command.
|
1 2 3 4 |
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP xxx.xx.x.x |
Scenario-2: Cluster was Created using the IAM Role (For example “eks-role”)
Mainly there are four different way to setup the access via cli when cluster was created via IAM role.
1. Setting up the role directly in kubeconfig file.
In this case we do not have to make any assume role api call via cli manually, before running kubectl command because that will be automatically done by aws/aws-iam-authenticator set in the kube config file.
Lets say now we are trying to setup the access for the user eks-user the first make sure that user does have permission to assume the role eks-role
Add the assume role permission to the eks-user
|
1 2 3 4 5 6 7 8 9 10 11 |
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Resource": "arn:aws:iam::xxxxxxxxxxx:role/eks-role" } ] } |
Edit the trust relationship on the role so that it will allow the eks-user to assume the role.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
{ "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::xxxxxxxxxxx:user/eks-user" }, "Action": "sts:AssumeRole" } ] } |
Confirm that IAM user credentials are set properly on AWS cli who has created the cluster via running the command aws sts get-caller-identity. Important thing to remember it should show us the IAM user ARN not the IAM assumed ROLE ARN.
|
1 2 3 4 5 6 7 |
$ aws sts get-caller-identity { "Account": "xxxxxxxxxxxx", "UserId": "xxxxxxxxxxxxxxxxxxxxx", "Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user" } |
After that update the kubeconfig file using the below command
|
1 2 |
aws eks --region region-code update-kubeconfig --name cluster_name --role-arn arn:aws:iam::xxxxxxxxxxx:user/eks-role |
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
$ cat ~/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: CERT server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster contexts: - context: cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster kind: Config preferences: {} users: - name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: exec: apiVersion: client.authentication.k8s.io/v1alpha1 args: - --region - us-east-1 - eks - get-token - --cluster-name - eks-cluster - --role - arn:aws:iam::xxxxxxx:role/eks-role command: aws |
Once above setup is done you should be able to run the kubectl command.
|
1 2 3 4 |
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP xxx.xx.x.x |
2. If you have setup the AWS profile (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) on CLI and if you want to use that with the kube config.
Confirm that profile is set properly so that it can use the credentials for the eks-user
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
$ cat ~/.aws/config [default] output = json region = us-east-1 [eks] output = json region = us-east-1 [profile adminrole] role_arn = arn:aws:iam::############:role/eks-role source_profile = eks $ cat ~/.aws/credentials [default] aws_access_key_id = xxxxxxxxxxxx aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx [eks] aws_access_key_id = xxxxxxxxxxxx aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx |
Once this profile configuration is done please confirm that profile configuration is fine by running the command aws sts get-caller-identity --profile eks
|
1 2 3 4 5 6 7 |
$ aws sts get-caller-identity --profile eks { "Account": "xxxxxxxxxxxx", "UserId": "xxxxxxxxxxxxxxxxxxxxx", "Arn": "arn:aws:iam::xxxxxxxxxxx:user/eks-user" } |
After that update the kubeconfig file using the below command with the profile and please make sure we are not using the role here.
|
1 2 |
aws eks update-kubeconfig --name devel --profile eks |
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
$ cat ~/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: CERT server: https://xxxxx.sk1.us-east-1.eks.amazonaws.com name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster contexts: - context: cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster kind: Config preferences: {} users: - name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: exec: apiVersion: client.authentication.k8s.io/v1alpha1 args: - --region - us-east-1 - eks - get-token - --cluster-name - eks-cluster command: aws env: - name: AWS_PROFILE value: eks |
Once above setup is done you should be able to run the kubectl command.
|
1 2 3 4 |
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP xxx.xx.x.x |
3. Assume the role by any other way, For example we can attach the IAM role to the instance directly.
If role is directly attached to the instance profile then we can follow the similar steps as we followed while setting up the access for IAM user in Scenario-1
Verify that we have attached the correct role to EC2 instance and as this instance profile will come into least precedence, this step will also verify that there are no any other credentials setup on the instnace.
|
1 2 3 4 5 6 7 |
[ec2-user@ip-xx-xxx-xx-252 ~]$ aws sts get-caller-identity { "Account": "xxxxxxxxxxxx", "UserId": "xxxxxxxxxxxxxxxxxxxxx:i-xxxxxxxxxxx", "Arn": "arn:aws:sts::xxxxxxxxxxxx:assumed-role/eks-role/i-xxxxxxxxxxx" } |
After that update the kubeconfig file using the below command
|
1 2 |
aws eks --region region-code update-kubeconfig --name cluster_name |
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
$ cat ~/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: CERT server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster contexts: - context: cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster kind: Config preferences: {} users: - name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: exec: apiVersion: client.authentication.k8s.io/v1alpha1 args: - --region - us-east-1 - eks - get-token - --cluster-name - eks-cluster command: aws |
Once above setup is done you should be able to run the kubectl command.
|
1 2 3 4 |
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP xxx.xx.x.x |
4. Manually assuming the IAM role via aws sts assume-role command.
Assume the role eks-role manually by running the cli command.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
aws sts assume-role --role-arn arn:aws:iam::xxxxxxxxxxx:role/eks-role --role-session-name test { "AssumedRoleUser": { "AssumedRoleId": "xxxxxxxxxxxxxxxxxxxx:test", "Arn": "arn:aws:sts::xxxxxxxxxxx:assumed-role/eks-role/test" }, "Credentials": { "SecretAccessKey": "xxxxxxxxxx", "SessionToken": xxxxxxxxxxx", "Expiration": "xxxxxxxxx", "AccessKeyId": "xxxxxxxxxx" } } |
After that set the required environment variable using the value from above output so that we can use the correct credentials generated from the session.
|
1 2 3 4 |
export AWS_ACCESS_KEY_ID=xxxxxxxxxx export AWS_SECRET_ACCESS_KEY=xxxxxxxxxxx export AWS_SESSION_TOKEN=xxxxxxxxxx |
After that verify that we assumed the IAM role by running the command aws sts get-caller-identity.
$ aws sts get-caller-identity
{
“Account”: “xxxxxxxxxx”,
“UserId”: “xxxxxxxxxx:test”,
“Arn”: “arn:aws:sts::xxxxxxxxxx:assumed-role/eks-role/test”
}
After that update the kubeconfig file using the below command
|
1 2 |
aws eks --region region-code update-kubeconfig --name cluster_name |
Attaching the config file how it looks like once updated via above command. Please do not directly edit this file until and unless necessary.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
$ cat ~/.kube/config apiVersion: v1 clusters: - cluster: certificate-authority-data: CERT server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster contexts: - context: cluster: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster current-context: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster kind: Config preferences: {} users: - name: arn:aws:eks:us-east-1:xxxxxxx:cluster/eks-cluster user: exec: apiVersion: client.authentication.k8s.io/v1alpha1 args: - --region - us-east-1 - eks - get-token - --cluster-name - eks-cluster command: aws |
Once above setup is done you should be able to run the kubectl command.
|
1 2 3 4 |
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP xxx.xx.x.x |
NOTE:
I have try to cover major use case here but there might be other use case too where we need to setup the access to the cluster.
Also the above tests are mainly aiming at the first time setup of the EKS cluster and none of the above method is touching the aws-auth configmap yet.
But once you have given access to other IAM user/role to EKS cluster via aws-auth (https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html) file you can use the same set of commands for those users too.