Federate K3s cluster with AWS OIDC

Federate K3s cluster with AWS OIDC

Posted
Federated K3s with AWS OIDC
Federated K3s with AWS OIDC
By Yevhen Vasylenko
12 min read
Federate K3s cluster with AWS OIDC

Binding AWS IAM Roles to K3S Service accounts

In this short tutorial, you will learn how to configure the IAM roles for Service Account for a bare-metal cluster using k3s as an example. As public OIDC enpoint will be used public S3 bucket.

Provision K3S cluster

Provision on bare metal 

1

curl -sfL https://get.k3s.io | sh -

After cluster will start we need stop it,

1
2

systemctl stop k3s
/usr/local/bin/k3s-killall.sh

and edit it’s configuration. /etc/rancher/k3s/config.yaml We need add next values to kube-apiserver-arg:

1
2
3
4
5

kube-apiserver-arg:
  - 'service-account-issuer=https://[bucket name].s3.amazonaws.com'
  - 'service-account-jwks-uri=https://[bucket name].s3.amazonaws.com/openid/v1/jwks'
  - 'service-account-lookup=true'
  - 'api-audiences=k3s-cluster'

And now we should start k3s cluster

1

systemctl start k3s

Use k3d for local testing 

1
2
3
4

k3d cluster create test  --k3s-arg "--kube-apiserver-arg=--service-account-issuer=https://[bucket name].s3.amazonaws.com@server:0" \
--k3s-arg "--kube-apiserver-arg=--service-account-jwks-uri=https://[bucket name].s3.amazonaws.com/openid/v1/jwks@server:0" \
--k3s-arg "--kube-apiserver-arg=--service-account-lookup=true@server:0" \
--k3s-arg "--kube-apiserver-arg=--api-audiences=k3d-cluster@server:0"

Get OIDC configuration from k3s cluster

Now we need get configuration for OIDC from cluster:

1
2

kubectl get --raw /.well-known/openid-configuration > openid-configuration
kubectl get --raw /openid/v1/jwks > jwks

The content of the openid-configuration file should be:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

{
  "issuer": "https://[bucket name].s3.amazonaws.com",
  "jwks_uri": "https://[bucket name].s3.amazonaws.com/openid/v1/jwks",
  "response_types_supported": [
    "id_token"
  ],
  "subject_types_supported": [
    "public"
  ],
  "id_token_signing_alg_values_supported": [
    "RS256"
  ]
}

And for the jwks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  "keys": [
    {
      "use": "sig",
      "kty": "RSA",
      "kid": "ZO4TUgVjBzMWKVP8mmBwKLvsuyn8z-gfqUp27q9lO4w",
      "alg": "RS256",
      "n": "34a81xuM…",
      "e": "AQAB"
    }
  ]
}

Create s3 bucket and upload OIDC configurations

Furst of all we need create s3 bucket with Turned OFF Block Public Access option. Public S3 Bucket And add next bucket policy to make it public:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::[bucket name]/*"
        }
    ]
}

Upload the files on the S3 bucket with the following directory structure:

.well-known/openid-configuration for the configuration.

openid/v1/jwks for the configuration.

Test that the endpoints are publicly reachable:

1
2

curl https://[bucket name].s3.amazonaws.com/.well-known/openid-configuration
curl https://[bucket name].s3.amazonaws.com/openid/v1/jwks

Create an Open ID connect provider on AWS

On AWS console navigate to IAM -> Identity providers and push ADD Provider button. Fill in values that is required for creating OIDC provider Create OIDC

Create Role for SA

From the AWS console, navigate to IAM > Roles > click on the Create Role and choose Web identity role. Select OIDC provider that you have just create and also select audience (NOTE: It shoul be equal you put in k3s configuration)

  • Create a new role.
  • Type “web identity”.
  • Assign the policies.
  • Edit the trust relationships.

The content for the trust relationship should be the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::[aws account id]:oidc-provider/[bucket name].s3.amazonaws.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "[bucket name].s3.amazonaws.com:aud": "test", // <- this is the audience
                    "[bucket name].s3.amazonaws.com:sub": "system:serviceaccount:default:test" // <- optional
                }
            }
        }
    ]
}

Make a note of the Role ARN.

Test Integration

Apply next kubernetes manifest to cluster:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test
  namespace: default
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: "test-configmap"
data:
  AWS_DEFAULT_REGION: us-east-1
  AWS_ROLE_ARN: "arn:aws:iam::[aws account id]:role/[role name]"  # matches `RoleName` in the CloudFormation template
  AWS_WEB_IDENTITY_TOKEN_FILE: /var/run/secrets/kubernetes.io/serviceaccount/token
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: "aws-cli"
  labels:
    app: "aws-cli"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "aws-cli"
  template:
    metadata:
      labels:
        app: "aws-cli"
    spec:
      serviceAccountName: "test"
      containers:
        - name: "aws-cli"
          image: amazon/aws-cli:2.15.8
          command: ["sleep"]
          args: ["86400"]
          envFrom:
            - configMapRef:
                name: "test-configmap"

After deployment will be up and running it is time to check integration:

1kubectl exec -it [pod name] bash
2
3aws sts get-caller-identity
4{
5    "UserId": "AROAXXXXXXXXXXXXXXXXX:botocore-session-1608249397",
6    "Account": "000000000000",
7    "Arn": "arn:aws:iam::000000000000:role/blog-example-iam-role/botocore-session-1608249397"
8}
k3s kubernetes AWS OIDC