Scaleway k8s with google OIDC provider

OIDC in a managed K8s cluster means that the generated JWT token can be used to manage resources via kubectl; in other words, authorization to the API server. It is commonly used as a secure, simplified approach to authentication across multiple applications.

In our situation we used terraform to manage cluster infrastructure, so our example will include terraform code for the API server configuration, and some YAML code for RBAC with cluster manifests. Note that this was for a POC, and was designed to “get it working” as opposed to “production ready”.

This documentation does not go into cluster networking or any other specifics.

Setting up Google OIDC

1. Setup a new project for creating OAuth credentials (this may require filling out the OAuth consent page). 2. in api/credentials create a new OAuth 2.0 Client ID with CREATE CREDENTIALS -> OAuth Client ID
2. For application type, if other is not present, select Desktop App and provide a name for the client.
3. Once the client is created you will have a Client ID and Client Secret that will be later.

Configure API Server with OIDC config

Using the scaleway terraform module create or update the cluster resource scaleway_k8s_cluster with the open_id_connect_config

1. Update the terraform cluster configuration to include the OIDC component.

resource "scaleway_k8s_cluster" "poc" {
  name                        = "poc"
  version                     = "1.29.1"
  cni                         = "cilium"
  private_network_id          = scaleway_vpc_private_network.pn.id
  delete_additional_resources = false
  type                        = "kapsule-dedicated-4"
  open_id_connect_config {
    issuer_url = "https://accounts.google.com"
    client_id  = "xxxxxx.apps.googleusercontent.com"
    username_claim = "email"
    groups_claim = ["groups"]
  }
}

client_id should be the client ID from the google OAuth credentials username_claims = email: Tells the JWT token that the username will be the email address.

NOTE: This cluster resource assumes a scaleway vpc private network resource has been created, or will be along with this code snippit. Additional resources like node pools, gateways, etc. will probably be necessary to get a fully functional cluster.

2. Apply the configuration changes

Initial cluster access

If you already have access to the scaleway cluster and permissions to deploy RBAC manifests, you can skip this section. Otherwise, I will quickly go through my “hacky” way to get admin access.

1. You can download the kubeconfig file from the scaleway console or
2. Install the scaleway client (scw)
3. Create an API key in the scaleway console
4. Either export the keys as env variables or run scw init
5. Once the cli is setup, get the cluster ID with scw k8s cluster list
6. Then get the kubeconfig file using the cluster ID and save it to a file with scw k8s kubeconfig get <CLUSTERID> > scw_kubeconfig
7. Use the admin kubeconfig file to access the cluster KUBECONFIG=scw_kubeconfig kubectl get pods

Create RBAC user role and rolebinding

For the OIDC user to access resources in the cluster, they need a role that matches their email associated with the JWT token, and rolebinding for authorization to resources. For a quick example I created the following rbac.yaml file:

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-role
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: test-role-binding
subjects:
  - kind: User
    name: [email protected]
roleRef:
  kind: Role
  name: test-role
  apiGroup: rbac.authorization.k8s.io

This creates a role with mostly unlimited access at the namespace level, and binds it to a user with the name of my email address.

Apply the rbac.yaml file to the cluster: KUBECONFIG=scw_kubeconfig kubectl apply -f rbac.yaml

Authenticate to Google

There are multiple ways to authenticate the user with the cluster and google. There are a couple of 3rd party plugins that automate and simplify the connection between user to cluster with OIDC:

I chose to use kubelogin to automate the authentication. I installed it using the krew plugin manger. kubectl krew install oidc-login.

Then, I did the following to update my kubeconfig with the provider credentials:

KUBECONFIG=scw-kubeconfig kubectl config set-credentials "[email protected]" \
  --exec-api-version=client.authentication.k8s.io/v1beta1 \
  --exec-command=kubectl \
  --exec-arg=oidc-login \
  --exec-arg=get-token \
  --exec-arg=--oidc-extra-scope=email \
  --exec-arg=--oidc-issuer-url="https://accounts.google.com" \
  --exec-arg=--oidc-client-id="CLIENT_ID.apps.googleusercontent.com" \
  --exec-arg=--oidc-client-secret="CLIENT_SECRET"

The CLIENT_ID and CLIENT_SECRET are the OAuth credentials created in google

You may also manually add the user yourself to the kubeconfig, and have it look something like this:

users:
- name: [email protected]
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-extra-scope=email
      - --oidc-issuer-url=https://accounts.google.com
      - --oidc-client-id=XXXX.apps.googleusercontent.com
      - --oidc-client-secret=XXXXXXX
      command: kubectl
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

NOTE: Be sure you either specify the KUBECONFIG variable at the beginning of each command, export the variable name, or store the file in ~/.kube to ensure you’re using the desired context.

Now, when you attempt to access the cluster with the specified user, it should redirect you to sign-in to google and then provide you the results.

KUBECONFIG=scw-config kubectl get pods [email protected]

Conclusion

Again, I want to iterate that this was more of a “me learning OIDC and scaleway infrastructure” as opposed to “creating a production ready solution”. Enjoy :)

Headline photo by Katya Ross