kubeconfig – Brian Ragazzi

I confess, most of my experience with Kubernetes is with Pivotal Container Service (PKS) Enterprise. PKS makes it rather easy to get started and I found that I took some tasks for granted.

In PKS Enterprise, one can use the pks cli to not only life-cycle clusters, but to obtain the credentials to the cluster and automatically update the kubeconfig with the account information. So, administrative/operations users can run the command “pks get-credentials my-cluster” to have a kubeconfig updated with the authentication tokens and parameters to connect to my-cluster.

The PKS controller includes the User Account and Authentication (UAA) component, which is used to authenticate users into PKS Enterprise. UAA can also be easily configured to connect to an existing LDAP service – this is the desired configuration in most organizations so that users account exist in one place (Active Directory in my example).

So, I found myself wondering “I don’t want to provide the PKS CLI to developers, so how can they connect kubectl to the cluster?”

Assumptions:

PKS Enterprise on vSphere (with or without NSX-T)
Active Directory
Developer user account belongs to the k8s-devs security group in AD

Prerequisite configuration:

UAA on PKS configured a with UAA User Account Store: LDAP Server. This links UAA to LDAP/Active Directory
User Search Filter: userPrincipalName={0} This means that users can login as user@domain.tld
Group Search Filter: member={0} This ensures that AD groups may be used rather than specifying individual users
Configure created clusters to use UAA as the OIDC provider: Enabled This pre-configures the kubernetes API to use OpenID Connect with UAA. If not using PKS Enterprise, you’ll need to provide another OpenID Connect-Compliant endpoint (like Dex), link it to Active Directory and update the kubernetes cluster api manually to use the OpenID Authentication.

Operator: Create Role and RoleBinding:

While authentication is handled by OIDC/UAA/LDAP, Authorization must be configured on the cluster to provide access to resources via RBAC. This is done by defining a Role (or clusterRole) that indicates what actions may be taken on what resources and a RoleBinding which links the Role to one or more “subjects”.

Authenticate to kubernetes cluster with an administrative account (for example, using PKS cli to connect)

Create yaml file for our Role and RoleBinding:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: developers
rules:
- apiGroups: ["", "extensions", "apps"]
  resources: ["deployments", "replicasets", "pods"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] 
  # You can also use ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: developer-dev-binding
subjects:
- kind: Group
  name: k8s-devs
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: developers
  apiGroup: rbac.authorization.k8s.io

In the example above, we’re creating a Role named “developers”, granting access to the core, extensions and apps API groups and several actions against deployments, replicaSets and pods. Notice that developers in this role would have have access to secrets (for example)
The example RoleBinding binds a group named “k8s-devs” to the developers role. Notice that we have not created the k8s-devs group in Kubernetes or UAA; it exists in Active Directory

Use Kubectl to apply the yaml, creating the Role and Rolebinding in the targeted namespace

Creating the kubeconfig – the hard way

To get our developer connected with kubectl, they’ll need a kubeconfig with the authentication and connection details. The Hard way steps are:

Operator obtains the cluster’s certificate authority data. This can be done via curl or by copying the value from the existing kubeconfig.

Operator creates a template kubeconfig, replacing the value specified, then sends it to the developer user

apiVersion: v1
clusters:
- cluster:
  certificate-authority-data: <OBTAINED IN STEP 1 >
  server: < FQDN to Master Node. >
  name: PROVIDED-BY-ADMIN
contexts:
- context:
    cluster: PROVIDED-BY-ADMIN
    user: PROVIDED-BY-USER
  name:  PROVIDED-BY-ADMIN
current-context: PROVIDED-BY-ADMIN
kind: Config
preferences: {}
users:
- name: PROVIDED-BY-USER
  user:
    auth-provider:
      config:
        client-id: pks_cluster_client
        cluster_client_secret: ""
        id-token: PROVIDED-BY-USER
        idp-issuer-url: https://PROVIDED-BY-ADMIN:8443/oauth/token
        refresh-token:  PROVIDED-BY-USER
      name: oidc

The developer user obtains the id_token and refresh_token from UAA, via a curl command
curl 'https://PKS-API:8443/oauth/token' -k -XPOST -H 'Accept: application/json' -d "client_id=pks_cluster_client&client_secret=""&grant_type=password&username=UAA-USERNAME&response_type=id_token" --data-urlencode password=UAA-PASSWORD
The developer user updates the kubeconfig with the id_token and refresh token in the kubeconfig

Creating the kubeconfig – the easy way

Assuming the developer is using Mac or Linux…

Install jq on developer workstation
Download the get-pks-k8s-config.sh script, make it executable (chmod +x get-pks.k8s.config.sh)
Execute the script (replace the params with your own)
```
./get-pks-k8s-config.sh --API=api.pks.mydomain.com \
--CLUSTER=cl1.pks.mydomain.com \
--USER=dev1@mydomain.com \
--NS=scratch
```
- API – FQDN to PKS Controller, for UAA
- CLUSTER – FQDN to master node of k8s cluster
- USER – userPrincipalName for the user
- NS – Namespace to target; optional
After entering the user’s password, the script will set the params in the kubeconfig and switch context automatically

Try it out
Our developer user should able to “see” pods but not namespaces for example:

Creating the kubeconfig – the easiest way

Provide the developer with the PKS CLI tool, remember we have not added them to any group or role with PKS admin permissions.
Provide the developer with the PKS API endpoint FQDN and the cluster name
The developer may run this command to generate the updated kubeconfig and set the current context
pks get-kubeconfig CLUSTERNAME -a API -u USER -k
- CLUSTERNAME is the name of the cluster
- API – FQDN to PKS Controller
- USER – userPrincipalName for the user
You’ll be prompted for the account password. Once entered, the tool will fetch the user-specific kubeconfig.

Tag: kubeconfig

Logging into a Kubernetes cluster with an OIDC LDAP account