Archive

Posts Tagged ‘secret’

Adding a private Docker registry to a PKS 1.5 Windows Kubernetes cluster

Pivotal Container Service (PKS) 1.5 and Kubernetes 1.14 bring *beta* support for Workers running Windows.  This means that we can provide the advantages of Kubernetes to a huge array of applications running on Windows.  I see this especially useful for Windows applications that you don’t have the source code for and/or do not want to invest in reworking it for .NET core or languages that run on Linux.

In nearly all cases, you’ll need an image with your applications’ dependencies or configuration and in the real world, we don’t want those in the public space like dockerhub.  Enter Private Docker Repositories.

PKS Enterprise includes VMware Harbor as a private registry, it’s very easy to deploy alongside PKS and provides a lot of important functionality.  The Harbor interface uses TLS/SSL; you may use a self-signed, enterprise PKI-signed or public CA-signed certificate.  If you chose to not use a public CA-signed certificate ($!), the self-signed or PKI-signed certificate must be trusted by the docker engine on each Kubernetes worker node.

Clusters based on Ubuntu Xenial Stemcells:

  • The operator/administrator simply puts the CA certificate into the “Trusted Certificates” box of the Security section in Ops Manager.
  • When BOSH creates the VMs for kubernetes clusters, the trusted certificates are added to the certificate store automatically.
  • If using an enterprise PKI where all of the internal certificates are signed by the Enterprise CA, this method makes it very easy to trust and “un-trust” CAs.

Clusters based on Windows 2019 Stemcells:

This is one of those tasks that is easier to perform on Linux that it is on Windows.  Unfortunately, Windows does not automatically add the Trusted Certificates from Ops Manager to the certificate store, so extra steps are required.

    1. Obtain the Registry CA Certificate.  In Harbor, you may click the “REGISTRY CERTIFICATE” link while in a Project.  Save the certificate to where the BOSH cli is installed (Ops Manager typically).
    2.  Connect BOSH cli to director.  This may be done on the Ops Manager.
    3. List BOSH-managed vms to identify the service_instance deployment corresponding to the targeted K8s cluster by matching the VM IP address to the IP address of the master node as reported by PKS cluster.
    4. Run this command to copy the certificate to the Windows worker
      bosh -e ENV -d DEPLOYMENT scp root.cer WINDOWS-WORKER:/
      • ENV – your environment alias in the BOSH cli
      • DEPLOYMENT – the BOSH deployment that corresponds to the k8s cluster; ex: service-instance_921bd35d-c46d-4e7a-a289-b577ff743e15
      • WINDOWS-WORKER – the instance name of the specific Windows worker VM; ex: windows-worker/277536dd-a7e6-446b-acf7-97770be18144

      This command copies the local file named root.cer to the root folder on the Windows VM

    5. Use BOSH to SSH into the Windows Worker.
      bosh -e ENV -d DEPLOYMENT ssh WINDOWS-WORKER
      • ENV – your environment alias in the BOSH cli
      • DEPLOYMENT – the BOSH deployment that corresponds to the k8s cluster; ex: service-instance_921bd35d-c46d-4e7a-a289-b577ff743e15
      • WINDOWS-WORKER – the instance name of the specific Windows worker VM; ex: windows-worker/277536dd-a7e6-446b-acf7-97770be18144

      SSH into Windows node, notice root.cer on the filesystem

    6. In the Windows SSH session run “powershell.exe” to enter powershell
    7. At the PS prompt, enter
      Import-certificate -filepath .\root.cer -CertStoreLocation Cert:\LocalMachine\Root

      The example above imports the local file “root.cer” into the Trusted Root Certificate Store

    8. Type “exit” twice to exit PS and SSH
    9. Repeat steps 5-8 for each worker node.

Add docker-registry secret to k8s cluster

Whether the k8s cluster is running Windows workers or not, you’ll want to add credentials for authenticating to harbor.  These credentials are stored in a secret. To add the secret, use this command:

kubectl create secret docker-registry harbor \
--docker-server=HARBOR_FQDN \
--docker-username=HARBOR_USER \
--docker-password=USER_PASS \
--docker-email=USER_EMAIL
  • HARBOR_FQDN – FQDN for local/private Harbor registry
  • HARBOR_USER – name of user in Harbor with access to project and repos containing the desired images
  • USER_PASS – username for the above account
  • USER_EMAIL – email adddress for the above account

Note that this secret is namespaced; it needs to be added to the namespace of the deployments that will reference it

More info

Here’s an example deployment yaml for a Windows K8s cluster that uses a local private docker registry.  Note that Windows clusters cannot leverage NSX-T yet, so this example uses a NodePort to expose the service.

Advertisements