Configuring Backup in Tanzu SQL with MySQL for Kubernetes

Backup & Restore

Prerequisite: A reachable S3 endpoint. Can be local or remote, but the pods must be able to resolve its name or IP. Create or select and existing bucket for your database backups. In this case, I have a minio instance running on-prem with a bucket named backup-mysql.

Create a secret for the S3 endpoint credentials. This account will need to be able to write to the database backup bucket. Here’s an example:

apiVersion: v1
kind: Secret
  name: minio-creds
  # S3 Credentials
  accessKeyId: "MYACCESSKEY"
  secretAccessKey: "MYSECRETKEY"

Create a TanzuMySQLBackupLocation. In the example below, we’re not using SSL with the minio endpoint, so I’m explicitly using port 80. More examples and details are found here. I like to keep the backups organized, so I’ll create a backup location for each instance and specify an bucketPath for each.

kind: MySQLBackupLocation
  name: backuplocation-mysql-ha
    # For S3 or Minio:
      bucket: "backup-mysql-ha"
      bucketPath: "/mysql-ha/"
      # region: "us-east-1"
      endpoint:  "" # optional, default to AWS
      forcePathStyle: true
        name: minio-creds

Test with a one-off backup. Create and apply a yaml like the following to request a backup without a schedule. Here’s an example yaml for a one-off backup for the mysql-ha instance to its corresponding backup location:

kind: MySQLBackup
  name: backup-mysql-ha-1off
    name: backuplocation-mysql-ha
    name: mysql-ha

We can get the MySQLBackups to see that it has completed successfully:

Create a backup Schedule

Now that we’ve confirmed that the backup location and credentials work as expected, we should add a backup schedule. Here’s an example:

kind: MySQLBackupSchedule
  name: mysql-ha-daily
        name: backuplocation-mysql-ha
        name:  mysql-ha
  schedule: "@daily"

Apply this kubectl apply -n mysql-instances -f backupschedule-mysql-ha-daily.yaml

I found that (unlike Velero), when applying the MySQLBackupSchedule, a backup does not immediately begin. At the scheduled time however, a pod for the backup schedule will be created to run the backup job. This pod will remain intact to run subsequent backup jobs.

Backup Pods and created Backup objects

Lastly, regarding backups, keep in mind that the backup data on the S3 endpoint never expires, the backups will remain there until removed manually. This may be important if you have limited capacity.


From the docs:

MySQLRestores always restores to a new MySQL instance to avoid overwriting any data on an existing MySQL instance. The MySQL instance is created automatically when the restore is triggered. Tanzu MySQL for Kubernetes does not allow you to restore a backup to an existing MySQL instance. Although you can perform this manually by copying the MySQL data from the backup artifact onto an existing MySQL instance, VMware strongly discourages you from doing this because you might overwrite existing data on the MySQL instance.

So, we should not expect to restore directly to a running database instance. If we need to recover, we’ll create a new instance and restore the backup to it.

To create a restore, we’ll need the name of the MySQLBackup object to restore from and a name of a database to create from that backup as part of the restore. We’ll put that into a yaml like the one below. Notice that we provide a spec for a new database, I wanted a loadbalancer for it although we are able to repoint the existing loadbalancer to the new proxy nodes (for ha) or the new database node (for standalone)

kind: MySQLRestore
  name: restore-ha
    name: mysql-ha-daily-20210708-000005
      name: restored-mysql-database
      storageSize: 2Gi
      imagePullSecret: harbor
      serviceType: LoadBalancer
        enabled: true

Apply the yaml to create the restore kubectl apply -n mysql-instances -f ./restore-ha.yamlYou should see a new database pending and a MySQLRestore object running:

Job is running and instance is pending
Restore job succeeded and there is a new mysql instance

Now, the choice if yours to copy data from the restored database back to the original or to point the applications to the new database or to point the loadbalancer at the new database.

If you choose to repoint the existing load-balancer to the new database, here’s an example how to do that:

kubectl patch service -n mysql-instances mysql-ha -p '{"spec":{"selector":{"": "restored-mysql-database"}}}'

Resolutions for 2018

If I put it here, I’m much more likely to follow-through.  Like many, I work best under some pressure.  Here is a list of what I want to do differently (with regard to technology) next year.

  1. Do more blogging.  I can make a ton of excuses for not blogging as much this year.  I love sharing what I’ve learned; the more new stuff I learn, the more I share.  So….
  2. Do more for NSX for vSphere and NSX-T.  I feel strongly that SDN is critical to the future of how datacenters operate.  NSX is the logical leader in this space and will only grow in interest.  There is still a tendency to replicate what was done with pre-SDN technology and I’d like to see modern ways to solve problems while finding and pushing the limits of what can be done in SDN.
  3. PKS
    Do more with containers and PKS.  The technologies that Pivotal provides are cutting edge.  Already and continuing, containers and applications-as-code methods are growing and will define the datacenter of the future.  Just as a few years ago, we stopped thinking of hardware servers as single-purpose, we’ll embrace multiple workloads within a VM.
  4. Do more coding.  I love concourse and pipelines, but have a lot to learn.  Let’s find the limits of BOSH and pipelines.  Can we not only deploy, but automate the operation and maintenance of a PaaS solution?
  5. Do more coding.  I feel that as we move to “applications-as-code”, it’s important to understand what that means to developers and operators.  What sort of problems become irrelevant in this approach?  What molehills become mountains?

Hope to see you next year!


Use Cisco Nexus 1000V for virtual hosts in nested ESXi

The native VMware vSwitch and Distributed vSwitch do not use MAC-learning. This was removed because the vSwitches would be aware of the VMs attached to them and the MAC addresses in use. As a result, if you nest ESXi under a standard vSwitch and power-on VMs under the nested instance, those VMs will be unable to communicate because their MACs are masked by the virtual host and the vSwitch is not aware of them.

Workaround options:

  1. Enable Promiscuous mode on the vSwitch.
  2. This works but should never be used in production.  It adds a lot of unnecessary traffic and work to the physical NICs.  It makes troubleshooting difficult and is a security risk
  3. Attach your virtual hosts to a Cisco Nexus 1000V.
  4. The 1000V retains MAC-learning, so VMs on nested virtual ESXi hosts can successfully communicate because the switch learns the nested MAC addresses.
  5. If your physical servers support virtual interfaces, you can create additional “physical” interfaces and pass them through to the virtual instances.  This allows you to place the virtual hosts on the same switch as the physical hosts if you choose.  There is obviously a finite amount of virtual interfaces you can create in the service profile, but I think this is a clean, low-overhead solution for environments using Cisco UCS or HP C7000 or similar.


The Nexus 1000V brings back important functionality for nested ESXi environments, especially those environments that do not have access to features like virtual interfaces and service profiles.

Helpful links:

Standing Up The Cisco Nexus 1000v In Less Than 10 Minutes by Kendrick Coleman

Resolving vCAC Design Center Crash

After having installed the vCAC Design Center, every attempt to run it failed. The GUI would not load. In my case, the Windows Application log recorded the error:

.NET Runtime Event ID 1026 in Application CloudUtil.
Framework Version v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.Reflection.TargetInvocationException.

Only when I tried calling CloudUtil.exe from the command line did I get an error message indicating that the application could not establish a trust relationship with the vCAC MOdel Manager. This meant that the self-signed SSL cert was not trusted.

Logon to vCAC administration portal and import the SSL certificate into the “Trusted Root Certification Authorities\Local Computer” physical Certificate Store.

After this, I was able to successfully launch the vCAC Design Center.

WTF is wrong with vCenter Chargeback?!

Every time I try to install vCenter ChargeBack, it fails. Every. %^#$#*. Time.

The installation log – which isn’t visible until the installation ends – indicates that a batchfile is created and then cannot be run because it is in use by something else. As a result, the services don’t get created and because the installer cleans up after itself, I cannot go back to run the batchfiles.

Preinstallation Tasks

  • Create a vCenter Chargeback Manager database and database user.
    Done, created SQL database and user. Ran script to add the CB_ADMIN_ROLE, added user to dbo_admins for database. Database uses SQL_Latin1_General_CP1_CI_AS, so its not case-sensitive.
  • Set a static IP address on the machine
  • Ensure that MSI installation is enabled
    Installed the Microsoft Visual C++ 2005 Redistributable Package (x86) anyway.
  • Check whether the required ports are free
  • Ensure that port exceptions are configured on Windows FirewallWindows Firewall turned off.
  • Ensure that system time of all the servers and databases are in sync Done. All members of same AD domain.
  • Ensure that the Windows user has the Log on as a service permission Done. Added admins, domain admins, administrator.
  • Ensure that the SQL Server Browser service is running It is.
  • Ensure that the vCenter Chargeback Manager database is case-insensitive.See first bullet.

I’m attempting the installation under the domain administrator account. Windows 2008 R2, SQL 2008 R2. I’ve exhausted everything I know to try and am open to suggestions…

Ok, I started from scratch with a new Windows VM, built from scratch. Not yet joined to the domain. So far, so good. Switching to Decaf….


Windows 8 – How to install the vSphere Client 5.0 Update1a

If you’re considering a machine with Windows 8, you may notice that the installer for VMware vSphere client 5.01a (latest release) complains that it can only be installed on “Windows XP SP2 or higher” and quits. For now, here’s the work-around I used:

  1. Windows8 vSphere Client Compatibility Settings
    Windows8 vSphere Client 5.01a Compatibility Settings

    Use 7-zip to open the ~350MB executable and extract the contents

  2. In Windows 8, run dotnetfx35.exe – it will prompt you to download and install .NET 2.0. I let it download and install the files it needed.
  3. Next, run vjredist64.exe (Assuming Windows 8 x64)
  4. Right-Click VMware-viclient.exe, choose “Properties”
  5. On the “Compatibility” tab, check the “Run this program in compatibility mode for:” and choose “Windows 7” from the list. Click “Ok”
  6. Run the VMware-viclient installer as you normally would.

Something New!

After 15 great years at Venture Technologies, I’m moving on to boost my career. I’ve been given an opportunity to join EMC Consulting as an Advisory Solutions Architect. Everyone I’ve met there has been very friendly and knowledgable. I’m excited to join a great team and make the most of this opportunity. I’ll continue to tweet and blog about the things that interest me and may be a help to others.

EMC World Coming Up!

I’m excited to be returning to EMC World! If you’re going to be there, let me know via twitter @brianragazzi so we can meet. I try to schedule as many sessions as I can for the conference, but there’s a couple of openings. I’m scheduled to give a small, 10-minute presentation at VCE’s booth #410 on Tuesday at 6:20 PM and again on Wednesday at 2:10 PM. If you happen to see me great, just don’t tell you you’re going to come by. I’ll get all nervous and stuttery, it won’t be pretty.

Completed EMC Technology Architect Exams!

I’m excited to have passed the last exam for the EMC Technology Architect certification.  There is a tremendous amount of material to study (~1600 pages).  EMC has so many product lines, that even with that much material, it still feels like you’re only scratching the surface.  There’s many, many different replication technologies to accommodate any scenario, so the challenge is know which is appropriate for which situation.

This certification is a necessary requirement for VCE partners and is only available to EMC partners.