Extending vCAC IaaS to fix an annoyance

Background: When provisioning a Windows VM using the Clone Workflow and a vSphere customization specification that joins the computer to an active directory domain, the computer object is placed in the “Computers” container. I want to change that. ūüôā

Solution Overview:
Modify the built-in Stub workflow to execute a Powershell script that moves the computer object based on the Business Group.

Preparation:

  1. Created a new Build Profile with the ActiveDirectoryCleanupPlugin, MiscellaneousVrmProperties, RemoteDesktopProtocolProperties and VMwareWindows2008R2_64Properties Property Sets.

    vCAC Build Profile Properties
    vCAC Build Profile Properties
  2. Created a new Windows 2008 R2 VM from a vSphere template, did not power-on. Took a snapshot
  3. Created a new shared vSphere Linked Clone Blueprint, included a customization specification that joins the machine to the domain
    vCAC Windows Blueprint Information
    vCAC Windows Blueprint Information

    vCAC Windows Blueprint Build information
    vCAC Windows Blueprint Build information
  4. Created a Business Group, Created a reservation for them, entitled the Business Group to the service and catalog item for the Windows Server
  5. Tested requesting a new machine, it was provisioned, sysprepped and joined the domain correctly. I was annoyed that the computer object was in the “Computers” container.
  6. Installed the VMware vCloud Automation Center Designer (found at https://your-vcac-server:5480/i) on the IaaS Server.
  7. Installed Active Directory module for Windows PowerShell part of RSAT on the IaaS Server

Steps

  1. We’ll need to indicate where we want the Computer Object moved to, so we’ll add that property. Since I wanted all of my Business Group’s computer objects in the same place, I added a property named targetOU to the Business Group and assigned the distinguishedName of the OU.

    targetOU property added to Business Group
    targetOU property added to Business Group
  2. Save the PS script to C:\scripts\movecomputer.ps1

    Import-Module ActiveDirectory
    write "VM Name - $vmName" | out-file c:\scripts\invoketest.txt
    write "Target OU - $targetOU" | out-file c:\scripts\invoketest.txt -Append
    Get-ADComputer $vmName | Move-ADObject -TargetPath $targetOU

    This script will write out our variables to a text file, so we can verify that they’re getting passed correctly. Then it performs the move. Please note that this will be executed by the DEM, so make sure the execution account has permissions to perform this action in AD.

  3. Launch the vCAC Designer, Load the WFStubMachineProvisioned workflow from the list
    vCAC Designer Workflows
    vCAC Designer Workflows
  4. In the “Machine Provisioned” try loop, locate and double-click on the “Custom-Code” item.

    Custom Code section in workflow
    Custom Code section in workflow
  5. From the toolbox, under DynamicOps.Cdk.Activities, drag the GetMachineName element into the Custom Code box
  6. From the toolbox, under DynamicOps.Cdk.Activities, also drag the GetMachineProperty and InvokePowerShell elements into the Custom Code box, near GetMachineName
  7. Drag a connection from one of the “tabs” on the Start element to the GetMachineName element, from GetMachineName to GetMachineProperty and from GetMachineProperty to InvokePowerShell

    vCAC Designer - Workflow Custom Code Wiring
    vCAC Designer – Workflow Custom Code Wiring
  8. While still in the Custom Code element, click “Variables” (near the bottom), click Create Variable and enter vmName for the name, leave the variable type as String. Repeat with a variable named targetOU. These are going to hold the values we want to work with through the workflow.

    Custom Code Variables
    Custom Code Variables
  9. Select the GetMachineName element. On the Properties pane to the right, enter VirtualMachineId in the MachineId field. In the MachineName field, enter vmName. Ok, so where do these come from?!
    If you click on “Arguments” while in the GetMachineName element, you’ll see two, VirtualMachineId and ExternalWorkflowId. These are standard internal values that are used in these external workflows. So, we’re providing the VirtualMachine Id GUID to the system to look up the Virtual Machine Name. The “vmName” value is the name of the variable we assigned a moment ago and the GetMachineName element enters the retrieved Name into the vmName variable.

    GetMachineName Properties
    GetMachineName Properties
  10. Now select the GetMachineProperty element and work with its properties. Just like before, set the MachineId to VirtualMachineId. Here, we want to retrieve the value in the “targetOU” property and set it in the targetOU variable. So set the PropertyValue to targetOU without quotes and the PropertyName to "targetOU" WITH QUOTES.

    GetMachineProperty Properties
    GetMachineProperty Properties
  11. Select the InvokePowerShell element. Notice there are several more properties in with this one – don’t worry, we’re only going to use a few. In my case, I chose to use a PS script instead of a one-liner. This way, I could modify the script without modifying the workflow. So, check the box labelled “IsScript” and set the CommandText to the full path of the PS script in quotes. In this case, use "C:\scripts\movecomputer.ps1".

    InvokePowerShell Properties
    InvokePowerShell Properties
  12. Our script expects two variables to be provided; $vmName and $targetOU, so click the ellipsis beside PowerShellVariables. Click Create Argument to add a new variable. Set the name to vmName, leave the direction as In and the type as String, set the value also to vmName” no quotes. Repeat for targetOU. Here, we’re telling it to create PowerShell Variables and set their values to the values of the workflow. Click Ok

    Powershell Variables
    PowerShell Variables
  13. Click “Send” to upload the modified workflow to the Model Manager. Now that we’ve created the workflow, we need to make sure it fires when we want it to.
  14. Back in vCAC Infrastructure, modify the Windows blueprint by adding a property named ExternalWFStubs.MachineProvisioned. No value needed. This way, when this shared blueprint is used by any Business Group, the computer object will be moved to
    the OU given in the Business Group’s targetOU property.

    Property Added to blueprint to call customized workflow
    Property Added to blueprint to call customized workflow

Results
When an entitled member of Business Group 1 requests a VM from the Windows 2008 R2 catalog item, the VM is correctly created as a linked clone, assigned an IP address from the network profile and its Computer Object moved as expected.

I probably should have broken this into multiple parts…

References:
I would still be twiddling my thumbs if it weren’t for the following enormously helpful bloggers:

Use Cisco Nexus 1000V for virtual hosts in nested ESXi

The native VMware vSwitch and Distributed vSwitch do not use MAC-learning. This was removed because the vSwitches would be aware of the VMs attached to them and the MAC addresses in use. As a result, if you nest ESXi under a standard vSwitch and power-on VMs under the nested instance, those VMs will be unable to communicate because their MACs are masked by the virtual host and the vSwitch is not aware of them.

Workaround options:

  1. Enable Promiscuous mode on the vSwitch.
  2. This works but should never be used in production.  It adds a lot of unnecessary traffic and work to the physical NICs.  It makes troubleshooting difficult and is a security risk
  3. Attach your virtual hosts to a Cisco Nexus 1000V.
  4. The 1000V retains MAC-learning, so VMs on nested virtual ESXi hosts can successfully communicate because the switch learns the nested MAC addresses.
  5. If your physical servers support virtual interfaces, you can create additional “physical” interfaces and pass them through to the virtual instances.  This allows you to place the virtual hosts on the same switch as the physical hosts if you choose.  There is obviously a finite amount of virtual interfaces you can create in the service profile, but I think this is a clean, low-overhead solution for environments using Cisco UCS or HP C7000 or similar.

Conclusion

The Nexus 1000V brings back important functionality for nested ESXi environments, especially those environments that do not have access to features like virtual interfaces and service profiles.

Helpful links:

Standing Up The Cisco Nexus 1000v In Less Than 10 Minutes by Kendrick Coleman

VMware Horizon View Network Ports Illustrated pt. 2

As Carl pointed out, I left HTML Access (aka Blast) off of the first round of View network port diagrams.  So after going through the documents and making various connections while running TCPView, here’s the updated diagrams including the networks ports used by HTML Access and the Blast Secure Gateway.

Here’s the PDF

HTML Access & Blast Secure Gateway without Security Server
HTML Access Direct Connect without Security Server
HTML Access & Blast Secure Gateway with Security Server
HTML Access & Blast Secure Gateway with Security Server

Eww, curved, intersecting lines.

Once again, here’s the PDF

VMware Horizon View Network Ports Illustrated

I’ve recently had to attempt to describe the ports used in the various connection scenarios used by VMware View and found that a diagram really helped clear things up and have aided in producing accurate firewall rules.

A couple notes about the connections depicted though;

  • I only included v5.x. Previous versions behave differently, so use caution if you reference these in an environment where v4.x components are reused.
  • I did not depict the connection from the vCenter Server to hosts, LDAP etc. These diagrams are View-centric
  • Although the View client may connect to the View Connection Server over HTTP/TCP80, I did not depict it because we strongly prefer the HTTPS, encrypted connection.
  • Wyse MMR is not included since it is not supported on Windows 7 Virtual Desktops
  • Black arrows indicate TCP connection request and direction
  • Green arrows represent UDP traffic flow

Here’s the PDF

PCoIP with Secure Gateway on Security Server
PCoIP with Secure Gateway on Security Server

PCoIP with Secure Gateway on Connection Server
PCoIP with Secure Gateway on Connection Server

RDP with Secure Tunnel on Connection Server
RDP with Secure Tunnel on Connection Server

RDP with Secure Tunnel on Security Server
RDP with Secure Tunnel on Security Server

RDP Direct Connection without Security Server
RDP Direct Connection without Security Server

PCoIP Direct Connection without Security Server
PCoIP Direct Connection without Security Server

Once again, here’s the PDF

Expanding a VMDK for OpenFiler

In my lab, I have an OpenFiler 2.99.1 VM running on the physical host providing storage via iSCSI to my virtual hosts.

Increasing the size of the VMDK used by the OpenFiler VM does not equate to more storage shared by the OpenFiler. I banged my head against the wall for a few hours figuring it out; here’s how I did it.

  1. Expand VMDK
  2. Download GParted Live CD
  3. Stop anything consuming storage provided by OpenFiler
  4. Shut Down OpenFiler VM
  5. Boot OpenFiler from GParted Live CD
  6. Create additional LVM2 PV in the unused storage
  7. Apply changes
  8. Unmount Gparted ISO, reboot OpenFiler
  9. In the OpenFiler Web Interface, navigate to Volume Groups
  10. Add new PV to the Volume Group
  11. Navigate to Manage Volumes
  12. Select the VG, Edit the Volume, enter the new size (same as the volume group’s total space) in my case
  13. Restart iSCSI service
  14. In vSphere, view the properties of the iSCSI datastore to increase its size

What a pain, why is this necessary?
There is apparently an uncorrected bug in OpenFiler in that it will not create additional partitions on a block device. Attempting to create the PV/Partition from the CLI using parted will not accept the cylinders I provide, instead attempting to make the volume half as big as asked. – If someone knows why this is and how to correct, please comment.
In the future, if my OpenFiler needs more storage to share, I’ll just add a new VMDK, create the PV on it, add it to the Volume Group and increase the volume that way.

Order of Operations when Upgrading vCloud Director to 5.1

This should be a short article. I’m assuming you have vCD 1.5 running on vSphere 5.0 with vShield Manager 5.0.

Starting with:

  • vCenter Orchestrator 5.0
  • vCenter Server 5.0
  • vShield Manager 5.0.x
  • ESXi 5.0
  • vCloud Director 1.5.x

We want to end up with everything on v5.1.x

  1. Update Orchestrator to 5.1 – it’s compatible with vCenter 5.0 and 5.1
  2. Update vShield Manager to 5.0.2.  This is important, this version works with both vCenter 5.0 & 5.1 as well as vCD 1.5 and 5.1. Then redeploy your vShield Edge appliances
  3. Update vCloud Director to 5.1.  Be sure to back up your database before upgrading in case you have to roll-back
  4. Update vCenter Server to 5.1.  Install your Single-Sign On Service at this step too.
  5. Update vShield Manager to vCenter Security & Networking 5.1.  Then redeploy your vShield Edge appliances.
  6. Update VMware Update Manager to 5.1
  7. Use VUM to upgrade ESXi to 5.1

Note: If you’ve deployed VMware View, you’ll want to upgrade it to v5.1.2 before upgrading vCenter.

Windows 8 – How to install the vSphere Client 5.0 Update1a

If you’re considering a machine with Windows 8, you may notice that the installer for VMware vSphere client 5.01a (latest release) complains that it can only be installed on “Windows XP SP2 or higher” and quits. For now, here’s the work-around I used:

  1. Windows8 vSphere Client Compatibility Settings
    Windows8 vSphere Client 5.01a Compatibility Settings

    Use 7-zip to open the ~350MB executable and extract the contents

  2. In Windows 8, run dotnetfx35.exe – it will prompt you to download and install .NET 2.0. I let it download and install the files it needed.
  3. Next, run vjredist64.exe (Assuming Windows 8 x64)
  4. Right-Click VMware-viclient.exe, choose “Properties”
  5. On the “Compatibility” tab, check the “Run this program in compatibility mode for:” and choose “Windows 7” from the list. Click “Ok”
  6. Run the VMware-viclient installer as you normally would.

Use Alarms to monitor your vSphere environment

There have been a number of KB articles, blog posts, best practices documentation and more suggesting vSphere admins use the Alarms feature in vSphere to proactively monitor their environment.  This is another one.

I was recently asked about a tool that could let someone know when memory or CPU utilization gets too high on a VM. ¬†I suggested vCOps, but after a moment, I realized that vCOps is a fantastic tool for analysis and trending, but it’s not really meant to let someone know that a threshold has been reached and some action may or may not be needed.

For that, I suggest using the built-in and easy-to-use Alarms. ¬† There are a number (54 in my case) of built-in Alarms that come pre-configured, but none of them – by default – will notify you via email when they’ve fired. ¬†We’ll modify a built-in Alarm to send an email when its status changes.

Modify default Alarm

On event that deserves notification is a vSphere HA error. Namely, when vSphere HA on a host is in an error/alert state. ¬†The Alarm to modify is named “vSphere HA host status”. ¬†Find it on the Alarms tab while your vCenter server is selected in the treeview of the vSphere Client.

This One
This One

Right-click the Alarm and choose “Edit Settings”. ¬†The “Alarm Settings” window has four tabs; General, Triggers, Reporting and Actions. ¬†For this section, ¬†we are concerned about the Actions tab, but you can check out the other tabs too. ¬†Go ahead, I’ll be right here. ¬†You’ll notice on the Actions tab, there are no actions listed, so when the status changes, the icon on the host will change.

No actions defined
No actions defined

Unless you are watching the vSphere client, you may not notice that anything has changed. ¬†Let’s fix that. ¬†Click the “Add” button. ¬†It adds a new action item which, conveniently enough is “Send a notification email”, once, ¬†when the status is “Alert”. ¬†All we have to do is click in the “Configuration” field and type in the comma-separated email addresses for notifications to be sent to. ¬†You have options to send the message repeatedly and when the status changes back to warning or normal – there depend on your environment and just how much email you want to send. ¬†Please note that there’s no way – that I’m aware of – to modify the content of the notification email messages. ¬†They just say “Alarm status on XX host changed from warning to alert” or similar. ¬†It’s just enough information to know THAT something happened, but not WHY it happened.

There are a lot of valuable Alarms already configured that just need actions assigned. For example, there’s an alarm for Virtual Machine Memory Usage that will let you know when a VM is running out of vRAM – this is handy because you may not know about the high memory utilization without a notification.

Prerequisites:

The Alarm notifications are sent by the vCenter Server, so it has to be configured with valid SMTP settings:

vCenter Server SMTP Configuration
Configure SMTP settings for vCenter

Troubleshooting vMotion error

After recovering from a power failure, I noticed VMs were not migrating to one of the vSphere 5 hosts in a HA/DRS cluster. ¬†That host had a status message that read “Unable to apply DRS resource settings on host”. ¬†Oddly enough, that status message would go away and reappear periodically. ¬†Trying to manually vMotion a VM to the host would fail with the helpful message “A specified parameter was not correct. ¬†” Oh! ¬†Why didn’t you say so?! Ergh.

I tried putting the host in maintenance mode, removing it from the cluster and even removing it from vCenter Server.  None of these steps helped.  Time to get in the weeds!

I collected the logs from the host (“Administration|Export System Logs”) and started perusing. ¬†In the hostd.log file, this is what caught my eye:

2012-06-05T19:30:15.309Z [FFBB0B90 info ‘TaskManager’ opID=f2ea5ac7] Task Created : haTask-ha-root-pool-vim.ResourcePool.createResourcePool-125613723
2012-06-05T19:30:15.310Z [35AB7B90 error ‘ResourcePool ha-root-pool’ opID=f2ea5ac7] Duplicate name ‘Server Virtualization’
2012-06-05T19:30:15.310Z [35AB7B90 info ‘Default’ opID=f2ea5ac7] AdapterServer caught exception: vim.fault.DuplicateName
2012-06-05T19:30:15.310Z [35AB7B90 info ‘TaskManager’ opID=f2ea5ac7] Task Completed : haTask-ha-root-pool-vim.ResourcePool.createResourcePool-125613723 Status error

Okay, so now I have something to go on, the resource pools didn’t get removed properly and couldn’t be recreated because of a duplicate name. ¬†This is easy enough to fix.

  1. Enable and start the DCUI, SSH  and ESXi Shell
  2. Either get on the console or connect via SSH to the shell as root (Alt+F1 to get to tech support mode)
  3. Run this command

    # mv /etc/vmware/hostd/pools.xml /tmp

  4. Return to the DCUI (Alt+F2 from console, use “dcui” from SSH), logon as root
  5. Restart management Agents (under “Troubleshooting Options”)
  6. Logout
  7. Stop SSH & ESXi Shell

References:

Applying DRS Settings to an ESX Host Generates an Error

Using ESXi Shell in ESXi 5.0

One way to update a vSphere 5 host from CLI

Every time a new version comes out, I have to dig up this procedure, so I figured I’d post it for quicker reference next time.

If you have a vSphere Update Manager in your environment, use it, if not, here is one method to update a host: (using ESXi-5.0.0-20120302001-standard.zip in this case)

  1. Locate the Offline Bundle with the desired version or create an image profile from it that includes additional vibs you may need.
  2. Copy that zip file to a datastore that the host to be upgraded has access to.
  3. Use the vSphere CLI on a local machine and vMA to run this command to determine the profiles contained in the bundle

    esxcli --server=<host IP> --username=root --password=<password> software sources profile list --depot="[DATASTORE]ESXi-5.0.0-20120302001-standard.zip"

    Returns ‚ÄúESXi-5.0.0-20120302001-standard‚ÄĚ, ‚ÄúESXi-5.0.0-20120302001-no-tools‚ÄĚ; we‚Äôre going to use ‚Äústandard‚ÄĚ in this case.

  4. Put the host in maintenance mode

    vicfg-hostops.pl --server=<host IP> --username=root --password=<password> --operation enter

  5. Apply update to host using profile

    esxcli --server=<host IP> --username=root --password=<password>software profile update --depot="[DATASTORE] update-from-esxi5.0-5.0_update01.zip" --profile="ESXi-5.0.0-20120302001-standard"

  6. Reboot the host

    vicfg-hostops.pl --server=<host IP> --username=root --password=<password> --operation reboot