Hands-on with VxRail v4.7.100

Recently (yesterday!) upgraded the VxRail clusters in a lab my team uses for Pivotal Ready Architecture development and testing and immediately noticed many differences.

When trying to go to the VxRail Manager, I am redirected to the vSphere Web Client This was concerning at first, but quickly realized that the VxRail Manager interface is now embedded in the vSphere Web Client (hence the redirection!)

In this environment, we have a “management” cluster using the VxRail-managed vCenter Server. On this cluster is also another vCenter Server that is used by three additional VxRail clusters – when the “Availability Zone” clusters are deployed, they use this shared “external” vCenter Server.

After upgrading the management cluster, the VxRail extension was registered in the HTML5 vSphere Web Client.

From here, when selecting the “VxRail” choice, you’ll see the VxRail dashboard.  This allows you to see a quick status of the selected VxRail cluster, some support information and recent messages from the VxRail community.

Configure/VxRail

The most important features of the VxRail extension is found under the VxRail section of the Configure tab of a selected VxRail vSphere cluster:

  • The System section displays the currently-installed VxRail version and has a link to perform an update.  Clicking the “Update” link will launch a new browser tab and take you to the VxRail Manager web gui where you can perform the bundle update.
  • Next, the Market item will also launch a new browser tab on the VxRail Manager web gui where you can download available Dell EMC applications.  For now, it lists Data Domain Virtual Edition and Isilon SD Edge.
  • The Add VxRail Hosts item will display any newly-discovered VxRail nodes and allow you to add those nodes to a cluster
  • The Hosts item displays the hosts in the cluster.  One interesting feature here is that it displays the Service Tag and Appliance ID right here! You may not need this information often, but when you do, it’s super-critical.
    You’ll notice the “Edit” button on the hosts list; this allows you to set/change the host’s name and management IP.

Monitor/VxRail

  • On the Monitor tab for the selected vSphere VxRail cluster, the Appliances item provides a link to the VxRail Manager web gui where the hardware is shown and will highlight any components in need of attention.  Any faults will also be in the “All Issues” section of the regular web client, so the hardware detail will provide visual clues if needed.

Congratulations to the VxRail team responsible for this important milestone that brings another level of integration with vSphere and a “single-pane-of-glass”!

Configuring vRealize Log Insight Agent on MS SQL Server 2012

There’s a lot of information available about the SQL Server Content Pack for Log Insight, but actually  trying to use it reveals a lot of gaps.  So, here’s what I’ve done to get my SQL Server 2012 R2 instances to correctly report their information to vRealize Log Insight and show up in the Content Pack filters.

To ensure the correct log directories are used and allow reuse of filelog sections, I suggest creating an agent group for each SQL Server.  I’m also a fan of the server-side agent configuration, so if the agent gets totally removed and reinstalled, it’ll acquire the correct configuration again.

Steps:

  1. Install the Log Insight Agent for Windows on the SQL Server machine, be sure to use the FQDN of the Log Insight VIP
  2. While logged on to the SQL Server, locate the Logs folder for each SQL server instance. Note that this is not the database transaction logs folder – if it contains *.ldf files, it’s the wrong folder.   Specifically, you want the folder that contains the ERRORLOG file.  To be certain follow these steps:
    • Launch SQL Server Configuration Manager
    • Select “SQL Server Services” from the left pane
    • Right-click “SQL Server (INSTANCENAME)” from the right page, choose “Properties”.
    • Select the “Startup Parameters” tab
    • In the Existing Parameters box, make a note of the parameter that starts with “-e”
    • The filepath in that parameter (minus “ERRORLOG”) is the folder we need.
    • DO NOT CHANGE ANY VALUES, Cancel Properties, Close Configuration Manager

Startup Params in ConfigurationManager

  1. Logon to Log Insight as an administrator
  2. Check the Content Package area to be sure the “Microsoft – SQL Server” Content Pack is installed
  3. In the next steps, we’ll configure the agent from the server-side. It is possible to do from the client-side.
  4. In the Administration|Agents section of Log Insight select Microsoft – SQL Server from Available Templates drop-down list
  5. Click the “Copy Template” button. In the Copy Agent Group box that appears, change the name to “Microsoft – SQL Server – SERVERNAME”.  (replace SERVERNAME with the actual server’s name).  Click “Copy”.
  6. Now, the dropdown list should indicate that you are editing the new Agent Group
  7. In the Filter box, edit the parameters to create a filter that identifies only the desired server. For example “Hostname | starts with | SERVERNAME”
  8. In the Agent Configuration section
    • Update the section header from “[filelog|MSSQL]” to [filelog|MSSQL-SERVERNAME-INSTANCENAME]” replace SERVERNAME and INSTANCENAME with the correct, actual values. Be sure to keep “MSSQL” as the prefix, since the dashboard elements key on that.
    • Update the directory value to where the ERRORLOG file was found
    • Update the exclude value to read *.trc;*.xel;*.mdmp
    • Example:

[filelog|MSSQL-COREDB1-MSQLSERVER]
; IMPORTANT: Change the directory as per the environment
directory=C:\MSSQL\DATA\MSSQL12.MSSQLSERVER\MSSQL\Log
tags={“ms_product”:”mssql”}
charset=UTF-16LE
exclude=*.trc;*.xel;*.mdmp

  1. If there are multiple instances of SQL server on the host server, copy the entire section and edit the section header with the Instance Name. Example of configuration for multiple instances:

[filelog|MSSQL-COREDB1-MSSQLSERVER]
; IMPORTANT: Change the directory as per the environment
directory=C:\MSSQL\DATA\MSSQL12.MSSQLSERVER\MSSQL\Log
tags={“ms_product”:”mssql”}
charset=UTF-16LE
exclude=*.trc;*.xel;*.mdmp

[filelog|MSSQL-COREDB1-WORKLOAD]
; IMPORTANT: Change the directory as per the environment
directory=C:\MSSQL\DATA\MSSQL12.WORKLOAD\MSSQL\Log
tags={“ms_product”:”mssql”}
charset=UTF-16LE
exclude=*.trc;*.xel;*.mdmp

  1. Click Save New Group.
  2. On the SQL Server, restart the “VMware vRealize Log Insight Agent”
  3. Navigate to %ALLUSERSPROFILE%\VMware\Log Insight Agent and open liagent-effective.ini with notepad. Check that the sections added above appear in this file.
    • If they do not, you may have to adjust the filters on the Agent Group

Resolving Site Recovery Manager Error “Unable to create protection group. No VRM registered with vCenter Server”

Just a quick note here about this error.  I felt kinda silly once I realized what I’d done.

Scenario:

  1. Installed a vCenter Server at both sites
  2. Deployed a vSphere Replication 6.1 appliance at both sites.  Connected each to its corresponding PSC and vCenter Server
  3. Installed Site Recovery Manager 6.1 on each site, registered to corresponding vCenter Server
  4. Complete mappings and basic configurations in SRM, confirm VR is available at each site

    SRM configured with VR
    SRM configured with VR
  5. Attempt to create a protection group
  6. Receive Error “Unable to create protection group. No VRM registered with vCenter Server …”
  7. Google frantically

Its not clear from the documentation, but you have to configure at least one VM for replication using vSphere Replication section of Web Client first.  Totally counter-intuitive.  Just right-click the target VM and select “All vSphere Replication Actions | Configure Replication” and walk through the wizard.  Once complete, you can return to the SRM configuration and succesfully set up a Protection Group and subsequent Recovery Plan.

Deploy vCenter Log Insight Windows Agent using GPO

The VMware documentation covers this, but I thought I’d add my “insight” (get it? har har)

Prerequsities

  1. Get Microsoft Orca.  It’s part of the SDK found here.
  2. Get the vCenter Log Insight Windows Agent from my.vmware.com if you haven’t already
  3. An appropriate network location for GPO-delivered installers containing the vCLI Agent MSI

Orca Steps

  1. Launch Orca, choose file|open and select thevCLI Agent MSI (VMware-vCenter-Log-Insight-Agent-2.0.3-1879692_1.msi in this case)

    vCLI MSI open in Orca
    vCLI MSI open in Orca
  2. Within Orca, click Transform|New Transform

    Create new transform
    Create new transform
  3. Click the “Property” table to load its rows

    Property Table
    Property Table
  4. Right-click under the populated rows and choose “Add Row”
  5. In the “Add Row” dialog, enter the property as “SERVERHOST” and the value as theFQDN of yourvCenter Log Insight server, click OK.  Notice thenew record has a green box around it?  That means it’ll be included in the transform file.

    Add SERVERHOST Property
    Add SERVERHOST Property
  6. On the menu, choose Transform|Generate Transform.  Put it in the same folder as the vCLI Agent MSI, give it a descriptive name.
  7. Once the mst is saved, you can close Orca.

GPO Steps

  1. Create a new or open an existing GPO in Group Policy Management Editor
  2. Expand Computer Configuration|Policies|Software Settings, right-click Software installation and choose New|Package

    Add software installation package to GPO
    Add software installation package to GPO
  3. Select thevCLI MSI on the network share, select the “Advanced” option on the Deploy Software dialog, click OK.

    Choose the Advanced Option
    Choose the Advanced Option
  4. On the VMware vCenter Log Insight Agent Properties dialog, navigate to the Modifications tab.
  5. Click the “Add” button and select themst you created earlier. Click OK.  Make other changes to the package as appropriate for your environment and click OK to save.

    Modifications Tab
    Modifications Tab
  6. Like all other GPOs, link it to the appropriate OU(s) containing the computers you want the agent deployed on.

Handy VMKB for SRM & VR 5.1

VMKB 1009562  has a lot of good information, I’m not going to repeat it here, but it is a great resource for determining what network ports have to be open between what devices when using SRM & vSphere Replication.

Also, this diagram is surprisingly complicated… (reminds me of a dream-catcher)

SRM & VR ports

vCloud Director Database Issues

We had a situation recently where vCloud Director 1.5.1 (that’s one-dot-five-dot-one, not five-dot-one) failed to delete an external network, giving this error:

Could not execute JDBC batch update
– The DELETE statement conflicted with the REFERENCE constraint “allocated_ip_add_fk”. The conflict occurred in database “vCloud”, table “dbo.network_assigned_ip”, column ‘allocated_ip_add_id’.

This indicates that the record in the network_assigned_ip table cannot be deleted because it has a reference to a record in another table.

I had previously deleted all the related vApp networks and org networks and removed any stranded items.

To find the erroneous record, we first obtained the logical network id based on the network name:

Select * from [vCloud].[dbo].[logical_network] where name like ‘%BAD_NET%’ –note the network name has been set to “BAD_NET”

This returned one record, the first column value is the logical network ID, so we’re going to use this value to identify any assigned IP addresses in that logical networks via this query:

SELECT * from network_assigned_ip where logical_net_id = 0xLOGICALNETWORKID –replace with correct Logical network id value from previous query

Yes, I know how to do an INNER JOIN, but that’s not the point…

At this point, we had no left over IP addresses for that network, so we deleted the errant record from the logical_network table.

Now, the external network no longer appears, but everything’s not rosy.

vCD will only let you bind one network to a given port group and the database indicates that the port group I need to bind to a new external network is still in use. In my case this query was expected to have a record for the port group but did not:

Select * from [dbo].[ui_portgroups_avail_list_view

This information is stored in two different tables; vlan_in_use and real_network. Searching the vlan_in_use table for my VLAN ID did not return any records, so there must be a lingering item in real_network. I found it by querying the real_networks table and deleted that record.

Now ui_portgroups_avail_list_view includes a record for my port group, so I’m good-to-go.

Hopefully this information will be helpful to someone. I strongly suggest working with VMware support and having a case open and a very recent database backup before making any changes to the vCloud database.

vCloud Suite 5.1 upgrade suggestions

This information will probably be outdated quickly, but for now (9/14/2012), here are my suggestions and tips concerning vCloud Suite 5.1

  • If you plan to use Microsoft SQL Server for your SSO database, update the service configuration to listen on a static port.  vCenter Server and VUM use the .NET native SQL client, so they can deal with dynamic TCP ports on the SQL service, but SSO uses the Java SQL driver, which will require a static port (TCP 1433 by default)
  • Many administrators have had problems logging into their vCenter Server after installing SSO and upgrading vCenter Server to 5.1.  It appears that if the vCenter Server is a member of the domain, it may still try to use local credentials as the initial/default SSO source.  If your vCenter Server is a member of an Active Directory domain, be sure to locate/create a local admin account before upgrading vCenter Server to 5.1 so you can log on if you are affected by this issue
  • If you are running EMC Powerpath/VE on your ESXi hosts, do NOT upgrade to ESXi 5.1 yet.  The version of Powerpath/VE that works with ESXi 5.1 has not yet been released.
  • Despite the naming similarities, VMware View 5.1 is not compatible with vSphere 5.1.  Do not yet upgrade to vSphere 5.1 in your View environment.
  • The vCloud API for vCloud Director 5.1 is different from the API for vCloud Director 1.5.1.  If you have developed anything that consumes the vCloud API, you will probably have to make significant changes.  If you have vCO packages/workflows that use the vCloud API, you may be affected and should upgrade vCO and vCloud in a lab/test environment where you can update the objects appropriately first.
  • There is a new version of the Cisco Nexus 1000V for vSphere 5.1 that should be factored into your upgrade plan if your environment uses that component.
  • In vCenter Server 2.5-4.0, host passwords are encrypted in the database using an encryption key of 512-to-1024 bits. However, in 5.1, the encryption key must be 2048 bits. If it is not, the vCenter Services may not start. Check this KB article to see how to check and correct this issue.

Use Alarms to monitor your vSphere environment

There have been a number of KB articles, blog posts, best practices documentation and more suggesting vSphere admins use the Alarms feature in vSphere to proactively monitor their environment.  This is another one.

I was recently asked about a tool that could let someone know when memory or CPU utilization gets too high on a VM.  I suggested vCOps, but after a moment, I realized that vCOps is a fantastic tool for analysis and trending, but it’s not really meant to let someone know that a threshold has been reached and some action may or may not be needed.

For that, I suggest using the built-in and easy-to-use Alarms.   There are a number (54 in my case) of built-in Alarms that come pre-configured, but none of them – by default – will notify you via email when they’ve fired.  We’ll modify a built-in Alarm to send an email when its status changes.

Modify default Alarm

On event that deserves notification is a vSphere HA error. Namely, when vSphere HA on a host is in an error/alert state.  The Alarm to modify is named “vSphere HA host status”.  Find it on the Alarms tab while your vCenter server is selected in the treeview of the vSphere Client.

This One
This One

Right-click the Alarm and choose “Edit Settings”.  The “Alarm Settings” window has four tabs; General, Triggers, Reporting and Actions.  For this section,  we are concerned about the Actions tab, but you can check out the other tabs too.  Go ahead, I’ll be right here.  You’ll notice on the Actions tab, there are no actions listed, so when the status changes, the icon on the host will change.

No actions defined
No actions defined

Unless you are watching the vSphere client, you may not notice that anything has changed.  Let’s fix that.  Click the “Add” button.  It adds a new action item which, conveniently enough is “Send a notification email”, once,  when the status is “Alert”.  All we have to do is click in the “Configuration” field and type in the comma-separated email addresses for notifications to be sent to.  You have options to send the message repeatedly and when the status changes back to warning or normal – there depend on your environment and just how much email you want to send.  Please note that there’s no way – that I’m aware of – to modify the content of the notification email messages.  They just say “Alarm status on XX host changed from warning to alert” or similar.  It’s just enough information to know THAT something happened, but not WHY it happened.

There are a lot of valuable Alarms already configured that just need actions assigned. For example, there’s an alarm for Virtual Machine Memory Usage that will let you know when a VM is running out of vRAM – this is handy because you may not know about the high memory utilization without a notification.

Prerequisites:

The Alarm notifications are sent by the vCenter Server, so it has to be configured with valid SMTP settings:

vCenter Server SMTP Configuration
Configure SMTP settings for vCenter

VMware I/O Analyzer 1.0.0 Review

Since I’m in the middle of replacing one storage array (EMC Celerra NS-120) with another (EMC VNX 5500), I’m interested in comparisons of I/O performance between the arrays and between the various LUN configurations.  VMware made the I/O Analyzer Fling available on December 5, 2011.  Hopefully, you know how to deploy the OVA, so I won’t bore you with those details and will simply start with “okay, I’ve got it installed per the instructions, now what?”

  • Using your vSphere Client, open a console on the IOAnalyzer VM you imported.  Logon as “root”, using “vmware” as the password.  This will take you to an ordinary gray screen.

    IO Analyzer Grey Screen
  • Back on the vSphere Client, make a note of the IP Address of the IO Analyzer VM and the host on which it’s running.
  • Launch Firefox or Chrome and point it to the IP address of the IO Analyzer.  It should come up with the “Home” screen.

    I/O Analyzer Home Screen
  • Click “|| Setup & Run ||” at the top to go to that area.
  • On the Setup & Run page, we’ll set the host to gather ESXtop statistics on and the parameters to pass to IOMeter running in the IO Analyzer VM.
  • Under “Host Credentials”, enter the host name or IP address of the host where the IO Analyzer VM is running, provide the root password and click “Add Host” to save the details.
  • Under “Guest/Worker to Workload Binding”, select the host under the Host Name dropdown.  Supposedly, the “Select VM” dropdown will enumerate the VMs on the selected host, but I never saw it behave like this.  Just choose one of the VM names listed – the name is just used in the formatting report and has no bearing on the results.
  • The “Workload Spec” dropdown lists the access pattern you want to use with IOMeter.  There are some that represent a Workstation, SQL or Exchange Server in addition to the percent read/percent random ones – pick one.
  • In the VM IP Address, provide the IP address of the IO Analyzer VM (same IP address your browser is pointed to) and click “Add Worker”
  • You may add additional workers/access patterns, but when I tried, these additional workers were not spawned on the VM and only used the last workload defined.
  • Enter a duration for the test to run in seconds – usually 120 and click “Run”.
  • Flip over to the console on the IO Analyzer VM and watch IOMeter launch

    IOMeter running on the VM
  • In IOMeter, you can choose the “Results Display” tab, set the Results since to “Last Update” and the Update Frequency to 2 to see the results while the test is progressing.  This is great for those of us with little to no patience.
  • When finished, IOMeter will close and the “Setup & Run” page of the Web UI will state “Run complete.  Goto results tab.”
  • Click the “|| Results ||” link at the top, select the most recent item from the Result Summary drop down and click “View Summary” to load the details.

    IO Analyser Results View
  • The IOMeter Results summary is obtained from the IOMetere running within the VM and provides Read & Write IOPS and MBPS.
  • The Host Results Summary is obtained from ESXtop against the specified host

You can compare the IOPS and MBPS from IOMeter and ESXtop to see what additional load the host is enduring.

To measure the performance of another LUN, simply use Storage vMotion to migrate the IO Analyzer VM elsewhere and rerun it.

My Observations

The VM dropdown thing is annoying, but not really crucial.  The tool will let you select a host unrelated to the VM and confound yourself.  You must logon to the VM on the console in order for it to fire off IOMeter and obtain the results.  It’s a little clunky; multiple workers are not spawned on the VM when selected in the Web GUI.  But, you can simply run IOMeter yourself in the VM and set your own params.

Coming Up

I’ll post my performance statistics for various LUNs and later compare them to the storage on the VNX5500  with and without- FAST cache.

vSphere 5 Licensing impact on VDI

Very quietly, VMware released some information about vSphere for Desktop. This is the licensing model you’re going to want… especially for VDI.

Here’s a link to the details:vSphere Desktop FAQ

There are a number of caveats explained in this FAQ, but the licensing strategy is beginning to make sense.

In short, if you run VMware View, you can get an unlimited vRAM entitlement.  You cannot upgrade to vSphere 5 for Desktop and if you’re running a third-party VDI solution on vSphere, you will have to use the “non-Desktop” editions and be subjected to vRAM license limits.

This is just my humble opinion and does not necessarily reflect the opinion of my employer or partners or anyone else for that matter