How UCS QOS System Class can affect VMware NSX VXLANs

In a recent, ongoing, installation we encountered a wide variety of sporadic network traffic issues  affecting the VMs connected to NSX Logical Switches (VXLANs).

Some of the symptoms were:

• Can ping a device, but not get a complete tracert
• Can connect to a server over HTTPS, but not its neighbor (both webservers)
• vCAC Gugent cannot connect to vCAC server
• Were unable to perform a vmkping using the VXLAN TCP stack with more than 1470 bytes.

The last bullet made it pretty clear that the issue was related to the MTU.  We had no visibility into the configuration of the north-south layer 3 devices, but had been assured that they were configured for 1600 byte frames.

In the NSX for vSphere implementation of VXLAN, the packets sent by devices get a new additional header, increasing its overall size beyond 1500 bytes (up to ~1540 bytes or so).

I checked the UCS service profiles and the vNIC Templates, it looked something like this:

UCS vNIC template

It certainly looks like its set for jumbo frames, but also notice the second red ellipse there; QoS Policy. If you pay attention to things like that, you might also notice the warning about the MTU size.

The QoS policy assigned to the vNIC (template) is uses an egress priority based on a QoS System Class.

UCS QoS Policy

The QoS System Classes specify not only a class-of-service and a weight, but also MTU! In my case, I changed the vNIC Template QoS Policy to one with a System Class whose MTU is  9216. Once this change was made, the VMs behaved as expected.

UCS System Class

A couple of notes:

• If your vNIC (templates) do not specify a QoS Policy, UCS appears to use the MTU given
• If you do not have an enabled QoS System Class with an MTU of 9216, you’ll have to type it in, the dropdown list only contains “normal” and “fc”

This is another of those posts where I just stumbled upon the fix and needed to write it up before I forgot. Hopefully this will some someone a lot of time later.

Advertisement

Use Cisco Nexus 1000V for virtual hosts in nested ESXi

The native VMware vSwitch and Distributed vSwitch do not use MAC-learning. This was removed because the vSwitches would be aware of the VMs attached to them and the MAC addresses in use. As a result, if you nest ESXi under a standard vSwitch and power-on VMs under the nested instance, those VMs will be unable to communicate because their MACs are masked by the virtual host and the vSwitch is not aware of them.

Workaround options:

1. Enable Promiscuous mode on the vSwitch.
2. This works but should never be used in production.  It adds a lot of unnecessary traffic and work to the physical NICs.  It makes troubleshooting difficult and is a security risk
3. Attach your virtual hosts to a Cisco Nexus 1000V.
4. The 1000V retains MAC-learning, so VMs on nested virtual ESXi hosts can successfully communicate because the switch learns the nested MAC addresses.
5. If your physical servers support virtual interfaces, you can create additional “physical” interfaces and pass them through to the virtual instances.  This allows you to place the virtual hosts on the same switch as the physical hosts if you choose.  There is obviously a finite amount of virtual interfaces you can create in the service profile, but I think this is a clean, low-overhead solution for environments using Cisco UCS or HP C7000 or similar.

Conclusion

The Nexus 1000V brings back important functionality for nested ESXi environments, especially those environments that do not have access to features like virtual interfaces and service profiles.

Helpful links:

Standing Up The Cisco Nexus 1000v In Less Than 10 Minutes by Kendrick Coleman

Experience in upgrading UCS to 1.4(1j)

The UCS deployment in the Mobile VCE is different from many deployments because it does not employ many of the redundant and fault-tolerant options and doesn’t run a production workload.  So, I have the flexibility to bring it down at almost any time for as long a duration as needed.

All this aside, it IS a complete Cisco UCS deployment with all the same behavior as if it were in production.  This means, I can perform an upgrade or configuration change in this environment first and work through all the ramifications before performing the same action on a production environment.

There’s a lot of excitement around the web about the new features in this upgrade and I’ve been looking forward to installing it.  For me, I’m excited about the lengthy list of fixes, the ability to integrate the management of the C-series server with the UCS Manager, FC port-channels and user-labels.

To start, I used the vSphere client to shut down the VMs I could and moved those I couldn’t to the C-series.  Please note that I have not yet connected the C-series to the Fabric Interconnect for integration yet (that’s another post).  Then I shutdown the blades themselves.

For the actual upgrade, I simply followed the upgrade guide – there’s no reason to go through the details of that here.

However, the experience was not exactly stress-free.

Although it makes sense, when the IO Module is rebooted, the Fabric Interconnect loses connection to the Chassis.  It cycled through a sequence of heart-stopping error messages before finally rediscovering the chassis and servers and stabilizing.  During this phase, it’s best to not look – the error messages led me to believe the IOM had become incompatible with the Fabric Interconnect.  Like I said, after a few minutes, the error messages all resolved and every component was successfully updated to 1.4.1.

GUI changes after upgrade

Nodes on the Equipment tree for Rack-Mounts/FEX and Rack-Mounts/Servers

 

 

 

 

 

 

User labels (Yes! )

Summary
I’ll be connecting the C-series to the Fabric Interconnect soon and am looking forward to setting up the FC port channel.

Resolve Hardware Status Alert SEL_FULLNESS

I noticed an alert on two UCS B250M2 hosts in the vSphere Client.  The alert Name was “Status of other host hardware objects”.  This isn’t helpful.  To get more information, you have to navigate to the Hardware Status tab of the host properties.  Here I saw more information about the alert.  It’s cryptically named “System Board 0 SEL_FULLNESS”.

SEL_FULLNESS alert in vSphere Client

This points to the System Event Log for the UCS blade itself.  Luckily, this is easily cleared by using the UCS Manager to navigate to the management Logs tab of the Server properties under Equipment.

Clear management Log for UCS Blade

Once there, you can back up and clear the SEL.  Within a few minutes, the vSphere sensors will update and the alert will be gone.

UPDATE:  Once UCSM has been updated to 1.4.1, the “Management Logs” tab is named “SEL Logs”