I’ve spent the past several weeks testing, trying to understand Cross-vCenter NSX and make it work in a useful way. Here’s what I’ve learned so far.
Environment and Set up:
I knew I needed two “sites” with discrete storage, but had a few physical limitations. I.e; I only own one managed router (Cisco C2821) and one managed switch (Force 10 S50). I trunked the management and vtep network to all the hosts, but configured a discrete transit network for each “site”. I hope to work with a much larger lab environment and do a thorough review of the setup and configuration. In the meantime, here’s the basics.
- Two-host cluster, NSX manager and vCenter Server for primary site
- 1 host cluster, NSX manager and vCenter Server for recovery site
- vSphere 6.0 U1b
- NSX for vSphere 6.2.1
- Universal Transport Zone includes both clusters, configured for Local Egress.
- VTEPs in same L3 network (for simplicity’s sake)
- Edge per site, discrete transit networks
- Universal DLR for tenant networks, uplinked to both site ESGs
- OSPF area for each ESG->UDLR
Objectives
- Eliminate need for manual synchronization between sites/NSX instances
- Success! The distributed firewall and universal objects are respected by the NSX manager at both sites. Universal Security Groups and Logical Switches are usable at either end.
- Span VXLANs between sites
- Success! This is not really a surprise. As long as the clusters share a Transport Zone and teh VTEPs can route to each other, this works great.
- Minimize network alterations when used with Site Recovery Manager to failover protected VMs between vCenter Servers.
- Partial success. A placeholder/recovered VM will be in the same Universal Security Groups as the protected VM and the distributed firewall rules will apply as expected without any changes needed. However, even with Local Egress enabled, you’ll have to apply some sort of route updates so that traffic destined for the recovered VMs can get there. It looks like the route redistribution for egress us handled automatically though. This configuration pushed the limits of what I can do with my lab.
- Make use of site-to-site microsegmentation (my definition: application of security policy regardless of L3 network scope)
- Partial Success. I was really disappointed here. When the document states that you can add IP Sets, MAC Sets and other Universal Security Groups to a Universal Security Group, it means that’s ALL you can add. I’ve blogged a few times about adding VMs to a Security Group – even doing it as a Resource Action in vRA. That won’t work with Universal Security Groups! The only workaround I see is to add the desired VM’s IP or MAC to a Set and include the Set in the Security group. Blech!
- Retain ability to assign security policy via security group membership through vRA
- FAILURE. As noted above, we cannot assign Universal security group membership to a VM using any of the existing custom properties.
Notes and Observations
I guess I get it. The vm id between vCenter Servers will differ, so saying that “vm-123 is a member of securitygroup-456” is not valid on a different vCenter Server. But a table of IPs and MACs would be universally valid. I’m hoping that in a near-future version, the universal security group membership capabilities can be extended; perhaps with a shared or replicated vCenter Inventory Service.
Brian Ragazzi 