UCS and Mixed vPC / Non vPC VLANs
A great feature added to the Cisco UCS V2+ is the ability to map VLANs to specific Port Channel uplinks and/or Ethernet uplinks. This seems pretty straight forward, but this was not available in V1 and I am finding customers who are not aware of this capability…
Many deployments of UCS use vPC links from Nexus core networks to the UCS Fabric Interconnects, and by default, both vPC trunks presented to the UCS are used for all VLAN traffic without any pinning of VLANs to specific Port Channels or Ethernet uplinks.
Due to a “feature” of the Nexus platform, there is an issue using vPC backed VLAN(s) for routing Edge traffic from an NSX environment and the solution is to add non vPC uplinks from the Nexus core to the UCS Fabric Interconnects. As I have seen, if VLAN pinning is not being used, then adding a new uplink without preparation will create a “black hole” for all vPC VLANs given the new non vPC uplink(s) are automatically another path to use for all VLANs.
Brad Hedlund does a great job of describing the issues surrounding routing across vPC trunks and the requirement to provide a non vPC back uplinks: http://lostdomain.org/2014/02/13/design-guide-deploying-vmware-nsx-with-cisco-ucs-and-nexus-7000/
Attached UCS configuration guide speaks to pinning traffic: http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/sw/gui/config/guide
So simply, before creating new uplinks for non vPC traffic on the UCS, ensure existing vPC VLANs are mapping to existing vPC Port Channels or Ethernet interfaces. Once this is in place, adding a new uplink will not create a “black hole” for existing VLAN’s, and the new VLAN required for Edge Traffic can then be created and mapped to the two (Fabric A and B) non vPC uplinks/port groups.
If you have not mapped VLAN’s to Port Croups/Ethernet Interfaces, the LAN Uplinks Manager configuration option available by right clicking on any Port Channel or Ethernet Interface under LAN configuration:
From LAN Uplinks Manager, you can then select VLAN Manager:
Remember to perform this configuration for both Fabric A and B Port Channels/Ethernet Interfaces. In the example below, VLAN’s have been mapped to Ethernet Interfaces Fabric A 1/17, 1/18, Fabric B 1/17, 1/18.
UCS is a awesome platform, but care must be taken to ensure you fully understand how VLANs map from the Fabric Interconnects to the core network before changes are made to enable new uplinks.
Hi Dwayne:
I have some tips for those implementing this into an existing UCS domain.
1. In order to implement disjoint L2 without affecting existing vNIC pinning, you first need to create a VLAN group called “default” or whatever you want to name it, and assign that VLAN group to the existing uplinks using VLAN manager. This ensures that bringing additional uplinks online do not immediately become available for vNIC pinning.
2. Create your second VLAN group, and assign it to the new uplinks.
3. If vNICs participating in your secondary path are ESX trunks, you MUST be sure that you do not trunk VLANs that belong to two separate uplinks, otherwise you’ll fail to pin. Futhermore, if you do not specify a default VLAN your DVS health checks will fail, and you can’t send UCS “default” to both uplink groups. Therefore for each set of disjoint links you create I’ve found it good practice to create a new “default” VLAN for those links. It doesn’t really have to go anywhere, it just needs to exist.
4. If you need a host, perhaps say an “edge” cluster used for NSX Edge services, be sure you provision a pair of vNICs per disjoint link or DVS that it needs to participate it. Remember, mixing VLANs that belong to two different uplinks will lead to bad times.
Awesome feedback. Thanks Tommy!
– Dwayne
Hello,
I have 3 servers with ESXi 5.5 and I want to create 2 vlans for storage propose but I don’t have blade servers.
Do you know if its possible configure only vlans for use UCS Fabric Interconnects 6100 40XP to connect different vnics for use vMotion and vSan without blades? My networks cards are Intel Adapter X710-DA4
In case that I can, can you explain me how to do it, please?
Thanks