Jumbo Frames and Network Virtualization Overlay Networks

In data networking, the standard Maximum Transmission Unit (MTU) for IP frames is 1500 Bytes. Jumbo Frames are MTU’s larger than 1500 Bytes although when we reference Jumbo Frames, we are normally referencing a Frame with an MTU of 9000 Bytes.

Network Virtualization and the addition of an overlay (encapsulation) networking technology (VXLAN, STT, GRE, Geneve) to the data center requires some consideration as to what MTU to use due to the overhead of overlay transport technologies.

VMware NSX using the overlay technology VXLAN requires an MTU of 100 bytes greater than the networks used for Virtual Machines – In a traditional data center using a 1500 byte MTU for virtual machines, this would require the use of a 1600 Byte MTU for the L2 or L3 network(s) associated with the overlay or NSX Transport network…

Why 1600 Bytes for the overlay network? Around 50 Bytes for the encapsulation header (VXLAN), and extra 16 to 32 Bytes for future IPv6 possibilities (IPv4 Transport/IPv6 Payload, IPv6 Transport/IPv4 Payload, and IPv6 Transport/IPv6 Payload) then round up to nice 1600. Again, this assumes virtual machines and networks are configured for an MTU of 1500 bytes.

So who uses 1500 Byte MTU’s for physical and virtual machines? Pretty much everyone I have worked with in recent years. I have seen 9K MTU used for the network fabric and for IP ISCSI and NFS implementations, but it is very uncommon for physical server and virtual machines together with virtual machine networks to use an MTU greater than 1500 Bytes.

The Modern Data Center Network Fabric

With physical servers (hypervisors) and network fabrics capable of running 10GbE, 40GbE, and 100GbE, it is becoming a requirement for the modern data center to move up to Jumbo Frames to take advantage of the benefits of minimizing fragmentation. Deploying or updating a network fabric to 9000 Byte MTU is relatively easy and many environments are already running 9000 Bytes across the network fabric… The hard part is the physical and virtual machines.

Path MTU Discovery

Path MTU Discovery (PMTUD) is a mechanism used by IP to discover the end to end MTU across a network. Unfortunately, depending on the application and network boxes between source and destination, PMTUD can be problematic. Any security service that filters ICMP Fragmentation Needed will break PMTUD, and PMTUD cannot mitigate against problems caused by mixed MTU sizes on the same Layer 2 network.

In a VMware environment, I have personally worked on issues with vMotion timeouts associated with mixed MTU sizes between Hypervisor vMotion networks.

Jumbo Frames on the Physical Servers, Blade Chassis and Virtual Machines

For a green field environment, deploying a network fabric with 9000 Byte MTU support together with a Jumbo Frame for physical and virtual machines is a relatively easy proposition. In a brownfield environment, the implementation of 9000 Byte MTU on the network fabric is straight forward, but for Physical and Virtual Machines, requires careful planning, testing and validation on a Layer 2 Segment by Layer 2 Segment basis…

The most important question to consider before attempting to deploy Jumbo Frames for Physical and Virtual Machines is what size to use? 9000 Bytes??

The Overlay Network in a Jumbo Frame MTU Data Center…

As I stated at the beginning of this blog, an overlay network such as VXLAN together with IPv6 futures requires a overhead of around 100 bytes. If there is a plan on using virtual networking within the data center and also a plan on using Jumbo Frames, an MTU of 8900 Bytes would be the largest MTU that could be used by Physical and Virtual Machines together with Physical and Virtual Machine networks to allow for the overhead of the overlay technology.

Gotcha’s – Although a network fabric may support an MTU greater than 9200 Bytes, vSphere Distributed Switch (vDS) only supports an MTU up to 9000 Bytes, implementing vDS with VXLAN would again require Virtual Machines to have an MTU no greater than 8900 Bytes.

NFS and iSCSI IP Storage?

vSphere Distributed Switch Network IO Controls together with QoS ensure that IP Storage is not impacted by the use of Jumbo Frames for all data center IP communications.

Related Posts
NSX and Virtual Machine Sprawl
Logical Network to Physical Network Mapping
Installing or Upgrading to ESXi 5.5 Best Practices