Back to the basics - VMware vSphere networking

As a software-defined networking (VMware NSX) is getting more and more traction I have been recently often asked to explain the basics of VMware vSphere networking to networking experts who do not have experience with VMware vSphere platform. First of all, networking team should familiarize them self with vSphere platform at least from a high level. Following two videos can help them to understand what vSphere platform is.

vSphere Overview Video
https://youtu.be/EvXn2QiL3gs

What is vCenter (Watch the first two minutes)
https://youtu.be/J0pQ2dKFLbg

When they understand basic vSphere terms like vCenter and  ESXi we can start talking about virtual networking.

First thing first, VMware vSwitch is not a switch. Let me repeat it again ...

VMware vSwitch is not a typical ethernet switch.

It is not a typical network (ethernet) switch because not all switch ports are equal. In VMware vSwitch you have to configure switch uplinks (physical NICs) and internal switch ports (software constructs). If the ethernet frame is coming from the physical network via uplink, vSwitch will never forward such frame to any other uplink but only to internal switch ports, where virtual machines are connected. This behavior guarantees that vSwitch will never cause the L2 loop problem.  It also means that vSwitch does not need to implement and participate in spanning tree protocol (STP) usually running in your physical network. Another different vSwitch behavior compared to traditional ethernet switch is that vSwitch does not learn external MAC addresses. It only knows about MAC addresses of virtual machines running on particular ESXi host (hypervisor). Such devices are often called port extenders. For example, CISCO FEX (fabric extender) is a physical device having the same behavior.

Now let's talk about network redundancy. In production environments, we usually have a redundant network where multiple NICs are connected to different physical switches.

Each NIC connected to different physical switch

vSwitch network redundancy is achieved by NIC teaming. NIC teaming is also known as link aggregation, link bundling, port channeling, ethernet bonding or NIC teaming. VMware is using the term Network teaming or NIC teaming. So what teaming options do we have in VMware vSphere platform? It depends on what edition (license) you have and what vSwitch you want to use. VMware offers two types of vSwitches.

• VMware vSphere standard switch (aka vSwitch or vSS)
• VMware vSphere distributed virtual switch (aka dvSwitch or vDS)

Let's start with VMware's standard switch available on all editions.

VMware vSphere standard switch (vSS)

VMware vSphere standard switch supports multiple switch independent active/active and active/standby teaming methods and also one switch dependent active/active teaming method.

The standard switch can use following switch independent load balancing algorithms:

• Route based on originating virtual port - (default) switch independent active/active teaming where the traffic is load balanced in round-robin fashion across all active network adapters (NICs) based on internal vSwitch port id where virtual machine vNIC's or ESXi vmKernel ports are connected.
• Route based on source MAC hash - switch independent active/active teaming where the traffic is load balanced in round-robin fashion across all active network adapters (NICs) based on source MAC address identified in standard vSwitch.
• Use explicit failover order - is another switch independent teaming but active/passive. Only one adapter from all active adapters is used and if it fails the next one is used. In other words, it always uses the highest order uplink from the list of Active adapters which passes failover detection criteria.

and only one switch dependent load balancing algorithm

• Route based on IP hash - switch dependent active/active teaming where the traffic is load balanced based on a hash of the source and destination IP addresses of each packet. For non-IP packets, whatever is at those offsets is used to compute the hash. This is switch dependent teaming, therefore, the static port-channel (aka ether-channel) has to be configured on the physical switch side otherwise, it will not work.

It is worth to mention that for all active/active teaming methods you can add additional standby adapters which are used just in case the active adapter fails and you can also define unused adapters which you do not want to use at all. For further information, you can read VMware vSphere documentation.

VMware vSphere distributed switch (vDS)

If you have vSphere Enterprise Plus license or VSAN license you are eligible to use VMware vSphere distributed switch. VMware distributed switch key advantages are

• centralized management
• advanced enterprise functionality

When you use virtual distrubuted switch, you do not need to configure each vSwitch individually but instead, you have single distributed vSwitch across multiple ESXi hosts and you can manage it centrally. On top of centralized management you will get following advanced enterprise functionalities:

• NIOC (Network I/O Control) which allows QoS and marking (802.1p tagging, DSCP)
• PVLAN
• LBT (Load Based Teaming) - official name is "Route based on physical NIC load"
• LACP - dynamic switch dependent teaming
• Route based on physical NIC load - another switch independent teaming with optimized load balancing
• ACLs - Access Control Lists
• LLDP
• Port mirroring
• NetFlow
• Configuration backup and restore
• and more

So in vDS you have two additional teaming options

1. Switch Independent - LBT
2. Switch Dependent - LACP

LBT is proprietary VMware solution where vmkernel checks every 30sec the stats from the relevant pNIC's and the calculation use the stats for the 30sec interval and get an average over the 30sec (to normalize the average and to eliminate spikes). If the bandwidth is above 75%, it will mark the pNIC as saturated. If a pNIC have been marked as saturated, the vmkernel will not move any more traffic onto the saturated pNIC. Read blog post "LBT (Load Based Teaming) explained" for more details (see links to references below this article).

LACP is standardized link aggregation protocol. It is worth to mention, that when LACP is used you can leverage significantly enhanced load balancing algorithms to more optimal bandwidth usage of physical NICs. Theoretically, you can use for single VM more bandwidth than is the bandwidth of single pNIC. But to see it in the real world, multiple flows must be initiated from that single VM and LACP the result of the hash algorithm must route it across multiple links bundled in a LAG.

vSphere 6.0 LACP supports following twenty (20) hash algorithms:

1. Destination IP address
2. Destination IP address and TCP/UDP port
3. Destination IP address and VLAN
4. Destination IP address, TCP/UDP port and VLAN
5. Destination MAC address
6. Destination TCP/UDP port
7. Source IP address
8. Source IP address and TCP/UDP port
9. Source IP address and VLAN
10. Source IP address, TCP/UDP port and VLAN
11. Source MAC address
12. Source TCP/UDP port
13. Source and destination IP address
14. Source and destination IP address and TCP/UDP port
15. Source and destination IP address and VLAN
16. Source and destination IP address, TCP/UDP port and VLAN
17. Source and destination MAC address
18. Source and destination TCP/UDP port
19. Source port ID
20. VLAN

Note: Advanced LACP settings are available via esxcli commands.  
esxcli network vswitch dvs vmware lacp
esxcli network vswitch dvs vmware lacp config get
esxcli network vswitch dvs vmware lacp status get
esxcli network vswitch dvs vmware lacp timeout set

Hope this was informative and useful.

References to other useful resources

• VMware : vSphere Networking (official documentation)
• James Green : LAG vs. LBT for vSwitch Uplinks in vSphere
• Hugo Strydom : LBT (Load Based Teaming) explained - Part 1, Part 2,  Part 3
• Chris Wahl : Exploring Enhanced LACP Support with vSphere 5.5 
• Steven Kang : ADVANCED LACP CONFIGURATION USING ESXCLI

ESXi Host Power Management

I have just listened to Qasim Ali's  VMworld session "INF8465 - Extreme Performance Series: Power Management's Impact on Performance" about ESXi Host Power Management (P-States, C-States, TurboMode and more) and here are his general recommendations

• Configure BIOS to allow ESXi host the most flexibility in using power management features offered by the hardware
• Select "OS Control mode", "Performace per Watt", or equivalent 
• Enable everything P-States, C-States and Turbo mode
• To achieve the best performance per watt for most workloads, leave the power policy at default which is "Balanced"
• For applications that require maximum performance, switch to "High Performance" from within ESXi host

Ali's VMworld session linked above is really worth to watch. I encourage you to watch it by yourself.