2-Node vSAN Direct Connect and LACP

One of my customers is using 2-node vSANs on multiple branch offices. One of many reasons of using 2-node vSAN is the possibility to leverage existing 1 Gb network and use 25 Gb Direct Connect between ESXi hosts (vSAN nodes) without the need of 25 Gb Ethernet switches. Generally they have very good experience with vSAN, but recently they have experienced vSAN Direct Connect outages when testing the network resiliency. The resiliency test was done by administrative shutdown of one vmnic (physical NIC port) on one vSAN node. After further troubleshooting, they realized their particular NICs (Network Adapters) do not propagate link down state to the physical link, when vmnic is administratively disabled by command "esxcli network nic down -n vmnic2". 

It is worth to mention, that such network outage does not mean 2-node vSAN outage because that's the reason why we have vSAN witness, however, vSAN is in degraded state and cannot provide mirror (RAID1) protection of vSAN objects.

Such network behavior is definitely strange and we have opened discussion and root cause analysis with hardware vendor, however, we have also started the internal discussion about design alternatives we have to mitigate such weird situations and increase resiliency and the overall availability of vSAN system.

Here are three design options how to implement direct connect networking between two ESXi hosts.

Design Option 1 - Switch independent teaming with explicit fail-over

Option 1 is using single VMkernel interface (vmk2) connected to single vSwitch portgroup which is using two uplinks with explicit fail-over teaming where vmnic2 is the explicit active uplink and vmnic3 will be used only in case vmnic2 is not available.

 

This design option is generally recommended by VMware.

Benefits: simple configuration, highly available solution

Drawbacks: in case of link state hardware problem, you can be in situation when one vSAN node is using VMkernel interface via vmnic2 uplink and 

Design Option 2 - Link Aggregation (LACP)

Option 2 is using single VMkernel interface (vmk2) connected to single vSwitch portgroup having single logical uplink (LAG) which is backed by two uplinks (vmnic2, vminc3) bonded into the port-channel. In such network configuration, both uplinks are active. It is worth to mention, that in 2-node configuration, LACP load balancing algorithm can help with load balancing of vSAN traffic across both uplinks, but the benefit of LACP is periodical heart beating (sending LACPDU) which is by default done every 30 seconds (slow LACP). For more information LACP timers read this blog post.

Benefits: LAG virtual interface with LACPDU heart beating can mitigate the risk of black hole scenario in case of problems with link state.

Drawbacks: 

• LACP configuration is more complicated than switch independent teaming, therefore it has a negative impact on manageability. 
• Network availability is not guaranteed with multiple vmknics in some asymmetric failures, such as one NIC failure on one host and another NIC failure on another host. However, more bundled links can increase vSAN traffic availability, because vSAN L3 connectivity would stay up and running until single L1 link is up.
  

Useful LACP commands

• esxcli network vswitch dvs vmware lacp status get
• esxcli network vswitch dvs vmware lacp stats get
• esxcli network nic down -n vmnic2 
• esxcli network nic up -n vmnic2

Design Option 3 - Two vSAN Air Gap Network

Two vSAN Air Gap Networks actually means two vSAN vmkernel interfaces connected to two totally independent (air gap) networks.

Benefits: Little bit easier configuration than LACP.

Drawbacks: 

• Setup is complex and error prone, so troubleshooting is more complex. 
• Requires multiple L3 VMkernel interfaces for vSAN traffic. 
  
• Network availability is not guaranteed with multiple vmknics in some asymmetric failures, such as one NIC failure on one host and another NIC failure on another host. 
• Source: Pros and Cons of Air Gap Network Configurations with vSAN
  

Conclusion and design decision

In this blog post, I have described three different option of network configuration for vSAN direct connect. I personally believe, the design option 2 (LACP for vSAN Direct Connect) is the optimal design decision, especially if NIC link state propagation is not reliable as is the case for my customer. However, the design option 2 is solving the issue as well. The final design decision is on the customer.