NSX VTEP encapsulation significantly benefits from physical NIC offload capabilities. In this blog post, I will show how to identify NIC capabilities.
Check NIC type and driver
-->
HCL device identifiers
Check TSO configuration
Check LRO configuration
Check Checksum offload configuration
Check VXLAN offloading
1.1.7 Check VMDq (NetQueue)
1.1.8 Dynamic NetQ
VXLAN PERFORMANCE
Check NIC type and driver
esxcli network nic get -n vmnic4
[dpasek@esx01:~] esxcli network nic get -n vmnic4
Advertised Auto Negotiation: false
Advertised Link Modes: 10000BaseT/Full
Auto Negotiation: false
Cable Type: FIBRE
Current Message Level: 0
Driver Info:
Bus Info: 0000:05:00.0
Driver: bnx2x
Firmware Version: bc 7.13.75
Version: 2.713.10.v60.4
Link Detected: true
Link Status: Up
Name: vmnic4
PHYAddress: 1
Pause Autonegotiate: false
Pause RX: true
Pause TX: true
Supported Ports: FIBRE
Supports Auto Negotiation: false
Supports Pause: true
Supports Wakeon: false
Transceiver: internal
Virtual Address: 00:50:56:59:d8:8c
Wakeon: None
[dpasek@czchoesint203:~]
esxcli software vib list | grep bnx2x
[dpasek@esx01:~] esxcli software vib list | grep bnx2x
net-bnx2x 2.713.10.v60.4-1OEM.600.0.0.2494585 QLogic VMwareCertified 2017-05-10
[dpasek@czchoesint203:~]
Driver parameters can be listed by command …
esxcli system module parameters list -m bnx2x
[dpasek@esx01:~] esxcli system module parameters list -m bnx2x
Name Type Value Description
------------------------------------ ------------ ----- -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
RSS int Controls the number of queues in an RSS pool. Supported Values 2-4.
autogreeen uint Set autoGrEEEn (0:HW default; 1:force on; 2:force off)
bnx2x_vf_passthru_wait_event_timeout uint For debug purposes, set the value timeout value on VF OP to complete in ms
debug uint Default debug msglevel
debug_unhide_nics int Force the exposure of the vmnic interface for debugging purposes[Default is to hide the nics]1. In SRIOV mode expose the PF
disable_feat_preemptible int For debug purposes, disable FEAT_PREEMPTIBLE when set to value of 1
disable_fw_dmp int For debug purposes, disable firmware dump feature when set to value of 1
disable_iscsi_ooo uint Disable iSCSI OOO support
disable_rss_dyn int For debug purposes, disable RSS_DYN feature when set to value of 1
disable_tpa uint Disable the TPA (LRO) feature
disable_vxlan_filter int Enable/disable vxlan filtering feature. Default:1, Enable:0, Disable:1
dropless_fc uint Pause on exhausted host ring
eee set EEE Tx LPI timer with this value; 0: HW default; -1: Force disable EEE.
enable_default_queue_filters int Allow filters on the default queue. [Default is disabled for non-NPAR mode, enabled by default on NPAR mode]
enable_geneve_ofld int Enable/Disable GENEVE offloads. 1: [Default] Enable GENEVE Offloads. 0: Disable GENEVE Offloads.
enable_live_grcdump int Enable live GRC dump 0x0: Disable live GRC dump, 0x1: Enable Parity/Live GRC dump [Enabled by default], 0x2: Enable Tx timeout GRC dump, 0x4: Enable Stats timeout GRC dump
enable_vxlan_ofld int Allow vxlan TSO/CSO offload support.[Default is enabled, 1: enable vxlan offload, 0: disable vxlan offload]
heap_initial int Initial heap size allocated for the driver.
heap_max int Maximum attainable heap size for the driver.
int_mode uint Force interrupt mode other than MSI-X (1 INT#x; 2 MSI)
max_agg_size_param uint max aggregation size
max_vfs array of int Number of Virtual Functions: 0 = disable (default), 1-64 = enable this many VFs
mrrs int Force Max Read Req Size (0..3) (for debug)
multi_rx_filters int Define the number of RX filters per NetQueue: (allowed values: -1 to Max # of RX filters per NetQueue, -1: use the default number of RX filters; 0: Disable use of multiple RX filters; 1..Max # the number of RX filters per NetQueue: will force the number of RX filters to use for NetQueue
native_eee int
num_queues int Set number of queues (default is as a number of CPUs)
num_queues_on_default_queue int Controls the number of RSS queues ( 1 or more) enabled on the default queue. Supported Values 1-7, Default=4
num_rss_pools int Control the existance of an RSS pool. When 0,RSS pool is disabled. When 1, there will be an RSS pool (given that RSS>0).
poll uint Use polling (for debug)
pri_map uint Priority to HW queue mapping
psod_on_panic int PSOD on panic
rss_on_default_queue int RSS feature on default queue on eachphysical function that is an L2 function. Enable=1, Disable=0. Default=0
skb_mpool_initial int Driver's minimum private socket buffer memory pool size.
skb_mpool_max int Maximum attainable private socket buffer memory pool size for the driver.
To get driver parameter value
esxcfg-module --get-options bnx2x
[dpasek@esx01:~] esxcfg-module -g bnx2x
bnx2x enabled = 1 options = ''
HCL device identifiers
vmkchdev -l | grep vmnic
[dpasek@esx01:~] vmkchdev -l | grep vmnic
0000:02:00.0 14e4:1657 103c:22be vmkernel vmnic0
0000:02:00.1 14e4:1657 103c:22be vmkernel vmnic1
0000:02:00.2 14e4:1657 103c:22be vmkernel vmnic2
0000:02:00.3 14e4:1657 103c:22be vmkernel vmnic3
0000:05:00.0 14e4:168e 103c:339d vmkernel vmnic4
0000:05:00.1 14e4:168e 103c:339d vmkernel vmnic5
0000:88:00.0 14e4:168e 103c:339d vmkernel vmnic6
0000:88:00.1 14e4:168e 103c:339d vmkernel vmnic7
So, in case of vmknic4 there is
· VID:DID SVID:SSID
· 14e4:168e 103c:339d
Check TSO configuration
esxcli network nic tso get
[dpasek@esx01:~] esxcli network nic tso get
NIC Value
------ -----
vmnic0 on
vmnic1 on
vmnic2 on
vmnic3 on
vmnic4 on
vmnic5 on
vmnic6 on
vmnic7 on
esxcli system settings advanced list -o /Net/UseHwTSO
[dpasek@esx01:~] esxcli system settings advanced list -o /Net/UseHwTSO
Path: /Net/UseHwTSO
Type: integer
Int Value: 1
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: When non-zero, use pNIC HW TSO offload if available
If you want disable TSO, use following commands …
esxcli network nic software set --ipv4tso = 0 -n vmnicX
esxcli network nic software set --ipv6tso = 0 -n vmnicX
Guest OS TSO settings in Linux OS can be changed by command …
ethtool -K ethX tso on/ off
Check LRO configuration
esxcli system settings advanced list -o /Net/TcpipDefLROEnabled
[dpasek@esx01:~] esxcli system settings advanced list -o /Net/TcpipDefLROEnabled
Path: /Net/TcpipDefLROEnabled
Type: integer
Int Value: 1
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: LRO enabled for TCP/IP
vmxnet settings can be validated by command …
esxcli system settings advanced list -o /Net/Vmxnet3HwLRO
[dpasek@esx01:~] esxcli system settings advanced list -o /Net/Vmxnet3HwLRO
Path: /Net/Vmxnet3HwLRO
Type: integer
Int Value: 1
Default Int Value: 1
Min Value: 0
Max Value: 1
String Value:
Default String Value:
Valid Characters:
Description: Whether to enable HW LRO on pkts going to a LPD capable vmxnet3
Set guest OS LRO settings in linux OS …
ethtool -k ethx lro on/ off.
Check Checksum offload configuration
ESX settings
esxcli network nic cso get
[dpasek@esx01:~] esxcli network nic cso get
NIC RX Checksum Offload TX Checksum Offload
------ ------------------- -------------------
vmnic0 on on
vmnic1 on on
vmnic2 on on
vmnic3 on on
vmnic4 on on
vmnic5 on on
vmnic6 on on
vmnic7 on on
[dpasek@czchoesint203:~]
The following command can be used for disabling CSO for a specific pNIC:
esxcli network nic cso set -n vmnicX
Check VXLAN offloading
vsish -e get /net/pNics/vmnic4/properties
dpasek@esx01:~] vsish -e get /net/pNics/vmnic4/properties
properties {
Driver Name:bnx2x
Driver Version:2.713.10.v60.4
Driver Firmware Version:bc 7.13.75
System Device Name:vmnic4
Module Interface Used By The Driver:vmklinux
Device Hardware Cap Supported:: 0x493c032b -> VMNET_CAP_SG VMNET_CAP_IP4_CSUM VMNET_CAP_HIGH_DMA VMNET_CAP_TSO VMNET_CAP_HW_TX_VLAN VMNET_CAP_HW_RX_VLAN VMNET_CAP_SG_SPAN_PAGES VMNET_CAP_IP6_CSUM VMNET_CAP_TSO6 VMNET_CAP_TSO256k VMNET_CAP_ENCAP VMNET_CAP_GENEVE_OFFLOAD VMNET_CAP_SCHED
Device Hardware Cap Activated:: 0x403c032b -> VMNET_CAP_SG VMNET_CAP_IP4_CSUM VMNET_CAP_HIGH_DMA VMNET_CAP_TSO VMNET_CAP_HW_TX_VLAN VMNET_CAP_HW_RX_VLAN VMNET_CAP_SG_SPAN_PAGES VMNET_CAP_IP6_CSUM VMNET_CAP_TSO6 VMNET_CAP_TSO256k VMNET_CAP_SCHED
Device Software Cap Activated:: 0x30800000 -> VMNET_CAP_RDONLY_INETHDRS VMNET_CAP_IP6_CSUM_EXT_HDRS VMNET_CAP_TSO6_EXT_HDRS
Device Software Assistance Activated:: 0 -> No matching defined enum value found.
PCI Segment:0
PCI Bus:5
PCI Slot:0
PCI Fn:0
Device NUMA Node:0
PCI Vendor:0x14e4
PCI Device ID:0x168e
Link Up:1
Operational Status:1
Administrative Status:1
Full Duplex:1
Auto Negotiation:0
Speed (Mb/s):10000
Uplink Port ID:0x0400000a
Flags:: 0x41e0e -> DEVICE_PRESENT DEVICE_OPENED DEVICE_EVENT_NOTIFIED DEVICE_SCHED_CONNECTED DEVICE_USE_RESPOOLS_CFG DEVICE_RESPOOLS_SCHED_ALLOWED DEVICE_RESPOOLS_SCHED_SUPPORTED DEIVCE_ASSOCIATED
Network Hint:
MAC address:9c:dc:71:db:d0:38
VLanHwTxAccel:1
VLanHwRxAccel:1
States:: 0xff -> DEVICE_PRESENT DEVICE_READY DEVICE_RUNNING DEVICE_QUEUE_OK DEVICE_LINK_OK DEVICE_PROMISC DEVICE_BROADCAST DEVICE_MULTICAST
Pseudo Device:0
Legacy vmklinux device:1
Respools sched allowed:1
Respools sched supported:1
}
VXLAN offload capability is called 'VMNET_CAP_ENCAP'. That's what you need to look for.
vsish -e get /net/pNics/vmnic4/properties | grep VMNET_CAP_ENCAP
[dpasek@esx01:~] vsish -e get /net/pNics/vmnic4/properties | grep VMNET_CAP_ENCAP
Device Hardware Cap Supported:: 0x493c032b -> VMNET_CAP_SG VMNET_CAP_IP4_CSUM VMNET_CAP_HIGH_DMA VMNET_CAP_TSO VMNET_CAP_HW_TX_VLAN VMNET_CAP_HW_RX_VLAN VMNET_CAP_SG_SPAN_PAGES VMNET_CAP_IP6_CSUM VMNET_CAP_TSO6 VMNET_CAP_TSO256k VMNET_CAP_ENCAP VMNET_CAP_GENEVE_OFFLOAD VMNET_CAP_SCHED
1.1.7 Check VMDq (NetQueue)
esxcli network nic queue filterclass list
This esxcli command shows information about the filters supported per vmnic and used by NetQueue.
[dpasek@esx01:~] esxcli network nic queue filterclass list
NIC MacOnly VlanOnly VlanMac Vxlan Geneve GenericEncap
------ ------- -------- ------- ----- ------ ------------
vmnic0 false false false false false false
vmnic1 false false false false false false
vmnic2 false false false false false false
vmnic3 false false false false false false
vmnic4 true false false false false false
vmnic5 true false false false false false
vmnic6 true false false false false false
vmnic7 true false false false false false
1.1.8 Dynamic NetQ
The following command will output the queues for all vmnics in your ESXi host.
esxcli network nic queue count get
[dpasek@esx01:~] esxcli network nic queue count get
NIC Tx netqueue count Rx netqueue count
------ ----------------- -----------------
vmnic0 1 1
vmnic1 1 1
vmnic2 1 1
vmnic3 1 1
vmnic4 8 5
vmnic5 8 5
vmnic6 8 5
vmnic7 8 5
It is possible to disable NetQueue on a ESXi host level using the following command:
esxcli system settings kernel set --setting =" netNetqueueEnabled" --value =" false"
VXLAN PERFORMANCE
RSS can help in case VXLAN is used because VXLAN traffic can be distributed among multiple hardware queues. NICs that offer RSS have a throughput around 9 Gbps but NICs that do not only have a throughput of around 6 Gbps. Therefore, the right choice of physical NIC is critical.
1 comment:
Aw, this was an incredibly good post. Spending some time and actual effort to make
a top notch article… but what can I say… I hesitate a whole lot and
don't seem to get nearly anything done.
Post a Comment