ISCSI Best Practices

General ISCSI Best Practices

• Separate VLAN for iSCSI traffic. 
• Two separate networks or VLANs for multipath iSCSI. 
• Two separate IP subnets for the separate networks or VLANs in multipath iSCSI. 
• Gigabit (or better) Full Duplex connectivity between storage targets (storage front-end ports) and all storage initiators (server ports) 
• Auto-Negotiate for all switches that will correctly negotiate Full Duplex 
• Full Duplex hard set for all iSCSI ports for switches that do not correctly negotiate 
• Bi-Directional Flow Control enabled for all Switch Ports that servers or controllers are using for iSCSI traffic. 
• Bi-Directional Flow Control enabled for all ports that handle iSCSI traffic. This includes all devices between two sites that are used for replication. 
• Unicast storm control disabled on every switch that handles iSCSI traffic. 
• Multicast disabled at the switch level for any iSCSI VLANs. 
• Broadcast disabled at the switch level for any iSCSI VLANs. 
• Routing disabled between the regular network and iSCSI VLANs. 
• Do not use Spanning Tree (STP or RSTP) on ports that connect directly to end nodes (the server or storage iSCSI ports.) If you must use it, enable the Cisco PortFast option or equivalent on these ports so that they are configured as edge ports. 
• Ensure that any switches used for iSCSI are of a non-blocking design. 
• When deciding which switches to use, remember that you are running SCSI traffic over it. Be sure to use a quality managed enterprise-class networking equipment. It is not recommended to use SBHO (small business/home office) class equipment outside of lab/test environments. 

For Jumbo Frame Support

• Some switches have limited buffer sizes and can only support Flow Control or Jumbo Frames, but not both at the same time. It is strongly recommended to choose Flow Control. 
• All devices connected through iSCSI need to support 9k jumbo frames. 
• All devices used to connect iSCSI devices need to support it. 
• This means every switch, router, WAN Accelerator, and any other network device that will handle iSCSI traffic needs to support 9k Jumbo Frames. 
• If it is not 100% positive that every device in the iSCSI network supports 9k Jumbo Frames, then NOT turn on Jumbo Frames. 
• Because devices on both sides (server and SAN) need Jumbo Frames enabled, change disable to enable Jumbo Frames is recommended during a maintenance window. If servers have it enabled first, the Storage System will not understand their packets. If Storage System enables it first, servers will not understand its packets.

VMware ESXi iSCSI tunning

• Disabling "TCP Delayed ACK" (esxcli iscsi adapter param set -A vmhba33 -k DelayedAck -v 0 - command not tested)
• Adjust iSCSI Login Timeout (esxcli iscsi adapter param set -A vmhba33 -k LoginTimeout -v 60)
• Disable large receive offload (LRO) (esxcli system settings advanced set -o /Net/TcpipDefLROEnabled 0 or esxcfg-advcfg -s 0 /Net/TcpipDefLROEnabled)
• Set up Jumbo Frames is configured end to end (esxcli network vswitch standard set -m 9000 -v vSwitch2 and  esxcli network ip interface set -m 9000 -i vmk1)
• Set up appropriate multi pathing based on iSCSI storage system
• FlowControl is enabled on ESXi by default. To display FlowControl settings use ethtool --show-pause vmnic0 or esxcli system module parameters list --module e1000 | grep "FlowControl"

If you know about some other best practice, tuning setting or recommendation don't hesitate to leave a comment below this blog post. 

 

Related documents:

[1] VMware. Best Practices For Running VMware vSphere On iSCSI. In: core.vmware.com, URL: https://core.vmware.com/resource/best-practices-running-vmware-vsphere-iscsi