The ZALMAN ZM VE200 SATA hard disk caddy with DVD/HDD/FDD emulation

I have just bought external USB drive with DVD emulation from ISO file. That's should be pretty handy for OS installs. I'm looking forward for first ESXi installation directly from ISO file.

Here is nice and useful tutorial how to use it.

Announcing the VMware Learning Zone

As a VMware vExpert I had a chance to use beta access to VMware Learning Zone. I blogged about my experience here. VMware Learning Zone has been officially announced today.

VMware Learning Zone is a new subscription-based service that gives you a full year of unlimited, 24/7 access to official VMware video-based training. Top VMware experts and instructors discuss solutions, provide tips and give advice on a variety of advanced topics. Your VMware Learning Zone subscription gives you:

• Easy to consume training on the latest products and technologies
• Powerful search functionality to find the answers you need fast
• Content that delivers exactly the knowledge you need
• Mobile access for on the go viewing
• Much more

Learn more here.

ISCSI Best Practices

General ISCSI Best Practices

• Separate VLAN for iSCSI traffic. 
• Two separate networks or VLANs for multipath iSCSI. 
• Two separate IP subnets for the separate networks or VLANs in multipath iSCSI. 
• Gigabit (or better) Full Duplex connectivity between storage targets (storage front-end ports) and all storage initiators (server ports) 
• Auto-Negotiate for all switches that will correctly negotiate Full Duplex 
• Full Duplex hard set for all iSCSI ports for switches that do not correctly negotiate 
• Bi-Directional Flow Control enabled for all Switch Ports that servers or controllers are using for iSCSI traffic. 
• Bi-Directional Flow Control enabled for all ports that handle iSCSI traffic. This includes all devices between two sites that are used for replication. 
• Unicast storm control disabled on every switch that handles iSCSI traffic. 
• Multicast disabled at the switch level for any iSCSI VLANs. 
• Broadcast disabled at the switch level for any iSCSI VLANs. 
• Routing disabled between the regular network and iSCSI VLANs. 
• Do not use Spanning Tree (STP or RSTP) on ports that connect directly to end nodes (the server or storage iSCSI ports.) If you must use it, enable the Cisco PortFast option or equivalent on these ports so that they are configured as edge ports. 
• Ensure that any switches used for iSCSI are of a non-blocking design. 
• When deciding which switches to use, remember that you are running SCSI traffic over it. Be sure to use a quality managed enterprise-class networking equipment. It is not recommended to use SBHO (small business/home office) class equipment outside of lab/test environments. 

For Jumbo Frame Support

• Some switches have limited buffer sizes and can only support Flow Control or Jumbo Frames, but not both at the same time. It is strongly recommended to choose Flow Control. 
• All devices connected through iSCSI need to support 9k jumbo frames. 
• All devices used to connect iSCSI devices need to support it. 
• This means every switch, router, WAN Accelerator, and any other network device that will handle iSCSI traffic needs to support 9k Jumbo Frames. 
• If it is not 100% positive that every device in the iSCSI network supports 9k Jumbo Frames, then NOT turn on Jumbo Frames. 
• Because devices on both sides (server and SAN) need Jumbo Frames enabled, change disable to enable Jumbo Frames is recommended during a maintenance window. If servers have it enabled first, the Storage System will not understand their packets. If Storage System enables it first, servers will not understand its packets.

VMware ESXi iSCSI tunning

• Disabling "TCP Delayed ACK" (esxcli iscsi adapter param set -A vmhba33 -k DelayedAck -v 0 - command not tested)
• Adjust iSCSI Login Timeout (esxcli iscsi adapter param set -A vmhba33 -k LoginTimeout -v 60)
• Disable large receive offload (LRO) (esxcli system settings advanced set -o /Net/TcpipDefLROEnabled 0 or esxcfg-advcfg -s 0 /Net/TcpipDefLROEnabled)
• Set up Jumbo Frames is configured end to end (esxcli network vswitch standard set -m 9000 -v vSwitch2 and  esxcli network ip interface set -m 9000 -i vmk1)
• Set up appropriate multi pathing based on iSCSI storage system
• FlowControl is enabled on ESXi by default. To display FlowControl settings use ethtool --show-pause vmnic0 or esxcli system module parameters list --module e1000 | grep "FlowControl"

If you know about some other best practice, tuning setting or recommendation don't hesitate to leave a comment below this blog post. 

 

Related documents:

[1] VMware. Best Practices For Running VMware vSphere On iSCSI. In: core.vmware.com, URL: https://core.vmware.com/resource/best-practices-running-vmware-vsphere-iscsi

How to quickly get changed ESXi advanced settings?

Below is esxcli command to list ESXi Advanced Settings that have changed from the system defaults:

esxcli system settings advanced list -d

Here is real example form my ESXi host in lab ...

~ # esxcli system settings advanced list -d
   Path: /UserVars/SuppressShellWarning
   Type: integer
   Int Value: 1
   Default Int Value: 0
   Min Value: 0
   Max Value: 1
   String Value:
   Default String Value:
   Valid Characters:
   Description: Don't show warning for enabled local and remote shell access

You can see that I'm suppressing Shell Warning because I really want to have SSH enabled and running on my lab ESXi all the time.

If you want list kernel settings there is another command

esxcli system settings kernel list

and you can also used option -d to get just changed settings from default.

Virtualisation Design & Project Framework

Gareth Hogarth wrote excellent high level plan (aka methodology, framework) how to properly deliver virtualization project as a turn key solution. I used very similar approach and not only for virtualization project but to any IT project where I have a role of Leading Architect. I have never written a blog post about this particular topic because it is usually internal intellectual property  of any consulting organization. So if you have never seen any similar methodology look at Gareth's post to get an idea of project phases and overall project process. It is good to note that all these methodologies are just frameworks and frameworks are usually good starting points which doesn't stop you to improve it to fulfill all specific project requirements and make your project successful.

40Gb over existing LC fiber optics

Do you know DELL has QSFP+ LM4 transciever allowing 40Gb traffic up to 160m on LC OM4 MMF (multi mode fiber) or up to 2km on LC SMF (single mode fiber)?

Use Case:  

This optic has an LC connection and is ideal for customers who want to use existing LC fiber.  It can be used for 40GbE traffic up to 160m on MultiMode Fiber OR 2km on Single Mode fiber.

Specification

Periferal Type: DELL QSFP+ LM4

Connection: LC Connection, Dulplex Multi-Mode Fiber or Dulpex Single-Mode Fiber
Max Distance: 140m OM3 or 160m OM4 MMF, 2km SMF
Transmitter Output Wavelength (nm): 1270 to 1330
Transmit Output Power (dBm): -7.0 to 3.5 [avg power per lane]
Receive Input Power (dBm): -10.0 to 3.5 [avg power per lane]
Temperature: 0 to 70C
Power:  3.5W max

Based on wavelength range 1270 to 1330 I assume 40Gb is achieved as 4 x 10Gb leveraging wavelength-division multiplexing (CWDM) on following wave lengths:

• 1270 nm
• 1290 nm
• 1310 nm
• 1330 nm

ESXi Network Troubleshooting

Introduction

As VMware vExpert, I had a chance and privilege to use VMware Learning Zone. There are excellent training videos. Today I would like to blog about useful commands trained on video training “Network Troubleshooting at the ESXi Command Line”.  If you ask me I have to say that Vmware Learning Zone has very valuable content and it comes really handy during real troubleshooting. 

UPDATE 2020-10-17: I have just found the blog post "ESXi Network Troubleshooting Tools" containing a lot of useful tools and insights.

NIC Adapters Information

To see Network Interface Cards Information you can run following command

~ # /usr/lib/vmware/vm-support/bin/nicinfo.sh | more

Network Interface Cards Information.

Name    PCI Device     Driver  Link  Speed  Duplex  MAC Address        MTU   Description

----------------------------------------------------------------------------------------

vmnic0  0000:001:00.0  bnx2    Up     1000  Full    14:fe:b5:7d:8d:05  1500  Broadcom Corporation Broadcom NetXtreme II BCM5709S 1000Base-SX

vmnic1  0000:001:00.1  bnx2    Up     1000  Full    14:fe:b5:7d:8d:07  1500  Broadcom Corporation Broadcom NetXtreme II BCM5709S 1000Base-SX

vmnic2  0000:002:00.0  bnx2    Up     1000  Full    14:fe:b5:7d:8d:6d  1500  Broadcom Corporation Broadcom NetXtreme II BCM5709S 1000Base-SX

vmnic3  0000:002:00.1  bnx2    Up     1000  Full    14:fe:b5:7d:8d:6f  1500  Broadcom Corporation Broadcom NetXtreme II BCM5709S 1000Base-SX

NIC:  vmnic0

NICInfo:

   Advertised Auto Negotiation: true

   Advertised Link Modes: 1000baseT/Full, 2500baseT/Full

   Auto Negotiation: true

   Cable Type: FIBRE

   Current Message Level: -1

   Driver Info:

      NICDriverInfo:

         Bus Info: 0000:01:00.0

         Driver: bnx2

         Firmware Version: 7.8.53 bc 7.4.0 NCSI 2.0.13

         Version: 2.2.3t.v55.7

   Link Detected: true

   Link Status: Up

   Name: vmnic0

   PHY Address: 2

   Pause Autonegotiate: false

   Pause RX: true

   Pause TX: true

   Supported Ports: TP, FIBRE

   Supports Auto Negotiation: true

   Supports Pause: true

   Supports Wakeon: true

   Transceiver: internal

   Wakeon: MagicPacket(tm)

Ring parameters for vmnic0:

Pre-set maximums:

RX:             4080

RX Mini:        0

RX Jumbo:       16320

TX:             255

Current hardware settings:

RX:             255

RX Mini:        0

RX Jumbo:       0

TX:             255

…

Output above is sniped just for vmnic0. You can see useful information like PCI Device ID, Driver, Link Status, Speed, Duplex and MTU for each vmnic.

It also shows detail driver information, FlowControl (Pause Frame) status, cable type. etc.

To find particular vmnic PCI Vendor ID's use command vmkchdev

~ # vmkchdev -l | grep vmnic0

0000:01:00.0 14e4:163a 1028:02dc vmkernel vmnic0

PCI Slot: 0000:01:00.0

VID (Vendor ID): 14e4

DID (Device ID): 163a

SVID (Sub-Vendor ID): 1028

SSID (Sub-Device ID): 02dc

You can use PCI devices Vendor ID’s  to find the latest drivers at VMware Compatibility Guide (http://www.vmware.com/go/hcl/).

Below is another command how to find full details of all PCI devices.

esxcli hardware pci list

If you are interested just for particular vmnic PCI details command below can be used.

~ # esxcli hardware pci list | grep -B 6 -A 29 vmnic0

000:001:00.0

   Address: 000:001:00.0

   Segment: 0x0000

   Bus: 0x01

   Slot: 0x00

   Function: 0x00

   VMkernel Name: vmnic0

   Vendor Name: Broadcom Corporation

   Device Name: Broadcom NetXtreme II BCM5709S 1000Base-SX

   Configured Owner: Unknown

   Current Owner: VMkernel

   Vendor ID: 0x14e4

   Device ID: 0x163a

   SubVendor ID: 0x1028

   SubDevice ID: 0x02dc

   Device Class: 0x0200

   Device Class Name: Ethernet controller

   Programming Interface: 0x00

   Revision ID: 0x20

   Interrupt Line: 0x0f

   IRQ: 15

   Interrupt Vector: 0x2b

   PCI Pin: 0x75

   Spawned Bus: 0x00

   Flags: 0x0201

   Module ID: 4125

   Module Name: bnx2

   Chassis: 0

   Physical Slot: 0

   Slot Description: Embedded NIC 1

   Passthru Capable: true

   Parent Device: PCI 0:0:1:0

   Dependent Device: PCI 0:0:1:0

   Reset Method: Link reset

   FPT Sharable: true

Note: same command can be used for HBA cards by substituting vmnic0 by vmhba0

VLAN Sniffing

The commands below enable VLAN statistics collection on particular vmnic which can be shown and used for troubleshooting.  

esxcli network nic vlan stats set --enabled=true -n vmnic0

~ # esxcli network nic vlan stats get -n vmnic0

VLAN 0

   Packets received: 22

   Packets sent: 0

VLAN 22

   Packets received: 21

   Packets sent: 10

VLAN 201

   Packets received: 28

   Packets sent: 0

VLAN 202

   Packets received: 28

   Packets sent: 0

VLAN 204

   Packets received: 5

   Packets sent: 0

VLAN 205

   Packets received: 5

   Packets sent: 0

Don’t forget to disable VLAN statistics after troubleshooting.

esxcli network nic vlan stats set --enabled=false -n vmnic0

VMkernel Arp Cache

To work with ESXi ARP cache you can use command

esxcli network ip neighbor  

Below is example how to list ARP entries …

~ # esxcli network ip neighbor list

Neighbor   Mac Address        Vmknic   Expiry  State  Type

---------  -----------------  ------  -------  -----  -------

10.2.22.1  5c:26:0a:ae:5a:c6  vmk0    933 sec         Unknown

You can see there just default gateway 10.2.22.1

Let’s ping some other device in the same broadcast domain and look at ARP entries again.

~ # ping 10.2.22.51

PING 10.2.22.51 (10.2.22.51): 56 data bytes

64 bytes from 10.2.22.51: icmp_seq=0 ttl=128 time=0.802 ms

~ # esxcli network ip neighbor list

Neighbor    Mac Address        Vmknic    Expiry  State  Type

----------  -----------------  ------  --------  -----  -------

10.2.22.51  00:0c:29:4a:5b:ba  vmk0    1195 sec         Unknown

10.2.22.1   5c:26:0a:ae:5a:c6  vmk0     878 sec         Unknown

Now you can see  entry for device 10.2.22.51 in ARP table as well. Below is another command to remove ARP entry from ARP table.

~ # esxcli network ip neighbor remove -v 4 -a 10.2.22.51

… and let’s check if ARP entry has been removed.

~ # esxcli network ip neighbor list

Neighbor   Mac Address        Vmknic   Expiry  State  Type

---------  -----------------  ------  -------  -----  -------

10.2.22.1  5c:26:0a:ae:5a:c6  vmk0    817 sec         Unknown

Note: ESXi ARP timeout is 1200 second therefore remove command can be handy in some situations.

VMkernel Routing

Since vSphere 5.1 it is possible to have more than one networking stack. Normally you work with default networking stack.

To show ESXi routing table you can use command

esxcli network ip route ipv4 list  

~ # esxcli network ip route ipv4 list

Network    Netmask        Gateway    Interface  Source

---------  -------------  ---------  ---------  ------

default    0.0.0.0        10.2.22.1  vmk0       MANUAL

10.2.22.0  255.255.255.0  0.0.0.0    vmk0       MANUAL

You can see default gateway 10.2.22.1 used for default networking stack.

Command esxcli network ip connection list shows all IP network connections from and to ESXi host.

~ # esxcli network ip connection list

Proto  Recv Q  Send Q  Local Address                    Foreign Address     State        World ID  CC Algo  World Name

-----  ------  ------  -------------------------------  ------------------  -----------  --------  -------  ---------------

tcp         0       0  127.0.0.1:8307                   127.0.0.1:54854     ESTABLISHED     34376  newreno  hostd-worker

tcp         0       0  127.0.0.1:54854                  127.0.0.1:8307      ESTABLISHED    570032  newreno  rhttpproxy-work

tcp         0       0  127.0.0.1:443                    127.0.0.1:54632     ESTABLISHED    570032  newreno  rhttpproxy-work

tcp         0       0  127.0.0.1:54632                  127.0.0.1:443       ESTABLISHED   1495503  newreno  python

tcp         0       0  127.0.0.1:8307                   127.0.0.1:61173     ESTABLISHED     34806  newreno  hostd-worker

tcp         0       0  127.0.0.1:61173                  127.0.0.1:8307      ESTABLISHED    570032  newreno  rhttpproxy-work

tcp         0       0  127.0.0.1:80                     127.0.0.1:60974     ESTABLISHED     34267  newreno  rhttpproxy-work

tcp         0       0  127.0.0.1:60974                  127.0.0.1:80        ESTABLISHED     35402  newreno  sfcb-vmware_bas

tcp         0       0  10.2.22.101:80                   10.44.44.110:50351  TIME_WAIT           0

tcp         0       0  127.0.0.1:5988                   127.0.0.1:14341     FIN_WAIT_2      35127  newreno  sfcb-HTTP-Daemo

tcp         0       0  127.0.0.1:14341                  127.0.0.1:5988      CLOSE_WAIT    1473527  newreno  hostd-worker

tcp         0       0  127.0.0.1:8307                   127.0.0.1:45011     ESTABLISHED     34806  newreno  hostd-worker

tcp         0       0  127.0.0.1:45011                  127.0.0.1:8307      ESTABLISHED    570032  newreno  rhttpproxy-work

NetCat

Netcat program (nc) is available on ESXi and it can test TCP connectivity to some IP target.

~ # nc -v 10.2.22.100 80

Connection to 10.2.22.100 80 port [tcp/http] succeeded!

TraceNet

Tracenet is very handy program available in ESXi to identify also latencies inside vmkernel IP stack.

~ # tracenet 10.2.22.51

Using interface vmk0 ...

Time         0.068 0.023 0.019 ms

Location:    ESXi-Firewall

Time         0.070 0.025 0.020 ms

Location:    VLAN_InputProcessor@#

Time         0.073 0.027 0.022 ms

Location:    vSwitch0: port 0x2000004

Time         0.089 0.030 0.024 ms

Location:    VLAN_OutputProcessor@#

Time         0.090 0.031 0.025 ms

Location:    DC01

Endpoint:       10.2.22.51

Roundtrip Time: 0.417 0.195 0.196 ms

Dropped packets

In this section are commands to verify dropped packets on different places of VMkernel Ip stack.

Command net-stats –l list all devices (Clients – nic-ports,vmk-ports, vm-ports) connected to VMware switch. You can simply identify to which vSwitch port number (PortNum) is device connected.

~ # net-stats -l

PortNum          Type SubType SwitchName       MACAddress         ClientName

33554434            4       0 vSwitch0         14:fe:b5:7d:8d:05  vmnic0

33554436            3       0 vSwitch0         14:fe:b5:7d:8d:05  vmk0

33554437            5       9 vSwitch0         00:0c:29:4a:5b:ba  DC01

33554438            5       9 vSwitch0         00:0c:29:f0:df:4c  VC01

Note: SubType is VM Hardware Version

vSwitch port numbers are important for following commands.

Command esxcli network port stats get –p shows statistics for particular vSwitch port.

~ # esxcli network port stats get -p 33554434

Packet statistics for port 33554434

   Packets received: 2346445

   Packets sent: 5853

   Bytes received: 295800113

   Bytes sent: 1225842

   Broadcast packets received: 1440669

   Broadcast packets sent: 336

   Multicast packets received: 896958

   Multicast packets sent: 120

   Unicast packets received: 8818

   Unicast packets sent: 5397

   Receive packets dropped: 0

   Transmit packets dropped: 0

You can also show filter statistics for ESXi firewall by command esxcli network port filter stats get –p 33554436

~ # esxcli network port filter stats get -p 33554436

Filter statistics for ESXi-Firewall

   Filter direction: Receive

   Packets in: 5801

   Packets out: 5660

   Packets dropped: 141

   Packets filtered: 150

   Packets faulted: 0

   Packets queued: 0

   Packets injected: 0

   Packet errors: 0

Filter statistics for ESXi-Firewall

   Filter direction: Transmit

   Packets in: 4893

   Packets out: 4887

   Packets dropped: 6

   Packets filtered: 6

   Packets faulted: 0

   Packets queued: 0

   Packets injected: 0

   Packet errors: 0

To show physical NIC statistics you have to use command esxcli network nic stats get –n vmnic0

~ # esxcli network nic stats get -n vmnic0

NIC statistics for vmnic0

   Packets received: 2350559

   Packets sent: 8083

   Bytes received: 312690659

   Bytes sent: 5791889

   Receive packets dropped: 0

   Transmit packets dropped: 0

   Total receive errors: 0

   Receive length errors: 0

   Receive over errors: 0

   Receive CRC errors: 0

   Receive frame errors: 0

   Receive FIFO errors: 0

   Receive missed errors: 0

   Total transmit errors: 0

   Transmit aborted errors: 0

   Transmit carrier errors: 0

   Transmit FIFO errors: 0

   Transmit heartbeat errors: 0

   Transmit window errors: 0

Packet Capture

If you want to do deeper network troubleshooting you can do packet capturing on ESXi host. You have two tools available for packet capturing

•  tcpdump-uw (example: tcpdump-uw –I vmk0 –s0 –C100M –W 10 –w /var/tmp/test.pcap)
• pktcap-uw

pktcap Examples:

• pktcap-uw –uplink vmnicX –capture UplinkRcv
• pktcap-uw –uplink vmnicX –capture UplinkSnd
•    you can filter for icmp –proto 0x01 or beacon probes –ethtype 0x8922

Other example based on [SOURCE] https://kb.fortinet.com/kb/documentLink.do?externalID=FD47845

In case of connectivity issue between a VM and other VM/s it is worth sniffing traffic on the hypervisor side in order to isolate the issue.

In order to sniff traffic on ESXi server, it is necessary to perform the steps below:

- Enable ssh access on ESXi.

- Ssh to ESXi.

- Run in CLI net-stats -l | grep <VM name> in order to find virtual switchport of the VM.

- In vSphere 6.5 or earlier it is necessary to specify direction of sniffing (either input or output).

- Switchport number for particular VM can be found using net-stats command.

- 'O' defines path where pcap file will be created and specify file name.

- dir specify direction (either input or output):

    pktcap-uw --switchport 123 -o /tmp/in.pcap --dir input

    pktcap-uw --switchport 123 -o /tmp/out.pcap --dir output

- In vSphere 6.7 or later it is possible to sniff traffic in both directions by setting --dir 2:

    pktcap-uw --switchport 123 -o /tmp/both.pcap --dir 2

- Run Ctrl-C in CLI order to stop sniffing.

- Download created pcap file/s over ssh from ESXi.