ESXi sched-stats command

Frank Denneman has shared on twitter very interesting ESXi command to show CPU scheduling statistics and information.

@FrankDenneman tweet

There are not so much information about this command so one have to rely on command help ...

[root@esx01:~] sched-stats -h
Usage:
-c   : use vsi-cache instead of live kernel
-t   : specify the output type from the following list
          :    vcpu-state-times
          :    vcpu-run-times
          :    vcpu-state-counts
          :    vcpu-run-states
          :    vcpu-alloc
          :    vcpu-migration-stats
          :    vcpu-load
          :    vcpu-relations
          :    vcpu-comminfo
          :    ncpus
          :    cpu
          :    pcpu-stats
          :    pcpu-load
          :    overhead-histo
          :    sys-service-stats
          :    run-state-histo
          :    wait-state-histo
          :    coSched-stats
          :    groups
          :    worldlet-state-times
          :    worldlet-state-counts
          :    worldlet-mig-state
          :    worldlet-load
          :    worldlet-relations
          :    worldlet-comminfo
          :    power-pstates
          :    power-cstates
          :    numa-clients
          :    numa-migration
          :    numa-cnode
          :    numa-pnode
          :    numa-global
-f          : ignore version check
-w          : only show stats of the specified world
-p          : only show stats of the specified pcpu
-m          : only show stats of the specified module
-r          : reset scheduler statistics
-s : 1 to enable advanced cpu sched stats gathering, 0 to disable.
-l , : comma separated list of ids to restrict the report to;
                (not supported by all reports)
-k          : check the correctness of scheduling stats
-v          : verbose
-h          : print friendly help message

Note:
Sched-stats reads the stats data from vmkernel for each vcpu one
by one via the VSI interface. Since the scheduling stats may
continue to change during the VSI calls, what's reported by
sched-stats is not a consistent snapshot of the kernel stats.
But the inconsistency is expected to be small.

The OVF package is invalid or could not be read.

I have just tried to deploy NSX Manager 6.2.4 virtual appliance downloaded from VMware site through WebClient. Following error message popup ...

"The OVF package is invalid or could not be read."

It sounds like corrupted file but it is very rare as it was successfully downloaded  directly from my.vmware.com.

I have double checked download and quickly realize what is wrong. OVF file should be XML file with references to vmdk files (disk images) but this OVF file was just a single large file. The single large file is typically OVA which is an archive of OVF.  OVA is nothing else then zipped TAR (aka tgz). I tried to rename file to .tgz, extract the file and voila ... there was real OVF and VMDK file. See. screenshot below

OVF directory

Deployment of real OVF (ovf descriptor + vmdk files) was successful so it is workaround #1.

Workaround #2  is just renaming file extension of downloaded OVF file (VMware-NSX-Manager-6.2.4-4292526.ovf) to OVA (VMware-NSX-Manager-6.2.4-4292526.ova) and deploy OVF/OVA again.

Hope this helps others struggling with virtual appliance deployment when OVF and OVA file extensions will be interchanged.

Note: I have double checked my.vmware.com and NSX Manager 6.2.4 is downloadable as file with .OVA extension so it was most probably already identified and repaired by VMware or something else weird happened during my download several days ago. 

PowerCLI script to report VMtools version(s)

UPDATE 2018-02-05: I have just been told about very nice PowerCLI command-lets allowing to manage VMtools.  Leveraging command-lets Get-VMToolsInfo, Get-VMToolsGuestInfo and Get-VMByToolsInfo is definitely the better way then my script below. All VMtools management command-lets are available on GitHub here
https://github.com/vmware/PowerCLI-Example-Scripts/blob/master/Modules/VMToolsManagement/VMToolsManagement.psm1

But if you are interested in my approach and simple script below I will be more than happy if it helps you.

###################################################################

This week I had a talk with one of my customers about VMtools versions and benefits of VMtools updates/upgrades. Few months ago I wrote another blog post discussing centralized version of VMtools repository (aka shared productLocker) so you can read it here.

Nevertheless, in large environments with multiple vCenters and thousands of VMs it is pretty handy to have single report with all VMtools versions to be able to asset and plan potential VMtools update/upgrade.

The problem with VMtools versions is that APIs reports just a single integer number of VMtools identification like 9354, 10246, etc. These numbers are also visible as Tools version on NGC/VI Client. However, vSphere admins are more familiar with human readable VMtools version also visible in guest OS which looks like 8.3.19, 9.4.6, 10.0.8, etc. For all VMtools version numbers and mapping you can check text file at https://packages.vmware.com/tools/versions

I have invested an hour or so to write simple PowerCLI script reporting VMtools identification and also human readable version for all VMs in particular vCenter or even across multiple vCenters. I have also included VM hardware version which can be handy information as well.

So initial PowerCLI code is below and the latest version will always appear on github here.

 ######################################################################################################################################  
 # Author: David Pasek  
 # E-mail: david.pasek@gmail.com  
 # Twitter: david_pasek  
 # Creation Date: 2016-11-25  
 #  
 # Use case:  
 #  Key use case of this script is to report VMtools from all VMs in vCenter  
 #  
 # Disclaimer:  
 #  Use it on your own risk. Author is not responsible for any impacts caused by this script.   
 ######################################################################################################################################  
 #   
 # CHANGE FOLLOWING VARIABLES BASED ON YOUR SPECIFIC REQUIREMENTS   
 # vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv  
 #  
 # Report type - table, grid, file, csv-file  
  $REPORT_TYPE = "csv-file"  
 # Report file name without file extension. Extension is automatically added. File is created in current working directory.   
  $REPORT_FILE_NAME = "report-vmtools"  
 ######################################################################################################################################  
 Clear-Host  
 # We need VMware PowerCLI snapin  
 $o = Add-PSSnapin VMware.VimAutomation.Core  
 $o = Set-PowerCLIConfiguration -InvalidCertificateAction Ignore -Confirm:$false  
 # Connect to vCenter  
 Write-Host "Connecting to vCenter ..."  
 $VC = Read-Host "Enter one vCentre Server or multiple vCenter servers delimited by comma."  
 Write-Host "Enter vCenter credentials ..."  
 $CRED = Get-Credential  
 Connect-VIServer -Server $VC -Credential $CRED -ErrorAction Stop | Out-Null  
 # Add new property (ToolsVersion) to VM  
 New-VIProperty -Name ToolsVersion -ObjectType VirtualMachine -ValueFromExtensionProperty 'Config.tools.ToolsVersion' -Force | Out-Null  
 # Initalize report  
 $Report = @()  
 foreach ($vm in Get-VM) {  
  # Numbers mapping is from https://packages.vmware.com/tools/versions  
  Switch ($vm.ToolsVersion) {  
      7302 {$GuestToolsVersion = "7.4.6"}  
      7303 {$GuestToolsVersion = "7.4.7"}  
   7304 {$GuestToolsVersion = "7.4.8"}  
   8192 {$GuestToolsVersion = "8.0.0"}  
   8194 {$GuestToolsVersion = "8.0.2"}  
   8195 {$GuestToolsVersion = "8.0.3"}  
   8196 {$GuestToolsVersion = "8.0.4"}  
   8197 {$GuestToolsVersion = "8.0.5"}  
   8198 {$GuestToolsVersion = "8.0.6"}  
   8199 {$GuestToolsVersion = "8.0.7"}  
   8290 {$GuestToolsVersion = "8.3.2"}  
   8295 {$GuestToolsVersion = "8.3.7"}  
   8300 {$GuestToolsVersion = "8.3.12"}  
   8305 {$GuestToolsVersion = "8.3.17"}  
   8306 {$GuestToolsVersion = "8.3.18"}  
   8307 {$GuestToolsVersion = "8.3.19"}  
   8384 {$GuestToolsVersion = "8.6.0"}  
   8389 {$GuestToolsVersion = "8.6.5"}  
   8394 {$GuestToolsVersion = "8.6.10"}  
   8395 {$GuestToolsVersion = "8.6.11"}  
   8396 {$GuestToolsVersion = "8.6.12"}  
   8397 {$GuestToolsVersion = "8.6.13"}  
   8398 {$GuestToolsVersion = "8.6.14"}  
   8399 {$GuestToolsVersion = "8.6.15"}  
   8400 {$GuestToolsVersion = "8.6.16"}  
   8401 {$GuestToolsVersion = "8.6.17"}  
   9216 {$GuestToolsVersion = "9.0.0"}  
   9217 {$GuestToolsVersion = "9.0.1"}  
   9221 {$GuestToolsVersion = "9.0.5"}  
   9226 {$GuestToolsVersion = "9.0.10"}  
   9227 {$GuestToolsVersion = "9.0.11"}  
   9228 {$GuestToolsVersion = "9.0.12"}  
   9229 {$GuestToolsVersion = "9.0.13"}  
   9231 {$GuestToolsVersion = "9.0.15"}  
   9232 {$GuestToolsVersion = "9.0.16"}  
   9233 {$GuestToolsVersion = "9.0.17"}  
   9344 {$GuestToolsVersion = "9.4.0"}  
   9349 {$GuestToolsVersion = "9.4.5"}  
   9350 {$GuestToolsVersion = "9.4.6"}  
   9354 {$GuestToolsVersion = "9.4.10"}  
   9355 {$GuestToolsVersion = "9.4.11"}  
   9356 {$GuestToolsVersion = "9.4.12"}  
   9359 {$GuestToolsVersion = "9.4.15"}  
   9536 {$GuestToolsVersion = "9.10.0"}  
   9537 {$GuestToolsVersion = "9.10.1"}  
   9541 {$GuestToolsVersion = "9.10.5"}  
   10240 {$GuestToolsVersion = "10.0.0"}  
   10245 {$GuestToolsVersion = "10.0.5"}  
   10246 {$GuestToolsVersion = "10.0.6"}  
   10247 {$GuestToolsVersion = "10.0.8"}  
   10249 {$GuestToolsVersion = "10.0.9"}  
   10252 {$GuestToolsVersion = "10.0.12"}  
   10272 {$GuestToolsVersion = "10.1.0"}  
   0   {$GuestToolsVersion = "Not installed"}  
   2147483647 {$GuestToolsVersion = "3rd party - guest managed"}  
      default {$GuestToolsVersion = "Unknown"}  
      }  
  $vminfo = New-Object -Type PSObject -Property @{  
             Name = $vm.Name  
     VMhardwareVersion = $vm.Version  
           ToolsVersion = $vm.ToolsVersion  
           GuestToolsVersion = $GuestToolsVersion  
      }  
  $Report += $vminfo  
 }  
 # Show report  
 Switch ($REPORT_TYPE) {  
      "grid"   { $Report | select Name,VMhardwareVersion,ToolsVersion,GuestToolsVersion | Out-GridView }  
   "file"   { $Report | select Name,VMhardwareVersion,ToolsVersion,GuestToolsVersion | Out-File -FilePath "$REPORT_FILE_NAME.txt" }  
   "csv-file" { $Report | select Name,VMhardwareVersion,ToolsVersion,GuestToolsVersion | export-csv "$REPORT_FILE_NAME.csv" }  
   default  { $Report | select Name,VMhardwareVersion,ToolsVersion,GuestToolsVersion | Format-Table }  
 }  
 Disconnect-VIserver -Server $VC -Force -Confirm:$false  

The script can be configured for different report types by changing variable $REPORT_TYPE

Following report types are supported:

• Standard PowerShell table report to output terminal (table)
• PowerShell GridView (grid)
• Standard text file including same content as table (file)
• Comma separated values file (csv-file)

Different sample reports are visible in screenshots below

CSV-FILE

FILE

GRID

Hope this helps some other folks in VMware community. Any feedback or feature requests are welcome.

Related links:

• VMware Tools 10 and "shared productLocker"
• VMware Tools 10.0.8 is now GA
• Powershell Script Upgrade VMware tools without a reboot - looks interesting but I did not test it so far.

vSphere 6.5 announced so what is coming?

vSphere 6.5 has been announced on VMworld 2016 so you can ask yourself what it brings and why consider upgrade or at least upgrade plan.

It is obvious and expected that almost all vSphere 6.5 scalability limits will be increased. Configuration maximums like hosts per vCenter, powered on VMs per vCenter, hosts per cluster, VMs per cluster, vCenters in linked mode, etc are expected to increase. Theses limits are no longer limits for me but if you need it, just wait for vSphere 6.5 GA and double check well known document vsphere-65-configuration-maximums.pdf

However, vSphere users are usually looking for new features. So here they are ...

vCenter features

• vCenter Server Appliance (aka VCSA) will be recommended as "First Choice" because the coolest new features are available just in VCSA. 
• Platform Service Controller (PSC) will have out-of-the-box high availability for VCSA using PSC's. You will be able to achieve PSC RTO lower then 5 minutes. << UPDATE: unfortunately, this feature was not released in vSphere 6.5 release so let's hope it will be released in future vSphere 6.5 Updates.  
• VCSA supports native High Availability support of vCenter service with RTO lower then 15 minutes.
• VCSA has embedded vCenter Appliance Monitoring and Management to gain visibility into VCSA performance and capacity management including embedded vPostgreSQL database service.
• VMware Update Manager (VUM) is be fully integrated into VCSA.
• File level vCenter Server Backup and Restore is complementary backup method to existing VDP image backup. It will be possible to restore vCenter file level backup to fresh VCSA.
• Content Libraries in vSphere 6.5 have additional features including the option to mount an ISO from a Content Library, update existing templates, and apply guest OS Customization Specifications during VM deployments. If Content Libraries reside on VCSA then you can also make use of vCenter HA, and native Backup and Restore, both new features to vSphere 6.5 mentioned above.

vSphere HA Cluster features

• vSphere HA cluster wide restart ordering ability with intra-app dependencies during failover. It allows multi-tier application consistency during VM fail-overs.  It is also known as "vSphere HA Orchestrated Restart" because you can create VM to VM dependencies which will force specified VMs to perform HA restarts before others. You can also choose in your vSphere HA settings, when the next VM should begin restarting. At the power-on initiated command, when resources allocated, VMware Tools heartbeats,  etc. You can also set additional timeouts and delays if needed.
• vSphere HA Admission Control - default Admission Control policy has changed from Slot Policy (Default until 6.5), to ‘Cluster Resource Percentage’. Any time you add or remove a host from the cluster, the failover capacity percentages will update, and the amount of resources required on each host will also be updated automatically.
• Proactive HA - it integrates with the Server vendor’s monitoring software, via a Web Client plugin, which will pass detailed server health status/alerts to DRS, and DRS will react based on the health state of the host’s hardware. Yes, even the name is "Proactive HA" it is DRS functionality. Confusing? The name was chosen because it has positive impact on availability.

vSphere DRS Cluster features

• Predictive DRS - it integrates DRS with vROps to provide placement and balancing decisions.
• Network-Aware DRS - DRS takes physical NIC utilization in to consideration. Once a target host has been chosen for placement/load-balancing, DRS will then check to see if that host’s network is saturated (default is 80% utilization of connected uplinks, but can be configured with ‘NetworkAwareDrsSaturationThresholdPercent’. If the host is considered saturated, it will use a different target host
• DRS Additional Option : VM Distribution - even distribution of VMs across cluster
• DRS Additional Option : Memory Metric for Load Balancing - usage of active versus consumed memory for DRS recommendations
• DRS Additional Option : CPU Over-Commitment - limit the number of vCPUs per pCPU in particular DRS cluster. Specific vCPU:pCPU ratio is set as advanced DRS option MaxVcpusPerClusterPct.

ESXi features and improvements

• ESXi is pretty stable and best in class hypervisor. However even in this component you can expect some improvements.  For example I/O improvements because of RDMA / PVRDMA. PVRDMA (para-virtualized RDMA) is industry first virtualized RDMA and it allows virtualization of applications which require ultra low latency. And it supports live vMotion which SR-IOV does not.
• ESXi core storage improvements - Support for 4K Native Drives in 512e mode, SE Sparse Default for VMFS, Automatic Space Reclamation, Support for 1024 devices and 4096 paths (versus 256 and 1024 in the previous versions)

vSphere management features

• Auto Deploy and Image Builder will be full integrated into WebClient and Host profiles will be improved to smoothly support auto deploy.
• vSphere Web Client usability and performance will be improved again. It is pretty important because C# client is not available for vSphere 6.5 so vSphere admins will rely on web client. HTML5-based vSphere Client should be included in 6.5 release.
• Content library improvements - mount ISO directly from content library, customization during VM deployment, improved scale and performance, high availability along with VCSA
• vSphere 6.5 introduces new REST-based APIs for VM Management

Storage related features

• VVOLs 2.0 will bring data protection and replication along with support for MSCS, Oracle RAC, NFS 4.1 and SMP-FT.
• VSAN - Virtual SAN iSCSI Service
• VSAN - 2-Node Direct Connect with witness Traffic Separation for ROBO
• VSAN - 512e drive support (still waiting for 4K native support)

Security related features

• VM Encryption will be new feature to protect your VM data with tenants keys. It enables encryption on a per VM as well as per VMDK basis. It can be integrated with 3rd party Key Management Servers (KMS).
• vSphere 6.5 also delivers enhanced audit-quality logging capabilities that provide more forensic information about user actions.

WebClient related features

• In vSphere 6.5, the vSphere Web Client will have no dependency on Client Integration Plug-in (as it exists before).  For the Use Window Session Authentication functionality, you will need the new slimmed down Enhanced Authentication Plug-in, but the other functions (File upload/download, Deploy OVA/OVF) are replicated without CIP.

Conclusion

vSphere 6 is already very mature virtualization platform but vSphere 6.5 brings some very interesting enterprise features if you ask me. The most interesting features for me personally are

• VCSA and PSC high availability
• VVOLs 2.0
• VM Encryption
• REST-based APIs for vSphere Management

but all other features are cool and very handy as well. 

It is very common practice to wait for Update 1 before upgrading production environments but our labs and test environments are good candidates for vSphere 6.5 release when available. I'm eagerly waiting for GA.

Other related blog posts and resources:

• VMware : What's New in vSphere 6.5
• VMware : VMware vSphere 6.5 Release Notes
• Charu Chaubal : Introducing vSphere 6.5
• Charu Chaubal : What’s New in vSphere 6.5: vCenter Server
• Charu Chaubal : What’s New in vSphere 6.5: Host & Resource Management and Operations
• Brian Graf : vSphere 6.5 – Everything You Need To Know
• Brian Graf : vSphere 6.5: vSphere HA What’s New – Part 1 – UI and also read parts 2,3,4
• Brian Graf : vSphere 6.5: DRS what’s new – Part 1 and also read parts 2 and 3 
• Eric Siebert : Quick summary of What’s New in vSphere 6.5
• Duncan Epping : vSphere 6.5 what’s new – VMFS 6 / Core Storage
• Duncan Epping : What is new for Virtual SAN 6.5?
• Duncan Epping : vSphere 6.5 what’s new – DRS
• Cormac Hogan : What’s new in vSphere 6.5 Core Storage
• Chris Wahl : Big Improvements to vSphere HA and DRS in 6.5 Release

VMware SIOC quick configuration in datacenter scale

I'm currently troubleshooting one weird high kernel latency (KAVG) issue and there is a suspicion that the issue can be somehow related to VMware SIOC which is widely use in customer's environment. To confirm or disprove the issue is really related to SIOC we can simply disable SIOC on all datastores and observe if it has positive impact on kernel latency.

Customer has lot of production datastores grouped in datastore clusters so following PowerCLI one liners can help with quick configuration and validation of SIOC settings across whole datacenter.

SIOC current state for all datastores in datastore clusters

 Get-DatastoreCluster | Get-Datastore | select-object name,type,StorageIOControlEnabled | Format-List -Property *  

Disable SIOC for all datastores in datastore clusters

 Set-Datastore (Get-DatastoreCluster | Get-Datastore) -StorageIOControlEnabled $false | select-object name,type,StorageIOControlEnabled

Enable SIOC for all datastores in datastore clusters

 Set-Datastore (Get-DatastoreCluster | Get-Datastore) -StorageIOControlEnabled $true | select-object name,type,StorageIOControlEnabled  

Thanks PowerCLI!

Metro Cluster High Availability or SRM Disaster Recovery?

Several years I continuously try to explain my customers that metro cluster is not disaster recovery. I have finally found some time and summarize my thoughts into slide deck which I published on SlideShare. I'm planning to present it at Czech VMUG local meeting on 6 December this year. More info about this particular Czech VMUG event is here.

The goal of my presentation is to explain the difference between multi site high availability (aka metro cluster) and disaster recovery. General concepts are same for any products but presentation is obviously more tailored for specific VMware products and technologies.

You can look at presentation here on SlideShare ...

Metro Cluster High Availability or SRM Disaster Recovery? from David Pasek

It would be great to see you at the event if you will be in the town. But in the meantime don't hesitate to write any comment or feedback here and we can have good discussion as there are still two months till the event.

BTW: Kudos to Stanislav Jurena @stan_jurena who already did several reviews and gave me some comments and feedback before first public release.  

Leveraging VMware LogInsight for VM hardware inventory

There is no doubt that VMware LogInsight is a must for any properly managed vSphere environment. I'm explaining LogInsight benefits to all my customers. The main use case for LogInsight is troubleshooting but there are infinite number of other use cases where LogInsight can help.

During last LogInsight presentation to one of my customers I have got an interesting question if LogInsight can be used for change management. Change management is very broad term but all VM hardware changes should be in log files therefore there is potential to prepare specific interactive analytics to track VM hardware changes.

You can very simply find reconfiguration tasks for particular VMs logged by vCenter or ESXi hosts.

In ESXi logs reconfiguration log entries looks like

 2016-10-01T21:01:27.126Z esx01.home.uw.cz Hostd: info hostd[31C40B70] [Originator@6876 sub=Vmsvc.vm:/vmfs/volumes/d22648f3-b9848863/test_cmdb/test_cmdb.vmx opID=7c3fe7f0-976b-4afa-a182-6d9f074a6446-45173-ngc-25-78-fc98 user=vpxuser:UW.CZ\cdave] Send config update invoked  

And in vCenter tasks it looks like

 2016-10-01 21:01:27.434 vc01.home.uw.cz vcenter-server: Reconfigured test_cmdb on esx01.home.uw.cz in home  

The problem is that even we know there were some reconfigurations, there are no VM reconfiguration details like CPU, RAM, disks and NICs. This is the current state in vSphere 6.0 U2 and it would be great if logging of such details would be improved in future vSphere version but this is what it is at the moment.

However, lot of VM hardware details should be in vmware.log which is located in home directory of each VM. The problem is that these logs are placed on VMFS filesystem and not forwarded to syslog so LogInsight cannot be used for log analysis.

Fortunately, there is the blog of the best VMware virtualization blogger William Lam and more specifically his blog post "A Hidden vSphere 5.1 Gem – Forwarding Virtual Machine Logs (vmware.log) to Syslog Part 1". William very nicely explains how to redirect VM logs also to syslog.

You have to change following two advanced settings for each Virtual Machine you would like to redirect VM hardware logs to syslog server.

vmx.log.destination=syslog-and-disk 

vmx.log.syslogID=[YOUR-VM-NAME-OR-OTHER-ID]

And when VM hardware logs are available in LogInsight you can prepare custom analytics to visualize VM hardware changes over time and put it on to dashboard for future use. You can see my dashboard on screenshot below.

VMware LogInsight dashboard - VM Hardware Inventory 

I have exported my VM Hardware Inventory dashboard as LogInsight content pack and shared it here on github so you can download and import it in to your LogInsight and test it.

Content Pack source code from GitHub is embedded below ...
[ Download Content Pack from GitHub ]

And as always, any comments are more then welcome.

How to change default policy of VMware Virtual Distributed Switch (VDS)

The main advantage of VMware virtual distributed switch (VDS) over VMware virtual standard switch (VSS) is the centralized configuration which is pushed to ESXi hosts. This centralized management provides uniform virtual switch configuration across all ESXi hosts in VDS scope. Virtual switch specific settings can be generally reconfigured for each port-group. In other words, port-group is a virtual switch construct which defines how particular group of ports will behave. Port-groups inherit default settings from virtual switch.

So far so good. However, VDS and SVS differ. In VDS, you can change default settings of the switch and all port-groups on that particular switch inherit these settings. However, this is not possible on VDS. Actually, it is not possible through vSphere Client. Through, GUI you can change just settings of particular port-group as you can see on screenshot below.

DVS port-group various settings.

It is very annoying and time consuming to change settings for each port-group manually. There is also very high probability of inconsistent settings across port-groups. To mitigate challenges described above, you can leverage PowerCLI and change default VDS settings (policy) as you can see in the example below. This will change default VDS settings and all new port-groups will be configured consistently and as required by particular design.

PowerCLI script below shows how to change default settings for traffic Shaping (in/out) and uplink teaming policy to LACP. This particular example shows how to configure LAG teaming with IP src/dst load balancing but any other teaming and load balancing can be used as well.

 $vDSName = “vDS01”  
 $vds = Get-VDSwitch $vDSName  
 $spec = New-Object VMware.Vim.DVSConfigSpec  
 $spec.configVersion = $vds.ExtensionData.Config.ConfigVersion  
 $spec.defaultPortConfig = New-Object VMware.Vim.VMwareDVSPortSetting

  
 $inShapingPolicy = New-Object VMware.Vim.DVSTrafficShapingPolicy  
 $inShapingPolicy.Enabled = New-Object VMware.Vim.BoolPolicy  
 $inShapingPolicy.AverageBandwidth = New-Object VMware.Vim.LongPolicy  
 $inShapingPolicy.PeakBandwidth = New-Object VMware.Vim.LongPolicy  
 $inShapingPolicy.BurstSize = New-Object VMware.Vim.LongPolicy  
 $inShapingPolicy.Enabled.Value = $true  
 $inShapingPolicy.AverageBandwidth.Value = 1000000000  
 $inShapingPolicy.PeakBandwidth.Value = 5000000000  
 $inShapingPolicy.BurstSize.Value = 19200000000  
 $inShapingPolicy.Inherited = $false  

 $outShapingPolicy = New-Object VMware.Vim.DVSTrafficShapingPolicy  
 $outShapingPolicy.Enabled = New-Object VMware.Vim.BoolPolicy  
 $outShapingPolicy.AverageBandwidth = New-Object VMware.Vim.LongPolicy  
 $outShapingPolicy.PeakBandwidth = New-Object VMware.Vim.LongPolicy  
 $outShapingPolicy.BurstSize = New-Object VMware.Vim.LongPolicy  
 $outShapingPolicy.Enabled.Value = $true  
 $outShapingPolicy.AverageBandwidth.Value = 1000000000  
 $outShapingPolicy.PeakBandwidth.Value = 5000000000  
 $outShapingPolicy.BurstSize.Value = 19200000000  
 $outShapingPolicy.Inherited = $false  

 $uplinkTeamingPolicy = New-Object VMware.Vim.VmwareUplinkPortTeamingPolicy  
 $uplinkTeamingPolicy.policy = New-Object VMware.Vim.StringPolicy  
 $uplinkTeamingPolicy.policy.inherited = $false  
 $uplinkTeamingPolicy.policy.value = “loadbalance_ip”  
 $uplinkTeamingPolicy.uplinkPortOrder = New-Object VMware.Vim.VMwareUplinkPortOrderPolicy  
 $uplinkTeamingPolicy.uplinkPortOrder.inherited = $false  
 $uplinkTeamingPolicy.uplinkPortOrder.activeUplinkPort = New-Object System.String[] (1) # designates the number of uplinks you will be specifying.  
 $uplinkTeamingPolicy.uplinkPortOrder.activeUplinkPort[0] = "LAG01"  

 $spec.DefaultPortConfig.InShapingPolicy = $inShapingPolicy  
 $spec.DefaultPortConfig.OutShapingPolicy = $outShapingPolicy  
 $spec.DefaultPortConfig.UplinkTeamingPolicy = $uplinkTeamingPolicy  

 $vds.ExtensionData.ReconfigureDvs_Task($spec)  

Another example reconfigures default DVS policy to VMware Load Based Teaming (aka LBT).

 Import-Module VMware.VimAutomation.vds  
 $vDSName = “test”   
 $vds = Get-VDSwitch $vDSName   
 $spec = New-Object VMwareåç.Vim.DVSConfigSpec   
 $spec.configVersion = $vds.ExtensionData.Config.ConfigVersion   
 $spec.defaultPortConfig = New-Object VMware.Vim.VMwareDVSPortSetting  
 $uplinkTeamingPolicy = New-Object VMware.Vim.VmwareUplinkPortTeamingPolicy   
 $uplinkTeamingPolicy.policy = New-Object VMware.Vim.StringPolicy   
 $uplinkTeamingPolicy.policy.value = “loadbalance_loadbased”   
 $spec.DefaultPortConfig.UplinkTeamingPolicy = $uplinkTeamingPolicy  
 $vds.ExtensionData.ReconfigureDvs_Task($spec)  

VDS supports following teaming policies (source: vSphere Documentation)

• failover_explicit
• loadbalance_ip
• loadbalance_loadbased
• loadbalance_srcid
• loadbalance_srcmac

Please, note that examples above have to be altered to fulfill your particular needs but now you should have a pretty good idea how to change the default DVS.

VMworld 2016 US sessions worth to watch

Here is the list of VMworld 2016 sessions from US event I watched or still have to watch during next days and weeks. After watching the session I do categorization and brief description of sessions. I'm also assigning category labels and technical level to each session.

Category labels:

• Strategy
• Architecture
• Operations
• High Level Product Overview
• Deep Dive Product Overview
• Technology Preview
• Idea for Improvement

Technical levels:

• Basic
• Middle
• Advanced

Note: If you will have trouble with re-play particular session from my links below go to OFFICIAL VMWORLD SITE and search for particular session by session code. You will need to register there but it still should be free of charge.

Already watched and categorized sessions

vCENTER

Session name: Journey to the vCenter Server Appliance: A Migration Story

Session code: INF9044
Speaker(s): Emad Younis

Category labels: Technology Preview

Technical level: Basic-Middle
Brief session summary: Introduction and demo of vCenter migration tool for easy migration from Windows based vCenter to vCenter Server Appliance (VCSA).

Session name: Automated Deployment and Configuration of the vCenter Server Appliance

Session code: INF8260
Speaker(s): William Lam, Alan Renouf

Category labels: Technology Preview

Technical level: Middle-Advanced
Brief session summary: William explains current automation possibilities of automated VCSA deployment and Alan presents what is coming soon. In nutshell, REST API is coming and it is really great if you ask me.

Session name: Extreme Performance Series: vCenter Performance Deep Dive

Session code: INF8108
Speaker(s): Ravi Soundararajan, Priya Sethuraman

Category labels: Deep Dive Product Overview

Technical level: Advanced
Brief session summary: At the begging there are presented some numbers explaining vCenter Server Appliance performance benefits over Windows based vCenter. At 00:07:00 vCenter deep dive starts with explanation of vCenter internal architecture (vsphere-client, sso, directory-service, vpxd, vpxd-svcs,  vmware-sps [aka storage profile services], perfcharts, eam [esx agent manager] ). The new info for me was that vCenter does not use inventory services anymore and since 2015 it is replaced by vpxd-svcs. Then vsphere-client plugin architecture is explained.

Single vCenter Internal Architecture

Later search use cases and consequences for different vCenter/SSO topologies are explained.

Search in Single Site multi vCenter Topology

Search in Multi Site multi vCenter Topology

In next section, other PSC and vCenter performance considerations are discussed. For PSC 2 vCPU and 4 GB is sufficient for any environment.

vCenter 6  has several hard limits good to know:

• 640 concurrent operations before incoming requests are queued
• 2000 concurrent sessions (user sessions + incoming requests +   remote console sessions)

ESXi host limits

• A host can perform up to 8 provisioning operations at once (provisioning = clone, vMotion, relocate, snapshot, etc.)
• If host is source and destination then host can only do 4 operations at once

Datastore limits

• A datastore can perform up to 128 vMotions at once
• A datastore can perform up to 8 Storage vMotions at once

In presentation is highlighted the fact that, vSphere 6 supports dedicated vmknic with special Provisioning TCP/IP Stack for Cold Migration, Cloning, and Snapshots. I did not know that! It is pretty cool management and potential performance optimization trick.

Dedicated vmknic configuration for provisioning

Higher latency between vCenter and ESXi does NOT have so big impact on vCenter operations (up to 100ms is good) but latency between vCenter and database is critical. Therefore embedded database is recommended and preferred configuration.

vCenter performance stats levels have big impact on vCenter performance. The biggest performance drop (4x) is between level 1 and level 2. Therefore, there is recommendation to keep stats on level 1 and use some external monitoring solution (like vROps) for historical performance monitoring and capacity planning or size your database performance appropriately.

If you have VCSA you can use tools like vimtop and cloudvm-ram-size for performance troubleshooting and tuning.

At the end of presentation, there are presented general conclusions. Generally, more hardware resources improves vCenter 6 performance significantly more then vCenter 5. Additional vCenter plugins and extensions can require more hardware resources.  Future releases of vCenter provide better scale and performance. Use the appliance (VCSA) to achieve better performance and simplified management.

vSPHERE HA

Session name: vSphere High Availability Best Practices and Tech Preview

Session code: INF8045
Speaker(s): Manoj Krishnan, Matthew Meyer

Category labels: Technology Preview

Technical level: Middle-Advanced
Brief session summary: At the beginning of presentation, vSphere HA basics (master/slaves, heart beating, host failure scenarios, partitioning, host isolation, ) are explained by Matthew and Manoj. After basic HA intro Matthew explains VM Component Protection (HA APD and PDL responses). Matthew explanation of PDL (00:12:20) is little bit strange because he is explaining it on example with LUN masking on fibre channel switches which is not the right case. LUN masking is done on storage arrays. In fibre channel switches we usually do zoning which is different technique. Matthew claims ESXi host will get PDL after wrong SAN fabric reconfiguration which is IMHO not the case because PDL has to be sent from storage through each particular path. Therefore if there is not path between ESXi host (initiator) and storage front-end port (target) ESXi cannot get PDL from storage array. Yes, you can get PDL from storage array when you will do LUN masking mistake on storage array. Slides are correct, just explanation example is not correct. The more interesting is Manoj explanation of APD timeouts (00:13:45).  APD VM recovery (VMCP APD response) is executed after ESXi APD Timeout (140 seconds) + VMCP APD Timeout (default is 3 minutes).  VMCP APD Timeout can be configured through vSphere Web Client. Another option is "Response for APD recovery after APD timeout". This response (reset or no) is done in case datastore appears back to ESXi host during "VMCP APD Timeout" period. In such case affected VMs are not restarted on different host in cluster but they are restarted on the same ESXi host. This prevents potential VM panic state inside VM guest OS'es because of storage unavailability. Next presented topic are networking and storage recommendations. Presentation continues with explanation of HA differences when using conventional storage or VSAN. When conventional storage is used then management network is used for network heart beating, any datastore connected to more then one host can be used for storage heartbeat and host is claimed as isolated when isolation address cannot be pinged. When VSAN is used, then VSAN network for network heart beating, storage heart beating can be used only if traditional datastores exists as well and host isolation is claimed when host cannot ping isolation addresses over VSAN network. At 00:28:43 presentation covers Admission Control details and HA integration with DRS. HA can ask DRS for cluster defragmentation in case there is not any available space for failed VMs. At 00:34:31 starts technology preview of new things VMware is working on - vSphere HA priorities (5 priorities instead of 3), HA orchestrated restart (dependencies among VMs ), FTT Admission Control (Percentage based but calculated automatically), Proactive HA (vMotion in case of hardware health degradation).

ESXi

Session name: vSphere 6.x Host Resource Deep Dive
Session code: INF8430
Speaker(s): Frank Denneman, Niels Hagoort
Category labels: Architecture
Technical level: Advanced
Brief session summary: Frank is presenting his findings about NUMA and implications for vSphere Architecture. Frank is also explaining the difference between RAM, local NVMe and external storage proxiimity and impact on access times. Niels is very nicely explaining how hardware features like VXLAN, RSS, VMDq, device drivers and drivers parameters impacts performance of virtual overlay networking (VXLAN).   

VIRTUAL SAN

Session name: VSAN Networking Deep Dive and Best Practices
Session code: STO8165R
Speaker(s): Ankur Pai, John Nicholson
Category labels: Architecture, Deep Dive Product Overview
Technical level: Advanced
Brief session summary: John starts presentation and he discuss what kind of virtual switch to use. You can use standard or distributed. Distributed virtual switch is highly recommended because of configuration consistency and advanced features like NIOC, LACP, etc. Next there is discussion about multicast used for VSAN. Multicast is used just for state and metadata. VSAN data traffic is transfer via unicast. IGMP snooping for L2 should be always configured to optimize metada traffic and PIM just in case the VSAN traffic should go over L3. VSAN uses two multicast addresses - 224.2.3.4 port 23451 (Agent Group Multicast Address) and 224.1.2.3 port 12345 (Master Group Multicast Address). These default multicast addresses has to be changed in case two VSAN clusters are running in the same broadcast domain. John is sharing esxcli commands how to change default multicast addresses but generally it is recommended to keep each VSAN cluster in dedicated non routable VLAN (single broadcast domain). John explains that VSAN is supported on both L2 and L3 topologies but keep it in L2 topology is significantly simpler. Physical network equipment, cabling and subscription ratios matters. John warns against CISCO FEX (port extender) and Blade Chassis switches where oversubscription is usually expected. Next topic is troubleshooting and presenters introduce VSAN health-check plugin which removes the need for cli (Ruby vSphere Console aka RVC) troubleshooting commands. VSAN does not need Jumbo Frames but you can use it if dictated by some other requirement. LLDP/CDP should be enabled for better visibility and easier troubleshooting.  After 24 minute John pass the presentation to Ankur. Ankur is presenting Virtual SAN stretched cluster networking considerations. He starts with high level overview of stretched VSAN topology and witness requirement in third location. L2 stretch network is recommended even L3 is supported but more complex. Cross site round trip has to be less then 5 ms and recommended bandwidth is 10Gbps. Witness can be connected over L3 with roundtrip less then 200ms and bandwidth 100Mbps (2Mbps per 1000 VSAN components). Communication between the witness and main site is L3 unicast - tcp 2233 (IO) and udp 23451 (cluster) both directions. Periodic heartbeating between the witness and the main site occurs every second. Failure is declared after 5 consecutive failures. Static routes have to be configured between data hosts and witness host because custom ESXi TCP/IP stack for VSAN is not supported at the moment. Ankurs presents three form factors of witness appliance (Large, Medium, Tiny). Later Ankurs explains bandwidth calculations for cross site interconnect and another calculation for witness network bandwidths requirements. At the end there is Q&A for about 5 minutes.   

Session name: Extreme Performance Series: Virtual SAN Performance Troubleshooting
Session code: STO8743
Speaker(s): Zach Shen, Ruijin Zhou
Category labels: Deep Dive Product Overview
Technical level: Advanced
Brief session summary: Zach and Ruijin are from VMware Performance Engineering team. Zach started presentation by quick VSAN architecture overview depicted below.

VSAN Architecture

VSAN Architecture - LSOM detail

At 7:10 Ruijin takes the stage and explains what VSAN Observer is. VSAN observer has to be initiated from RVC (Ruby vSphere Console) and then you can access live data via web URL: https://vc-hostname:8010. Ruijin goes through VSAN Observer GUI and explains what is what. At 25:25 presentation continues by troubleshooting tool #2, which is VSAN Performance Service.  VSAN Perfo rmance Service is integrated in vSphere Web Client. One can monitor common metrics (IOPS, latency, throughput) in 5 minutes granularity. Single plot range 24 hours and 90 days are available before roll over.

VIRTUAL VOLUMES (VVOLs)

Session name: Top 10 Thing You MUST Know Before Implementing Virtual Volumes
Session code: STO5888
Speaker(s): Eric Siebert
Category labels: Architecture, Deep Dive Product Overview
Technical level: Middle
Brief session summary: Eric Siebert from HPE explains VVOLs 1.0 architecture in pretty nice detail with some HP 3PAR specific implementation details which is very good for understanding how particular VVOL implementation fits into VMware VVOL general framework.

BUSINESS CRITICAL APPLICATIONS

Session name: Performance Tuning and Monitoring for Virtualized  Database Servers
Session code: VIRT7511
Speaker(s): David Klee, Thomas LaRock
Category labels: Architecture, Operations
Technical level: Middle
Brief session summary: TBD

vSPHERE CONFIGURATION MANAGEMENT

Session name: Enforcing a vSphere Cluster Design with PowerCLI Automation
Session code: INF8036
Speaker(s): Duncan Epping, Chris Wahl
Category labels: Idea for Improvement
Technical level: Middle to Advanced
Brief session summary: TBD


SITE RECOVERY MANAGER

Session name: SRM with NSX: Simplifying Disaster Recovery Operations & Reducing RTO
Session code: STO7977
Speaker(s): Rumen Colov, Stefan Tsonev
Category labels: High Level Product Overview
Technical level: Basic-Medium
Brief session summary: Rumen Colov is SRM Product Manager so his part of session is more about High Level introduction to SRM and NSX integration. Session really starts from 8:30 min. Before it is just introduction. Then Rumen explains main SRM use cases - Disaster Recovery, Datacenter Migration, Disaster Avoidance - and differentiation among them. The latest SRM versions supports zero-downtime VM mobility in Active/Active datacenter topology.  Then there is explained Storage Policy-based Management integration with SRM and NSX 6.2 interoperability with NSX stretched logical wire (NSX Universal Logical Switch) across two vCenters. Ruman also covers product licensing editions required for integration.  In 0:22:53 Rumen says that network inventory auto-mapping for NSX stretched networking works only with storage based replication (SBR) and not with host based replication (aka HBR or vSphere Replication). It seems to me the reason is because for SBR information about network connectivity are saved in VMX file which is replicated by storage but for HBR we would need manual inventory mappings. In 0:24:00 presentation is handed over to Stefan Tsonev (Director R&D).  Stefan shows several screenshots with SRM and NSX integration. It seem that Stefan is very deeply SRM oriented with basic NSX knowledge but it is very nice introduction into SRM and NSX integration basics. 

LOG INSIGHT

Session name: Insight into the World of Logs with VMware vRealize Log Insight
Session code: MGT7685R
Speaker(s): Iwan Rahabok, Karl Fultz, Manny Sidhu
Category labels: Operations, Deep Dive Product Overview
Technical level: Middle
Brief session summary: Very nice walkthrough VMware LogInsight use cases and real examples how to analyze logs. At the end of presentation are explained some architecture topologies for log management in financial institution.

Session name: vSphere Logs Grow Up! Tech Preview of Actionable Logging with vRealize Log Insight
Session code: INF8845
Speaker(s): Mike Foley, Antoan Arnaudov
Category labels: Technology Preview
Technical level: Basic
Brief session summary: In this session, Antoan and Mike will show you what is coming in next release of vSphere from logging perspective. We can expect significantly improved details in vCenter events which are sent to syslog server and if you use something like LogInsight you can do a lot of magics with these details. It can significantly helps with security audits, configuration management, etc. 

PSO and CoE

Session name: How to Relocate Your Physical Data Center Without Downtime in a Few Clicks, Thanks to Automation
Session code: INF8837
Speaker(s): Rene-Francois Mennecier (PSO), Constantin Natchev (CoE)
Category labels: Operations, Idea for Improvement
Technical level: Middle
Brief session summary: Rene-Francois and Constantin presents the datacentre migration project they have delivered to one of the largest banks in Europe. The project was fully automated by vRealize Orchestrator and enhanced storage vMotion (vMotion share nothing) was leveraged for workload migrations.

Still have to watch and categorize sessions below

STO7650 - Duncan Epping, Lee Dilworth : Software-Defined Storage at VMware Primer
Downtime in a Few Clicks, Thanks to Automation
INF8858 - vSphere Identity: Multifactor Authentication Deep Dive
INF9089 - Managing vCenter Server at Scale? Here's What You Need to Know
INF8553 - The Nuts and Bolts of vSphere Resource Management
INF8959 - Extreme Performance Series: DRS Performance Deep Dive—Bigger Clusters, Better Balancing, Lower Overhead
VIRT8530R - Deep Dive on pNUMA & vNUMA - Save Your SQL VMs from Certain DoomA!
INF8780R - vSphere Core 4 Performance Troubleshooting and Root Cause Analysis, Part 1: CPU and RAM
INF8701R - vSphere Core 4 Performance Troubleshooting and Root Cause Analysis, Part 2: Disk and Network
INF9205R - Troubleshooting vSphere 6 Made Easy: Expert Talk
INF8755R - Troubleshooting vSphere 6: Tips and Tricks for the Real World
INF8850 - vSphere Platform Security
VIRT8290R - Monster VMs (Database Virtualization) Doing IT Right
VIRT7654 - SQL Server on vSphere: A Panel with Some of the World's Most Renowned Experts
VIRT7621 - Virtualize Active Directory, the Right Way!
INF8469 - iSCSI/iSER: HW SAN Performance Over the Converged Data Center
SDDC7808-S - How I Learned to Stop Worrying and Love Consistency: Standardizing Datacenter Designs
INF9048 - An Architect's Guide to Designing Risk: The VCDX Methodology
INF8644 - Getting the Most out of vMotion: Architecture, Features, Performance and Debugging
INF8856 - vSphere Encryption Deep Dive: Technology Preview
INF8914 - Mastering the VM Tools Lifecycle in your vSphere Data Center
INF7825 - vSphere DRS and vRealize Operations: Better Together
INF7827 - vSphere DRS Deep Dive: Understanding the Best Practices, Advanced Concepts, and Future Direction of DRS
INF8275R - How to Manage Health, Performance, and Capacity of Your Virtualized Data Center Using vSphere with Operations Management
INF9047 - Managing vSphere 6.0 Deployments and Upgrades
STO7965 - VMware Site Recovery Manager: Technical Walkthrough and Best Practices
STO7973 - Architecting Site Recovery Manager to Meet Your Recovery Goals
STO8344 - SRM with vRA 7: Automating Disaster Recovery Operations
STO8246R - Virtual SAN Technical Deep Dive and What’s New
STO8750 - Troubleshooting Virtual SAN 6.2: Tips & Tricks for the Real World
STO7904 - Virtual SAN Management Current & Future
STO8179R - Understanding the Availability Features of Virtual SAN
STO7557 - Successful Virtual SAN 6 Stretched Clusters
STO7645r - Virtual Volumes Technical Deep Dive
INF8038r - Getting Started with PowerShell and PowerCLI for Your VMware Environment
NET7858R - Reference Design for SDDC with NSX and vSphere: Part 2
CNA9993-S - Cloud Native Applications, What it means & Why it matters
CNA7741 - From Zero to VMware Photon Platform
CNA7524 - Photon Platform, vSphere, or Both?
SDDC7502 - On the Front Line: A VCDX Perspective Working in VMware Global Support Services
VIRT9034 - Oracle Databases Licensing on a Hyper-Converged Platform
VIRT9009 - Licensing SQL Server and Oracle on vSphere
INF9083 - Ask the vCenter Server Experts Panel
INF9151 - Getting to Zero: Zero Downtime, Zero Data loss with vSphere Fault Tolerance
INF9119 - How To manage PSC like Batman
INF8225 - The vCenter Server and PSC guide to the Gallaxy
INF9144 - An Overview of vCenter Server Appliance management interface
INF9128 - Day 2 operations: A vCenter Server Administrator's Diary
INF8172 - vSphere Client Roadmap: Host Client, HTML 5 Client, and web Client
INF8631 - VMware Certificate Management for Mere Mortals
INF8465 - Power Management's Impact on Performance
INF8089 - vSphere Compute and Memory
VIRT7598 - Monster VM Database Performance

All VMworld US 2016 Breakout Sessions

All VMworld US 2016 session are listed here.

If you cannot play session from list above go to OFFICIAL SITE of VMWORLD 2016 US and search for particular session. You will need to register there but it is free of charge.

And here is another repository of all videos for VMworld 2016 US.