How to remove VMFS datastore and reuse local disks for vSAN

I'm upgrading the hardware in my home lab to to leverage vSAN. I have 4x Dell PowerEdge R620, each having 2x 500 GB SATA disks but no SSD for cache disks. The cost is always the constraint for any home lab but I've recently found the M.2 NVMe PCI-e adapter for M.2 NVMe SSD in my local computer shop. The total cost of 1x M.2 NVMe PCI-e adapter + 1x M.2 NVMe 512 GB SSD is just $100.

Such hardware upgrade for only $400 would allow me to have vSAN datastore with almost 4 TB raw space because I would have 4-node HYBRID vSAN where each node has 1x NVMe disk as a cache disk and 2x 500 GB SATA disks as capacity disks. The vSAN raw space will be probably 4TB - 10% after disks format but 3.6 TB raw space and 2 TB usable space after decreasing 25% slack space and an additional 25% for RAID 5 protection is still a pretty good deal.

The issue I'm describing in this blog post usually happens in environments where you use local disks as backing storage for local VMFS datastores. Local VMFS datastores work perfectly fine until you would like to remove VMFS datastore and reuse these local disks for example for vSAN. That was exactly my case in my home lab where I have four ESXi hosts each with 2x 500 GB SATA disks having local VMFS datastore on two disks in each ESXi host.

When I tried to remove local datastore (ESX22-Local-SATA-01) it fails with the following error message:

The resource 'Datastore Name: ESX22-Local-SATA-01 VMFS uuid: 5c969e10-1d37088c-3a57-90b11c142bbc' is in use.

Why is the datastore in use? Well, it can be from several reasons. All these reasons are very well described back in 2014 on Virten blog post "Cannot remove datastore * because file system is busy."

Here is Virten's LUN removal checklist:

• No virtual machine, template, snapshot or CD/DVD image resides on the datastore
• The datastore is not part of a Datastore Cluster
• Storage I/O Control is disabled for the datastore
• The datastore is not used for vSphere HA heartbeat
• The LUN is not used as an RDM
• The Datastore is not used as a scratch location
• The Datastore is not used as VMkernel Dump file location (/vmkdump/)
• The Datastore is not used as active vsantraced location (/vsantrace/)
• The Datastore is not used as Scratch location
• The Datastore is not used to store VM swap files.

The root cause of my issue was the usage of "scratch location". I was blogging about this topic back in 2012 here "Set the Scratch Partition from the vSphere Client".

When you have another datastore available on ESXi host, the solution is very easy. You can simply change "the scratch location". It is much more tricky, in case you do not have any alternative datastore. Fortunately enough, in my home lab, I have three Synology NAS boxes leveraged as shared datastores over NFS and iSCSI, so the fix was quick. If you would need to do it for more then few ESXi hosts, PowerCLI script can be handy.

In case, you do not have any other datastore and you need to remove VMFS datastore you have two options

1. Reboot the computer to some alternative system (linux, FreeBSD, etc.) and destroy MBR or GPT partition on a particular disk device. Something like gpart destroy -F /dev/ad0 in FreeBSD.
2.  Physically remove the disk from your computer and when you boot it up VMware should automatically default back to temp scratch location (assuming you don't have any other available datastores on that box). You can then reinsert the disk and correctly remove Datastore from the ESXi host.

VMware NSX-T and vIDM licensing question

Time to time customers are asking the following NSX-T & vIDM question ...

Do I need license for VMware Identity Manager? The aim for using VIDM is RBAC for NSX-T.

There is the community discussion in the VMTN at https://communities.vmware.com/thread/616803 with the correct answer ...

You may use vIDM for free with NSX-T if you bought NSX. No license required. vIDM may not be deployed for other solutions in this case.

However, it would be nice to know some official VMware documents supporting the answer.

Here it is ...
https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/downloads/eula/product-guide/2019/vmware-product-guide-19-may.pdf

On page 16 is written ...

VMware Identity Manager feature.A license to use VMware NSX Data Center (any edition) or NSX Cloud (any edition) includes an entitlement to use the VMware Identity Manager feature, but only for the following functionalities:

• directory integration functionality of VMware Identity Manager to authenticate users in a user directory such as Microsoft Active Directory or LDAP

• conditional access policy

• single-sign-on integration functionality with third party Identity providers to allow third party identity providers’ users to single-sign-on into NSX

• two-factor authentication solution through integration with third party systems. VMware Verify, VMware’s multi-factor authentication solution, received as part of VMware Identity Manager, may not be used as part of NSX, and single-sign-on functionality to access VMware products that support single-sign-on capabilities.

Restriction on Features.Each edition of the Software (Standard, Professional, Advanced, Enterprise or Enterprise Plus) offers different features and functionality. You may access and use only the features and functionality that
correspond with the edition of Software You have validly licensed. Those features and functionality are detailed at the following links: http://kb.vmware.com/kb/2145269 and https://kb.vmware.com/s/article/52462. You agree to not access or use any feature or functionality of an edition of the Software to which You are not entitled based on the edition of Software You have validly licensed. 

Hope this helps.

vSAN Encryption and Core Dumps

I was on three days vSAN training (VMware vSAN: Deploy and Manage [V6.7]) which is very useful even for someone like me, who is observing vSAN since the beginning (2013) and did a lot of self-study and home lab practicing during the last year or so. The trainer (Jiri Viktorin) is very knowledgable and ready to answer any question. To be honest, I personally prefer class trainings over on-line trainings as personal contact and student discussions are sometimes even more valuable than the official training content. And that was exactly the case in this particular training.

One training attendee asked the trainer if DEK keys (Data Encryption Key) used for vSAN Encryption are included in core dumps and shared with VMware Support (GSS) in case of core dump analysis. This would be a significant security risk especially in environments with strict security requirements.

In the beginning, I've also assumed that DEK's are most probably included in a memory dump because they are in ESXi host RAM, right? However, any assumption should be validated and clarified if possible, therefore I did some quick research in product documentation and also on google but I did not find any relevant info. Fortunately enough, as a VMware employee, I can use internal slack channels and ask our VMware SME's for answers to these deep-dive questions. The well-known authority in terms of vSphere security is Mike Foley and he replied back in a few minutes. Thanks, Mike! Really appreciated.

Mike's answer was even better what I was expecting.

VMware Encryption technology is smart enough to not include DEK keys (Data Encryption Key) into core dumps. Keys live in the key cache. The key cache is not included in a core dump. 

That's great! ESXi core dumps can be shared with VMware GSS without security risk of unveiling Data Encryption Keys. That's an awesome message for customer's security officers and it shows that VMware takes vSphere security seriously.

Hope this information will help other folks in the VMware community.