Since January 3, 2017, the whole IT industry is mitigating the impact of SPECTRE and MELTDOWN vulnerabilities and administrators are updating their infrastructures.
Three different CVEs have been identified related to the media described issues:
VMware infrastructure is, obviously, impacted as well. It is very important to patch the whole infrastructure stack. All subsystems in the infrastructure stack described below must be patched or upgraded for effective mitigation. The virtual infrastructure stack is composed of
HYPERVISOR
Hypervisor remediation can be classified into the two following categories:
For Hypervisor-Assisted Guest Remediation, all relevant patches exist and are described in security advisory VMSA-2018-0004.
PERFORMANCE IMPACT
HOW TO UPDATE?
An ESXi host that is running both a patched CPU microcode and a patched hypervisor will see new CPU features that were not previously available. These new features will be exposed to all Virtual Hardware Version 9+ VMs that are powered-on by that host. Because these VMs now see additional CPU features, vMotion to ESXi without the microcode or hypervisor patches applied will be prevented. There are three new CPU Features
OTHER VENDORS RESPONSES AND MITIGATIONS
Intel official response is available here
https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr
Intel Analysis of Speculative Execution Side Channels is available here https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/Intel-Analysis-of-Speculative-Execution-Side-Channels.pdf
DELL official response: Meltdown and Spectre Vulnerabilities
DELL KB article with technical information: http://www.dell.com/support/article/SLN308588
HP response I have found so far: Resources to help mitigate Speculative Execution vulnerability in Intel and other processors
HP Support Customer Bulletin: a00039267en_us
Hewlett Packard Enterprise Product Security Vulnerability Alerts: http://h22208.www2.hpe.com/eginfolib/securityalerts/SCAM/Side_Channel_Analysis_Method.html
RedHat: Kernel Side-Channel Attacks - CVE-2017-5754 CVE-2017-5753 CVE-2017-5715
FreeBSD: How to patch Meltdown and Spectre vulnerabilities on FreeBSD
If you know links to other or better server vendors responses, please leave it in comments.
Hope this article is helpful. And as always, if you see some wrong information or you just have a question or some other experience, do not hesitate to write the comment. I will try to update this article in case something new is available.
Three different CVEs have been identified related to the media described issues:
- CVE-2017-5753 (Spectre - Variant 1) - Branch target injection
- CVE-2017-5715 (Spectre - Variant 2) - Bounds check bypass
- CVE-2017-5754 (Meltdown - Variant 3) - Rogue data cache load
VMware infrastructure is, obviously, impacted as well. It is very important to patch the whole infrastructure stack. All subsystems in the infrastructure stack described below must be patched or upgraded for effective mitigation. The virtual infrastructure stack is composed of
- Operating systems (OS)
- Virtual Machines / Virtual Appliances
- Hypervisors
- CPU microcode (usually part of BIOS)
GUEST OS
For guest OS you should obtain patches from your OS software vendors. However, guest operating systems are using new CPU instructions (capabilities IBRS, IBPB, STIBP) to mitigate Spectre/Meltdown vulnerabilities. These capabilities are exposed from physical CPU microcode, therefore, patches from top to down (OS-HYPERVISOR-CPU microcode) is necessary for successful vulnerability mitigation. To expose those new CPU capabilities to guest OS, Virtual Machine hardware 9 and newer must be used otherwise those new capabilities are masked by VM hardware and Guest OS kernel would not be able to use it.
For guest OS you should obtain patches from your OS software vendors. However, guest operating systems are using new CPU instructions (capabilities IBRS, IBPB, STIBP) to mitigate Spectre/Meltdown vulnerabilities. These capabilities are exposed from physical CPU microcode, therefore, patches from top to down (OS-HYPERVISOR-CPU microcode) is necessary for successful vulnerability mitigation. To expose those new CPU capabilities to guest OS, Virtual Machine hardware 9 and newer must be used otherwise those new capabilities are masked by VM hardware and Guest OS kernel would not be able to use it.
VIRTUAL APPLIANCES
Another consideration should be given to VMware Virtual Appliances. VMware officially released a new Knowledge Base article regarding the investigation of the impact on the VMware Virtual Appliances (KB 52264) and there is also VMware Security Advisory VMSA-2018-0007.
HYPERVISOR
Hypervisor remediation can be classified into the two following categories:
- Hypervisor-Specific Remediation (documented in VMSA-2018-0002) - Result of exploitation may allow for information disclosure from one Virtual Machine to another Virtual Machine that is running on the same host.
- Hypervisor-Assisted Guest Remediation (documented in VMSA-2018-0004) - This issue may allow for information disclosure between processes within the VM.
For Hypervisor-Specific remediation, all relevant patches already exist and are described in security advisory VMSA-2018-0002. VMware recommends applying described patches.
For Hypervisor-Assisted Guest Remediation, all relevant patches exist and are described in security advisory VMSA-2018-0004.
Rogue Data Cache Load or “Meltdown” (CVE-2017-5754) does not affect ESXi, Workstation, and Fusion because ESXi does not run untrusted user mode code, and Workstation and Fusion rely on the protection that the underlying operating system provides.
PERFORMANCE IMPACT
The virtualization specific performance impact of these mitigations should be negligible but performance impact of workloads inside guest OS is expected but workload dependent. There are rumors it might be somewhere between 10% and 30%.
The VMware performance team evaluated the performance costs of the Meltdown/Spectre mitigations for vSphere and official messaging of performance impact is published in VMware KB 52337.
Customers having vROps 6.6.0 and above can leverage specific dashboards to assess performance impact of Spectre & Meltdown patches using vRealize Operations. For more information read this blog post.
The VMware performance team evaluated the performance costs of the Meltdown/Spectre mitigations for vSphere and official messaging of performance impact is published in VMware KB 52337.
Customers having vROps 6.6.0 and above can leverage specific dashboards to assess performance impact of Spectre & Meltdown patches using vRealize Operations. For more information read this blog post.
HOW TO UPDATE?
Please note, the update order is important.
To enable hardware support for branch target mitigation in vSphere, you should apply these steps, in the order shown below:
- Ensure you have EVC enabled on vSphere clusters
- For vCenter 6.x, in case you have external PSC, update all external PSCs to the latest patches of PSC 5.5 – 6.5
- Update to the latest patches of vCenter 5.5 – 6.5
- Apply the latest ESXi 5.5 – 6.5 patches
- Apply the CPU Microcode by BIOS or alternatively by VMware patch mentioned in VMSA-2018-0004 where VMware distributes CPU Microcode. Please note, CPU Microcode distributed by BIOS is highly recommended.
An ESXi host that is running both a patched CPU microcode and a patched hypervisor will see new CPU features that were not previously available. These new features will be exposed to all Virtual Hardware Version 9+ VMs that are powered-on by that host. Because these VMs now see additional CPU features, vMotion to ESXi without the microcode or hypervisor patches applied will be prevented. There are three new CPU Features
- IBRS
- IBPB
- STIBP
These three new CPU capabilities are not known to existing EVC baselines, therefore it seems to ESXi hosts as newer CPU model and prevents vMotion migration. EVC baselines are defined in vCenter and that's the reason why vCenter patch must be applied before ESXi patch and CPU microcode update. Patched EVC on the latest vCenter Server is leveraged to temporarily mask new CPU Features until all hosts in the cluster have new CPU Microcode. This allows vMotion of VMs from unpatched to patched ESXi hosts.
HOW TO VERIFY YOUR UPDATE WAS SUCCESSFUL AND YOU ARE NOT VULNERABLE?
You should verify it in different layers.
Hypervisor level
- You can verify it manually in vmware.log of particular VM as is described in KB 52085
- William Lam has published blog post "Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI" with PowerCLI script to test vulnerability status on particular ESXi host or vSphere Cluster.
- Vikas Shitole has published the blog post "PowerCLI script : How to confirm a vCenter server, ESXi VMware hypervisor & microcode patches are applied?: Spectre vulnerability" with PowerCLI automation script to test all ESXi host within vCenter.
If you want to know if you have Intel microcode updated you can run following shell command on your ESXi host
if [ `vsish -e get /hardware/msr/pcpu/0/addr/0x00000048 2&>1 > /dev/null ;echo $?` -eq 0 ]; then echo -e "\nIntel Security Microcode Updated\n";else echo -e "\nIntel Security Microcode NOT Updated\n";fi
On Intel machine with microcode update, you get the output "Intel Security Microcode Updated".
On Intel machine with microcode update, you get the output "Intel Security Microcode NOT Updated".
If you want to do the same for AMD microcode following command should work
On Intel machine with microcode update, you get the output "Intel Security Microcode NOT Updated".
If you want to do the same for AMD microcode following command should work
if [ $(($((`vsish -e get /hardware/cpu/cpuList/0 | grep -i EBX | head -5 | tail -1 | awk -F: '{print $2}'` & 0x00001000)) >> 12)) -eq 1 ]; then echo -e "\nAMD Security Microcode Updated\n";else echo -e "\nAMD Security Microcode NOT Updated\n";fiGuest OS - Linux
- Very nice blog post with the shell script is here “How to check Linux for Spectre and Meltdown vulnerability". Please note, that if you run the shell script inside guest OS on top of update ESXi host, the script will report that CPU microcode is updated even the physical CPU microcode is vulnerable. However, the information if Guest OS is patched is still valuable.
Guest OS - Microsoft Windows
- Microsoft has published following article "Windows Client Guidance for IT Pros to protect against speculative execution side-channel vulnerabilities" with PowerShell module SpeculationControl where it the command-let Get-SpeculationControlSettings.
OTHER VENDORS RESPONSES AND MITIGATIONS
Intel official response is available here
https://security-center.intel.com/advisory.aspx?intelid=INTEL-SA-00088&languageid=en-fr
Intel Analysis of Speculative Execution Side Channels is available here https://newsroom.intel.com/wp-content/uploads/sites/11/2018/01/Intel-Analysis-of-Speculative-Execution-Side-Channels.pdf
DELL official response: Meltdown and Spectre Vulnerabilities
DELL KB article with technical information: http://www.dell.com/support/article/SLN308588
HP response I have found so far: Resources to help mitigate Speculative Execution vulnerability in Intel and other processors
HP Support Customer Bulletin: a00039267en_us
Hewlett Packard Enterprise Product Security Vulnerability Alerts: http://h22208.www2.hpe.com/eginfolib/securityalerts/SCAM/Side_Channel_Analysis_Method.html
RedHat: Kernel Side-Channel Attacks - CVE-2017-5754 CVE-2017-5753 CVE-2017-5715
FreeBSD: How to patch Meltdown and Spectre vulnerabilities on FreeBSD
If you know links to other or better server vendors responses, please leave it in comments.
Hope this article is helpful. And as always, if you see some wrong information or you just have a question or some other experience, do not hesitate to write the comment. I will try to update this article in case something new is available.
Sources:
- VMSA-2018-0002 - VMware ESXi, Workstation and Fusion updates address side-channel analysis due to speculative execution - https://www.vmware.com/security/advisories/VMSA-2018-0002.10.html
- VMSA-2018-0004 - VMware vSphere, Workstation and Fusion updates add Hypervisor-Assisted Guest Remediation for speculative execution issue - https://www.vmware.com/security/advisories/VMSA-2018-0004.10.html
- VMSA-2018-007 - VMware Virtual Appliance updates address side-channel analysis due to speculative execution - https://www.vmware.com/us/security/advisories/VMSA-2018-0007.10.html
- VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52245) - https://kb.vmware.com/kb/52245
- Hypervisor-Assisted Guest Mitigation for branch target injection (52085) - https://kb.vmware.com/s/article/52085
- VMware Virtual Appliances and CVE-2017-5753, CVE-2017-5715 (Spectre), CVE-2017-5754 (Meltdown) (52264) - https://kb.vmware.com/s/article/52264
- VMware Performance Impact for CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) (52337) - https://kb.vmware.com/kb/52337
- Intel Sightings in ESXi Bundled Microcode Patches for VMSA-2018-0004 (52345) - https://kb.vmware.com/s/article/52345
- Assess performance impact of Spectre & Meltdown patches using vRealize Operations Manager - https://blogs.vmware.com/management/2018/01/assess-performance-impact-spectre-meltdown-patches-using-vrealize-operations-manager.html
- WHY RASPBERRY PI ISN’T VULNERABLE TO SPECTRE OR MELTDOWN
- Controlling the Performance Impact of Microcode and Security Patches for CVE-2017-5754 CVE-2017-5715 and CVE-2017-5753 using Red Hat Enterprise Linux Tunables
2 comments:
The link in "Check VMware KB 52345." is actually just "http://www.processors impacted by intel microcode sightings are.com"
Thanks Daren. The link should be fixed now.
Post a Comment