This will be a relatively short blog post. The whole industry is aware of Spectre/Meltdown security vulnerabilities. I wrote recently the blog post "VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown)".
A few days ago NCF announced additional CPU vulnerabilities (CVE-2018-3639 and CVE-2018-3640) and VMware released yesterday the official response in following documents:
What does it mean for IT infrastructure practitioners / VMware vSphere administrators?
Well, actually nothing new. The update process is the same as for previous Spectre/Meltdown remediations. VMware vSphere administrator must apply following update procedure.
Conclusion
So even all patches exist and the update process is well known, it is definitely not a simple project, especially in large organizations where collaboration among multiple teams and departments is required.
It is obvious that remediations have some negative performance impact on applications, however, all these remediations can be disabled in operating systems, therefore hardware and vSphere layers can be patched and application owner can decide between security and performance. However, please note that even disabling security remediation has a positive impact on the performance, the final performance can still be worse than the original performance on unpatched systems.
A few days ago NCF announced additional CPU vulnerabilities (CVE-2018-3639 and CVE-2018-3640) and VMware released yesterday the official response in following documents:
- VMware Security Advisory VMSA-2018-0012
- Source of Truth KB: https://kb.vmware.com/kb/54951
- VMware Response to Speculative Execution security issues, CVE-2018-3639 and CVE-2018-3640: https://kb.vmware.com/kb/54951
- Patching/Upgrade Guide: https://kb.vmware.com/kb/55111
- Performance Info: https://kb.vmware.com/kb/55210
What does it mean for IT infrastructure practitioners / VMware vSphere administrators?
Well, actually nothing new. The update process is the same as for previous Spectre/Meltdown remediations. VMware vSphere administrator must apply following update procedure.
- Update vCenter to apply patches to EVC. Note: Patches add new CPU features (IBRS, IBPB, STIBP) into existing EVC baselines.
- (optional but recommended) Validate that EVC is enabled on vSphere Clusters. Note: Without EVC you can experience vMotion issues of newly Powered On VMs within vSphere Cluster.
- Update the latest BIOS with patched CPU microcode. Note: VMware delivers ESXi patch with updated CPU microcode but CPU microcode from hardware vendor is recommended.
- Apply appropriate ESXi security patches
- Validate VM hardware is at least in version 9 (PCID enabled) but for better performance VM hardware 11 is recommended because Virtual Hardware Version 11 supports INVPCID.
- Apply all applicable security patches for your Guest OS which have been made available from the OS vendor.
- Power Off / Power On VMs (VM restart is not sufficient)
- It has some hardly predictable negative performance impact which is workload specific, therefore application owners have to evaluate the specific impact on their application.
- IT management is afraid of the unpredictable performance impact, lack of computing resources and tremendous impact on capacity planning.
- If VM hardware upgrade is required, a maintenance window with application owners. Note: Virtual hardware upgrade can bring a certain risk because you are actually changing motherboard and chipset.
- Power Off / Power On VMs is required, therefore maintenance window must be planned by or with application owners.
Conclusion
So even all patches exist and the update process is well known, it is definitely not a simple project, especially in large organizations where collaboration among multiple teams and departments is required.
It is obvious that remediations have some negative performance impact on applications, however, all these remediations can be disabled in operating systems, therefore hardware and vSphere layers can be patched and application owner can decide between security and performance. However, please note that even disabling security remediation has a positive impact on the performance, the final performance can still be worse than the original performance on unpatched systems.
