January 3, 2018 8:42 pm
A number of serious security vulnerabilities affecting many CPU architectures were disclosed this week (CVE-2017-5753, CVE-2017-5715, and CVE-2017-5754) by Google’s Project Zero team and others. Our team is working with vendors and our own engineers to determine the implications for our platform, but the expectation is that a fleet-wide reboot will be necessary to protect against these issues.
While we work through our response plan, please understand that due to the nature and seriousness of these issues a rapid response may be necessary. As always, we will provide as much advance notice as possible. We will communicate directly to you, with scheduling information, if a reboot of your Linodes is necessary.
We will continue to keep you updated here as more information becomes available.
Information regarding these vulnerabilities can be found on the following sites:
Update: January 4, 2018
We are continuing to investigate this issue and wanted to provide a brief update as to where we are:
- We are postponing all unrelated maintenances to focus our efforts and resources on mitigating this issue.
- As discussed by the Scaleway team earlier today, due to the incomplete information provided by hardware manufacturers, we joined forces with other potentially impacted cloud hosting providers including Scaleway, Packet, and OVH. We’ve created a dedicated communications channel to share information and work together to address the Meltdown & Spectre vulnerabilities.
- We are continuing internal evaluation and testing of mitigations.
- We have discussions set for tomorrow for a deeper dive with the hardware providers.
We will continue providing updates here as appropriate.
Update: January 5, 2018
We’re continuing to make progress, and wanted to share the latest with you:
- The latest stable and longterm Linux kernels were released today with the KPTI / Meltdown patches in place. As such, we have made the 4.14.12 kernel available to you and have set it as the latest. If you are leveraging a Linode kernel, upon your next reboot your Linode will be upgraded to this version. This doesn’t fully mitigate you from the Meltdown and Spectre vulnerabilities, but provides us a good foundation to work with while planning for full remediation.
- We’ve had planning sessions with our hardware providers and have been working on implementation plans for kernel, hypervisor, and firmware updates. All of these will be required to get us into a remediated state, but not all of these are available.
We are not expecting much movement on this over the weekend while we wait on external dependencies, but will certainly provide updates here if there is. If not, more updates will be shared here on Monday next week.
Update: January 8, 2018
We are continuing to make progress with our internal testing, but are still waiting for microcode updates from our hardware providers. Both the microcode update and the kernel update are required in order to ensure there is proper mitigation to the three variants of Meltdown and Spectre.
Update: January 9, 2018
We spent today preparing the plan for deploying the Meltdown mitigations across Linode’s fleet. Over the course of the next day we will be implementing fixes to a subset of the fleet, monitoring for impact, and then continuing the rollout to the rest. The Meltdown mitigation requires reboots of our physical hardware which will reboot the Linodes hosted on them. A subset of Linodes in the Tokyo 1, Frankfurt, and Singapore data centers will be rebooted as part of this initial group. For those affected, you will receive a support ticket and email with scheduling information.
The reboots for this week address Meltdown only. We have testing and planning occurring in parallel to address Spectre. Additional reboots over the coming weeks will be required to properly mitigate all Spectre variants.
Update: January 10, 2018
The rollout to the subset of our fleet has gone well so far for the Meltdown mitigation. We are continuing with this plan and will be conducting reboots over the course of the next several days for the rest of the fleet. Customers affected will receive support ticket and emails with the reboot window for their Linodes with 24 hours of minimum notice.
- Due to the ongoing nature of this issue, the following status page has been created.
- We are posting a document soon that better talks about Meltdown and Spectre to show what it means to you and what you can do to prepare for it on your Linodes. We’ll share a link to this on an upcoming blog post.
Update: January 11, 2018
The mitigation process is continuing forward for Meltdown and we’re making progress each day across the fleet. There is a new guide available with more information on these vulnerabilities and how you can protect your Linode: What You Need to do to Mitigate Meltdown and Spectre.
Update: January 12, 2018
We are continuing forward with the Meltdown mitigation process and have reboots scheduled over the weekend. Our schedule runs through January 18th. We are going to pause our daily blog updates until this is complete unless other actionable news becomes available.
Update: February 8, 2018
As a reminder, all of our KVM hosts are now properly mitigated for Meltdown. We are continuing to work towards a proper mitigation for the Spectre vulnerability and will be providing an updated plan on our blog once one is available.
For more information on these vulnerabilities, the status of our fleet, and how to protect your Linode; please refer to the Meltdown & Spectre guide.
Filed under: announcements, security by Tory Kulick
72 Comments »
We’re celebrating our 14th birthday!
Happy Valentine’s Day!
We are proud to announce the opening of our ninth datacenter: Tokyo 2. There has been high demand for additional Linode capacity in this region and we’re excited to open up availability.