9 ms·

RHEL/CentOS state: Red Hat built their own mitigation for Meltdown and Spectre and were the first distribution to have an updated kernel right when the embargo

by lima 9y ago

RHEL/CentOS state: Red Hat built their own mitigation for Meltdown and Spectre and were the first distribution to have an updated kernel right when the embargo was lifted. As far as I can tell, their meltdown mitigations are similar to what's in the upstream kernel now, and the Spectre mitigations relies on Intel's new MSRs (the microcode update that had since been retracted - new microcode_ctl packages remove it). Interestingly, you can enable/disable the mitigations at runtime with Red Hat, which is useful for performance testing. I found no way to do this with the upstream kernel. I'm not happy that Intel, Google, Red Hat and a few others have known about the bug for months and the upstream kernel devs are scrambling to build mitigations and still don't support the microcode mitigations. Google apparently patched their stuff in September, while everyone else is left out in the cold. This is a real mess. Surely there must be a better way than telling a few select companies? The Red Hat guys in particular surely would've loved to collaborate with the upstream kernel. Must be frustrating not to be able to tell anyone.

pqh 9y ago
They really could have done a tiered release. Keep the knowledge of the vulnerability as close as possible for the ~6 months they had it, then release it to a greater (but still restricted) set of people for another ~6 months. the worst case would have been someone in the latter set leaking the info, which is just what we have now. So it could have been potentially better. There was really no reason to drop a bomb like this.
- rplnt 9y ago
  It didn't survive even the first 6 months and was published shortly before the embargo was over.
  - Aissen 9y ago
    Nitpick: it actually survived 7 months. From June 1st to January ~1st
tener 9y ago
I'm afraid this is the kind of problem where there is no easy solution. Is there any specific improvement you would like to propose?
- masklinn 9y ago
  > I'm afraid this is the kind of problem where there is no easy solution. Issues on a similar scale have been properly handled in the past: http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-... > The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. Nobody knew what was coming aside from the people who needed to know; and the people who needed to know had months of warning. > Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered.
masklinn 9y ago
> I'm not happy that Intel, Google, Red Hat and a few others have known about the bug for months and the upstream kernel devs are scrambling to build mitigations and still don't support the microcode mitigations. Google apparently patched their stuff in September, while everyone else is left out in the cold. This is a real mess. Surely there must be a better way than telling a few select companies? I think Colin's take[0] was interesting and insightful there, especially in the light of him having handled a very similar issue a decade earlier: > The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. > Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered. > […] > To make things worse, the Linux community was notified and couldn't keep their mouths shut. Standard practice for multi-vendor advisories like this is that an embargo date is set, and nobody does anything publicly prior to that date. People don't publish advisories; they don't commit patches into their public source code repositories; and they definitely don't engage in arguments on public mailing lists about whether the patches are needed for different CPUs. (emphasis his) [0] http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-... final section on vulnerability disclosures
- ddevault 9y ago
  I think lots of people (particularly security people) tend to fundamentally misunderstand how the Linux community works. It's not a single organization calling the shots where they can have a meeting behind closed doors about the vulnerability and internally incubate a patchset. That's simply not how Linux development works. Linux is a loosely organized community of developers who collaborate in public. They don't have any ubiquitous communication channels established other than mailing lists. If you attempted to establish a private security mailing list, the list of people who get to go on it is (1) hard to define (2) very large and (3) frequently changing. The affected subsystems have lots of disparate maintainers and interested parties who need to get their eyes on any patches that go into them. It might not be ideal from a security person's perspective but from everyone else's perspective this model has been the driving factor behind the most successful collaborative project of all time.
lmm 9y ago
As soon as changes hit the mainline kernel that started the clock. The blogpost that made this all public was just someone noticing the kernel changes and speculating about how they must be in response to an embargoed security bug. So no, there's no better way. You tell every organization that's capable of patching their kernels privately and keeping the vulnerability under wraps - something that mainline Linux can't or won't do, for better or worse. This is like complaining that you found something in the last place you looked - the linux mainline were the last people who heard about it before it went public because as soon as the linux mainline heard about it, it became public.
- masklinn 9y ago
  So… the better way is to just not contact the Linux mainline folks, as contacting them is the same as lifting/breaking the embargo.
  - deleted 9y ago
    [deleted]
- paulmd 9y ago
  I'm not going to beat around the bush here, there was speculation about what might be behind this sudden push for isolating kernel page-tables, but what really blew the lid was AMD outlining the specific details of the vulnerability in a pull request. Everyone here acted responsibly except AMD, who broke embargo so they could force a response from Intel at CES and sell more processors.
amluto 9y ago
> As far as I can tell, their meltdown mitigations are similar to what's in the upstream kernel now, I haven't checked, but I'd be surprised. Basically all the backports except 4.14's are quite different from upstream's.

by lima 9y ago

pqh 9y ago

They really could have done a tiered release. Keep the knowledge of the vulnerability as close as possible for the ~6 months they had it, then release it to a greater (but still restricted) set of people for another ~6 months. the worst case would have been someone in the latter set leaking the info, which is just what we have now. So it could have been potentially better. There was really no reason to drop a bomb like this.

rplnt 9y ago
It didn't survive even the first 6 months and was published shortly before the embargo was over.
- Aissen 9y ago
  Nitpick: it actually survived 7 months. From June 1st to January ~1st

tener 9y ago

I'm afraid this is the kind of problem where there is no easy solution. Is there any specific improvement you would like to propose?

masklinn 9y ago
> I'm afraid this is the kind of problem where there is no easy solution. Issues on a similar scale have been properly handled in the past: http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-... > The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. Nobody knew what was coming aside from the people who needed to know; and the people who needed to know had months of warning. > Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered.

masklinn 9y ago

> I'm not happy that Intel, Google, Red Hat and a few others have known about the bug for months and the upstream kernel devs are scrambling to build mitigations and still don't support the microcode mitigations. Google apparently patched their stuff in September, while everyone else is left out in the cold. This is a real mess. Surely there must be a better way than telling a few select companies? I think Colin's take[0] was interesting and insightful there, especially in the light of him having handled a very similar issue a decade earlier: > The way these issues were handled was a mess; frankly, I expected better of Google, I expected better of Intel, and I expected better of the Linux community. When I found that Hyper-Threading was easily exploitable, I spent five months notifying the security community and preparing everyone for my announcement of the vulnerability; but when the embargo ended at midnight UTC and FreeBSD published its advisory a few minutes later, the broader world was taken entirely by surprise. > Contrast that with what happened this time around. Google discovered a problem and reported it to Intel, AMD, and ARM on June 1st. Did they then go around contacting all of the operating systems which would need to work on fixes for this? Not even close. FreeBSD was notified the week before Christmas, over six months after the vulnerabilities were discovered. > […] > To make things worse, the Linux community was notified and couldn't keep their mouths shut. Standard practice for multi-vendor advisories like this is that an embargo date is set, and nobody does anything publicly prior to that date. People don't publish advisories; they don't commit patches into their public source code repositories; and they definitely don't engage in arguments on public mailing lists about whether the patches are needed for different CPUs. (emphasis his) [0] http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-... final section on vulnerability disclosures

ddevault 9y ago
I think lots of people (particularly security people) tend to fundamentally misunderstand how the Linux community works. It's not a single organization calling the shots where they can have a meeting behind closed doors about the vulnerability and internally incubate a patchset. That's simply not how Linux development works. Linux is a loosely organized community of developers who collaborate in public. They don't have any ubiquitous communication channels established other than mailing lists. If you attempted to establish a private security mailing list, the list of people who get to go on it is (1) hard to define (2) very large and (3) frequently changing. The affected subsystems have lots of disparate maintainers and interested parties who need to get their eyes on any patches that go into them. It might not be ideal from a security person's perspective but from everyone else's perspective this model has been the driving factor behind the most successful collaborative project of all time.

lmm 9y ago

As soon as changes hit the mainline kernel that started the clock. The blogpost that made this all public was just someone noticing the kernel changes and speculating about how they must be in response to an embargoed security bug. So no, there's no better way. You tell every organization that's capable of patching their kernels privately and keeping the vulnerability under wraps - something that mainline Linux can't or won't do, for better or worse. This is like complaining that you found something in the last place you looked - the linux mainline were the last people who heard about it before it went public because as soon as the linux mainline heard about it, it became public.

masklinn 9y ago
So… the better way is to just not contact the Linux mainline folks, as contacting them is the same as lifting/breaking the embargo.
- deleted 9y ago
  [deleted]
paulmd 9y ago
I'm not going to beat around the bush here, there was speculation about what might be behind this sudden push for isolating kernel page-tables, but what really blew the lid was AMD outlining the specific details of the vulnerability in a pull request. Everyone here acted responsibly except AMD, who broke embargo so they could force a response from Intel at CES and sell more processors.

amluto 9y ago

> As far as I can tell, their meltdown mitigations are similar to what's in the upstream kernel now, I haven't checked, but I'd be surprised. Basically all the backports except 4.14's are quite different from upstream's.