8 ms·

Can somebody here explain why Intel and kernel devs are still debating about solutions in late January? I haven't followed this stuff closely, but I was under

Can somebody here explain why Intel and kernel devs are still debating about solutions in late January? I haven't followed this stuff closely, but I was under the impression that Intel and others had been informed about these vulnerabilities 6 months ago and everybody had fixes ready by early January.

gtirloni 9y ago
Botched disclosure process. More details at the end here: http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-...
- ustolemyname 9y ago
  How do you mean? The formal disclosure was supposed to be earlier this month. [Edit] I see, link implies the problem is that the 6 month disclosure process lacked urgency, and many affected parties were notified late in the cycle. Perhaps this is an argument against agreements to keep vulnerabilities secret for a long time, as an attitude of, "there's enough time to address this," can be counterproductive.
- sschueller 9y ago
  I am sorry but after 6 months of Intel/AMD doing nothing I don't expect the Linux community or anyone else to keep their mouth shut. Also because of 5 days? Embargo was for the 9th of Jan. and it leaked on the 5th after 6 months! What are you going to do in 5 days that you didn't in the last 6 months.
  - phire 9y ago
    The botched embargo destroyed their chances of managing the PR like they thought they would be able to do. Intel thought they could roll Spector and Meltdown into a single massive bug, that all CPUs were vulnerable to. Instead, AMD leaked the fact that AMD CPUs didn't need the KPTI patch, which meant the very first thing people knew was "it's an Intel bug". This very much put the Intel PR effort on a back foot. Then Intel thought they could just release their microcode update (hell, they even coordinated with AMD to release the same microcode patches), and tell linux "here are the patches to use it, you must merge it". I assume the plan involved disclosing on the 9th and essentially bully Linus into merging the patches by the 10th. They wanted people to thank them for spending millions to develop the new microcode and once-and-for-all solving this problem. They wanted the headlines "Intel releases microcode which fixes Spectre/Meltdown flaw" so that people would stop demanding intel replaces their broken silicon. Then google releases details about their retpoline idea (I'm assuming it was developed not long before the embargo ended). It achieves the same thing, but with less performance penalty and no microcode update needed. So now the Linux mailing list has something to debate. The Intel engineers are very motivated to get their patches merged, because they spent millions developing that microcode and they want the PR wins for "fixing the problem". Watch how the immediately segway into "oh, but retpolines don't work on skylake" The Linux engineers are not happy with this, they are angry at Intel for being the source of the problem and probably see through Intel's little PR plan. They are triple checking intel's work and seeing if they can come up with an even better solution that works on skylake too.
    - walterbell 9y ago
      Is Microsoft relying on Intel & AMD microcode to mitigate Spectre, or could they also use the approach now being proposed by the Linux kernel developers? If they don’t, will Windows performance be worse than Linux on the same hardware, for a comparable workload?
      - sgc 9y ago
        It seems pretty obvious that they will be following what happens at the very least, and won't allow themselves to fall into irrelevance if better classes of solutions come along.
        walterbell 9y ago
        Since Microsoft is in the Linux Foundation and runs Linux guests on Windows, they could contribute to the public discussion on Spectre mitigation in Linux.
      - phire 9y ago
        Yes, RHEL too (and probably apple). The Intel engineer in that email seems to be bragging about it: I think we also want to expose IBRS to VM guests, even if we don't use it ourselves. Because Windows guests (and RHEL guests; yay!) do use it.
        bonzini 9y ago
        RHEL is going to switch to retpolines too. But it will take some time.
      - forgot-my-pw 9y ago
        I hope so. Their first round of bug fixes caused pretty significant slowdown on 2015 and older CPUs. > With Windows 10 on older silicon (2015-era PCs with Haswell or older CPU), some benchmarks show more significant slowdowns, and we expect that some users will notice a decrease in system performance. > With Windows 8 and Windows 7 on older silicon (2015-era PCs with Haswell or older CPU), we expect most users to notice a decrease in system performance. > Windows Server on any silicon, especially in any IO-intensive application, shows a more significant performance impact when you enable the mitigations to isolate untrusted code within a Windows Server instance. This is why you want to be careful to evaluate the risk of untrusted code for each Windows Server instance, and balance the security versus performance tradeoff for your environment. https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/understanding-the-performance-impact-of-spectre-and-meltdown-mitigations-on-windows-systems/ https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/...
- kbenson 9y ago
  Whether it was botched or not (from the position of this article) is based on whether you think it was important that FreeBSD have been notified ahead of time. Obviously the FreeBSD devs think they should have been notified. Since this basically affects everyone that runs x86 code, and not everyone could be notified ahead of time and still keep any form of secrecy, some groups will need to be notified and others not. I think a case can be made that as operating systems go, FreeBSD is at a much lower threat level than Linux, MacOS/iOS, Windows (i.e. desktops that run unvetted code) and the VPS platforms that run host operating systems that need to worry about guests breaking isolation (Google, Amazon, others). Given all that, while it wasn't done perfectly (it was leaked a few days early and some cloud providers had little warning, especially Joyent which also runs a different OS), I'm not sure I would call it botched entirely since it is also likely the largest exploit in history. Others disagree with some or all of this.
  - deleted 9y ago
    [deleted]
  - qubex 9y ago
    I read somewhere (didn't verify, wasn't obviously substantiated) that the OpenBSD folks were beside themselves with anger, and that in turn others shot back that they had demonstrated a very poor track record with respecting embargoes in the past and had let slip “seven of the last five” major vulnerabilities they had been pre-warned of. EDIT: For reference, consider some of the comments in this thread: https://news.ycombinator.com/item?id=16110750&ref=hvper.com https://news.ycombinator.com/item?id=16110750&ref=hvper.com
    - jasonkostempski 9y ago
      Seven of the last five? Sounds like they have some out-of-order execution issues of their own.
      - qubex 9y ago
        It’s a common British idiom, to opine against eternal doomsayers. :)
- arto 9y ago
  Echoing the sentiments from OpenBSD's Theo de Raadt as well: https://www.itwire.com/security/81338-handling-of-cpu-bug-disclosure-incredibly-bad-openbsd-s-de-raadt.html https://www.itwire.com/security/81338-handling-of-cpu-bug-di...
davidlt 9y ago
IIUC, a limited number of people from kernel community knew about this and NDAs didn't exactly allow all the parties work together. The first solutions/mitigations exist (might not be nice), but now embargo is lifter and the communities (incl. companies) can work on better solutions together.
- jasonkostempski 9y ago
  I would hope people from the Linux and BSD kernel communities arent signing NDAs. I would expect them to refuse on principal. Do they not?
  - bonzini 9y ago
    I don't sign them (in general), but my employer has agreements and I like my job enough to accept that secrecy and embargoes are on occasion a necessary evil. In this case, however, it was handled horribly. During the embargo period, we kept telling Intel and AMD what they were doing wrong, and they wouldn't listen (or alternatively, they made clear enough that they wouldn't listen that we didn't even try). But really there's little more than I can do than hope that the next time they listen to us.
    - chris_wot 9y ago
      If only one of them had listened, it might have been an impetus to stick with that manufacturer.
  - ambulancechaser 9y ago
    wouldn't the NDA in a case like this just be about the embargo? If this is the case, what strong argument against an NDA is there given that agreement not to disclose the bug is necessary until a good mitigation strategy is in place?
    - walterbell 9y ago
      In the Spectre case, the 6-month NDA did not result in a good mitigation strategy. A few weeks of public discussion has identified better mitigations for some customers.

gtirloni 9y ago

Botched disclosure process. More details at the end here: http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-spectre-and-meltdown.html http://www.daemonology.net/blog/2018-01-17-some-thoughts-on-...

ustolemyname 9y ago
How do you mean? The formal disclosure was supposed to be earlier this month. [Edit] I see, link implies the problem is that the 6 month disclosure process lacked urgency, and many affected parties were notified late in the cycle. Perhaps this is an argument against agreements to keep vulnerabilities secret for a long time, as an attitude of, "there's enough time to address this," can be counterproductive.
sschueller 9y ago
I am sorry but after 6 months of Intel/AMD doing nothing I don't expect the Linux community or anyone else to keep their mouth shut. Also because of 5 days? Embargo was for the 9th of Jan. and it leaked on the 5th after 6 months! What are you going to do in 5 days that you didn't in the last 6 months.
- phire 9y ago
  The botched embargo destroyed their chances of managing the PR like they thought they would be able to do. Intel thought they could roll Spector and Meltdown into a single massive bug, that all CPUs were vulnerable to. Instead, AMD leaked the fact that AMD CPUs didn't need the KPTI patch, which meant the very first thing people knew was "it's an Intel bug". This very much put the Intel PR effort on a back foot. Then Intel thought they could just release their microcode update (hell, they even coordinated with AMD to release the same microcode patches), and tell linux "here are the patches to use it, you must merge it". I assume the plan involved disclosing on the 9th and essentially bully Linus into merging the patches by the 10th. They wanted people to thank them for spending millions to develop the new microcode and once-and-for-all solving this problem. They wanted the headlines "Intel releases microcode which fixes Spectre/Meltdown flaw" so that people would stop demanding intel replaces their broken silicon. Then google releases details about their retpoline idea (I'm assuming it was developed not long before the embargo ended). It achieves the same thing, but with less performance penalty and no microcode update needed. So now the Linux mailing list has something to debate. The Intel engineers are very motivated to get their patches merged, because they spent millions developing that microcode and they want the PR wins for "fixing the problem". Watch how the immediately segway into "oh, but retpolines don't work on skylake" The Linux engineers are not happy with this, they are angry at Intel for being the source of the problem and probably see through Intel's little PR plan. They are triple checking intel's work and seeing if they can come up with an even better solution that works on skylake too.
  - walterbell 9y ago
    Is Microsoft relying on Intel & AMD microcode to mitigate Spectre, or could they also use the approach now being proposed by the Linux kernel developers? If they don’t, will Windows performance be worse than Linux on the same hardware, for a comparable workload?
    - sgc 9y ago
      It seems pretty obvious that they will be following what happens at the very least, and won't allow themselves to fall into irrelevance if better classes of solutions come along.
      - walterbell 9y ago
        Since Microsoft is in the Linux Foundation and runs Linux guests on Windows, they could contribute to the public discussion on Spectre mitigation in Linux.
    - phire 9y ago
      Yes, RHEL too (and probably apple). The Intel engineer in that email seems to be bragging about it: I think we also want to expose IBRS to VM guests, even if we don't use it ourselves. Because Windows guests (and RHEL guests; yay!) do use it.
      - bonzini 9y ago
        RHEL is going to switch to retpolines too. But it will take some time.
    - forgot-my-pw 9y ago
      I hope so. Their first round of bug fixes caused pretty significant slowdown on 2015 and older CPUs. > With Windows 10 on older silicon (2015-era PCs with Haswell or older CPU), some benchmarks show more significant slowdowns, and we expect that some users will notice a decrease in system performance. > With Windows 8 and Windows 7 on older silicon (2015-era PCs with Haswell or older CPU), we expect most users to notice a decrease in system performance. > Windows Server on any silicon, especially in any IO-intensive application, shows a more significant performance impact when you enable the mitigations to isolate untrusted code within a Windows Server instance. This is why you want to be careful to evaluate the risk of untrusted code for each Windows Server instance, and balance the security versus performance tradeoff for your environment. https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/understanding-the-performance-impact-of-spectre-and-meltdown-mitigations-on-windows-systems/ https://cloudblogs.microsoft.com/microsoftsecure/2018/01/09/...
kbenson 9y ago
Whether it was botched or not (from the position of this article) is based on whether you think it was important that FreeBSD have been notified ahead of time. Obviously the FreeBSD devs think they should have been notified. Since this basically affects everyone that runs x86 code, and not everyone could be notified ahead of time and still keep any form of secrecy, some groups will need to be notified and others not. I think a case can be made that as operating systems go, FreeBSD is at a much lower threat level than Linux, MacOS/iOS, Windows (i.e. desktops that run unvetted code) and the VPS platforms that run host operating systems that need to worry about guests breaking isolation (Google, Amazon, others). Given all that, while it wasn't done perfectly (it was leaked a few days early and some cloud providers had little warning, especially Joyent which also runs a different OS), I'm not sure I would call it botched entirely since it is also likely the largest exploit in history. Others disagree with some or all of this.
- deleted 9y ago
  [deleted]
- qubex 9y ago
  I read somewhere (didn't verify, wasn't obviously substantiated) that the OpenBSD folks were beside themselves with anger, and that in turn others shot back that they had demonstrated a very poor track record with respecting embargoes in the past and had let slip “seven of the last five” major vulnerabilities they had been pre-warned of. EDIT: For reference, consider some of the comments in this thread: https://news.ycombinator.com/item?id=16110750&ref=hvper.com https://news.ycombinator.com/item?id=16110750&ref=hvper.com
  - jasonkostempski 9y ago
    Seven of the last five? Sounds like they have some out-of-order execution issues of their own.
    - qubex 9y ago
      It’s a common British idiom, to opine against eternal doomsayers. :)
arto 9y ago
Echoing the sentiments from OpenBSD's Theo de Raadt as well: https://www.itwire.com/security/81338-handling-of-cpu-bug-disclosure-incredibly-bad-openbsd-s-de-raadt.html https://www.itwire.com/security/81338-handling-of-cpu-bug-di...

davidlt 9y ago

IIUC, a limited number of people from kernel community knew about this and NDAs didn't exactly allow all the parties work together. The first solutions/mitigations exist (might not be nice), but now embargo is lifter and the communities (incl. companies) can work on better solutions together.

jasonkostempski 9y ago
I would hope people from the Linux and BSD kernel communities arent signing NDAs. I would expect them to refuse on principal. Do they not?
- bonzini 9y ago
  I don't sign them (in general), but my employer has agreements and I like my job enough to accept that secrecy and embargoes are on occasion a necessary evil. In this case, however, it was handled horribly. During the embargo period, we kept telling Intel and AMD what they were doing wrong, and they wouldn't listen (or alternatively, they made clear enough that they wouldn't listen that we didn't even try). But really there's little more than I can do than hope that the next time they listen to us.
  - chris_wot 9y ago
    If only one of them had listened, it might have been an impetus to stick with that manufacturer.
- ambulancechaser 9y ago
  wouldn't the NDA in a case like this just be about the embargo? If this is the case, what strong argument against an NDA is there given that agreement not to disclose the bug is necessary until a good mitigation strategy is in place?
  - walterbell 9y ago
    In the Spectre case, the 6-month NDA did not result in a good mitigation strategy. A few weeks of public discussion has identified better mitigations for some customers.