8 ms·

What I find odd is that after the initial Spectre attacks, there have been a long string of these attacks discovered by outside researchers and then patched by

What I find odd is that after the initial Spectre attacks, there have been a long string of these attacks discovered by outside researchers and then patched by the chipmakers. In principle it seems like the chipmakers should hold all the cards when it comes to discovery: they are experts in speculative execution, know exactly how their chips work and have massive existing validation suites, simulators and internal machine-readable specifications for the low-level operations of these chips. Outside researches need to reverse all this by probing a black box (plus a few much-worse-than-insider sources like patents). Yet years after the initial disclosures it's still random individuals or groups who are discovering these? Perhaps pre-Spectre this attack vector wasn't even considered, but once the general mechanism was obvious did the chip-makers not simply set their biggest brains down and say "go through this with a fine-toothed comb looking for other Spectre attacks"? Maybe they did and are well aware of all these attacks but to save face and performance hits they simply hold on to them hoping nobody makes them public?

DropInIn 3y ago
Tinfoil hat time: They know at release about many of these bugs but between incentivizing upgrades and selling the resultant bugs as backdoor to 'three letter agencies' there's simply too much money to be made by not disclosing/patching the problems before third parties release discovery of the problems.
infinityio 3y ago
This could be a case of survivorship bias - we don't know how many spectre-like bugs did get patched, because they never made it to the public
- hinkley 3y ago
  Or the problem could be with methodology, and the wrong people are in charge of the right people left, and so the mindset for testing is just wrong. Also you’re dealing with a company that has been running to stand still for a long time. There’s been a lot of pressure to meet numbers that they simply cannot keep up with. At some point people cheat, unconsciously or consciously, to achieve the impossible.
  - BeeOnRope 3y ago
    > Also you’re dealing with a company that has been running to stand still for a long time I'm not just talking about Intel, but also Arm and AMD. As far as I know none of these has obviously been making proactive Spectre fixes.
- BeeOnRope 3y ago
  I considered this, but we have pretty good evidence that the chipmakers have not been busily secretly patching Spectre attacks: 1) Microcode updates are visible and Spectre fixes are hard to hide: most have performance impacts and most require coordination from the kernel to enable or make effective (which are visible for Linux). There have been a large number of microcode changes tied to published attacks and corresponding fixes, but no corresponding "mystery" updates and hidden kernel fixes to my knowledge which have a similar shape to Spectre fixes. It's possible they could wait to try to bundle these fixes into a microcode update that arrives for another reason, but the performance impacts and kernel-side changes are harder to hide. 2) If this were the case, we'd expect independent researches to be at least in part re-discovering these attacks, rather than finding completely new ones. This would lead to a case where an attack was already resolved in a release microcode version. To my knowledge this hasn't really happened.
  - mabbo 3y ago
    You don't need to patch the errors found during the initial R&D. The patch was made before the hardware was ever sold. What's left are the attack vectors that Intel's greatest minds didn't discover.
    - BeeOnRope 3y ago
      That's true, but it leads the odd assumption that the vendor managed to fix N side-channel attacks before release but 0 thereafter, while random individuals fixed M thereafter over a period of years with N >> M. This seems to be much less likely than the conclusion that vendors are not in fact fixing many prior to the release and then stopping "cold turkey" after that. Especially since these attacks seem to cross chip versions, in many cases 6+ generations of chips: if vendors had substantial and increasing efforts on new chip versions they'd also be catching issues that applied to old released chips as well. We don't see that happening.
      - Panzer04 3y ago
        If a bug is not known, most of the incentives to the vendor are to not bother investigating, I suspect. You could spend arbitrary amounts of time looking for these bugs and find nothing. Simpler and easier to offer a bounty or something and fix it then. If no one publicly finds the bug it doesn't matter to the mfg (and it wouldn't surprise me if there's truth to your supposition that they know about the bug but wait to fix until someone reports it - no public backlash so long as the bug is unknown) Fixing bugs prior to release seems easy and free, though, especially since many more eyes would also be on the "new" in progress architecture, and proper hardware mitigations that don't cost a lot of performance can be made.
- api 3y ago
  We also don't know how many are still out there unreported and part of the secret zero-day caches of various intelligence agencies.
  - thumbuddy 3y ago
    There's probably thousands if not more. The way I always imagined this working was. agency and company work together to leave gaps in our hardware and software under the ruse of "it must be secure enough for our use." Agencies get a preemptive six months or so to find enough zero days to do what they want. The engineers at the company screaming about the issues are ignored. Eventually a foreign adversary or domestic hacker finds one that can cause a lot of harm. As soon as they find one a DOD funded student simultaneously discovers it. Alternatively, if documents leak showing how these exploits could happen, same scenario. Not to say all bugs are known, but I'd imagine a fair deal of them certainly are.
Method-X 3y ago
Like how car companies will do a cost/benefit analysis to see if a recall is worth it.
- david-gpu 3y ago
  Having worked in the industry, my gut feeling is that chipmakers don't invest all that much in looking for and preventing these sorts of attacks. When working on a new feature, you are desperately trying to deliver on time something that adds value in the sorts of scenarios that it was designed for. And that is already hard enough.
  - pjmlp 3y ago
    The industry as a whole is yet to have the same liability laws as the car industry. When it comes, and it will come, that will change.
hackermatic 3y ago
A fundamental problem is that the attack surface is so, so huge. Even if their security researchers are doing blue-sky research on both very small and very broad areas of processor functionality, they're going to miss a lot. And in line with that and >Maybe they _did_ and are well aware of all these attacks but to save face and performance hits they simply hold on to them hoping nobody makes them public? ... maybe they have patched a number of issues and just never announced them.
- BeeOnRope 3y ago
  > A fundamental problem is that the attack surface is so, so huge. Even if their security researchers are doing blue-sky research on both very small and very broad areas of processor functionality, they're going to miss a lot. Sure. If they had patched a bunch of Spectre vulnerabilities and independent researchers had discovered a few more that would be one thing, but as far as I can tell they have patched _zero_ while independent researches have found many and it has been years since the initial attack. Many of these follow very similar patterns and "in what cases is protected data exposed via speculative execution" is something that an architect or engineer could definitely assess.
- yencabulator 3y ago
  Generally all these workarounds have a measurable slowdown associated with them. This mitigation apparently has an up to 50% cost. It's unlikely many of them have been silently fixed without people noticing.
zamadatix 3y ago
The logic in this comment rubs me the wrong way. You could use the same train of thought to postulate programmers that have made 2 memory safety errors are nefarious instead of simply human. When billions use something I expect them to find more problems, flaws, and exploits in it than the creator/manufacturer did. The presence of this does nothing to indicate (or refute) any further conclusion about why.
- BeeOnRope 3y ago
  I think the comparison between CPU and software exploits holds at a very high level, but in the case of software the gap between internal and external researches seems lower. Much software is open-source, in which case the play field is almost level and even closed source software is available in assembly which exposes the entire attack surface in a reasonably consumable form. Software reverse-engineering is a hugely popular, fairly accessible field with good tools. Hardware not so much. > When billions use something I expect them to find more problems, flaws, and exploits in it than the creator/manufacturer did. The presence of this does nothing to indicate (or refute) any further conclusion about why. To be very clear none of these errors have been found by billions of random users but by a few interested third parties: many of them working as students with microscopic funding levels and no apparent inside information. I'm not actually suggesting that the nefarious explanation holds: I'm genuinely curious.
  - zamadatix 3y ago
    I'm not sure I agree significant gaps in the playing fields are really there. By significant I mean something that explains it should be e.g. 10x harder or something to the point it's supposed to be suspicious how the ratio is indicative of something off. Sure, you don't get to see how they laid out the transistors of the CPU but that's not how these attacks work it's by some oversight in memory handling not that different from software. They analyze how individual assembly instructions behave in certain scenarios vs how they are designed to behave. Compare this to the process of attacking closed source software like Windows, sift through a bunch of assembly in a debugger and see what gets left behind or compared incorrectly, and it's not glaringly different just because hardware is involved. Difficult, sure, but far from anything to suggest it should be uncommon. More importantly, by definition you don't really get to see all of the things they do catch. Maybe they are getting 90% of other related vulnerabilities with the patches but you only hear about the 10% that weren't covered by it because that's the only thing someone is going to publish/get a cve for/make the front page of HN. The point isn't that billions of users all actively try to exploit software it's that if you have billions of users then even if 0.01% try to then that's still a hell of a lot more external bug finders than internal bug finders. Yeah nothing against you or genuine curiosity it's just when a comment sets up a series of logic and concludes with a leading question then the conversation is damned to largely revolve around the leading question instead of genuine answers.
justinator 3y ago
>In principle it seems like the chipmakers should hold all the cards when it comes to discovery: they are experts in speculative execution, know exactly how their chips work and have massive existing validation suites, simulators and internal machine-readable specifications for the low-level operations of these chips. I hope you're not in charge of hiring QA. Bugs are often found by people who AREN'T thinking like the developers whichstart wearing blinders on how they're stuff should work and stop trying stupid things.
- BeeOnRope 3y ago
  I'm not directly in charge of hiring QA, no! I think this sort of excuses the initial blindness to Spectre style attacks in the first place, but once the basic pattern was clear it doesn't excuse the subsequent lack of discovering any of the subsequent issues. It is as if someone found a bug by examining the assembly language of your process which was caused by unsafe inlined `strcpy` calls (though they could not see the source, so they had no idea strcpy was the problem), and then over a the subsequent 6 years other people slowly found more strcpy issues in your application using brute-force black-box engineering and meanwhile you never just grepped your (closed) source for strcpy uses and audited them or used many of the compiler or runtime mitigations against this issue.
  - justinator 3y ago
    That still seems like a job for QA and you do have a good point. If there's an attack using one technique, there should be an audit to make sure that technique can't be modified to be used in other nasty ways, then have those tests part of the test suite.
- margalabargala 3y ago
  > I hope you're not in charge of hiring QA. This seems unnecessarily harsh. The whole post would be improved by removing that sentence IMO.
  - justinator 3y ago
z3t4 3y ago
You are describing a creator bias, there are probably a better name, where you think the ones who created something knows it best. For example, you could create a programming language, a game, or anything, and you think that you know it better then someone who use it for several hours every day. The larger the user base the less likely you are better or know it better then all users.
- weebull 3y ago
  The fans know Star Wars far better than George Lucas.
bsder 3y ago
The problem is that nobody wants to admit that the old, stodgy mainframe guys were right 30 years ago and that sharing anything always results in an exfiltration surface. Nobody wants to be the first to take proper steps because they have to either: 1) Partition hardware properly so that users are genuinely isolated. This costs silicon area that nobody wants to pay for. 2) Stop all the speculative bullshit. This throws performance into rewind and will put chip performance back a decade or two that nobody wants to pay for. Until people are willing to value security enough to put real money behind it, this will continue ad nauseam.
- tracker1 3y ago
  I'd add that the status quo has done pretty well, and many of these exploits are fixed. It's also worth noting that a lot of the exploits in question may be known, but the people working on them couldn't theorize a practical exploit. How many web browser sandbox breaches have there been over the years? Far less than the CPU exploits in the past several years. The latter can have a much bigger impact though. The biggest risk target seems to be shared servers, and you often don't know who you're sharing with, so is it worth trying? It seems to be usually, no... in a specific target, maybe.
  - saagarjha 3y ago
    Web browser sandboxes are broken all the time using conventional bugs.
- pnut 3y ago
  Bad news, 1993 was 30 years ago. Maybe 50+ years ago at this point.
  - bsder 3y ago
    No, 1993 is about right. The mainframe guys were screaming their heads off as these dumbass, insecure, non-ECC x86 processors were eating up more and more computing--before that the mainframe guys kind of dismissed x86 as simply toys.
quickthrower2 3y ago
My guess: There are a lot more researchers on the outside than inside. And also incentives - you would need a red team at Intel who tries to attack their own chips, and those people would be even fewer. And the best people may want to stay independent.
TZubiri 3y ago
It's probably that they are aware of several vulnerabilities, such is life. But they are unable to prioritize them and assess which to solve first, they need external auditors using black box techniques to help them identify which are exploitable by external attackers, so they can fix that one, and not the other non issues
efficax 3y ago
an easier explanation is that modern chipsets are incredibly complex, side channel attacks like these are hard to reason about and traditionally, processors themselves have not been the target of these kinds of attacks, so i don’t think the engineers working on them are accustomed to thinking about them as attack surfaces in this way.
- CanaryLayout 3y ago
  You didn't have to think about this when you ran your code on your own chips; so it's your fault for backdooring the front end into the datacenter. But now we have the 21st-century mainframe we like to call the cloud, where everything is shared. So I upload a container image with the vampire vuln intending to read all the activity on the host. Other customers jobs, the OS itself, even steal internal keys used at Amazon. The motivation to do this kind of attack now is incalculable.
  - CanaryLayout 3y ago
    I kid. The entire Zen class of AMD processors are vuln to Inception so you're not safe anywhere. There is the "just use OS/2" strategy of the 1990s (referring to the period from 1995-2002 where you could sit happy in your niche OS that researchers and BHs were not targeting): in this case s390x probably has the same problem but no one is paying attention to it (other than a 3 letter agency, of course), so the chances of a payload reaching that arch are very close to zero.
adgjlsfhk1 3y ago
I think you're last statement is half right. They probably don't bother looking for them that hard because if they look for them they might find them and then have to make their CPUs slower before launch.
deleted 3y ago
[deleted]
CTDOCodebases 3y ago
And maybe just maybe when the Snowden revelations started to come out some people woke up and realised that the companies who design the processors used in the vast majority of computers are from the US. https://www.theverge.com/2013/12/20/5231006/nsa-paid-10-million-for-a-back-door-into-rsa-encryption-according-to https://www.theverge.com/2013/12/20/5231006/nsa-paid-10-mill...
- CanaryLayout 3y ago
  Since Applied Cryptograhy and everyday since publication it's been well-known and well-understood that NSA's never-ending efforts to weaken systems to make it easier for them to do their work wrecks havoc and costs billions to everyone else. Luckily their attempt to get everyone to adopt a PRNG that was broken on-purpose was thwarted. But who's to say that any chipmaker gets bribed to backdoor their design to allow the reading of any page of RAM from any protection level doesn't happen? It would make their job super easy.
  - CTDOCodebases 3y ago
    Some people might say “don’t attribute to malice what can be attributed to incompetence” but introducing bugs via bribing an insider or even getting one of your people a job at Intel or AMD would be a very clever way to give yourself the keys to (nearly) all the castles. Just don’t forget to patch the microcode on your own systems.
  - mnw21cam 3y ago
    It wreaks havoc. I think wrecking havoc is probably the opposite to what you meant.
rvba 3y ago
Maybe Intel cut corners wherever possible - so their processors are few percent faster due to each bug, but much less safe. But hey they were faster than AMD!
- CanaryLayout 3y ago
  shoulda bought the threadripper the intern recommended than buying that vulnshit your ISV recommended to you
chrsw 3y ago
There's more money to be made in performance and efficiency than security. If chipmakers could design and build the perfect processor, they would. But like anything else complex, there are compromises everywhere.
- BeeOnRope 3y ago
  Granted but the vendors accept these are serious problems given that they are immediately patched and the mitigation all enabled by default even at significant performance cost (most chip generations are down double digit perf % based versus "zero mitigations" at this int). So they don't need to build the "perfect processor" but why aren't they discovering any of these issues themselves?
raggi 3y ago
The intersection of probabilistic optimization and timing based side channels is a gift that will never ever fully go away. Everything _really fast_ which is approximately any very mature systems gear has probabilistic optimization in it now, and that's where a great deal of modern performance comes from. Even thermals and power draws produce side channels. Eradicating every side channel is untenable, research is required to understand what side channels are tenable, then we have to patch them up as best we can. I'm waiting for the one where we find that binning is involved in an integrated way, and that some arbitrary sub-population of popular chips turn out to much more exploitable than others due to particular paths being disabled and leaking extra timing info. That'll be a really fun and awful day.
jgalt212 3y ago
> Outside researches need to reverse all this by probing a black box The kids are calling this activity "prompt engineering".
toyg 3y ago
Maybe they're simply victims of Kernighan's Law of Debugging: "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" There is no doubt that Intel make chips "as clever as they can". Hence, by definition, they can't fully debug them.
- djbusby 3y ago
  If debugging is what we call it when fixing things, does that mean we're "bugging" when we make it?
  - yaris 3y ago
    Indeed! "Every program can be reduced to a single line of code, which will have a bug". For hardware this also holds, I believe.
  - Hunpeter 3y ago
    "If debugging is the process of removing bugs, then programming must be the process of putting them in" - Edsger W. Dijkstra
yyyk 3y ago
The chipmakers don't have an incentive to look too hard for speculation security issues beyond a bit of PR. If they succeed, they lose money and marketshare, while their 'insecure' opponents gain and at most patches later. And in fairness, a lot of these bugs are rather theoretical. Until buyers take these bugs much more seriously, this isn't going to change.
- peddling-brink 3y ago
  Clouds take these vulns seriously, and have a lot to lose, and have deep wallets. I'd be surprised if this topic didn't come up when large purchases are discussed. Not that there are many alternatives..
  - CanaryLayout 3y ago
    They have workarounds. If you prevent multi-tenant from sharing their threads on the same core, that eliminates the most desirable goal of an attacker. However it does not eliminate the vulnerability within a single tenants own threads. You also have to think about all the web shops that are out there that are just running proxmox or a cloud reseller who reintroduces the vulnerability in their multicore VPS setup
  - ComodoHacker 3y ago
    And this flaw was found by researcher working for a cloud vendor. I wouldn't be surprised if they already have some special long-term agreements.
  - yyyk 3y ago
    Big clouds have the exact same incentive issue - if a customer is really paranoid, the customer can pay the cloud extra and ensure their own exclusive infra. For regular users, the clouds can mitigate a bit at scale (but not care much about this in practice, it's not as important as cheaper faster processors).
cjbprime 3y ago
I think there are simply very few humans performing competent vulnerability research (proactive discovery, not reactive patching) in public.
dougall 3y ago
Yeah... I don't know if you saw Rodrigo Branco's damning "The Microarchitectures That I Saw And The Ones That I Hope To One Day See": https://www.youtube.com/watch?v=WlcQrx7VK00 https://www.youtube.com/watch?v=WlcQrx7VK00 https://hardwear.io/usa-2023/presentation/the-microarchitectures-that-I-saw-and-the-ones-that-I-hope-to-one-day-see.pdf https://hardwear.io/usa-2023/presentation/the-microarchitect... But it definitely seems to be a culture/disclosure problem. (Also, hi - hope things are going well! We miss you on Mastodon)
radium3d 3y ago
Do "they know exactly how their chips work"?
_lvbh 3y ago
NSA Backdoor \s
whoisthemachine 3y ago
It's the same as any product, the product team wants a faster, cheaper product, yesterday. Security and trust is secondary, because if you're lucky enough, that will fall on the next product team. Beyond that, processors contain billions (or trillions?) of possible outcomes from a set of inputs. Testing for all of these just to verify reliability and stamp out logic bugs is really hard due to the combinatorial explosion. Putting security testing on top just complicates matter further. The best they can probably do is map out potential ways in which their general purpose processors could be used for specific nefarious uses.
thayne 3y ago
> Maybe they did and are well aware of all these attacks but to save face and performance hits they simply hold on to them hoping nobody makes them public? Or they chose not to look for these types of bugs in the first place, for those reasons.
throwawaylinux 3y ago
There are security teams and/or methodologies inside all major CPU designers today that look at speculation and other side channels. Although these might still not be up to quite the kind of rigor that you see in traditional verification. That is to say, their unit tests and fuzzers and formal analysis and proving methodologies are all well set up to verify that architectural results are correct, my guess is that they don't all verify intermediate results or side effects can't be observed by different privilege contexts. In many ways it is a much more difficult problem too. Going back to first principles, what execution in one context could have an affect on a subsequent context? The answer is just about everything. If you really wanted to be sure you couldn't leak information between contexts, you could only ever run one privilege level on the machine at any time. Not just core, entire machine. When switching contexts, you would have to stop everything, stop all memory and IO traffic, flush all caches and queues, and idle all silicon until all temperatures had reached ambient, voltages, fans, frequency scaling had settled. Then you could start your next program. Even then there's probably things you've forgotten -- programs can access DRAM in certain ways to cause bitflips in adjacent cells for example, so you've probably got a bunch of persistent and unfixable problems there to if you're paranoid. That's not even starting on any of the OS, hypervisor or firmware state that the attacking program might have influenced. So the real answer is that you simply can't share any silicon, software, or wires whatsoever between different trust domains if you are totally paranoid. All of these things are well known about, but at some point you make your best estimation of whether something could realistically be exploited and that's very hard to actually prove one way or another. Multiply by all possible channels and techniques. That's probably why you see a side channel vulnerability discovered every month by outsiders, but very few architectural defects (Pentium FDIV type bugs). That said, this issue looks like a clear miss by their security process. Supplying data to an untrusted context, even if it can only be used speculatively, is clearly outside what is acceptable, and it is one of the things that could be discovered by an analysis of the pipeline. Contrast with the recent AMD branch prediction vulnerability, which could plausibly fall under teh category of it being a known risk but was not thought to be realistically exploitable. As others have said though, everyone makes mistakes, every CPU and program and every engineering project has bugs and mistakes. I don't know if you can deduce much about a CPU design company's internal process from looking at things like this.
IshKebab 3y ago
As I understand it this isn't really a Spectre style bug, it's just a straight up bug. It's pretty much the hardware equivalent of use-after-free. Simple to fix with a microcode update. No serious performance implications.
mike_hearn 3y ago
One possibility nobody mentioned yet: the chip vendors don't invest a ton of time looking for them because they don't actually matter that much. Bear in mind, security researchers are incentivized to find things to build their reputation. It's very often the case that they claim something is a world-shaking security vulnerability when in reality it doesn't matter much for real world attackers. Has anyone ever found a speculation attack in the wild? I think the answer might be no. In which case, why would chip vendors invest tons of money into this? Real customers aren't being hurt by it except in the sense that when an external researcher forces action, they're made to release new microcode that slows things down. Note how all their mitigations for these attacks always have off switches: not something you usually see in security fixes. It's because in many, many cases, these attacks just don't matter. All software running on the same physical core or even the same physical CPU is either running at the same trust level, or sandboxed so heavily it can't mount the attack.
- yencabulator 3y ago
  > Note how all their mitigations for these attacks always have off switches: not something you usually see in security fixes. It's because in many, many cases, these attacks just don't matter. They have off switches because they can have severe performance costs, which most security fixes don't have. > All software running on the same physical core or even the same physical CPU is either running at the same trust level, or sandboxed so heavily it can't mount the attack. Former is simply not true. For example, EC2 avoids mixing tenants on the same core, but even the AWS serverless stuff doesn't do that, for cost reasons. For most end user computers, this is blatantly not true. Operating system level sandboxing is largely irrelevant if the attacker can simply read secrets from other processes running on the same core at the CPU level. Most processes will have a way to smuggle the stolen goods out.
  - mike_hearn 3y ago
    By "sandboxed so heavily" I meant browsers, which don't allow shared memory multi-threading, tight control over CPU instructions or high resolution timers, so it's very hard to mount specex attacks there. I've seen claims it can be done but very few demos, and the only demo I remember trying didn't actually work. By "running on the same physical core" I meant simultaneously. You can usually wipe uarch state when switching between tenants pretty well. If AWS aren't doing this then that's something for them to solve but I'm pretty sure major cloud users don't have to worry about these attacks as so many only affect hyperthreads that are running concurrently and it's not cost prohibitive to avoid mixing tenants that way.
mad 3y ago
Company politics? According to this tweet, parts of Intel did know about this attack: https://twitter.com/bsdaemon/status/1688978152201015301 https://twitter.com/bsdaemon/status/1688978152201015301
weebull 3y ago
As a CPU engineer, I can say that spectre highlighted channels of information escape that weren't previously considered as vulnerable. That's why it kicked off a new batch of exploits. There was a new idea at the heart of it and others built on that idea. It's also important to say that these are not bugs. The design is behaving as intentioned. That the performance differs based off the previous code that the CPU has executed was understood and deameed acceptable because the cost of the alternative was considered too high (either in power, performance or area) than the alternative. That's what it means to be an engineer. You weigh up alternatives and make a choice. In this case, CPUs became fast enough that the fractional part of a bit per iteration became high enough bandwidth to be exploitable, but it needed someone to demonstrate it for it to be understood in the industry. That changed the engineering decision.

DropInIn 3y ago

Tinfoil hat time: They know at release about many of these bugs but between incentivizing upgrades and selling the resultant bugs as backdoor to 'three letter agencies' there's simply too much money to be made by not disclosing/patching the problems before third parties release discovery of the problems.

infinityio 3y ago

This could be a case of survivorship bias - we don't know how many spectre-like bugs did get patched, because they never made it to the public

hinkley 3y ago
Or the problem could be with methodology, and the wrong people are in charge of the right people left, and so the mindset for testing is just wrong. Also you’re dealing with a company that has been running to stand still for a long time. There’s been a lot of pressure to meet numbers that they simply cannot keep up with. At some point people cheat, unconsciously or consciously, to achieve the impossible.
- BeeOnRope 3y ago
  > Also you’re dealing with a company that has been running to stand still for a long time I'm not just talking about Intel, but also Arm and AMD. As far as I know none of these has obviously been making proactive Spectre fixes.
BeeOnRope 3y ago
I considered this, but we have pretty good evidence that the chipmakers have not been busily secretly patching Spectre attacks: 1) Microcode updates are visible and Spectre fixes are hard to hide: most have performance impacts and most require coordination from the kernel to enable or make effective (which are visible for Linux). There have been a large number of microcode changes tied to published attacks and corresponding fixes, but no corresponding "mystery" updates and hidden kernel fixes to my knowledge which have a similar shape to Spectre fixes. It's possible they could wait to try to bundle these fixes into a microcode update that arrives for another reason, but the performance impacts and kernel-side changes are harder to hide. 2) If this were the case, we'd expect independent researches to be at least in part re-discovering these attacks, rather than finding completely new ones. This would lead to a case where an attack was already resolved in a release microcode version. To my knowledge this hasn't really happened.
- mabbo 3y ago
  You don't need to patch the errors found during the initial R&D. The patch was made before the hardware was ever sold. What's left are the attack vectors that Intel's greatest minds didn't discover.
  - BeeOnRope 3y ago
    That's true, but it leads the odd assumption that the vendor managed to fix N side-channel attacks before release but 0 thereafter, while random individuals fixed M thereafter over a period of years with N >> M. This seems to be much less likely than the conclusion that vendors are not in fact fixing many prior to the release and then stopping "cold turkey" after that. Especially since these attacks seem to cross chip versions, in many cases 6+ generations of chips: if vendors had substantial and increasing efforts on new chip versions they'd also be catching issues that applied to old released chips as well. We don't see that happening.
    - Panzer04 3y ago
      If a bug is not known, most of the incentives to the vendor are to not bother investigating, I suspect. You could spend arbitrary amounts of time looking for these bugs and find nothing. Simpler and easier to offer a bounty or something and fix it then. If no one publicly finds the bug it doesn't matter to the mfg (and it wouldn't surprise me if there's truth to your supposition that they know about the bug but wait to fix until someone reports it - no public backlash so long as the bug is unknown) Fixing bugs prior to release seems easy and free, though, especially since many more eyes would also be on the "new" in progress architecture, and proper hardware mitigations that don't cost a lot of performance can be made.
api 3y ago
We also don't know how many are still out there unreported and part of the secret zero-day caches of various intelligence agencies.
- thumbuddy 3y ago
  There's probably thousands if not more. The way I always imagined this working was. agency and company work together to leave gaps in our hardware and software under the ruse of "it must be secure enough for our use." Agencies get a preemptive six months or so to find enough zero days to do what they want. The engineers at the company screaming about the issues are ignored. Eventually a foreign adversary or domestic hacker finds one that can cause a lot of harm. As soon as they find one a DOD funded student simultaneously discovers it. Alternatively, if documents leak showing how these exploits could happen, same scenario. Not to say all bugs are known, but I'd imagine a fair deal of them certainly are.

Method-X 3y ago

Like how car companies will do a cost/benefit analysis to see if a recall is worth it.

david-gpu 3y ago
Having worked in the industry, my gut feeling is that chipmakers don't invest all that much in looking for and preventing these sorts of attacks. When working on a new feature, you are desperately trying to deliver on time something that adds value in the sorts of scenarios that it was designed for. And that is already hard enough.
- pjmlp 3y ago
  The industry as a whole is yet to have the same liability laws as the car industry. When it comes, and it will come, that will change.

hackermatic 3y ago

A fundamental problem is that the attack surface is so, so huge. Even if their security researchers are doing blue-sky research on both very small and very broad areas of processor functionality, they're going to miss a lot. And in line with that and >Maybe they _did_ and are well aware of all these attacks but to save face and performance hits they simply hold on to them hoping nobody makes them public? ... maybe they have patched a number of issues and just never announced them.

BeeOnRope 3y ago
> A fundamental problem is that the attack surface is so, so huge. Even if their security researchers are doing blue-sky research on both very small and very broad areas of processor functionality, they're going to miss a lot. Sure. If they had patched a bunch of Spectre vulnerabilities and independent researchers had discovered a few more that would be one thing, but as far as I can tell they have patched _zero_ while independent researches have found many and it has been years since the initial attack. Many of these follow very similar patterns and "in what cases is protected data exposed via speculative execution" is something that an architect or engineer could definitely assess.
yencabulator 3y ago
Generally all these workarounds have a measurable slowdown associated with them. This mitigation apparently has an up to 50% cost. It's unlikely many of them have been silently fixed without people noticing.

zamadatix 3y ago

The logic in this comment rubs me the wrong way. You could use the same train of thought to postulate programmers that have made 2 memory safety errors are nefarious instead of simply human. When billions use something I expect them to find more problems, flaws, and exploits in it than the creator/manufacturer did. The presence of this does nothing to indicate (or refute) any further conclusion about why.

BeeOnRope 3y ago
I think the comparison between CPU and software exploits holds at a very high level, but in the case of software the gap between internal and external researches seems lower. Much software is open-source, in which case the play field is almost level and even closed source software is available in assembly which exposes the entire attack surface in a reasonably consumable form. Software reverse-engineering is a hugely popular, fairly accessible field with good tools. Hardware not so much. > When billions use something I expect them to find more problems, flaws, and exploits in it than the creator/manufacturer did. The presence of this does nothing to indicate (or refute) any further conclusion about why. To be very clear none of these errors have been found by billions of random users but by a few interested third parties: many of them working as students with microscopic funding levels and no apparent inside information. I'm not actually suggesting that the nefarious explanation holds: I'm genuinely curious.
- zamadatix 3y ago
  I'm not sure I agree significant gaps in the playing fields are really there. By significant I mean something that explains it should be e.g. 10x harder or something to the point it's supposed to be suspicious how the ratio is indicative of something off. Sure, you don't get to see how they laid out the transistors of the CPU but that's not how these attacks work it's by some oversight in memory handling not that different from software. They analyze how individual assembly instructions behave in certain scenarios vs how they are designed to behave. Compare this to the process of attacking closed source software like Windows, sift through a bunch of assembly in a debugger and see what gets left behind or compared incorrectly, and it's not glaringly different just because hardware is involved. Difficult, sure, but far from anything to suggest it should be uncommon. More importantly, by definition you don't really get to see all of the things they do catch. Maybe they are getting 90% of other related vulnerabilities with the patches but you only hear about the 10% that weren't covered by it because that's the only thing someone is going to publish/get a cve for/make the front page of HN. The point isn't that billions of users all actively try to exploit software it's that if you have billions of users then even if 0.01% try to then that's still a hell of a lot more external bug finders than internal bug finders. Yeah nothing against you or genuine curiosity it's just when a comment sets up a series of logic and concludes with a leading question then the conversation is damned to largely revolve around the leading question instead of genuine answers.

justinator 3y ago

>In principle it seems like the chipmakers should hold all the cards when it comes to discovery: they are experts in speculative execution, know exactly how their chips work and have massive existing validation suites, simulators and internal machine-readable specifications for the low-level operations of these chips. I hope you're not in charge of hiring QA. Bugs are often found by people who AREN'T thinking like the developers whichstart wearing blinders on how they're stuff should work and stop trying stupid things.

BeeOnRope 3y ago
I'm not directly in charge of hiring QA, no! I think this sort of excuses the initial blindness to Spectre style attacks in the first place, but once the basic pattern was clear it doesn't excuse the subsequent lack of discovering any of the subsequent issues. It is as if someone found a bug by examining the assembly language of your process which was caused by unsafe inlined `strcpy` calls (though they could not see the source, so they had no idea strcpy was the problem), and then over a the subsequent 6 years other people slowly found more strcpy issues in your application using brute-force black-box engineering and meanwhile you never just grepped your (closed) source for strcpy uses and audited them or used many of the compiler or runtime mitigations against this issue.
- justinator 3y ago
  That still seems like a job for QA and you do have a good point. If there's an attack using one technique, there should be an audit to make sure that technique can't be modified to be used in other nasty ways, then have those tests part of the test suite.
margalabargala 3y ago
> I hope you're not in charge of hiring QA. This seems unnecessarily harsh. The whole post would be improved by removing that sentence IMO.
- justinator 3y ago

z3t4 3y ago

You are describing a creator bias, there are probably a better name, where you think the ones who created something knows it best. For example, you could create a programming language, a game, or anything, and you think that you know it better then someone who use it for several hours every day. The larger the user base the less likely you are better or know it better then all users.

weebull 3y ago
The fans know Star Wars far better than George Lucas.

bsder 3y ago

The problem is that nobody wants to admit that the old, stodgy mainframe guys were right 30 years ago and that sharing anything always results in an exfiltration surface. Nobody wants to be the first to take proper steps because they have to either: 1) Partition hardware properly so that users are genuinely isolated. This costs silicon area that nobody wants to pay for. 2) Stop all the speculative bullshit. This throws performance into rewind and will put chip performance back a decade or two that nobody wants to pay for. Until people are willing to value security enough to put real money behind it, this will continue ad nauseam.

tracker1 3y ago
I'd add that the status quo has done pretty well, and many of these exploits are fixed. It's also worth noting that a lot of the exploits in question may be known, but the people working on them couldn't theorize a practical exploit. How many web browser sandbox breaches have there been over the years? Far less than the CPU exploits in the past several years. The latter can have a much bigger impact though. The biggest risk target seems to be shared servers, and you often don't know who you're sharing with, so is it worth trying? It seems to be usually, no... in a specific target, maybe.
- saagarjha 3y ago
  Web browser sandboxes are broken all the time using conventional bugs.
pnut 3y ago
Bad news, 1993 was 30 years ago. Maybe 50+ years ago at this point.
- bsder 3y ago
  No, 1993 is about right. The mainframe guys were screaming their heads off as these dumbass, insecure, non-ECC x86 processors were eating up more and more computing--before that the mainframe guys kind of dismissed x86 as simply toys.

quickthrower2 3y ago

My guess: There are a lot more researchers on the outside than inside. And also incentives - you would need a red team at Intel who tries to attack their own chips, and those people would be even fewer. And the best people may want to stay independent.

TZubiri 3y ago

It's probably that they are aware of several vulnerabilities, such is life. But they are unable to prioritize them and assess which to solve first, they need external auditors using black box techniques to help them identify which are exploitable by external attackers, so they can fix that one, and not the other non issues

efficax 3y ago

an easier explanation is that modern chipsets are incredibly complex, side channel attacks like these are hard to reason about and traditionally, processors themselves have not been the target of these kinds of attacks, so i don’t think the engineers working on them are accustomed to thinking about them as attack surfaces in this way.

CanaryLayout 3y ago
You didn't have to think about this when you ran your code on your own chips; so it's your fault for backdooring the front end into the datacenter. But now we have the 21st-century mainframe we like to call the cloud, where everything is shared. So I upload a container image with the vampire vuln intending to read all the activity on the host. Other customers jobs, the OS itself, even steal internal keys used at Amazon. The motivation to do this kind of attack now is incalculable.
- CanaryLayout 3y ago
  I kid. The entire Zen class of AMD processors are vuln to Inception so you're not safe anywhere. There is the "just use OS/2" strategy of the 1990s (referring to the period from 1995-2002 where you could sit happy in your niche OS that researchers and BHs were not targeting): in this case s390x probably has the same problem but no one is paying attention to it (other than a 3 letter agency, of course), so the chances of a payload reaching that arch are very close to zero.

adgjlsfhk1 3y ago

I think you're last statement is half right. They probably don't bother looking for them that hard because if they look for them they might find them and then have to make their CPUs slower before launch.

deleted 3y ago

[deleted]

CTDOCodebases 3y ago

And maybe just maybe when the Snowden revelations started to come out some people woke up and realised that the companies who design the processors used in the vast majority of computers are from the US. https://www.theverge.com/2013/12/20/5231006/nsa-paid-10-million-for-a-back-door-into-rsa-encryption-according-to https://www.theverge.com/2013/12/20/5231006/nsa-paid-10-mill...

CanaryLayout 3y ago
Since Applied Cryptograhy and everyday since publication it's been well-known and well-understood that NSA's never-ending efforts to weaken systems to make it easier for them to do their work wrecks havoc and costs billions to everyone else. Luckily their attempt to get everyone to adopt a PRNG that was broken on-purpose was thwarted. But who's to say that any chipmaker gets bribed to backdoor their design to allow the reading of any page of RAM from any protection level doesn't happen? It would make their job super easy.
- CTDOCodebases 3y ago
  Some people might say “don’t attribute to malice what can be attributed to incompetence” but introducing bugs via bribing an insider or even getting one of your people a job at Intel or AMD would be a very clever way to give yourself the keys to (nearly) all the castles. Just don’t forget to patch the microcode on your own systems.
- mnw21cam 3y ago
  It wreaks havoc. I think wrecking havoc is probably the opposite to what you meant.

rvba 3y ago

Maybe Intel cut corners wherever possible - so their processors are few percent faster due to each bug, but much less safe. But hey they were faster than AMD!

CanaryLayout 3y ago
shoulda bought the threadripper the intern recommended than buying that vulnshit your ISV recommended to you

chrsw 3y ago

There's more money to be made in performance and efficiency than security. If chipmakers could design and build the perfect processor, they would. But like anything else complex, there are compromises everywhere.

BeeOnRope 3y ago
Granted but the vendors accept these are serious problems given that they are immediately patched and the mitigation all enabled by default even at significant performance cost (most chip generations are down double digit perf % based versus "zero mitigations" at this int). So they don't need to build the "perfect processor" but why aren't they discovering any of these issues themselves?

raggi 3y ago

The intersection of probabilistic optimization and timing based side channels is a gift that will never ever fully go away. Everything _really fast_ which is approximately any very mature systems gear has probabilistic optimization in it now, and that's where a great deal of modern performance comes from. Even thermals and power draws produce side channels. Eradicating every side channel is untenable, research is required to understand what side channels are tenable, then we have to patch them up as best we can. I'm waiting for the one where we find that binning is involved in an integrated way, and that some arbitrary sub-population of popular chips turn out to much more exploitable than others due to particular paths being disabled and leaking extra timing info. That'll be a really fun and awful day.

jgalt212 3y ago

> Outside researches need to reverse all this by probing a black box The kids are calling this activity "prompt engineering".

toyg 3y ago

Maybe they're simply victims of Kernighan's Law of Debugging: "Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?" There is no doubt that Intel make chips "as clever as they can". Hence, by definition, they can't fully debug them.

djbusby 3y ago
If debugging is what we call it when fixing things, does that mean we're "bugging" when we make it?
- yaris 3y ago
  Indeed! "Every program can be reduced to a single line of code, which will have a bug". For hardware this also holds, I believe.
- Hunpeter 3y ago
  "If debugging is the process of removing bugs, then programming must be the process of putting them in" - Edsger W. Dijkstra

yyyk 3y ago

The chipmakers don't have an incentive to look too hard for speculation security issues beyond a bit of PR. If they succeed, they lose money and marketshare, while their 'insecure' opponents gain and at most patches later. And in fairness, a lot of these bugs are rather theoretical. Until buyers take these bugs much more seriously, this isn't going to change.

peddling-brink 3y ago
Clouds take these vulns seriously, and have a lot to lose, and have deep wallets. I'd be surprised if this topic didn't come up when large purchases are discussed. Not that there are many alternatives..
- CanaryLayout 3y ago
  They have workarounds. If you prevent multi-tenant from sharing their threads on the same core, that eliminates the most desirable goal of an attacker. However it does not eliminate the vulnerability within a single tenants own threads. You also have to think about all the web shops that are out there that are just running proxmox or a cloud reseller who reintroduces the vulnerability in their multicore VPS setup
- ComodoHacker 3y ago
  And this flaw was found by researcher working for a cloud vendor. I wouldn't be surprised if they already have some special long-term agreements.
- yyyk 3y ago
  Big clouds have the exact same incentive issue - if a customer is really paranoid, the customer can pay the cloud extra and ensure their own exclusive infra. For regular users, the clouds can mitigate a bit at scale (but not care much about this in practice, it's not as important as cheaper faster processors).

cjbprime 3y ago

I think there are simply very few humans performing competent vulnerability research (proactive discovery, not reactive patching) in public.

dougall 3y ago

Yeah... I don't know if you saw Rodrigo Branco's damning "The Microarchitectures That I Saw And The Ones That I Hope To One Day See": https://www.youtube.com/watch?v=WlcQrx7VK00 https://www.youtube.com/watch?v=WlcQrx7VK00 https://hardwear.io/usa-2023/presentation/the-microarchitectures-that-I-saw-and-the-ones-that-I-hope-to-one-day-see.pdf https://hardwear.io/usa-2023/presentation/the-microarchitect... But it definitely seems to be a culture/disclosure problem. (Also, hi - hope things are going well! We miss you on Mastodon)

radium3d 3y ago

Do "they know exactly how their chips work"?

_lvbh 3y ago

NSA Backdoor \s

whoisthemachine 3y ago

It's the same as any product, the product team wants a faster, cheaper product, yesterday. Security and trust is secondary, because if you're lucky enough, that will fall on the next product team. Beyond that, processors contain billions (or trillions?) of possible outcomes from a set of inputs. Testing for all of these just to verify reliability and stamp out logic bugs is really hard due to the combinatorial explosion. Putting security testing on top just complicates matter further. The best they can probably do is map out potential ways in which their general purpose processors could be used for specific nefarious uses.

thayne 3y ago

> Maybe they did and are well aware of all these attacks but to save face and performance hits they simply hold on to them hoping nobody makes them public? Or they chose not to look for these types of bugs in the first place, for those reasons.

throwawaylinux 3y ago

There are security teams and/or methodologies inside all major CPU designers today that look at speculation and other side channels. Although these might still not be up to quite the kind of rigor that you see in traditional verification. That is to say, their unit tests and fuzzers and formal analysis and proving methodologies are all well set up to verify that architectural results are correct, my guess is that they don't all verify intermediate results or side effects can't be observed by different privilege contexts. In many ways it is a much more difficult problem too. Going back to first principles, what execution in one context could have an affect on a subsequent context? The answer is just about everything. If you really wanted to be sure you couldn't leak information between contexts, you could only ever run one privilege level on the machine at any time. Not just core, entire machine. When switching contexts, you would have to stop everything, stop all memory and IO traffic, flush all caches and queues, and idle all silicon until all temperatures had reached ambient, voltages, fans, frequency scaling had settled. Then you could start your next program. Even then there's probably things you've forgotten -- programs can access DRAM in certain ways to cause bitflips in adjacent cells for example, so you've probably got a bunch of persistent and unfixable problems there to if you're paranoid. That's not even starting on any of the OS, hypervisor or firmware state that the attacking program might have influenced. So the real answer is that you simply can't share any silicon, software, or wires whatsoever between different trust domains if you are totally paranoid. All of these things are well known about, but at some point you make your best estimation of whether something could realistically be exploited and that's very hard to actually prove one way or another. Multiply by all possible channels and techniques. That's probably why you see a side channel vulnerability discovered every month by outsiders, but very few architectural defects (Pentium FDIV type bugs). That said, this issue looks like a clear miss by their security process. Supplying data to an untrusted context, even if it can only be used speculatively, is clearly outside what is acceptable, and it is one of the things that could be discovered by an analysis of the pipeline. Contrast with the recent AMD branch prediction vulnerability, which could plausibly fall under teh category of it being a known risk but was not thought to be realistically exploitable. As others have said though, everyone makes mistakes, every CPU and program and every engineering project has bugs and mistakes. I don't know if you can deduce much about a CPU design company's internal process from looking at things like this.

IshKebab 3y ago

As I understand it this isn't really a Spectre style bug, it's just a straight up bug. It's pretty much the hardware equivalent of use-after-free. Simple to fix with a microcode update. No serious performance implications.

mike_hearn 3y ago

One possibility nobody mentioned yet: the chip vendors don't invest a ton of time looking for them because they don't actually matter that much. Bear in mind, security researchers are incentivized to find things to build their reputation. It's very often the case that they claim something is a world-shaking security vulnerability when in reality it doesn't matter much for real world attackers. Has anyone ever found a speculation attack in the wild? I think the answer might be no. In which case, why would chip vendors invest tons of money into this? Real customers aren't being hurt by it except in the sense that when an external researcher forces action, they're made to release new microcode that slows things down. Note how all their mitigations for these attacks always have off switches: not something you usually see in security fixes. It's because in many, many cases, these attacks just don't matter. All software running on the same physical core or even the same physical CPU is either running at the same trust level, or sandboxed so heavily it can't mount the attack.

yencabulator 3y ago
> Note how all their mitigations for these attacks always have off switches: not something you usually see in security fixes. It's because in many, many cases, these attacks just don't matter. They have off switches because they can have severe performance costs, which most security fixes don't have. > All software running on the same physical core or even the same physical CPU is either running at the same trust level, or sandboxed so heavily it can't mount the attack. Former is simply not true. For example, EC2 avoids mixing tenants on the same core, but even the AWS serverless stuff doesn't do that, for cost reasons. For most end user computers, this is blatantly not true. Operating system level sandboxing is largely irrelevant if the attacker can simply read secrets from other processes running on the same core at the CPU level. Most processes will have a way to smuggle the stolen goods out.
- mike_hearn 3y ago
  By "sandboxed so heavily" I meant browsers, which don't allow shared memory multi-threading, tight control over CPU instructions or high resolution timers, so it's very hard to mount specex attacks there. I've seen claims it can be done but very few demos, and the only demo I remember trying didn't actually work. By "running on the same physical core" I meant simultaneously. You can usually wipe uarch state when switching between tenants pretty well. If AWS aren't doing this then that's something for them to solve but I'm pretty sure major cloud users don't have to worry about these attacks as so many only affect hyperthreads that are running concurrently and it's not cost prohibitive to avoid mixing tenants that way.

mad 3y ago

Company politics? According to this tweet, parts of Intel did know about this attack: https://twitter.com/bsdaemon/status/1688978152201015301 https://twitter.com/bsdaemon/status/1688978152201015301

weebull 3y ago

As a CPU engineer, I can say that spectre highlighted channels of information escape that weren't previously considered as vulnerable. That's why it kicked off a new batch of exploits. There was a new idea at the heart of it and others built on that idea. It's also important to say that these are not bugs. The design is behaving as intentioned. That the performance differs based off the previous code that the CPU has executed was understood and deameed acceptable because the cost of the alternative was considered too high (either in power, performance or area) than the alternative. That's what it means to be an engineer. You weigh up alternatives and make a choice. In this case, CPUs became fast enough that the fractional part of a bit per iteration became high enough bandwidth to be exploitable, but it needed someone to demonstrate it for it to be understood in the industry. That changed the engineering decision.