11 ms·

I've been saying this from the start: the well of issues is infinitely deep as soon as you decide that multiple tenants running on the same physical hardware in

I've been saying this from the start: the well of issues is infinitely deep as soon as you decide that multiple tenants running on the same physical hardware inferring something about another is a vulnerability. I assert, but cannot rigorously prove, that it is not possible to design a CPU such that execution of arbitrary instructions has no observable side-effects, especially if the CPU is speculating. I don't know what that spells for cloud hosting providers - maybe they have to buy a lot more CPUs so every client can have their own, or commission a special "shared" SKU of CPU that doesn't have any speculative execution - but I know for me, if I have untrusted code running on my CPU, I've already lost. I could then care less about information leakage between threads. We're going to wind up undoing the last 20 years of performance gains in the name of 'security', and it scares me.

MarkSweep 5y ago
> if I have untrusted code running on my CPU, I've already lost Don’t forget about JavaScript, a common way for people to run untrusted code on their computers. Not all of micro-architectural data sample are exploitable in JavaScript, but some are.
- baybal2 5y ago
  Chrome had 7 exploits caught in the wild within 7 weeks in 2020. I believe it is going towards JIT being disabled, or most severely limited.
  - userbinator 5y ago
    It sounds like a dream, but going back towards interpreted JS instead of JIT may finally stem the insanity of bloat that JS has evolved in an environment of increasingly fast implementations.
    - otabdeveloper4 5y ago
      The problem of Javascript bloat doesn't have a technical solution. Javascript bloat exists because of a social problem: the guy who fixes the corporate webpage's javascripts is called a "webdesigner", and "webdesigners" are the lowest rung on the corporate IT ladder, maybe only a bit above first-tier techsupport. If you want to make some sort of career you need to upgrade from "webdesigner" to "frontend developer", and that means cryptic, incomprehensible and pointless "frontend frameworks". It provides to value to business or users, but management puts up with it because it fixes the problem of employee churn. (Frontend positions are a big pain in the ass.)
      - tannhaeuser 5y ago
        No reason to insult web developers in general. The simpler explanation is that webapps exist because of an economical problem: that you can make more money (have lower barriers) by either recurring payments for services, or by selling your user's attention, or both.
        otabdeveloper4 5y ago
        I do lots of frontend development myself, so I'm a kind of "web developer" as well. And I'm insulting web frameworks, not webapps in general.
      - mort96 5y ago
        But JavaScript bloat is allowed to stay (by product managers, middle managers, UX designers, etc) because the website is still fast. If the JS bloat actually caused the site to become too slow on fast machines, people with power to change stuff would demand change.
        jarcane 5y ago
        People with the power to change stuff thought Java applets were a good idea in 1995, when most computers in wide circulation could barely run the JVM at any acceptable speed. Never trust the tech industry to make optimal decisions, you are only in for a bad time.
  - kaba0 5y ago
    That would pretty much instantly kill the web, no matter how lean they make websites.
    - account42 5y ago
      I assure you, there are large parts of the web that do not use excessive amounts of javascript.
      - kaba0 5y ago
        An insignificant portion of a very huge thing can be large.
- akersten 5y ago
  Yeah, JS is the only hairy part. I considered mentioning it, since I know it was going to come up. But luckily, all I've seen so far are basic demos (like leaky.page) that read data from a carefully-crafted array that the page itself populated. I've yet to be convinced that you could realistically exfiltrate meaningful data at any sort of scale with in-browser JS, especially now that more blatant bugs like Meltdown are fixed. If anyone can show a proof-of-concept ("this page grabs your password manager extension's data") I'll eat my words. But I feel confident that most of these issues are purely academic and, while interesting, serve more to provide content for PhD theses than represent urgent hazards on the web.
  - userbinator 5y ago
    Indeed, I've been feeling indifferent about all these timing sidechannels ever since the very first ones (Spectre/Meltdown). The PoCs have not been particularly convincing to me, given that they are extremely contrived and rely on knowing the exact details of the system being exploited to such an extent that someone with those details would be better off with other ways in, and assumes those details haven't changed at all during the amount of time required to do the attack --- the nature of these side-channels is such that even the smallest change in environment can completely change the results. In other words, if I choose a process on my system at random, and dump a few dozen bytes from it, I can technically claim to have leaked some data; but the use of that data to an attacker likely depends strongly on factors which are outside of the attacker's control. It's somewhat like finding a (real-world) key on the ground: you theoretically now have access to something you shouldn't have, but you have next to no idea what that something is. But I feel confident that these issues are purely academic and, while interesting, serve more to provide content for PhD theses than represent urgent hazards on the web. They also provide content for sensationalist clickbait articles and fuel the paranoia that drives society towards authoritarianism and furthers the war on general-purpose-computing, which IMHO is a much bigger issue to worry about.
  - brutal_chaos_ 5y ago
    To add more to the hairy part: WASM. And that's growing rapidly.
  - 5y ago
baybal2 5y ago
You are completely correct. The safe execution of any untrusted Turing complete code is a pipe dream. You, at least, need a clean sheet CPU design starting from ISA, and basic logic operations formally validated against instruction level analysis to have a fighting chance. But even such chip do get pwned, as shown by key recovery from credit cards in the wild.
- mikewarot 5y ago
  >The safe execution of any untrusted Turing complete code is a pipe dream. The safe execution of any code requires an operating environment that never trusts the code with more than the least privilege required to complete a task. It has worked in mainframes that way for decades. The IT zeitgeist these days makes me sad. Things can be better, but almost everyone is pushing in counterproductive directions, or has given up hope.
  - baybal2 5y ago
    > The safe execution of any code requires an operating environment that never trusts the code with more than the least privilege required to complete a task. It has worked in mainframes that way for decades. It has nothing to do with any OS level security features. We are talking about things happening below the level of what software can see. You just cannot see any sign of such attack by looking at any register the OS can see.
    - mikewarot 5y ago
      These are timing attacks, if you can guarantee code runs deterministically, and deny access to timing information, you can defeat the attack.
      - addaon 5y ago
        Timing attacks are only a subset of side channel attacks, though. One can also imagine thermal attacks -- the amount of power you consume leaks information about what you're doing. And if I share a processor with you, there's various ways I can imagine estimating your power usage. On a processor that has dynamic clocking, the clock speed I'm running at is an indicator of the operations you're doing. Even without dynamic clocking, the probability of an ECC error, for example, is likely to change with temperature. Eliminating timing vulnerabilities is necessary to allow potentially-hostile workloads to share hardware, but it is not sufficient.
paulmd 5y ago
I've been saying since this initially came up that big.LITTLE is the long-term solution for this. In the grand scheme of things, high-intensity tasks are only infrequently high-security tasks - those two sets of workloads are mostly disjoint. So the long-term solution is to have "fast cores" and "secure cores". The fast cores can have all the OoO, speculation, all of that good stuff. That's where you run anything that needs to go fast, or anything running "trusted" code. By and large, nobody cares if an ffmpeg process or HPC node might leak data. Databases? You control the queries that are running on them, right? There are some edge cases like video games where leaking data is moderately harmful (could be useful for exploits if you can reliably leak useful data) yet you still want maximum performance, but at the end of the day leaking data at a couple kB/s usually isn't going to be the end of the world especially if the data is rapidly changing. If the code is untrusted or user-generated, or the data is sufficiently sensitive, then run it on a "secure" core. The "secure" cores have to be in-order, non-speculative, all that crap. Probably non-SMT as that seems to be a bottomless pit of sidechannels as well. But usually, you aren't churning huge workloads in the "secure" situations. You can still have crypto acceleration instructions built into the cores, AVX, whatever, just not speculative. It's probably better to get them fully out of the "normal" cache hierarchy as well. There are a couple obvious problems here, but much smaller than trying to fix everything for every use-case. In particular web browsers are running untrusted code, and every single website is running 15 mb of shitty javascript code. It sucks but it's basically become an inner platform and you can't trust the code that it's bringing in, so that needs to be permanently isolated on its own secure cores. People will have to start paying attention to the performance of their javascript and optimizing out the real shitty bits. Another big one is shared hosting environments - VPS environments are a prime target for trying to leak data from other clients on the same core/cache hierarchy, so those either need to be moved to "secure" cores, or switched to a model of renting out a whole core (or moved to a "hard time slice" where when the slice goes active you get the whole core for X seconds, then the processor stops, flushes everything, then switches clients). But VPS could conceivably be moved to "arrays of little cores" (to the extent that they aren't already) and that won't pose much problem for a lot of typical "micro" use-cases as long as every instance doesn't hit the server at once. Maybe for people that need faster than a dedicated "little" core the next increment becomes leasing the whole core, or even the whole complex of cores on that cache hierarchy. Web application servers (not necessarily databases) are another one, unfortunately, since you can time web requests and use that to "leak" data down different code paths. If it's a directly user-facing service, probably best to get it onto a secure core. The big task for humans is going to be identifying what stuff is allowable to run on the "fast" cores, and then get the schedulers set up so they understand that some stuff can only run in certain processor domains. It's not insurmountable, it just is going to take some time to plug away at it. Perhaps distribute whitelists, and allow the end-user to manually override it if they're really sure. But yes I've been saying that too, my suspicion is that basically all of OoO and speculation is fundamentally incompatible with not leaking timing data between processes, and that the harder we tilt at this the more attacks we're going to turn up, it's going to turn into an endless game of whack-a-mole and it's going to eat up all the performance gains that we've spent the last 20 years building on the backs of OoO and speculation. AMD is quite well-placed for this imo since each CCX basically acts like its own NUCA (non-uniform cache architecture) domain and they just happen to share a memory controller. That's pretty much the design you need to make it work right, just with big and little CCXs instead of only big. They just have to come up with their own little cores. Intel is going to be harder because the classic Sandy Bridge architecture (which is largely unchanged today) has all the cores collectively sharing their last-level cache, and I think that's probably a problem in the long term too. I think Skylake-X still works on the principle of cache being attached to each core and them talking to each other to share it. AMD and Intel engineers, please make your consulting checks out to 'cash'. Thanks! ;)
titoCA321 5y ago
Stop spreading these ideas. Have to keep the academic gravy train churning. Have to publish glorified book report to finish PhD. Vulnerabilities do not stop at processor execution. I have published book report showing how to capture username and password logins from electromagnetic emissions. Please protect yourself and buy my faraday-cage-as-a-service subscription.
Salgat 5y ago
Will this even be an issue when we have CPUs with hundreds/thousands of cores that can just sandbox processes to their own set of cores/cache with exclusive unshared memory?
- MaxBarraclough 5y ago
  I think this idea could be taken further: just build physical machines with lower capacity (RAM, cores), rather than filling data-centers with top-spec hardware then dividing them up with virtualisation. On the face of it at least, this seems like an idea worth taking seriously. With the right form-factor, I imagine it shouldn't even have much of an impact on space efficiency or power efficiency. Perhaps the CPU companies just aren't interested in making such hardware?
  - jimktrains2 5y ago
    And "lower capacity" isn't even that low any more, just in comparison with top-of-the line. Think a raspberry pi or basically any cellphone's main logic board. My motorola g7, that I got for something like $150 new, has Snapdragon 632 processor with 1.8 GHz octa-core CPU and Adreno 506 GPU, 4 GB of ram, and 64 GB internal storage. A pi4, for under $100 has a Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz and up to 8GB or ram. Those specs far outclass most budget VMs and are more than adequate for the vast majority of workloads. All that either is missing is a proper storage port (i.e. not an sd card but something like sata or m3), but otherwise how many raspberry pis could fit in a 1u enclosure? Even being generous and giving half of the volume to disks, dual power, and cooling it's still quite a few. Yes, there are definitely workloads that will benefit from better hardware, e.g. video transcoding or pure number crunching, but i would contend that most websites, databasing, ci, &c could be done on something like a pi replacing a vm or 3 (of the same customer).
  - hderms 5y ago
    Wouldn't that presumably cause energy costs to skyrocket because of all the overhead you get from going from multitenant machines to dedicated ones? Even if the capex is compare I'd imagine it would be hard to get the opex to be competitive
ChuckMcM 5y ago
I think we will eventually see a return to company 'data centers' away from IaaS plays. That said, its a pretty amazing time if you're a computer architect since you now have the transistors to spend on pretty much any crazy scheme you can dream up. So perhaps we'll see 'code safe' computer architectures emerge.
- fauigerzigerk 5y ago
  > I think we will eventually see a return to company 'data centers' away from IaaS plays I don’t see why. Cloud providers have been offering dedicated hardware for a long time. If this problem isn‘t reliably fixable then more customers will make use of these options.
Bancakes 5y ago
20 years ago, computer magazines wrote about single-core 10GHz CPUs. Billions of transistors. What we have can barely be described as performance gains more than "add SIMD and more cores, and performance hacks".
- mhh__ 5y ago
  As opposed to the billion transistors we have now?
  - Bancakes 5y ago
    20 years later, yes. Silicon is endgame. We'll never see 1970s-1990s levels of progress again.
- yjftsjthsd-h 5y ago
  > 20 years ago, computer magazines wrote about single-core 10GHz CPUs Yeah, because they didn't realize how terribly the power consumption / heat output would scale; a 10GHz CPU will just melt itself.
  - iggldiggl 5y ago
    For a humorous take on that, see also James Mickens' The Slow Winter: https://scholar.harvard.edu/files/mickens/files/theslowwinter.pdf https://scholar.harvard.edu/files/mickens/files/theslowwinte...
  - peter_d_sherman 5y ago
    >Yeah, because they didn't realize how terribly the power consumption / heat output would scale; a 10GHz CPU will just melt itself. Tesla, it was claimed, discovered something called "cold electricity" -- that is (according to the claim) -- when you ran it through a circuit -- it cooled rather than heated the circuit! Now, today we have something sort of like this as thermocouples/Peltier Junctions (see https://en.wikipedia.org/wiki/Thermoelectric_cooling https://en.wikipedia.org/wiki/Thermoelectric_cooling for more/better info on this) -- although it is not known if Tesla's "cold electricity" -- was talking about this effect and/or related -- or not. Nonetheless -- it seems to me that IF (and it's a big if!) -- IF Tesla's "Cold Electricity" existed, IF it could be rediscovered, and IF it could somehow be integrated on a CPU either as part of or as auxilliary to the main CPU circuitry -- then the CPU cooling problem could be solved(!) -- or at least mitigated somewhat, to the point of allowing/permitting CPU's with higher thermal envelopes/tolerances/CPU speeds... Again, there are some seriously big IF's there -- but I think it would be a great place for someone to do more research, or for researchers that might have an interest in this area... It seems to me that Intel and AMD (or heck, any chipmaker for that matter!) -- might (or should!) -- have an interest for more research in this area...
indigochill 5y ago
> We're going to wind up undoing the last 20 years of performance gains in the name of 'security', and it scares me. This actually excites me. When the foundation is shown to be rotten, it's time for a new foundation. I'm optimistic, though, that the future holds a fork, with some devices insecure-but-fast and others secure-but-slow. Because there's a market for both. I don't care if my gaming hardware is vulnerable to Spectre because ideally there's nothing worth stealing there anyway. Email/messaging hardware can afford to be a -lot- slower than my gaming rig without any appreciable impact on the experience. Perhaps the future holds motherboards that look like the physical embodiment of Qubes OS, with secure and insecure chips running compartmentalized features based on their security/speed requirements. We already do something like this for performance with the divide between CPUs and GPUs.
- jka 5y ago
  Good thinking, although I'd be a little wary about making a clear distinction between those two classes of device. It'd seem both theoretically and practically possible to engineer hardware that could enable and disable certain optimizations and extensions dynamically. (note that energy consumption may also be a related factor here)
- creatonez 5y ago
  I have a feeling that the wide spectrum of different ways to mitigate CPU bugs will slow down the demand for a new "slow and steady" CPU architecture. Linux already comes with a feature to wipe the L1 cache on every context switch -- simply enabling this option will compete with brand new architectures for a while. A major advantage of not adopting new CPU designs for a while is that you get to keep insecure-but-fast and secure-but-slow behavior in the same CPU by simply tweaking mitigations.
tehbeard 5y ago
Honest question, what's the "Explain like I'm a Freshman" for what they can work out/leak from all the spectre stuff? I see alot about private keys etc etc, but is just a blind attack? Or do you need more info on the target? How quickly can you attack to get info? In essence, is this something Joe Public needs to worry about their $5 vps, or something nefarious using against $CORP's public cloud infrastructure?
varispeed 5y ago
The thing is most project could successfully run on a single dedicated server plus have a one or two spares. There is absolutely no need for a slow virtual nodes. I always thought of those cloud solutions as a clever scam.
- yuliyp 5y ago
  If you see them as a scam you're welcome to not use them. There are still colo facilities out there, and if you don't want to use that there are ISPs that will let you connect to the Internet.
Szpadel 5y ago
I think that cloud providers get dedicated SKUs. I can imagine if you give each VM dedicated cores and you can partition L3 cache per user, you could mitigate most of those issues.
api 5y ago
It’s also possible that we strip off a ton of complexity and then find new performance directions that are better. For my money we’d end up going toward many-core with loads of simple in-order cores on a die. It’d almost look like a GPU. With 5nm how many in-order ARM or RISC-V cores could you put on a chip? You’d also probably move away from shared caches toward each core having more cache and processes having stronger core affinity. That would be both faster and less likely to allow cache timing attacks. You’d have so many cores a core per process would be feasible with sharing only happening at saturation. Another direction would be to go back to trying to crank up clock speed with some new approaches. What could we do with today’s manufacturing techniques if we focused on faster transistors more than smaller ones? AFAIK almost nobody has been working on this since the game has been to use more transistors to implement more features and hacks instead. I read about 10ghz parts on the lab bench in the 2000s. That’s eternity ago in terms of semiconductor process. A 10ghz in-order core would be like a 4X parallel 2.5ghz core… roughly… but more secure and broadly faster on code that’s hard to parallelize. Get rid of speculation and instead give it low branch latency and a ton of on board cache.
- yjftsjthsd-h 5y ago
  > For my money we’d end up going toward many-core with loads of simple in-order cores on a die. That helps with multitasking and parallel-friendly workloads, but lots of stuff isn't easy to make multithreaded.
- ThrowawayR2 5y ago
  > "What could we do with today’s manufacturing techniques if we focused on faster transistors more than smaller ones? AFAIK almost nobody has been working on this since the game has been to use more transistors to implement more features and hacks instead." Plenty of smart people spent lots of money trying it and as it happens the physics doesn't work out. There are countless articles explaining why processor clock speed isn't increasing, e.g. - https://www.maketecheasier.com/why-cpu-clock-speed-isnt-increasing/ https://www.maketecheasier.com/why-cpu-clock-speed-isnt-incr... - https://software.intel.com/content/www/us/en/develop/blogs/why-has-cpu-frequency-ceased-to-grow.html https://software.intel.com/content/www/us/en/develop/blogs/w...
deleted 5y ago
[deleted]
devit 5y ago
It's pretty trivial to make such a CPU: just execute instructions in order and with no cache. The challenge is more how to make a fast CPU like that.
dreamcompiler 5y ago
It's not impossible but it does require some relatively unfamiliar architectural approaches coupled with a lot more use of formal methods. Completely agree about SpecEx. That's a misfeature that needs to die.
fulafel 5y ago
> is not possible to design a CPU such that execution of arbitrary instructions has no observable side-effects, especially if the CPU is speculating As a counter example, how about the 8086?
ForOldHack 5y ago
I agree completely.
tinus_hn 5y ago
It’s perfectly possible, you just can’t take shortcuts, you’ll lose quite a bit of performance. And of course if you really ‘share’ a resource the users are going to know a bit about each other. If each users share grows and shrinks as others are using more or less they are going to know that. But if you don’t want that you could just reserve a fixed part of the resource and they wouldn’t know.
freemint 5y ago
You could very much design and verify that a CPU such that execution of arbitrary code has no observable side effects. However attempting to do so is multi year, for modern CPUs certainly multi decade project. This is not feasible as long as Moore's Law goes on. Formal verification of kernels (SEL4) and a micro processors it runs on have been performed together to proof properties. Large ALU blocks and vector units (such as multipliers) can be formally verified. However efforts of end-to-end formal verification of entire processors won't happen unless there is demand to justify the huge investment of engineering resources and the market would be fine with chips many years behind. Unless everyone is running dozens of clients on all their hardware it is cheaper to give everyone their own machine instead of investing the engineering time in to verifying chips. The economic incentives for Intel that selling more machines for isolation brings them more revenue without having to make investments in to decade long verification projects means it is not a mathematical certainty that it can't happen but just an economic one. After all speculative execution is just extra state and extra logic which can be formally verified. After all programs are just bit patterns and all quarters can be addressed in the formalism of Quantified Boolean Formulas. But nobody is gonna undo 20 years of performance. You will just be told to buy more machines to isolate workloads.
- rsj_hn 5y ago
  The largest system I'm aware of that meets formal verification -- like CC EAL 7 -- are small smart card (Gemalto) Operating Systems. Has it been done on anything bigger?

MarkSweep 5y ago

> if I have untrusted code running on my CPU, I've already lost Don’t forget about JavaScript, a common way for people to run untrusted code on their computers. Not all of micro-architectural data sample are exploitable in JavaScript, but some are.

baybal2 5y ago
Chrome had 7 exploits caught in the wild within 7 weeks in 2020. I believe it is going towards JIT being disabled, or most severely limited.
- userbinator 5y ago
  It sounds like a dream, but going back towards interpreted JS instead of JIT may finally stem the insanity of bloat that JS has evolved in an environment of increasingly fast implementations.
  - otabdeveloper4 5y ago
    The problem of Javascript bloat doesn't have a technical solution. Javascript bloat exists because of a social problem: the guy who fixes the corporate webpage's javascripts is called a "webdesigner", and "webdesigners" are the lowest rung on the corporate IT ladder, maybe only a bit above first-tier techsupport. If you want to make some sort of career you need to upgrade from "webdesigner" to "frontend developer", and that means cryptic, incomprehensible and pointless "frontend frameworks". It provides to value to business or users, but management puts up with it because it fixes the problem of employee churn. (Frontend positions are a big pain in the ass.)
    - tannhaeuser 5y ago
      No reason to insult web developers in general. The simpler explanation is that webapps exist because of an economical problem: that you can make more money (have lower barriers) by either recurring payments for services, or by selling your user's attention, or both.
      - otabdeveloper4 5y ago
        I do lots of frontend development myself, so I'm a kind of "web developer" as well. And I'm insulting web frameworks, not webapps in general.
    - mort96 5y ago
      But JavaScript bloat is allowed to stay (by product managers, middle managers, UX designers, etc) because the website is still fast. If the JS bloat actually caused the site to become too slow on fast machines, people with power to change stuff would demand change.
      - jarcane 5y ago
        People with the power to change stuff thought Java applets were a good idea in 1995, when most computers in wide circulation could barely run the JVM at any acceptable speed. Never trust the tech industry to make optimal decisions, you are only in for a bad time.
- kaba0 5y ago
  That would pretty much instantly kill the web, no matter how lean they make websites.
  - account42 5y ago
    I assure you, there are large parts of the web that do not use excessive amounts of javascript.
    - kaba0 5y ago
      An insignificant portion of a very huge thing can be large.
akersten 5y ago
Yeah, JS is the only hairy part. I considered mentioning it, since I know it was going to come up. But luckily, all I've seen so far are basic demos (like leaky.page) that read data from a carefully-crafted array that the page itself populated. I've yet to be convinced that you could realistically exfiltrate meaningful data at any sort of scale with in-browser JS, especially now that more blatant bugs like Meltdown are fixed. If anyone can show a proof-of-concept ("this page grabs your password manager extension's data") I'll eat my words. But I feel confident that most of these issues are purely academic and, while interesting, serve more to provide content for PhD theses than represent urgent hazards on the web.
- userbinator 5y ago
  Indeed, I've been feeling indifferent about all these timing sidechannels ever since the very first ones (Spectre/Meltdown). The PoCs have not been particularly convincing to me, given that they are extremely contrived and rely on knowing the exact details of the system being exploited to such an extent that someone with those details would be better off with other ways in, and assumes those details haven't changed at all during the amount of time required to do the attack --- the nature of these side-channels is such that even the smallest change in environment can completely change the results. In other words, if I choose a process on my system at random, and dump a few dozen bytes from it, I can technically claim to have leaked some data; but the use of that data to an attacker likely depends strongly on factors which are outside of the attacker's control. It's somewhat like finding a (real-world) key on the ground: you theoretically now have access to something you shouldn't have, but you have next to no idea what that something is. But I feel confident that these issues are purely academic and, while interesting, serve more to provide content for PhD theses than represent urgent hazards on the web. They also provide content for sensationalist clickbait articles and fuel the paranoia that drives society towards authoritarianism and furthers the war on general-purpose-computing, which IMHO is a much bigger issue to worry about.
- brutal_chaos_ 5y ago
  To add more to the hairy part: WASM. And that's growing rapidly.
- 5y ago

baybal2 5y ago

You are completely correct. The safe execution of any untrusted Turing complete code is a pipe dream. You, at least, need a clean sheet CPU design starting from ISA, and basic logic operations formally validated against instruction level analysis to have a fighting chance. But even such chip do get pwned, as shown by key recovery from credit cards in the wild.

mikewarot 5y ago
>The safe execution of any untrusted Turing complete code is a pipe dream. The safe execution of any code requires an operating environment that never trusts the code with more than the least privilege required to complete a task. It has worked in mainframes that way for decades. The IT zeitgeist these days makes me sad. Things can be better, but almost everyone is pushing in counterproductive directions, or has given up hope.
- baybal2 5y ago
  > The safe execution of any code requires an operating environment that never trusts the code with more than the least privilege required to complete a task. It has worked in mainframes that way for decades. It has nothing to do with any OS level security features. We are talking about things happening below the level of what software can see. You just cannot see any sign of such attack by looking at any register the OS can see.
  - mikewarot 5y ago
    These are timing attacks, if you can guarantee code runs deterministically, and deny access to timing information, you can defeat the attack.
    - addaon 5y ago
      Timing attacks are only a subset of side channel attacks, though. One can also imagine thermal attacks -- the amount of power you consume leaks information about what you're doing. And if I share a processor with you, there's various ways I can imagine estimating your power usage. On a processor that has dynamic clocking, the clock speed I'm running at is an indicator of the operations you're doing. Even without dynamic clocking, the probability of an ECC error, for example, is likely to change with temperature. Eliminating timing vulnerabilities is necessary to allow potentially-hostile workloads to share hardware, but it is not sufficient.

paulmd 5y ago

I've been saying since this initially came up that big.LITTLE is the long-term solution for this. In the grand scheme of things, high-intensity tasks are only infrequently high-security tasks - those two sets of workloads are mostly disjoint. So the long-term solution is to have "fast cores" and "secure cores". The fast cores can have all the OoO, speculation, all of that good stuff. That's where you run anything that needs to go fast, or anything running "trusted" code. By and large, nobody cares if an ffmpeg process or HPC node might leak data. Databases? You control the queries that are running on them, right? There are some edge cases like video games where leaking data is moderately harmful (could be useful for exploits if you can reliably leak useful data) yet you still want maximum performance, but at the end of the day leaking data at a couple kB/s usually isn't going to be the end of the world especially if the data is rapidly changing. If the code is untrusted or user-generated, or the data is sufficiently sensitive, then run it on a "secure" core. The "secure" cores have to be in-order, non-speculative, all that crap. Probably non-SMT as that seems to be a bottomless pit of sidechannels as well. But usually, you aren't churning huge workloads in the "secure" situations. You can still have crypto acceleration instructions built into the cores, AVX, whatever, just not speculative. It's probably better to get them fully out of the "normal" cache hierarchy as well. There are a couple obvious problems here, but much smaller than trying to fix everything for every use-case. In particular web browsers are running untrusted code, and every single website is running 15 mb of shitty javascript code. It sucks but it's basically become an inner platform and you can't trust the code that it's bringing in, so that needs to be permanently isolated on its own secure cores. People will have to start paying attention to the performance of their javascript and optimizing out the real shitty bits. Another big one is shared hosting environments - VPS environments are a prime target for trying to leak data from other clients on the same core/cache hierarchy, so those either need to be moved to "secure" cores, or switched to a model of renting out a whole core (or moved to a "hard time slice" where when the slice goes active you get the whole core for X seconds, then the processor stops, flushes everything, then switches clients). But VPS could conceivably be moved to "arrays of little cores" (to the extent that they aren't already) and that won't pose much problem for a lot of typical "micro" use-cases as long as every instance doesn't hit the server at once. Maybe for people that need faster than a dedicated "little" core the next increment becomes leasing the whole core, or even the whole complex of cores on that cache hierarchy. Web application servers (not necessarily databases) are another one, unfortunately, since you can time web requests and use that to "leak" data down different code paths. If it's a directly user-facing service, probably best to get it onto a secure core. The big task for humans is going to be identifying what stuff is allowable to run on the "fast" cores, and then get the schedulers set up so they understand that some stuff can only run in certain processor domains. It's not insurmountable, it just is going to take some time to plug away at it. Perhaps distribute whitelists, and allow the end-user to manually override it if they're really sure. But yes I've been saying that too, my suspicion is that basically all of OoO and speculation is fundamentally incompatible with not leaking timing data between processes, and that the harder we tilt at this the more attacks we're going to turn up, it's going to turn into an endless game of whack-a-mole and it's going to eat up all the performance gains that we've spent the last 20 years building on the backs of OoO and speculation. AMD is quite well-placed for this imo since each CCX basically acts like its own NUCA (non-uniform cache architecture) domain and they just happen to share a memory controller. That's pretty much the design you need to make it work right, just with big and little CCXs instead of only big. They just have to come up with their own little cores. Intel is going to be harder because the classic Sandy Bridge architecture (which is largely unchanged today) has all the cores collectively sharing their last-level cache, and I think that's probably a problem in the long term too. I think Skylake-X still works on the principle of cache being attached to each core and them talking to each other to share it. AMD and Intel engineers, please make your consulting checks out to 'cash'. Thanks! ;)

titoCA321 5y ago

Stop spreading these ideas. Have to keep the academic gravy train churning. Have to publish glorified book report to finish PhD. Vulnerabilities do not stop at processor execution. I have published book report showing how to capture username and password logins from electromagnetic emissions. Please protect yourself and buy my faraday-cage-as-a-service subscription.

Salgat 5y ago

Will this even be an issue when we have CPUs with hundreds/thousands of cores that can just sandbox processes to their own set of cores/cache with exclusive unshared memory?

MaxBarraclough 5y ago
I think this idea could be taken further: just build physical machines with lower capacity (RAM, cores), rather than filling data-centers with top-spec hardware then dividing them up with virtualisation. On the face of it at least, this seems like an idea worth taking seriously. With the right form-factor, I imagine it shouldn't even have much of an impact on space efficiency or power efficiency. Perhaps the CPU companies just aren't interested in making such hardware?
- jimktrains2 5y ago
  And "lower capacity" isn't even that low any more, just in comparison with top-of-the line. Think a raspberry pi or basically any cellphone's main logic board. My motorola g7, that I got for something like $150 new, has Snapdragon 632 processor with 1.8 GHz octa-core CPU and Adreno 506 GPU, 4 GB of ram, and 64 GB internal storage. A pi4, for under $100 has a Quad core Cortex-A72 (ARM v8) 64-bit SoC @ 1.5GHz and up to 8GB or ram. Those specs far outclass most budget VMs and are more than adequate for the vast majority of workloads. All that either is missing is a proper storage port (i.e. not an sd card but something like sata or m3), but otherwise how many raspberry pis could fit in a 1u enclosure? Even being generous and giving half of the volume to disks, dual power, and cooling it's still quite a few. Yes, there are definitely workloads that will benefit from better hardware, e.g. video transcoding or pure number crunching, but i would contend that most websites, databasing, ci, &c could be done on something like a pi replacing a vm or 3 (of the same customer).
- hderms 5y ago
  Wouldn't that presumably cause energy costs to skyrocket because of all the overhead you get from going from multitenant machines to dedicated ones? Even if the capex is compare I'd imagine it would be hard to get the opex to be competitive

ChuckMcM 5y ago

I think we will eventually see a return to company 'data centers' away from IaaS plays. That said, its a pretty amazing time if you're a computer architect since you now have the transistors to spend on pretty much any crazy scheme you can dream up. So perhaps we'll see 'code safe' computer architectures emerge.

fauigerzigerk 5y ago
> I think we will eventually see a return to company 'data centers' away from IaaS plays I don’t see why. Cloud providers have been offering dedicated hardware for a long time. If this problem isn‘t reliably fixable then more customers will make use of these options.

Bancakes 5y ago

20 years ago, computer magazines wrote about single-core 10GHz CPUs. Billions of transistors. What we have can barely be described as performance gains more than "add SIMD and more cores, and performance hacks".

mhh__ 5y ago
As opposed to the billion transistors we have now?
- Bancakes 5y ago
  20 years later, yes. Silicon is endgame. We'll never see 1970s-1990s levels of progress again.
yjftsjthsd-h 5y ago
> 20 years ago, computer magazines wrote about single-core 10GHz CPUs Yeah, because they didn't realize how terribly the power consumption / heat output would scale; a 10GHz CPU will just melt itself.
- iggldiggl 5y ago
  For a humorous take on that, see also James Mickens' The Slow Winter: https://scholar.harvard.edu/files/mickens/files/theslowwinter.pdf https://scholar.harvard.edu/files/mickens/files/theslowwinte...
- peter_d_sherman 5y ago
  >Yeah, because they didn't realize how terribly the power consumption / heat output would scale; a 10GHz CPU will just melt itself. Tesla, it was claimed, discovered something called "cold electricity" -- that is (according to the claim) -- when you ran it through a circuit -- it cooled rather than heated the circuit! Now, today we have something sort of like this as thermocouples/Peltier Junctions (see https://en.wikipedia.org/wiki/Thermoelectric_cooling https://en.wikipedia.org/wiki/Thermoelectric_cooling for more/better info on this) -- although it is not known if Tesla's "cold electricity" -- was talking about this effect and/or related -- or not. Nonetheless -- it seems to me that IF (and it's a big if!) -- IF Tesla's "Cold Electricity" existed, IF it could be rediscovered, and IF it could somehow be integrated on a CPU either as part of or as auxilliary to the main CPU circuitry -- then the CPU cooling problem could be solved(!) -- or at least mitigated somewhat, to the point of allowing/permitting CPU's with higher thermal envelopes/tolerances/CPU speeds... Again, there are some seriously big IF's there -- but I think it would be a great place for someone to do more research, or for researchers that might have an interest in this area... It seems to me that Intel and AMD (or heck, any chipmaker for that matter!) -- might (or should!) -- have an interest for more research in this area...

indigochill 5y ago

> We're going to wind up undoing the last 20 years of performance gains in the name of 'security', and it scares me. This actually excites me. When the foundation is shown to be rotten, it's time for a new foundation. I'm optimistic, though, that the future holds a fork, with some devices insecure-but-fast and others secure-but-slow. Because there's a market for both. I don't care if my gaming hardware is vulnerable to Spectre because ideally there's nothing worth stealing there anyway. Email/messaging hardware can afford to be a -lot- slower than my gaming rig without any appreciable impact on the experience. Perhaps the future holds motherboards that look like the physical embodiment of Qubes OS, with secure and insecure chips running compartmentalized features based on their security/speed requirements. We already do something like this for performance with the divide between CPUs and GPUs.

jka 5y ago
Good thinking, although I'd be a little wary about making a clear distinction between those two classes of device. It'd seem both theoretically and practically possible to engineer hardware that could enable and disable certain optimizations and extensions dynamically. (note that energy consumption may also be a related factor here)
creatonez 5y ago
I have a feeling that the wide spectrum of different ways to mitigate CPU bugs will slow down the demand for a new "slow and steady" CPU architecture. Linux already comes with a feature to wipe the L1 cache on every context switch -- simply enabling this option will compete with brand new architectures for a while. A major advantage of not adopting new CPU designs for a while is that you get to keep insecure-but-fast and secure-but-slow behavior in the same CPU by simply tweaking mitigations.

tehbeard 5y ago

Honest question, what's the "Explain like I'm a Freshman" for what they can work out/leak from all the spectre stuff? I see alot about private keys etc etc, but is just a blind attack? Or do you need more info on the target? How quickly can you attack to get info? In essence, is this something Joe Public needs to worry about their $5 vps, or something nefarious using against $CORP's public cloud infrastructure?

varispeed 5y ago

The thing is most project could successfully run on a single dedicated server plus have a one or two spares. There is absolutely no need for a slow virtual nodes. I always thought of those cloud solutions as a clever scam.

yuliyp 5y ago
If you see them as a scam you're welcome to not use them. There are still colo facilities out there, and if you don't want to use that there are ISPs that will let you connect to the Internet.

Szpadel 5y ago

I think that cloud providers get dedicated SKUs. I can imagine if you give each VM dedicated cores and you can partition L3 cache per user, you could mitigate most of those issues.

api 5y ago

It’s also possible that we strip off a ton of complexity and then find new performance directions that are better. For my money we’d end up going toward many-core with loads of simple in-order cores on a die. It’d almost look like a GPU. With 5nm how many in-order ARM or RISC-V cores could you put on a chip? You’d also probably move away from shared caches toward each core having more cache and processes having stronger core affinity. That would be both faster and less likely to allow cache timing attacks. You’d have so many cores a core per process would be feasible with sharing only happening at saturation. Another direction would be to go back to trying to crank up clock speed with some new approaches. What could we do with today’s manufacturing techniques if we focused on faster transistors more than smaller ones? AFAIK almost nobody has been working on this since the game has been to use more transistors to implement more features and hacks instead. I read about 10ghz parts on the lab bench in the 2000s. That’s eternity ago in terms of semiconductor process. A 10ghz in-order core would be like a 4X parallel 2.5ghz core… roughly… but more secure and broadly faster on code that’s hard to parallelize. Get rid of speculation and instead give it low branch latency and a ton of on board cache.

yjftsjthsd-h 5y ago
> For my money we’d end up going toward many-core with loads of simple in-order cores on a die. That helps with multitasking and parallel-friendly workloads, but lots of stuff isn't easy to make multithreaded.
ThrowawayR2 5y ago
> "What could we do with today’s manufacturing techniques if we focused on faster transistors more than smaller ones? AFAIK almost nobody has been working on this since the game has been to use more transistors to implement more features and hacks instead." Plenty of smart people spent lots of money trying it and as it happens the physics doesn't work out. There are countless articles explaining why processor clock speed isn't increasing, e.g. - https://www.maketecheasier.com/why-cpu-clock-speed-isnt-increasing/ https://www.maketecheasier.com/why-cpu-clock-speed-isnt-incr... - https://software.intel.com/content/www/us/en/develop/blogs/why-has-cpu-frequency-ceased-to-grow.html https://software.intel.com/content/www/us/en/develop/blogs/w...

deleted 5y ago

[deleted]

devit 5y ago

It's pretty trivial to make such a CPU: just execute instructions in order and with no cache. The challenge is more how to make a fast CPU like that.

dreamcompiler 5y ago

It's not impossible but it does require some relatively unfamiliar architectural approaches coupled with a lot more use of formal methods. Completely agree about SpecEx. That's a misfeature that needs to die.

fulafel 5y ago

> is not possible to design a CPU such that execution of arbitrary instructions has no observable side-effects, especially if the CPU is speculating As a counter example, how about the 8086?

ForOldHack 5y ago

I agree completely.

tinus_hn 5y ago

It’s perfectly possible, you just can’t take shortcuts, you’ll lose quite a bit of performance. And of course if you really ‘share’ a resource the users are going to know a bit about each other. If each users share grows and shrinks as others are using more or less they are going to know that. But if you don’t want that you could just reserve a fixed part of the resource and they wouldn’t know.

freemint 5y ago

You could very much design and verify that a CPU such that execution of arbitrary code has no observable side effects. However attempting to do so is multi year, for modern CPUs certainly multi decade project. This is not feasible as long as Moore's Law goes on. Formal verification of kernels (SEL4) and a micro processors it runs on have been performed together to proof properties. Large ALU blocks and vector units (such as multipliers) can be formally verified. However efforts of end-to-end formal verification of entire processors won't happen unless there is demand to justify the huge investment of engineering resources and the market would be fine with chips many years behind. Unless everyone is running dozens of clients on all their hardware it is cheaper to give everyone their own machine instead of investing the engineering time in to verifying chips. The economic incentives for Intel that selling more machines for isolation brings them more revenue without having to make investments in to decade long verification projects means it is not a mathematical certainty that it can't happen but just an economic one. After all speculative execution is just extra state and extra logic which can be formally verified. After all programs are just bit patterns and all quarters can be addressed in the formalism of Quantified Boolean Formulas. But nobody is gonna undo 20 years of performance. You will just be told to buy more machines to isolate workloads.

rsj_hn 5y ago
The largest system I'm aware of that meets formal verification -- like CC EAL 7 -- are small smart card (Gemalto) Operating Systems. Has it been done on anything bigger?