4 ms·

This is a great paper, really well explained. The source code: https://github.com/FPSG-UIUC/lotr https://github.com/FPSG-UIUC/lotr The TL;DR is that the L3 cac

This is a great paper, really well explained. The source code: https://github.com/FPSG-UIUC/lotr https://github.com/FPSG-UIUC/lotr The TL;DR is that the L3 cache (LLC: Last Level Cache) is shared between all cores on the chip, but the L3 is composed of a number of slices colocated with each core: the L3 is actually CC-NUMA! There is contention when reading/writing to the L3 via the ring interconnect, which can be used to identify the memory addresses access patterns of the cache. There's a bit more covering how the L1/L2 caches, private to each core, are revealed via the set inclusivity properly of Intel's cache design. In monitoring LLC it is similar to https://eprint.iacr.org/2015/898.pdf https://eprint.iacr.org/2015/898.pdf though it doesn't reference them. It isn't a game over attack but it reinforces how sharing resources (cache, memory, cores) is a bad idea when that crosses a security boundary. VM exclusive cache regions (Intel CAT cache partitioning) and cache locking should help, but cache slice partitioning is reqd too.

db48x 6y ago
We should just stop running multiple programs on the same hardware.
- adgjlsfhk1 6y ago
  it seems like it might be totally feasible to make a CPU where one core runs all the Ring 0 stuff, and the rest don't.
  - toast0 6y ago
    You don't really need a CPU to do that. Just operating system decisions. a) route all hardware interrupts to CPU 0 only b) when a user thread does a syscall, immediately task switch to another thread, marking the current thread as entering/in the kernel. Only service threads entering/in the kernel on CPU 0. Technically, you need a little bit of ring 0 time to task switch. But if you make all of the syscalls amazingly slow, timing attacks are a lot harder.
    - angry_octet 6y ago
      There are some patches out there that allow you to lock kernel threads to particular cores. With all the cache flushes on context switch, making syscalls slow should be no problem at all.
      - KMag 6y ago
        My memory could very well be wrong here, but isn't this how multi-core was first implemented on FreeBSD, with all post-context-switch kernel code running on CPU0? It's certainly a much more simple design, easier to avoid data races and deadlocks. However, I believe the performance penalties were pretty apparent early on, even on 2-4 core systems.
        toast0 6y ago
        I wasn't that understanding of kernel stuff back then, but I don't recall seeing anything that the kernel was always running on CPU0; but I think everything was behind the GIANT lock, so there was effectively zero kernel concurrency.
    - samus 6y ago
      I like that idea! There's already a large slowdown because of communication and NUMA. The syscall arguments have to be transferred to CPU 0 memory as well.
  - elihu 6y ago
    I just skimmed a little bit of the paper so I'm probably missing a lot of context, but would restricting to one core even help in this instance? Information is leaking via the L3 cache, which is shared.
- 01100011 6y ago
  Perhaps we should stop running multiple programs on the same hardware at the same time while making sure information isn't leaked between domains? If the future provides us with large but thermally limited dies, I could see a scenario where CPU components are duplicated but not all running at once. A thread or set of threads could own a hierarchy of caches, for example. Caches are die-intensive, so I don't see this happening soon, but maybe at some point in the future.
- elihu 6y ago
  Alternatively: maybe eliminating observable side-effects from cache behavior by restricting accurate timing information to privileged processes is the way to go. Though I imagine that's a lot easier done at the level of, say, a Javascript interpreter than at the level of raw machine code. There's probably a lot of indirect ways of figuring out how long something took when you can run arbitrary instructions and/or you're allowed to run multiple threads that share memory. Even being able to launch two threads and find out which one finished first can be used to construct a crude stopwatch.
  - db48x 6y ago
    Yea, that's a good way to go in the medium term. I like the Mill design, where every fetch from memory can return NaR (Not a Result, which is a little like a floating–point NaN), and all operations on a NaR take the same amount of time as normal but pass the NaR through. Also, they made sure to put the memory protection _before_ the cache, so when you don't have access to the memory you get the NaR before it even consults the cache. Looks like there are some nice performance benefits too. In the long term though? Who knows. Separate cpu+cache+memory hardware for every process seems a little crazy, but the alternatives might be worse.
  - the8472 6y ago
    > or you're allowed to run multiple threads that share memory. Indeed, you can always spawn a helper thread that does nothing but incrementing a counter on shared memory and use that as clock substitute. That's why browsers disabled shared arrays after spectre was revealed. So this isn't practical for any programs that need fine-grained parallelism.
- twic 6y ago
  And for additional security, reduce the program count by one beyond that.
- izacus 6y ago
  Original iOS and iPads were right - you should only be able to see a single piece of software and everything else should stop. This will make you safe and secure.
est31 6y ago
> VM exclusive cache regions and cache locking should help, but cache slice partitioning is reqd too. Note that in this instance the shared region is the bus used to access the cache. Reserving a cache region won't help with that. One would have to reserve time slices on the bus.
- angry_octet 6y ago
  But if the cache is in a different set, and each security partition has exclusive access to particular slices, you can't get contention because access is forbidden. Also, time slicing might not be sufficient due to the queues in each LLC slice. If you can fill the queue the bus access will retry I guess.

db48x 6y ago

We should just stop running multiple programs on the same hardware.

adgjlsfhk1 6y ago
it seems like it might be totally feasible to make a CPU where one core runs all the Ring 0 stuff, and the rest don't.
- toast0 6y ago
  You don't really need a CPU to do that. Just operating system decisions. a) route all hardware interrupts to CPU 0 only b) when a user thread does a syscall, immediately task switch to another thread, marking the current thread as entering/in the kernel. Only service threads entering/in the kernel on CPU 0. Technically, you need a little bit of ring 0 time to task switch. But if you make all of the syscalls amazingly slow, timing attacks are a lot harder.
  - angry_octet 6y ago
    There are some patches out there that allow you to lock kernel threads to particular cores. With all the cache flushes on context switch, making syscalls slow should be no problem at all.
    - KMag 6y ago
      My memory could very well be wrong here, but isn't this how multi-core was first implemented on FreeBSD, with all post-context-switch kernel code running on CPU0? It's certainly a much more simple design, easier to avoid data races and deadlocks. However, I believe the performance penalties were pretty apparent early on, even on 2-4 core systems.
      - toast0 6y ago
        I wasn't that understanding of kernel stuff back then, but I don't recall seeing anything that the kernel was always running on CPU0; but I think everything was behind the GIANT lock, so there was effectively zero kernel concurrency.
  - samus 6y ago
    I like that idea! There's already a large slowdown because of communication and NUMA. The syscall arguments have to be transferred to CPU 0 memory as well.
- elihu 6y ago
  I just skimmed a little bit of the paper so I'm probably missing a lot of context, but would restricting to one core even help in this instance? Information is leaking via the L3 cache, which is shared.
01100011 6y ago
Perhaps we should stop running multiple programs on the same hardware at the same time while making sure information isn't leaked between domains? If the future provides us with large but thermally limited dies, I could see a scenario where CPU components are duplicated but not all running at once. A thread or set of threads could own a hierarchy of caches, for example. Caches are die-intensive, so I don't see this happening soon, but maybe at some point in the future.
elihu 6y ago
Alternatively: maybe eliminating observable side-effects from cache behavior by restricting accurate timing information to privileged processes is the way to go. Though I imagine that's a lot easier done at the level of, say, a Javascript interpreter than at the level of raw machine code. There's probably a lot of indirect ways of figuring out how long something took when you can run arbitrary instructions and/or you're allowed to run multiple threads that share memory. Even being able to launch two threads and find out which one finished first can be used to construct a crude stopwatch.
- db48x 6y ago
  Yea, that's a good way to go in the medium term. I like the Mill design, where every fetch from memory can return NaR (Not a Result, which is a little like a floating–point NaN), and all operations on a NaR take the same amount of time as normal but pass the NaR through. Also, they made sure to put the memory protection _before_ the cache, so when you don't have access to the memory you get the NaR before it even consults the cache. Looks like there are some nice performance benefits too. In the long term though? Who knows. Separate cpu+cache+memory hardware for every process seems a little crazy, but the alternatives might be worse.
- the8472 6y ago
  > or you're allowed to run multiple threads that share memory. Indeed, you can always spawn a helper thread that does nothing but incrementing a counter on shared memory and use that as clock substitute. That's why browsers disabled shared arrays after spectre was revealed. So this isn't practical for any programs that need fine-grained parallelism.
twic 6y ago
And for additional security, reduce the program count by one beyond that.
izacus 6y ago
Original iOS and iPads were right - you should only be able to see a single piece of software and everything else should stop. This will make you safe and secure.

est31 6y ago

> VM exclusive cache regions and cache locking should help, but cache slice partitioning is reqd too. Note that in this instance the shared region is the bus used to access the cache. Reserving a cache region won't help with that. One would have to reserve time slices on the bus.

angry_octet 6y ago
But if the cache is in a different set, and each security partition has exclusive access to particular slices, you can't get contention because access is forbidden. Also, time slicing might not be sufficient due to the queues in each LLC slice. If you can fill the queue the bus access will retry I guess.