6 ms·

This sounds interesting, but why should this be a new piece of hardware as opposed to a new OS service? Are these functions simply so specialized that implement

by shimon 11y ago

This sounds interesting, but why should this be a new piece of hardware as opposed to a new OS service? Are these functions simply so specialized that implementing them in the OS would be a bottleneck (even though the CPU has plenty of free cycles)?

jerf 11y ago
"This sounds interesting, but why should this be a new piece of hardware as opposed to a new OS service?" Because the entire point is that CPUs are too slow by themselves, even without the OS, let alone with it. While you were context-switching into this OS server you missed the chance to do 10 IOPs give or take an order of magnitude. Yet the OS really can't go anywhere. We can sometimes poke a hole here and sometimes poke a hole there but in general they're there for good reasons and not going anywhere, just as no matter what crazy things we bodge in to our computers "things like CPUs" aren't going anywhere either, and my guess is they're likely to stay pretty "central", too.
- amluto 11y ago
  > While you were context-switching into this OS server you missed the chance to do 10 IOPs give or take an order of magnitude. I'll believe that when I see real numbers. A system call takes something like 54 ns on my laptop. With pwritev or similar, you can do quite a few IOs in a system call. (Of course, pwritev is slower than 54 ns, but that's not a fundamental constraint.) An IO requires making the IO durable if you want it to be reliably persistent. So you have to do CLWB; SFENCE; PCOMMIT; SFENCE or whatever magic sequence you're using (depends on IO type and use of nontemporal instructions, (and you have to have hardware that supports that). If you're using NVMe instead of NVDIMMs, then you have to do an IO to sync with the controller, and that IO will be uncached. Uncached IO is slow. PCOMMIT has unknown performance since no one has the hardware yet. System calls are fast.
  - uxcn 11y ago
    One of the other constraints is the actual data copy. I don't have any benchmarks on hand, but you pay the penalty for the copy, potentially cache misses, and the potential TLB miss. Obviously, there are ways to avoid it without resorting to bypassing the kernel, but there's still a non-negligible cost. Maybe it would be beneficial to have a coherent interface as well, considering NVMe.
  - eloff 11y ago
    The syscall overhead isn't the problem, that's dirt cheap as you say. The problem is the context-switch overhead. Calling into the OS flushes a lot of data and instructions from the cache, and that lost performance after returning can easily add up to around 30µs.[1] [1] http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-ma...
    - CamperBob2 11y ago
      Sounds like multiple caches are an obvious solution. No code in the OS needs access to the user code or data cache, and vice versa. Smart cache management might be one of Intel's goals for the Altera buyout.
      - eloff 11y ago
        It's already here[1], but I don't think it can be used for cache partitioning between the kernel and userspace (but I could be wrong!) [1] http://danluu.com/intel-cat/ http://danluu.com/intel-cat/
        hadagribble 11y ago
        (Disclosure: one of the authors here) I don't think CAT can be used to partition kernel and userspace -- I'm not even sure how you'd go about doing that given that you can (and do) have shared pages between them. That being said, from our experiments, if you're using userspace network and NVMe drivers the context switch and associated cache pollution is not a problem, since it is happening pretty infrequently (primarily just timer interrupts, and those can be turned off, but we haven't needed to).
      - xxs 11y ago
        >> No code in the OS needs access to the user code or data cache This is not true for the data. How do you pass any data structures outside CPU registers then, say the path to a file to open. Normally it's a char*[0] (indeed passed in a register) but then the OS actually reads the data off the process memory (L1 data cache usually) [0]: http://linux.die.net/man/3/open http://linux.die.net/man/3/open
- xxs 11y ago
  You do not context switch to the OS, it's a mode switch x2 but it's cheaper than a context switch.
woodchuck64 11y ago
Well the article does say: "Our own experience has been that efforts to saturate PCIe flash devices often require optimizations to existing storage subsystems, and then consume large amounts of CPU cycles. "

by shimon 11y ago

jerf 11y ago

"This sounds interesting, but why should this be a new piece of hardware as opposed to a new OS service?" Because the entire point is that CPUs are too slow by themselves, even without the OS, let alone with it. While you were context-switching into this OS server you missed the chance to do 10 IOPs give or take an order of magnitude. Yet the OS really can't go anywhere. We can sometimes poke a hole here and sometimes poke a hole there but in general they're there for good reasons and not going anywhere, just as no matter what crazy things we bodge in to our computers "things like CPUs" aren't going anywhere either, and my guess is they're likely to stay pretty "central", too.

amluto 11y ago
> While you were context-switching into this OS server you missed the chance to do 10 IOPs give or take an order of magnitude. I'll believe that when I see real numbers. A system call takes something like 54 ns on my laptop. With pwritev or similar, you can do quite a few IOs in a system call. (Of course, pwritev is slower than 54 ns, but that's not a fundamental constraint.) An IO requires making the IO durable if you want it to be reliably persistent. So you have to do CLWB; SFENCE; PCOMMIT; SFENCE or whatever magic sequence you're using (depends on IO type and use of nontemporal instructions, (and you have to have hardware that supports that). If you're using NVMe instead of NVDIMMs, then you have to do an IO to sync with the controller, and that IO will be uncached. Uncached IO is slow. PCOMMIT has unknown performance since no one has the hardware yet. System calls are fast.
- uxcn 11y ago
  One of the other constraints is the actual data copy. I don't have any benchmarks on hand, but you pay the penalty for the copy, potentially cache misses, and the potential TLB miss. Obviously, there are ways to avoid it without resorting to bypassing the kernel, but there's still a non-negligible cost. Maybe it would be beneficial to have a coherent interface as well, considering NVMe.
- eloff 11y ago
  The syscall overhead isn't the problem, that's dirt cheap as you say. The problem is the context-switch overhead. Calling into the OS flushes a lot of data and instructions from the cache, and that lost performance after returning can easily add up to around 30µs.[1] [1] http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-ma...
  - CamperBob2 11y ago
    Sounds like multiple caches are an obvious solution. No code in the OS needs access to the user code or data cache, and vice versa. Smart cache management might be one of Intel's goals for the Altera buyout.
    - eloff 11y ago
      It's already here[1], but I don't think it can be used for cache partitioning between the kernel and userspace (but I could be wrong!) [1] http://danluu.com/intel-cat/ http://danluu.com/intel-cat/
      - hadagribble 11y ago
        (Disclosure: one of the authors here) I don't think CAT can be used to partition kernel and userspace -- I'm not even sure how you'd go about doing that given that you can (and do) have shared pages between them. That being said, from our experiments, if you're using userspace network and NVMe drivers the context switch and associated cache pollution is not a problem, since it is happening pretty infrequently (primarily just timer interrupts, and those can be turned off, but we haven't needed to).
    - xxs 11y ago
      >> No code in the OS needs access to the user code or data cache This is not true for the data. How do you pass any data structures outside CPU registers then, say the path to a file to open. Normally it's a char*[0] (indeed passed in a register) but then the OS actually reads the data off the process memory (L1 data cache usually) [0]: http://linux.die.net/man/3/open http://linux.die.net/man/3/open
xxs 11y ago
You do not context switch to the OS, it's a mode switch x2 but it's cheaper than a context switch.

woodchuck64 11y ago

Well the article does say: "Our own experience has been that efforts to saturate PCIe flash devices often require optimizations to existing storage subsystems, and then consume large amounts of CPU cycles. "