6 ms·

Here are some numbers quantifying the problem. Big caveats apply as they are very preliminary, but the hit due to the software patches looks extremely significa

• 0xcde4c3db 9y ago
  Superficially, it seems like the performance hit mostly scales with IOPS or transactions per second, which might have some pretty serious implications for performance/dollar in the kinds of intensive back-end applications where Intel currently dominates and AMD is trying to make inroads with EPYC.
  • mtanski 9y ago
    As I mentioned in the other thread yesterday database and database like applications are going to be hit particular hard. Even more so on fast flash storage. Double whammy compared to apps just doing network IO. And while databases try to minimize the number of syscalls they still end up doing a lot of them for read, writeout, flush.
    • surajrmal 9y ago
      Don't flash devices use nvme (eg userspace queues) now and avoid the kernel all together for read and write operations? Shouldn't they have no impact?
      • kllrnohj 9y ago
        Not by default they don't. That only works if you're willing to dedicate that entire physical drive to a single application anyway.
        • mtanski 9y ago
          In the future this possible on Linux with the filesystems that support DAX. Currently this all pretty experimental with lots of work being done in this space in the last two years. But this will require you to have the right kind of flash storage, right kind of fs, right kind mount options, and probably a different code path in userspace for DAX vs traditional storage. So we're a little ways away from this.
          • kllrnohj 9y ago
            DAX doesn't appear related here at all. That is about bypassing the page cache for block devices that don't need one. That doesn't move anything from kernel land into userspace, certainly not in the app's process in userspace anyway.
            • mtanski 9y ago
              If you bypass the page cache you do not have read()/write() and mmap you avoid the syscall overhead. This matters a lot for high IOPs devices. Also these new fangled devices claim support word cache line sync using normal cpu flush instructions. Also avoiding fsync syscall.
              • kllrnohj 9y ago
                One does not follow the other. Where are any references to how this will let you bypass read & write? User-space applications are still interacting with a filesystem, which they access via read/write and not a block device. There's no talk in the DAX information about how this results in a zero-syscall filesystem API, and I'm not seeing how that would ever work given there would then be zero protections on anything. You need a handle, and that handle needs security. All of that is done today via syscalls, and DAX isn't changing that interface at all. So where is the API to userspace changing?
                • mtanski 9y ago
                  Please re-read my above comment. There is no new API. The DAX userspace API is mmap. This work is experimental but you can mmap a single file on a filesystem on this device using new DAX capabilities. Most access will not longer require a syscall. This comes with all the usual semantics and trappings of mmap plus some additional caveats as to how the filesystem / DAX / hardware is implemented. Most reads/writes will not require a trip to the kernel using the normal read()/write() syscalls. Additionally, there is no RAM page cache baking this mmap instead the device is mapped directly at a virtual address (like DMA). Finally, flush for these kinds of devices is at the block level implemented using normal instructions and not fsync. Flush is going to be done using the CLWB instruction. See: https://software.intel.com/en-us/blogs/2016/09/12/deprecate-pcommit-instruction https://software.intel.com/en-us/blogs/2016/09/12/deprecate-... LWN.net has lots of articles and links in their archives from 2016/2017. It's a really good read. Sadly I do not have time to dig more of them up for you. Do a search for site:lwn.net and search for DAX or MAP_DIRECT.
      • wtallis 9y ago
        NVMe means each drive can have multiple queues, but they're still managed by a kernel driver. You may be thinking of SPDK, which includes a usermode NVMe driver but requires you to rewrite your application. And many systems are still using SAS or SATA SSDs.
    • 3pt14159 9y ago
      Do we have a performance estimate? I can eat 20 or 30%, but I can't eat 90%.
      • mtanski 9y ago
        This comment further down thread mentions it's 20% in Postgres. https://news.ycombinator.com/item?id=16061926 https://news.ycombinator.com/item?id=16061926
        • eropple 9y ago
          ...when running SELECT 1 over a loopback socket. The reply to that comment is accurate: that's a pathological case. Probably an order of magnitude off.
          • mtanski 9y ago
            We're still learning, but it looks like pgbench is 7% to 15% off: https://www.postgresql.org/message-id/20180102222354.qikjmf7dvnjgbkxe@alap3.anarazel.de https://www.postgresql.org/message-id/20180102222354.qikjmf7...
            • eropple 9y ago
              I've seen that message. It acknowledges the same problems: do-nothing problems over a local unix socket. Real-world use cases introduce much more latency from other sources in the first place. I'm sticking with an expectation in the 2%-5% range.
              • cookiecaper 9y ago
                Yep, this is getting blown way out of proportion by all of these tiny scripts that just sit around connecting to themselves. Even pgbench is theoretical and intended for tuning; you're not going to hit your max tps in your Real Code that is doing Real Work. In the real world, where code is doing real things besides just entering/exiting itself all day, I think it's going to be a stretch to see even a 5% performance impact, let alone 10%.
    • panarky 9y ago
      How would you trade this knowledge? Intel has already dropped and AMD is up. Maybe there's more to move, but first-order effects are at least partially priced in already. But what about second-order effects? Seems like virtualization should be vulnerable (VMWare and Citrix), but maybe they actually benefit as customers add more capacity. Software-defined networking and cloud databases should also suffer though it's unclear how to trade these. AWS, Google Cloud and Azure might benefit as customers add capacity but there's no way to trade the business units. So what about cloud customers where compute costs are already a large percentage of total revenue? Netflix should be OK but Snap and Twilio could get squeezed hard. Akamai and Cloudflare might have higher costs they can't pass through to customers. And where's the upside? Who benefits? If the performance hit causes companies to add capacity, maybe semiconductor and DRAM suppliers like Micron would benefit.
      • rufugee 9y ago
        Not sure why you are being downvoted. It's an interesting question. I'm in on AMD for the time, just to see how to flows.
      • mtanski 9y ago
        I think the model for cloud vendors would be quite complicated. Not every version of the CPU and not every application is impacted as much (new intel processors with PCID will suffer less). Add on top of that the fact that a lot cloud customers over provision (there's good scientific papers on how much spare CPU capacity there is). Cloud service providers that sell things on a per request / real CPU usage model (vs reserved capacity) prob benefit more. Also, you can't just separate trading in AWS or GCE from the rest of the core business. Potentially business units of DELL, HP, IBM, ... should do better as people use this as a justification to upgrade overdue hardware they should cover 5% to 10% lower performance (needing more units to cover that).
        • user5994461 9y ago
          Agree on that last paragraph. The only reasonable thing people can do is buy more hardware to cover the performance loss and/or buy more hardware that's not needed using the bug as a pretext to get the budget approved now.
  • eptcyka 9y ago
    It has very little to do with what kind of syscalls (I/O or other kinds) and all to do with how many syscalls a given application makes per given time period. Compute bound applications are already avoiding syscalls in their hotter parts. This will mostly be a blow to databases, caching servers and other such I/O limited applications.
    • bunderbunder 9y ago
      In other words, don't worry - pretty much all the key performance bottlenecks most of us deal with at work will be getting tighter, but at least our video games will still run OK.
      • josefx 9y ago
        Right now I am just hoping that it wont add significant overhead to OpenGL. My application already has a bottleneck on changing OpenGL states and issuing render commands and I have no idea how much of that time is spend making syscalls.
        • elFarto 9y ago
          OpenGL implementations shouldn't be effected by syscall overhead. Historically it's been DirectX that had a syscall per draw call, but I believe both now just write the command to GPU RAM directly.
          • bartread 9y ago
            That's at least somewhat encouraging. Nevertheless, it sounds rather like the old, "Yes, but will it run Crysis?" question will perhaps have renewed relevance.
      • user5994461 9y ago
        Well, sounds like that time when there is no water left but there are still beers in the fridge. At least we can play our sorrows away.
    • 9y ago
  • lulmerchant 9y ago
    All the Oracle DBAs out there are in for some suffering. Forget the cost of 30% extra compute, what about the 30% increase to Oracle licensing?
    • paulmd 9y ago
      "SPARC user: not affected!" --Oracle's marketing tomorrow, probably (to their credit, SPARC does fully isolate kernel and user memory pages, so they were ahead of the curve here... for all 10 of their users who run anything other than Oracle DB on their systems.)
• api 9y ago
  A lot of people have noticed that High Sierra is slower than Sierra, specifically for filesystem operations with APFS. I wonder if Apple knew about this ahead of time and this explains the overhead?
  • InTheArena 9y ago
    Probably not. APFS just does a lot more then HFS, so there is a huge performance impact on disk related issues before this change goes in. This is a all hands on deck kind of situation. Apple doesn't usually do well with security firedrills like this.
    • awesomevibes1 9y ago
      I hope somebody on Twitter has informed Apple about this.
      • saagarjha 9y ago
        Unless they live in a bubble, I’m sure most people at Apple are already aware of this.
        • collinvandyck76 9y ago
          It was a joke referencing the login vulnerability that was disclosed on twitter a couple of months ago.
    • baq 9y ago
      If work in NT and Linux kernels started on November, they must know. Intel must've told them, the alternative of Apple learning about this from a third party and grilling them over it would be too scary.
      • iainmerrick 9y ago
        I just wonder whether Apple is threatening to move all their Macs to AMD, or to ARM?
• cookiecaper 9y ago
  Phoronix strikes again! I admire Michael's consistency and dedication and their benchmarks have certainly gotten better over the years as PTS has matured, but everything on Phoronix still needs to be taken with a generous helping of salt. New readers generally learn this after a few months; it applies not only to their benchmarks, but also their "news". The most obvious issue with this benchmark is that Phoronix is testing the latest rcs, with all of their changes, against the last stable version [EDIT: I misread or this changed overnight, see below] that doesn't have PTI integrated, instead of just isolating the PTI patchset. The right way to do this would be to use the same kernel version and either cherry-pick the specific patches or trust that the `nopti` boot parameter sufficiently disables the feature. That alone makes the test worthless. There is no way this causes a universal 30% perf deduction, especially not for workloads that are IO-bound (i.e., most real-world workloads). This is a significant hit for Intel, but it's not going to reduce global compute capacity by 30% overnight. EDIT: Looking at the Phoronix page, the benchmark actually appears to use 4.15-rc5 as "pre" and 4.15-some-unspecified-git-pull-from-Dec-31-that-isn't-called-rc6 as "post". I thought I had read 4.14.8 there last night, but may not have. Regardless, the point stands -- these are different versions of the kernel and the tests do not reflect the impact of the PTI patchset.
  • ricardobeat 9y ago
    So you’re saying that the latest RCS, without the patch, was supposed to be slower than stable by at least 10%? How often do companies release performance downgrades of that scale? That’s also very unlikely.
    • cookiecaper 9y ago
      >So you’re saying that the latest RCS, without the patch, was supposed to be slower than stable by at least 10%? I'm saying that it's not a reliable measurement of the impact of the PTI patchset. There was a PgSQL performance anecdote [0] (actually tested with the real boot parameters instead of entirely different versions of the kernel) that showed 9% performance decrease posted to LKML, which Linus described as "pretty much in line with expectations". [1] Quoting further from that mail: > Something around 5% performance impact of the isolation is what people are looking at. > Obviously it depends on just exactly what you do. Some loads will hardly be affected at all, if they just spend all their time in user space. And if you do a lot of small system calls, you might see double-digit slowdowns. So in general, the hit should be around 5%, and "[y]ou might see double-digit slowdowns" seems like the hit on a worst-case workload is hovering closer to the 10% range than 30%. That's also what the anecdote from LKML shows, unlike Phoronix which shows 25%-30% or worse. This is more of an attrition thing than a staggering loss. With people saying MS patched this in November, it would be interesting to see if people saw a similar 5-10% degradation in Windows benchmarks since that time. >How often do companies release performance downgrades of that scale? I don't know which "company" you're referring to here, but substantial changes in kernel performance characteristics are pretty common during the Linux development/RC process, and yes, definitely some workloads will often see changes +/- 10% between the roughly bi-monthly stable kernel releases. If you're surprised that Linux development is so "lively", you're not alone. That's one of the selling points of other OSes like FreeBSD. [0] https://lkml.org/lkml/2018/1/2/678 https://lkml.org/lkml/2018/1/2/678 [1] https://lkml.org/lkml/2018/1/2/703 https://lkml.org/lkml/2018/1/2/703
• cmiles74 9y ago
  I wonder if we'll see some performance return as subsequent patches are produced. I can't tell from the coverage so far if this is possible.
• deleted 9y ago
  [deleted]