3 ms·

Phoronix strikes again! I admire Michael's consistency and dedication and their benchmarks have certainly gotten better over the years as PTS has matured, but e

Phoronix strikes again! I admire Michael's consistency and dedication and their benchmarks have certainly gotten better over the years as PTS has matured, but everything on Phoronix still needs to be taken with a generous helping of salt. New readers generally learn this after a few months; it applies not only to their benchmarks, but also their "news". The most obvious issue with this benchmark is that Phoronix is testing the latest rcs, with all of their changes, against the last stable version [EDIT: I misread or this changed overnight, see below] that doesn't have PTI integrated, instead of just isolating the PTI patchset. The right way to do this would be to use the same kernel version and either cherry-pick the specific patches or trust that the `nopti` boot parameter sufficiently disables the feature. That alone makes the test worthless. There is no way this causes a universal 30% perf deduction, especially not for workloads that are IO-bound (i.e., most real-world workloads). This is a significant hit for Intel, but it's not going to reduce global compute capacity by 30% overnight. EDIT: Looking at the Phoronix page, the benchmark actually appears to use 4.15-rc5 as "pre" and 4.15-some-unspecified-git-pull-from-Dec-31-that-isn't-called-rc6 as "post". I thought I had read 4.14.8 there last night, but may not have. Regardless, the point stands -- these are different versions of the kernel and the tests do not reflect the impact of the PTI patchset.

ricardobeat 9y ago
So you’re saying that the latest RCS, without the patch, was supposed to be slower than stable by at least 10%? How often do companies release performance downgrades of that scale? That’s also very unlikely.
- cookiecaper 9y ago
  >So you’re saying that the latest RCS, without the patch, was supposed to be slower than stable by at least 10%? I'm saying that it's not a reliable measurement of the impact of the PTI patchset. There was a PgSQL performance anecdote [0] (actually tested with the real boot parameters instead of entirely different versions of the kernel) that showed 9% performance decrease posted to LKML, which Linus described as "pretty much in line with expectations". [1] Quoting further from that mail: > Something around 5% performance impact of the isolation is what people are looking at. > Obviously it depends on just exactly what you do. Some loads will hardly be affected at all, if they just spend all their time in user space. And if you do a lot of small system calls, you might see double-digit slowdowns. So in general, the hit should be around 5%, and "[y]ou might see double-digit slowdowns" seems like the hit on a worst-case workload is hovering closer to the 10% range than 30%. That's also what the anecdote from LKML shows, unlike Phoronix which shows 25%-30% or worse. This is more of an attrition thing than a staggering loss. With people saying MS patched this in November, it would be interesting to see if people saw a similar 5-10% degradation in Windows benchmarks since that time. >How often do companies release performance downgrades of that scale? I don't know which "company" you're referring to here, but substantial changes in kernel performance characteristics are pretty common during the Linux development/RC process, and yes, definitely some workloads will often see changes +/- 10% between the roughly bi-monthly stable kernel releases. If you're surprised that Linux development is so "lively", you're not alone. That's one of the selling points of other OSes like FreeBSD. [0] https://lkml.org/lkml/2018/1/2/678 https://lkml.org/lkml/2018/1/2/678 [1] https://lkml.org/lkml/2018/1/2/703 https://lkml.org/lkml/2018/1/2/703

ricardobeat 9y ago

So you’re saying that the latest RCS, without the patch, was supposed to be slower than stable by at least 10%? How often do companies release performance downgrades of that scale? That’s also very unlikely.

cookiecaper 9y ago
>So you’re saying that the latest RCS, without the patch, was supposed to be slower than stable by at least 10%? I'm saying that it's not a reliable measurement of the impact of the PTI patchset. There was a PgSQL performance anecdote [0] (actually tested with the real boot parameters instead of entirely different versions of the kernel) that showed 9% performance decrease posted to LKML, which Linus described as "pretty much in line with expectations". [1] Quoting further from that mail: > Something around 5% performance impact of the isolation is what people are looking at. > Obviously it depends on just exactly what you do. Some loads will hardly be affected at all, if they just spend all their time in user space. And if you do a lot of small system calls, you might see double-digit slowdowns. So in general, the hit should be around 5%, and "[y]ou might see double-digit slowdowns" seems like the hit on a worst-case workload is hovering closer to the 10% range than 30%. That's also what the anecdote from LKML shows, unlike Phoronix which shows 25%-30% or worse. This is more of an attrition thing than a staggering loss. With people saying MS patched this in November, it would be interesting to see if people saw a similar 5-10% degradation in Windows benchmarks since that time. >How often do companies release performance downgrades of that scale? I don't know which "company" you're referring to here, but substantial changes in kernel performance characteristics are pretty common during the Linux development/RC process, and yes, definitely some workloads will often see changes +/- 10% between the roughly bi-monthly stable kernel releases. If you're surprised that Linux development is so "lively", you're not alone. That's one of the selling points of other OSes like FreeBSD. [0] https://lkml.org/lkml/2018/1/2/678 https://lkml.org/lkml/2018/1/2/678 [1] https://lkml.org/lkml/2018/1/2/703 https://lkml.org/lkml/2018/1/2/703