4 ms·

On the comments: "Intel CPUs were tested with Meltdown/Spectre patches, that's probably the discrepancy you're seeing."

by mping 8y ago

onli 8y ago
So every other reviewer is not using those patches (isn't Windows installing them automatically?), and the performance impact is that huge that it completely negates every performance advantage Intel had for now almost a decade?
- citilife 8y ago
  I'm not 100% sure about every other reviewer, but many times tests will be done with a base windows 10 install (as to avoid the need to look at comparisons between patches). Regarding performance impact, short answer is - yes. Basically, it removes most of the advantage of the cache for Intel.
  - blattimwind 8y ago
    > many times tests will be done with a base windows 10 install [without any updates/patches] So, even without Spectre & friends, these benchmarks are essentially meaningless?
    - jdietrich 8y ago
      It's a difficult problem. It's not practical to re-test all of the old CPUs or GPUs every time you review a new part. Either you test all components against a baseline OS and driver set, or you test components with whatever OS and driver releases are contemporaneous. Neither case is ideal. Reviewers have generally settled on the former option; OS and driver updates tend to increase performance, so testing with contemporaneous software might artificially inflate the apparent performance deltas in favour of newer components. Nobody wants to buy a new processor thinking that it's 40% faster than their old one, only to discover that most of the difference was just OS optimisations. You'd rather have your readers be pleasantly surprised than bitterly disappointed, so it makes sense to choose a benchmarking methodology that errs on the side of favouring older parts. Spectre is a weird exception to business as usual, because we saw a huge performance decrease in a single update that affects one particular optimisation in one particular processor generation. In this case, it probably makes sense for reviewers to re-run all of their old benchmarks and set a new baseline.
- mtgx 8y ago
  > isn't Windows installing them automatically? I thought the OS update was only part of the patch and that OEMs also needed to update their BIOS, which means users have to manually update their BIOS.
  - floatboth 8y ago
    Microcode is indeed distributed through Windows Update as well. And packages on other OSes. And applied at boot time.
- hajile 8y ago
  It takes a long time to run a comprehensive test suite against a single processor. Reviewers often have to ship the hardware back to the company afterward. The result is that they very frequently re-use the old benchmarks in newer tests. New software doesn't affect existing benchmarks.
bwat49 8y ago
I could be mistaken, but I recall seeing benchmarks that came out during the whole meltdown/spectre fiasco showed that the effects on games were pretty negligible. I thought it was mainly workloads relating to virtualization etc... that were effected significantly
- onli 8y ago
  Here are two benchmarks supporting this: https://www.eurogamer.net/articles/digitalfoundry-2018-does-patching-cpu-security-flaws-impact-gaming-performance https://www.eurogamer.net/articles/digitalfoundry-2018-does-..., https://www.tomshardware.com/reviews/gaming-performance-meltdown-spectre-intel-amd,5457-5.html https://www.tomshardware.com/reviews/gaming-performance-melt.... I remember it like that as well.
  - citilife 8y ago
    There have been many more updates since (for instance the first article you linked was from 01/10/18): January 29th Microsoft had to revert the Intel patch discribed: https://www.zdnet.com/article/windows-emergency-patch-microsofts-new-update-kills-off-intels-spectre-fix/ https://www.zdnet.com/article/windows-emergency-patch-micros... There's also implementations in March for the Spectre variant 2: https://www.digitaltrends.com/computing/microsoft-windows-patch-skylake-spectre-variant-2/ https://www.digitaltrends.com/computing/microsoft-windows-pa... There's much more in here.
    - bwat49 8y ago
      it looks like anandtech's methodology was flawed: https://www.anandtech.com/show/12678/a-timely-discovery-examining-amd-2nd-gen-ryzen-results/4 https://www.anandtech.com/show/12678/a-timely-discovery-exam...
dragontamer 8y ago
Which is wrong. Video Games don't actually make very many kernel system calls. Meltdown / Spectre patches have virtually no effect on video games. The majority of DirectX stuff in the innermost-loops automatically DMAs to the video card without even needing a kernel call. Video games care a lot about performance and did their best to never touch kernel-space anyway. Compilers, Web Servers, Databases, etc. etc. were affected very strongly. Compilers open files. Web Servers open Sockets. Databases communicate with mutexes / semaphores. All of these are OS-level features and require a kernel-mode switch. Meltdown means that every kernel-mode switch clears out the TLB-cache. So each time a compiler opens a file (MMap or otherwise), it basically loses its entire TLB cache. Every. Single. System call. That's why its a big deal for servers, but not a big deal for video games.
- lightedman 8y ago
  " Meltdown / Spectre patches have virtually no effect on video games." The amount of speculative-execution that games rely upon for many of their components is insane. With the patch applied, I went from a solid 60FPS at 1440x900 in GTAV down to about 48 FPS (as measured by the Steam FPS counter) on a 7th-gen i3.
- localhost 8y ago
  I wonder if moving to a different file format for compilation might minimize the impact of this? For example, what if source code were stored as a single SQLite file? What impact would that have?
  - dragontamer 8y ago
    If anything, it'd be slower. SQLite locks the database file (which would be another OS-level system call). Simple files are more straightforward. You do one MMap and then there wouldn't be any other system calls made. I'd bet that compilers are "slow" because they have to make many, many, many mmaps to do anything. Each #include turns into another MMap, which Meltdown mitigations cause a full TLB flush each time.
    - 8y ago

On the comments: "Intel CPUs were tested with Meltdown/Spectre patches, that's probably the discrepancy you're seeing."

by mping 8y ago

On the comments: "Intel CPUs were tested with Meltdown/Spectre patches, that's probably the discrepancy you're seeing."

onli 8y ago

So every other reviewer is not using those patches (isn't Windows installing them automatically?), and the performance impact is that huge that it completely negates every performance advantage Intel had for now almost a decade?

citilife 8y ago
I'm not 100% sure about every other reviewer, but many times tests will be done with a base windows 10 install (as to avoid the need to look at comparisons between patches). Regarding performance impact, short answer is - yes. Basically, it removes most of the advantage of the cache for Intel.
- blattimwind 8y ago
  > many times tests will be done with a base windows 10 install [without any updates/patches] So, even without Spectre & friends, these benchmarks are essentially meaningless?
  - jdietrich 8y ago
    It's a difficult problem. It's not practical to re-test all of the old CPUs or GPUs every time you review a new part. Either you test all components against a baseline OS and driver set, or you test components with whatever OS and driver releases are contemporaneous. Neither case is ideal. Reviewers have generally settled on the former option; OS and driver updates tend to increase performance, so testing with contemporaneous software might artificially inflate the apparent performance deltas in favour of newer components. Nobody wants to buy a new processor thinking that it's 40% faster than their old one, only to discover that most of the difference was just OS optimisations. You'd rather have your readers be pleasantly surprised than bitterly disappointed, so it makes sense to choose a benchmarking methodology that errs on the side of favouring older parts. Spectre is a weird exception to business as usual, because we saw a huge performance decrease in a single update that affects one particular optimisation in one particular processor generation. In this case, it probably makes sense for reviewers to re-run all of their old benchmarks and set a new baseline.
mtgx 8y ago
> isn't Windows installing them automatically? I thought the OS update was only part of the patch and that OEMs also needed to update their BIOS, which means users have to manually update their BIOS.
- floatboth 8y ago
  Microcode is indeed distributed through Windows Update as well. And packages on other OSes. And applied at boot time.
hajile 8y ago
It takes a long time to run a comprehensive test suite against a single processor. Reviewers often have to ship the hardware back to the company afterward. The result is that they very frequently re-use the old benchmarks in newer tests. New software doesn't affect existing benchmarks.

bwat49 8y ago

I could be mistaken, but I recall seeing benchmarks that came out during the whole meltdown/spectre fiasco showed that the effects on games were pretty negligible. I thought it was mainly workloads relating to virtualization etc... that were effected significantly

onli 8y ago
Here are two benchmarks supporting this: https://www.eurogamer.net/articles/digitalfoundry-2018-does-patching-cpu-security-flaws-impact-gaming-performance https://www.eurogamer.net/articles/digitalfoundry-2018-does-..., https://www.tomshardware.com/reviews/gaming-performance-meltdown-spectre-intel-amd,5457-5.html https://www.tomshardware.com/reviews/gaming-performance-melt.... I remember it like that as well.
- citilife 8y ago
  There have been many more updates since (for instance the first article you linked was from 01/10/18): January 29th Microsoft had to revert the Intel patch discribed: https://www.zdnet.com/article/windows-emergency-patch-microsofts-new-update-kills-off-intels-spectre-fix/ https://www.zdnet.com/article/windows-emergency-patch-micros... There's also implementations in March for the Spectre variant 2: https://www.digitaltrends.com/computing/microsoft-windows-patch-skylake-spectre-variant-2/ https://www.digitaltrends.com/computing/microsoft-windows-pa... There's much more in here.
  - bwat49 8y ago
    it looks like anandtech's methodology was flawed: https://www.anandtech.com/show/12678/a-timely-discovery-examining-amd-2nd-gen-ryzen-results/4 https://www.anandtech.com/show/12678/a-timely-discovery-exam...

dragontamer 8y ago

Which is wrong. Video Games don't actually make very many kernel system calls. Meltdown / Spectre patches have virtually no effect on video games. The majority of DirectX stuff in the innermost-loops automatically DMAs to the video card without even needing a kernel call. Video games care a lot about performance and did their best to never touch kernel-space anyway. Compilers, Web Servers, Databases, etc. etc. were affected very strongly. Compilers open files. Web Servers open Sockets. Databases communicate with mutexes / semaphores. All of these are OS-level features and require a kernel-mode switch. Meltdown means that every kernel-mode switch clears out the TLB-cache. So each time a compiler opens a file (MMap or otherwise), it basically loses its entire TLB cache. Every. Single. System call. That's why its a big deal for servers, but not a big deal for video games.

lightedman 8y ago
" Meltdown / Spectre patches have virtually no effect on video games." The amount of speculative-execution that games rely upon for many of their components is insane. With the patch applied, I went from a solid 60FPS at 1440x900 in GTAV down to about 48 FPS (as measured by the Steam FPS counter) on a 7th-gen i3.
localhost 8y ago
I wonder if moving to a different file format for compilation might minimize the impact of this? For example, what if source code were stored as a single SQLite file? What impact would that have?
- dragontamer 8y ago
  If anything, it'd be slower. SQLite locks the database file (which would be another OS-level system call). Simple files are more straightforward. You do one MMap and then there wouldn't be any other system calls made. I'd bet that compilers are "slow" because they have to make many, many, many mmaps to do anything. Each #include turns into another MMap, which Meltdown mitigations cause a full TLB flush each time.
  - 8y ago