4 ms·

So every other reviewer is not using those patches (isn't Windows installing them automatically?), and the performance impact is that huge that it completely ne

by onli 9y ago

So every other reviewer is not using those patches (isn't Windows installing them automatically?), and the performance impact is that huge that it completely negates every performance advantage Intel had for now almost a decade?

citilife 9y ago
I'm not 100% sure about every other reviewer, but many times tests will be done with a base windows 10 install (as to avoid the need to look at comparisons between patches). Regarding performance impact, short answer is - yes. Basically, it removes most of the advantage of the cache for Intel.
- blattimwind 9y ago
  > many times tests will be done with a base windows 10 install [without any updates/patches] So, even without Spectre & friends, these benchmarks are essentially meaningless?
  - jdietrich 9y ago
    It's a difficult problem. It's not practical to re-test all of the old CPUs or GPUs every time you review a new part. Either you test all components against a baseline OS and driver set, or you test components with whatever OS and driver releases are contemporaneous. Neither case is ideal. Reviewers have generally settled on the former option; OS and driver updates tend to increase performance, so testing with contemporaneous software might artificially inflate the apparent performance deltas in favour of newer components. Nobody wants to buy a new processor thinking that it's 40% faster than their old one, only to discover that most of the difference was just OS optimisations. You'd rather have your readers be pleasantly surprised than bitterly disappointed, so it makes sense to choose a benchmarking methodology that errs on the side of favouring older parts. Spectre is a weird exception to business as usual, because we saw a huge performance decrease in a single update that affects one particular optimisation in one particular processor generation. In this case, it probably makes sense for reviewers to re-run all of their old benchmarks and set a new baseline.
mtgx 9y ago
> isn't Windows installing them automatically? I thought the OS update was only part of the patch and that OEMs also needed to update their BIOS, which means users have to manually update their BIOS.
- floatboth 9y ago
  Microcode is indeed distributed through Windows Update as well. And packages on other OSes. And applied at boot time.
hajile 9y ago
It takes a long time to run a comprehensive test suite against a single processor. Reviewers often have to ship the hardware back to the company afterward. The result is that they very frequently re-use the old benchmarks in newer tests. New software doesn't affect existing benchmarks.

by onli 9y ago

citilife 9y ago

I'm not 100% sure about every other reviewer, but many times tests will be done with a base windows 10 install (as to avoid the need to look at comparisons between patches). Regarding performance impact, short answer is - yes. Basically, it removes most of the advantage of the cache for Intel.

blattimwind 9y ago
> many times tests will be done with a base windows 10 install [without any updates/patches] So, even without Spectre & friends, these benchmarks are essentially meaningless?
- jdietrich 9y ago
  It's a difficult problem. It's not practical to re-test all of the old CPUs or GPUs every time you review a new part. Either you test all components against a baseline OS and driver set, or you test components with whatever OS and driver releases are contemporaneous. Neither case is ideal. Reviewers have generally settled on the former option; OS and driver updates tend to increase performance, so testing with contemporaneous software might artificially inflate the apparent performance deltas in favour of newer components. Nobody wants to buy a new processor thinking that it's 40% faster than their old one, only to discover that most of the difference was just OS optimisations. You'd rather have your readers be pleasantly surprised than bitterly disappointed, so it makes sense to choose a benchmarking methodology that errs on the side of favouring older parts. Spectre is a weird exception to business as usual, because we saw a huge performance decrease in a single update that affects one particular optimisation in one particular processor generation. In this case, it probably makes sense for reviewers to re-run all of their old benchmarks and set a new baseline.

mtgx 9y ago

> isn't Windows installing them automatically? I thought the OS update was only part of the patch and that OEMs also needed to update their BIOS, which means users have to manually update their BIOS.

floatboth 9y ago
Microcode is indeed distributed through Windows Update as well. And packages on other OSes. And applied at boot time.

hajile 9y ago

It takes a long time to run a comprehensive test suite against a single processor. Reviewers often have to ship the hardware back to the company afterward. The result is that they very frequently re-use the old benchmarks in newer tests. New software doesn't affect existing benchmarks.