3 ms·

I'm not 100% sure about every other reviewer, but many times tests will be done with a base windows 10 install (as to avoid the need to look at comparisons betw

• blattimwind 9y ago
  > many times tests will be done with a base windows 10 install [without any updates/patches] So, even without Spectre & friends, these benchmarks are essentially meaningless?
  • jdietrich 9y ago
    It's a difficult problem. It's not practical to re-test all of the old CPUs or GPUs every time you review a new part. Either you test all components against a baseline OS and driver set, or you test components with whatever OS and driver releases are contemporaneous. Neither case is ideal. Reviewers have generally settled on the former option; OS and driver updates tend to increase performance, so testing with contemporaneous software might artificially inflate the apparent performance deltas in favour of newer components. Nobody wants to buy a new processor thinking that it's 40% faster than their old one, only to discover that most of the difference was just OS optimisations. You'd rather have your readers be pleasantly surprised than bitterly disappointed, so it makes sense to choose a benchmarking methodology that errs on the side of favouring older parts. Spectre is a weird exception to business as usual, because we saw a huge performance decrease in a single update that affects one particular optimisation in one particular processor generation. In this case, it probably makes sense for reviewers to re-run all of their old benchmarks and set a new baseline.