5 ms·

We need modern open source CPU benchmarks. PassMark and Geekbench are closed source. I don't know why I should trust them to e.g. treat fundamentally different

by lysace 1y ago

We need modern open source CPU benchmarks. PassMark and Geekbench are closed source. I don't know why I should trust them to e.g. treat fundamentally different kinds of devices in a sane and fair way. They have vastly different cooling mechanisms, for one. It matters if they can sustain a certain load for 5 minutes or for 5 hours.

Etheryte 1y ago
In some other context I'd be inclined to agree, but I swear the moment there's an open benchmark, it's going to get gamed by someone. So long as there's money to be made, the incentives don't align, look at Volkswagen for some inspiration.
- lysace 1y ago
  Is it really harder for CPU designers to game closed source benchmarks?
  - hu3 1y ago
    And they can also be gamed with money.
zamadatix 1y ago
If you have a specific use case in mind you might as well just test that use case directly. E.g. PTS (Phoronix Test Suite) is not a traditional benchmark, it's a workload tester, and it's right up that alley (though you then run into the problem of "the workload I care to compare doesn't run on an iPhone at all", at which point you either compare generalities again or don't care anymore).
- lysace 1y ago
  I suppose what I really mean is for the industry to agree on some benchmark or two. Like the 80s PC mag benchmarks. It obviously doesn't scale well if every potential end user has to benchmark every device.
wmf 1y ago
SPEC exists. Apparently reviewers who speak English can't afford it.
- lysace 1y ago
  Not really open source, no.
  - GeekyBear 1y ago
    https://www.spec.org/sources/ https://www.spec.org/sources/
- GeekyBear 1y ago
  Pour one out for Anandtech. SPEC has been the industry standard performance benchmark for comparing between different CPU architectures for decades. It takes all freaking day to run, but Anandtech published the benchmarks as soon as they got their grubby little hands on a new core design for every well known architecture. Is GeekerWan on YouTube the only outlet still doing this today, albeit in Chinese with English subtitles? You would think that Chips and Cheese, at least, would take up the gauntlet.

by lysace 1y ago

Etheryte 1y ago

In some other context I'd be inclined to agree, but I swear the moment there's an open benchmark, it's going to get gamed by someone. So long as there's money to be made, the incentives don't align, look at Volkswagen for some inspiration.

lysace 1y ago
Is it really harder for CPU designers to game closed source benchmarks?
- hu3 1y ago
  And they can also be gamed with money.

zamadatix 1y ago

If you have a specific use case in mind you might as well just test that use case directly. E.g. PTS (Phoronix Test Suite) is not a traditional benchmark, it's a workload tester, and it's right up that alley (though you then run into the problem of "the workload I care to compare doesn't run on an iPhone at all", at which point you either compare generalities again or don't care anymore).

lysace 1y ago
I suppose what I really mean is for the industry to agree on some benchmark or two. Like the 80s PC mag benchmarks. It obviously doesn't scale well if every potential end user has to benchmark every device.

wmf 1y ago

SPEC exists. Apparently reviewers who speak English can't afford it.

lysace 1y ago
Not really open source, no.
- GeekyBear 1y ago
  https://www.spec.org/sources/ https://www.spec.org/sources/
GeekyBear 1y ago
Pour one out for Anandtech. SPEC has been the industry standard performance benchmark for comparing between different CPU architectures for decades. It takes all freaking day to run, but Anandtech published the benchmarks as soon as they got their grubby little hands on a new core design for every well known architecture. Is GeekerWan on YouTube the only outlet still doing this today, albeit in Chinese with English subtitles? You would think that Chips and Cheese, at least, would take up the gauntlet.