3 ms·

In an ideal world yes, but in order for LLM authors to provide the evals they need to access the private set (and promise not to train on them or use that to in

by sshh12 2y ago

In an ideal world yes, but in order for LLM authors to provide the evals they need to access the private set (and promise not to train on them or use that to influence eval/training methods). Either the eval maintainers need to be given the closed source models (which will likely never happen) or the model authors need to be given the private evals to run themselves.

ipaddr 2y ago
So the entire benchmark scheme is worthless?
- thewanderer1983 2y ago
  Well it depends on how you define worthless. For you as an individual to ascertain truth, it may be useless. To build up a bloated AI Enterprise stock value. For false consensus narrative scripting. Very valuable.
- AznHisoka 2y ago
  It’s essentially worthless for you, as a consumer of them. The best way to see which one works best is to give a bunch of them a try for your specific use case
- Tostino 2y ago
  Not if you have a private eval you use for your own use cases.
- selcuka 2y ago
  Benchmarks have been close to worthless since Nvidia cheated 3DMark benchmarks by adding benchmark detecting code to their drivers in the early 2000s [1]. [1] https://www.eurogamer.net/news280503nvidia https://www.eurogamer.net/news280503nvidia
  - DecentShoes 2y ago
    That's a bit dramatic. One person cheating doesn't mean everybody is cheating. Cycling didn't become pointless after Lance Armstrong was caught doping. But it may be the tip of the iceberg and warrant a change in methodology or further investigation.
    - Tijdreiziger 2y ago
      NVIDIA isn’t the only one who’s been caught. https://www.anandtech.com/show/15703/mobile-benchmark-cheating-mediatek https://www.anandtech.com/show/15703/mobile-benchmark-cheati...
    - ushiroda80 2y ago
      Were there like 20 or 30 other cycling cheaters that were caught alongside lance Armstrong?
      - ipaddr 2y ago
        Alberto Contador (2010, ban upheld in 2012) – Stripped of his 2010 Tour de France title due to clenbuterol use. Jan Ullrich (2012) – Officially found guilty of doping in connection with Operation Puerto, though his offenses dated back to the 2000s. Frank Schleck (2012) – Tested positive for a banned diuretic during the Tour de France. Johan Bruyneel (2018) – Armstrong’s former team director was banned for life for his role in systematic doping. Chris Froome (2017 case, cleared in 2018) – Found with high levels of salbutamol; later cleared by the UCI and WADA. Jarlinson Pantano (2019) – Tested positive for EPO and received a four-year ban. Nairo Quintana (2022) – Disqualified from the Tour de France for tramadol use, though it was not classified as a doping offense. The sport was tainted before Lance and still is.
enugu 2y ago
> Either the eval maintainers need to be given the closed source models (which will likely never happen) Given that the models are released to public, the test maintainers can just run the private tests after release, either via the prompts or via an api. Cheating won't be easy.
- karparov 2y ago
  Closed source in this context literally means that they are not released.
  - enugu 2y ago
    They dont need them to be released(in the sense that you have a copy of the binary) to evaluate the model. The costly model training is useless unless access is given to people who pay for it. The models of Open AI, Claude and other major companies - are all available either for free or a small amount(200$ for OpenAI Pro). Anyone who can pay this, can run private tests and compare scores. So, the public does not need to rely on benchmark claims of OpenAI based on its pre-release arrangements with test companies.
    - nwiswell 2y ago
      > Anyone who can pay this, can run private tests and compare scores. Yes, by uploading the tests to a server controlled by OpenAI/Anthropic/etc
      - enugu 2y ago
        These prompts are fielding millions of queries. The test questions are a small part of them. Further, the server doesn't know if it got the right answer or not, so it can't even train on them. Whereas in the arrangement with the testing companies before release, they can potentially do so, as the they are given the scores.

by sshh12 2y ago

ipaddr 2y ago

So the entire benchmark scheme is worthless?

thewanderer1983 2y ago
Well it depends on how you define worthless. For you as an individual to ascertain truth, it may be useless. To build up a bloated AI Enterprise stock value. For false consensus narrative scripting. Very valuable.
AznHisoka 2y ago
It’s essentially worthless for you, as a consumer of them. The best way to see which one works best is to give a bunch of them a try for your specific use case
Tostino 2y ago
Not if you have a private eval you use for your own use cases.
selcuka 2y ago
Benchmarks have been close to worthless since Nvidia cheated 3DMark benchmarks by adding benchmark detecting code to their drivers in the early 2000s [1]. [1] https://www.eurogamer.net/news280503nvidia https://www.eurogamer.net/news280503nvidia
- DecentShoes 2y ago
  That's a bit dramatic. One person cheating doesn't mean everybody is cheating. Cycling didn't become pointless after Lance Armstrong was caught doping. But it may be the tip of the iceberg and warrant a change in methodology or further investigation.
  - Tijdreiziger 2y ago
    NVIDIA isn’t the only one who’s been caught. https://www.anandtech.com/show/15703/mobile-benchmark-cheating-mediatek https://www.anandtech.com/show/15703/mobile-benchmark-cheati...
  - ushiroda80 2y ago
    Were there like 20 or 30 other cycling cheaters that were caught alongside lance Armstrong?
    - ipaddr 2y ago
      Alberto Contador (2010, ban upheld in 2012) – Stripped of his 2010 Tour de France title due to clenbuterol use. Jan Ullrich (2012) – Officially found guilty of doping in connection with Operation Puerto, though his offenses dated back to the 2000s. Frank Schleck (2012) – Tested positive for a banned diuretic during the Tour de France. Johan Bruyneel (2018) – Armstrong’s former team director was banned for life for his role in systematic doping. Chris Froome (2017 case, cleared in 2018) – Found with high levels of salbutamol; later cleared by the UCI and WADA. Jarlinson Pantano (2019) – Tested positive for EPO and received a four-year ban. Nairo Quintana (2022) – Disqualified from the Tour de France for tramadol use, though it was not classified as a doping offense. The sport was tainted before Lance and still is.

enugu 2y ago

> Either the eval maintainers need to be given the closed source models (which will likely never happen) Given that the models are released to public, the test maintainers can just run the private tests after release, either via the prompts or via an api. Cheating won't be easy.

karparov 2y ago
Closed source in this context literally means that they are not released.
- enugu 2y ago
  They dont need them to be released(in the sense that you have a copy of the binary) to evaluate the model. The costly model training is useless unless access is given to people who pay for it. The models of Open AI, Claude and other major companies - are all available either for free or a small amount(200$ for OpenAI Pro). Anyone who can pay this, can run private tests and compare scores. So, the public does not need to rely on benchmark claims of OpenAI based on its pre-release arrangements with test companies.
  - nwiswell 2y ago
    > Anyone who can pay this, can run private tests and compare scores. Yes, by uploading the tests to a server controlled by OpenAI/Anthropic/etc
    - enugu 2y ago
      These prompts are fielding millions of queries. The test questions are a small part of them. Further, the server doesn't know if it got the right answer or not, so it can't even train on them. Whereas in the arrangement with the testing companies before release, they can potentially do so, as the they are given the scores.