3 ms·

I'm so incredibly tired of all of the BS claims. (I'm an AI/ML researcher) > has enabled open-source LLMs “to exceed human-level performance on two more of the

I'm so incredibly tired of all of the BS claims. (I'm an AI/ML researcher) > has enabled open-source LLMs “to exceed human-level performance on two more of the lab-bench tasks: doing scientific literature research and reasoning about DNA constructs” with only “modest compute budgets.” No. They did not. They just ran a crappy experiment and came up with an absurd result. As a community we need to invest much more effort into benchmarking as a science. Our space is full of garbage claims like this and it isn't doing us any favors. Eventually the hype will die down and people will realize that a lot of the claims were obvious falsehoods. Then we'll all get collectively punished for it.

jaggederest 2y ago
I'm reminded of the time I saw some A/B test results that didn't make much sense, but were highly significant[1] I asked how many A/B tests they were running... hundreds. Overlapping. At least they had a holdout group (that they mostly ignored, which indicated that all the A/B tests more or less made no difference) [1] P < 0.001 with a large effect size. No, your a/b test probably didn't break the laws of economics - you probably messed up your data.
- MostlyStable 2y ago
  If they were running many concurrent, overlapping A/B tests, then they didn't necessarily mess up their data. You are likely to get that result, honestly and truthfully, purely by chance, if you run enough tests. Unless the "running concurrent tests and not correcting your significance level", is what you meant by messing up their data, in which case yeah.
  - jaggederest 2y ago
    Yes precisely, the latter. I know there are techniques to disentangle that kind of thing but neither I nor they had them readily to hand.
kordlessagain 2y ago
The problem is with the reporting, if you want to call it that. This is more like a promotional piece. They write something someone said, and assume it’s true or useful to get attention. There’s not even a name attributed to the article.
mskar 2y ago
Do you have ideas for what would make a better experiment? The methodology for a literature search comparison, while simple, is the best I could come up with. We developed ~250 multiple choice questions which require a deep dive into a paper to answer, ideally with very convincing distractor answers. Then we gave 9 evaluators (post-docs and grad students in biology) a week to answer 40 questions each, without any limitations on their search. The evaluators were incentivized by providing a base pay per question completed, with a 50-100% bonus if they got enough questions correct. Under those circumstances, the evaluators had an answer precision of 73.8%, and the AI system (PaperQA2) was 85.2%. Both the evaluators and PaperQA2 could choose not to answer on a particular question. If you look at accuracy, which takes into account not answering a question, evaluators were 67.7% and PaperQA2 was 66%. So in terms of overall accuracy -- humans still did a touch better. But when actually answering, the AI was more precise. In terms of literature synthesis comparison, I think the methodology was pretty solid too, but would love more feedback. We had PaperQA2 write cited articles for ~19k human genes, of which there are (non-stub) Wikipedia articles for ~3.9k. It's worth noting that this is a particularly technical subset of Wikipedia articles. We sampled 300 articles that were in both sources, then extracted 500 statements from each (basically a paragraph block). These statements could be compound, or even multi-sentence statements. These statements were shuffled and obfuscated such that the origin could not be determined from the statement alone. The statements were given to a team of 4 evaluators, who were each asked to evaluate if the information was correct as cited, i.e. did the source actually support the statement. So they had to access (if they could) and actually read all the sources. After we got the evaluator gradings back, we could compile and map each statement back to its origin for comparison. Under these circumstance, the PaperQA2 written articles were 83% cited and supported, while the Wikipedia articles were 61.5% cited and supported. Wikipedia had comparatively more uncited claims, so if we eliminate those and only focus on the cited claims themselves, then PaperQA2 had 86.1% of claims that were supported by the source and Wikipedia had 71.2%. We did an analysis of every single un-supported claim, and on Wikipedia, claims are often attributed to arbitrary or really broad sources, like a landing page to a database. (here's the paper fwiw: https://arxiv.org/abs/2409.13740 https://arxiv.org/abs/2409.13740)
nsagent 2y ago
I'm also an AI researcher and I'm not hopeful that the hype will die down anytime soon. People have been touting insignificant results through shoddy science for a long time now. They've noticed it works well enough because current ML is still pretty much alchemy (Ali Rahimi's 2017 NIPS Test of Time [1] talk still resonates today) so people rarely spend the effort to effectively refute the bogus claims. As a result, I've opted out of the system and I'm working on trying ambitious ideas I have to try to upend the current paradigm of training on big data, which is truly insane (by age 4 the most erudite of children have probably only been exposed to 45 million words [2] and yet exhibit vastly more understanding and fluency than any language model trained on a similar amount of data). [1]: https://www.youtube.com/watch?v=Qi1Yry33TQE https://www.youtube.com/watch?v=Qi1Yry33TQE [2]: https://www.aft.org/sites/default/files/media/2014/TheEarlyCatastrophe.pdf https://www.aft.org/sites/default/files/media/2014/TheEarlyC...

jaggederest 2y ago

I'm reminded of the time I saw some A/B test results that didn't make much sense, but were highly significant[1] I asked how many A/B tests they were running... hundreds. Overlapping. At least they had a holdout group (that they mostly ignored, which indicated that all the A/B tests more or less made no difference) [1] P < 0.001 with a large effect size. No, your a/b test probably didn't break the laws of economics - you probably messed up your data.

MostlyStable 2y ago
If they were running many concurrent, overlapping A/B tests, then they didn't necessarily mess up their data. You are likely to get that result, honestly and truthfully, purely by chance, if you run enough tests. Unless the "running concurrent tests and not correcting your significance level", is what you meant by messing up their data, in which case yeah.
- jaggederest 2y ago
  Yes precisely, the latter. I know there are techniques to disentangle that kind of thing but neither I nor they had them readily to hand.

kordlessagain 2y ago

The problem is with the reporting, if you want to call it that. This is more like a promotional piece. They write something someone said, and assume it’s true or useful to get attention. There’s not even a name attributed to the article.

mskar 2y ago

Do you have ideas for what would make a better experiment? The methodology for a literature search comparison, while simple, is the best I could come up with. We developed ~250 multiple choice questions which require a deep dive into a paper to answer, ideally with very convincing distractor answers. Then we gave 9 evaluators (post-docs and grad students in biology) a week to answer 40 questions each, without any limitations on their search. The evaluators were incentivized by providing a base pay per question completed, with a 50-100% bonus if they got enough questions correct. Under those circumstances, the evaluators had an answer precision of 73.8%, and the AI system (PaperQA2) was 85.2%. Both the evaluators and PaperQA2 could choose not to answer on a particular question. If you look at accuracy, which takes into account not answering a question, evaluators were 67.7% and PaperQA2 was 66%. So in terms of overall accuracy -- humans still did a touch better. But when actually answering, the AI was more precise. In terms of literature synthesis comparison, I think the methodology was pretty solid too, but would love more feedback. We had PaperQA2 write cited articles for ~19k human genes, of which there are (non-stub) Wikipedia articles for ~3.9k. It's worth noting that this is a particularly technical subset of Wikipedia articles. We sampled 300 articles that were in both sources, then extracted 500 statements from each (basically a paragraph block). These statements could be compound, or even multi-sentence statements. These statements were shuffled and obfuscated such that the origin could not be determined from the statement alone. The statements were given to a team of 4 evaluators, who were each asked to evaluate if the information was correct as cited, i.e. did the source actually support the statement. So they had to access (if they could) and actually read all the sources. After we got the evaluator gradings back, we could compile and map each statement back to its origin for comparison. Under these circumstance, the PaperQA2 written articles were 83% cited and supported, while the Wikipedia articles were 61.5% cited and supported. Wikipedia had comparatively more uncited claims, so if we eliminate those and only focus on the cited claims themselves, then PaperQA2 had 86.1% of claims that were supported by the source and Wikipedia had 71.2%. We did an analysis of every single un-supported claim, and on Wikipedia, claims are often attributed to arbitrary or really broad sources, like a landing page to a database. (here's the paper fwiw: https://arxiv.org/abs/2409.13740 https://arxiv.org/abs/2409.13740)

nsagent 2y ago

I'm also an AI researcher and I'm not hopeful that the hype will die down anytime soon. People have been touting insignificant results through shoddy science for a long time now. They've noticed it works well enough because current ML is still pretty much alchemy (Ali Rahimi's 2017 NIPS Test of Time [1] talk still resonates today) so people rarely spend the effort to effectively refute the bogus claims. As a result, I've opted out of the system and I'm working on trying ambitious ideas I have to try to upend the current paradigm of training on big data, which is truly insane (by age 4 the most erudite of children have probably only been exposed to 45 million words [2] and yet exhibit vastly more understanding and fluency than any language model trained on a similar amount of data). [1]: https://www.youtube.com/watch?v=Qi1Yry33TQE https://www.youtube.com/watch?v=Qi1Yry33TQE [2]: https://www.aft.org/sites/default/files/media/2014/TheEarlyCatastrophe.pdf https://www.aft.org/sites/default/files/media/2014/TheEarlyC...