5 ms·

It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".

by jakear 3y ago

PaulHoule 3y ago
Except for the ones that conclude it is all a scam.
kibibu 3y ago
There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of their capabilities. This wouldn't pass peer review.
- jakear 3y ago
  Industry doesn't need peer review. The proof of the pudding is in the sales.
  - ModernMech 3y ago
    This is the final stage of capitalism, where market success is conflated with scientific rigor.
    - jakear 3y ago
      Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.
      - depereo 3y ago
        This doesn't really account for fraud or abuse. Plenty of profitable companies whose product is a scam, or has other factors that make it successful. Market success isn't the same thing as validity.
        jakear 3y ago
        Peer review isn't the same thing as validity either. Both are proxies, sales are simply a much better proxy. Especially long term sales from an established company. Look at it like this: a Peer Reviewed (tm) article comes out saying "Foos cannot Bar, it is impossible". The same day, Apple releases "Bar for Foos, $8/month". Over the next year you see media outlets discussing how well Foo Barring works. Online reviewers talk about how they've incorporated Barring their Foos into everyday life and it has benefitted them in all these great ways. Your colleagues at work mention their fruitful Foo Barring adventure over the past weekend. Routine posts on HN come up where hackers describe how they've incorporated the Foo Barring API's into their own products in some novel way. Your mother then calls you up to ask if she should get involved with this new Foo Barring thing. What do you say, can Foos Bar?
      - tsimionescu 3y ago
        A quick trip around audiophile companies should quickly disabuse you of the notion that high sales implies any kind of worth to a product. There are several companies making lots of money on selling gold HDMI, ethernet etc cables "to improve sound quality", and plenty of rubes buying them and "hearing the difference".
        memefrog 3y ago
        Real Hi-Fi companies like Sennheiser and Bose make far more money than people selling gold HDMI cables. And they've been making much more money for decades and will continue to make money for decades. Grifters don't win in the long-term.
        Qwertious 3y ago
  - kibibu 3y ago
    Then why does it dress its marketing up in academic paper format?
    - jakear 3y ago
      In want of sales.
    - kergonath 3y ago
      It gives a nice patina and some legitimacy. They have no need to publish quasi-scientific articles.
- somenameforme 3y ago
  You can often see a p-hacked study from a mile away because they measure large numbers of unnecessary variables. They then pick those 4 variables that yield a probably false signal and publish on it. One would think these would never pass peer review, but they're regularly peer reviewed, published, and then, shockingly enough, fail to replicate. Hypothesizing after the results are known falls in the same bucket here. This is why pre-publishing kicked off, yet it's also hardly a savior. The point I make is that peer review can not be guaranteed to 'fix' science in any way we might like. The Sokal Affair [1] has now been replicated repeatedly, including in peer reviewed journals. The most recent one even got quite cheeky and published it under the names "Sage Owens, Kal Avers-Lynde III" - Sokal III. [2] It always preys on the same weakness - bias confirmation. [1] - https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair [2] - https://www.nationalreview.com/news/academic-journal-publishes-apparent-hoax-paper-alleging-right-wing-donor-influence-in-universities/ https://www.nationalreview.com/news/academic-journal-publish...
- kjkjadksj 3y ago
  What is honestly a lot more compelling than peer review is multiple supporting pieces of evidence from different sources. One result might be spurious, but if you can find a couple independent studies showing as much its probably a real phenomenon.

It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".

by jakear 3y ago

It's the general public that equates "peer reviewed" with "definitely correct, does not need to be questioned".

PaulHoule 3y ago

Except for the ones that conclude it is all a scam.

kibibu 3y ago

There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of their capabilities. This wouldn't pass peer review.

jakear 3y ago
Industry doesn't need peer review. The proof of the pudding is in the sales.
- ModernMech 3y ago
  This is the final stage of capitalism, where market success is conflated with scientific rigor.
  - jakear 3y ago
    Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.
    - depereo 3y ago
      This doesn't really account for fraud or abuse. Plenty of profitable companies whose product is a scam, or has other factors that make it successful. Market success isn't the same thing as validity.
      - jakear 3y ago
        Peer review isn't the same thing as validity either. Both are proxies, sales are simply a much better proxy. Especially long term sales from an established company. Look at it like this: a Peer Reviewed (tm) article comes out saying "Foos cannot Bar, it is impossible". The same day, Apple releases "Bar for Foos, $8/month". Over the next year you see media outlets discussing how well Foo Barring works. Online reviewers talk about how they've incorporated Barring their Foos into everyday life and it has benefitted them in all these great ways. Your colleagues at work mention their fruitful Foo Barring adventure over the past weekend. Routine posts on HN come up where hackers describe how they've incorporated the Foo Barring API's into their own products in some novel way. Your mother then calls you up to ask if she should get involved with this new Foo Barring thing. What do you say, can Foos Bar?
    - tsimionescu 3y ago
      A quick trip around audiophile companies should quickly disabuse you of the notion that high sales implies any kind of worth to a product. There are several companies making lots of money on selling gold HDMI, ethernet etc cables "to improve sound quality", and plenty of rubes buying them and "hearing the difference".
      - memefrog 3y ago
        Real Hi-Fi companies like Sennheiser and Bose make far more money than people selling gold HDMI cables. And they've been making much more money for decades and will continue to make money for decades. Grifters don't win in the long-term.
        Qwertious 3y ago
- kibibu 3y ago
  Then why does it dress its marketing up in academic paper format?
  - jakear 3y ago
    In want of sales.
  - kergonath 3y ago
    It gives a nice patina and some legitimacy. They have no need to publish quasi-scientific articles.
somenameforme 3y ago
You can often see a p-hacked study from a mile away because they measure large numbers of unnecessary variables. They then pick those 4 variables that yield a probably false signal and publish on it. One would think these would never pass peer review, but they're regularly peer reviewed, published, and then, shockingly enough, fail to replicate. Hypothesizing after the results are known falls in the same bucket here. This is why pre-publishing kicked off, yet it's also hardly a savior. The point I make is that peer review can not be guaranteed to 'fix' science in any way we might like. The Sokal Affair [1] has now been replicated repeatedly, including in peer reviewed journals. The most recent one even got quite cheeky and published it under the names "Sage Owens, Kal Avers-Lynde III" - Sokal III. [2] It always preys on the same weakness - bias confirmation. [1] - https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair [2] - https://www.nationalreview.com/news/academic-journal-publishes-apparent-hoax-paper-alleging-right-wing-donor-influence-in-universities/ https://www.nationalreview.com/news/academic-journal-publish...
kjkjadksj 3y ago
What is honestly a lot more compelling than peer review is multiple supporting pieces of evidence from different sources. One result might be spurious, but if you can find a couple independent studies showing as much its probably a real phenomenon.