5 ms·

There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process inst

by kibibu 3y ago

There is genuine merit in peer review though. A lot of the marketing "papers" from OpenAI and Google would benefit from going through a peer review process instead of scholarwashing and uploading straight to Arxiv. One simple example, (from memory) the Bard paper doesn't include results for experiments in which GPT-4 outperforms it. As a result, people come away from these works with an inflated understanding of their capabilities. This wouldn't pass peer review.

jakear 3y ago
Industry doesn't need peer review. The proof of the pudding is in the sales.
- ModernMech 3y ago
  This is the final stage of capitalism, where market success is conflated with scientific rigor.
  - jakear 3y ago
    Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.
    - depereo 3y ago
      This doesn't really account for fraud or abuse. Plenty of profitable companies whose product is a scam, or has other factors that make it successful. Market success isn't the same thing as validity.
      - jakear 3y ago
        Peer review isn't the same thing as validity either. Both are proxies, sales are simply a much better proxy. Especially long term sales from an established company. Look at it like this: a Peer Reviewed (tm) article comes out saying "Foos cannot Bar, it is impossible". The same day, Apple releases "Bar for Foos, $8/month". Over the next year you see media outlets discussing how well Foo Barring works. Online reviewers talk about how they've incorporated Barring their Foos into everyday life and it has benefitted them in all these great ways. Your colleagues at work mention their fruitful Foo Barring adventure over the past weekend. Routine posts on HN come up where hackers describe how they've incorporated the Foo Barring API's into their own products in some novel way. Your mother then calls you up to ask if she should get involved with this new Foo Barring thing. What do you say, can Foos Bar?
    - tsimionescu 3y ago
      A quick trip around audiophile companies should quickly disabuse you of the notion that high sales implies any kind of worth to a product. There are several companies making lots of money on selling gold HDMI, ethernet etc cables "to improve sound quality", and plenty of rubes buying them and "hearing the difference".
      - memefrog 3y ago
        Real Hi-Fi companies like Sennheiser and Bose make far more money than people selling gold HDMI cables. And they've been making much more money for decades and will continue to make money for decades. Grifters don't win in the long-term.
        Qwertious 3y ago
        >Grifters don't win in the long-term. Plenty of religions suggest otherwise. (Dear reader: I'm not referring to your religion - I'm referring to the other religions.)
        memefrog 3y ago
        Wow so edgy and cool. Are you a teenager?
    - optionalsquid 3y ago
      You frequently see this kind of reasoning in medical quackery: > Butt-candling[1] must work, just look all these happy customers! But history is replete with ineffective or downright harmful treatments being popular long after the evidence showed them to be ineffective or harmful. Homeopathy is a prime example of this, seeing as those concoctions contain either no active ingredients or (in cases of low dilutions still labeled "homeopathic") contain ingredients picked based on notions of sympathetic magic ("like cures like"). [1] A hopefully fictional example.
      - jakear 3y ago
        This is a perfect example. In the real world, some placebos work. You take them, believe that they will work, and they do. Fantastic! Your problem has been solved, you share your story with friend(s), the cure spreads based on how well it works for them. It's only in academia that they must instead work in a sterile environment that some person who has never stepped foot off a school campus thinks is "more authentic" for them to be seen as legitimate.
    - deleted 3y ago
      [deleted]
- kibibu 3y ago
  Then why does it dress its marketing up in academic paper format?
  - jakear 3y ago
    In want of sales.
  - kergonath 3y ago
    It gives a nice patina and some legitimacy. They have no need to publish quasi-scientific articles.
somenameforme 3y ago
You can often see a p-hacked study from a mile away because they measure large numbers of unnecessary variables. They then pick those 4 variables that yield a probably false signal and publish on it. One would think these would never pass peer review, but they're regularly peer reviewed, published, and then, shockingly enough, fail to replicate. Hypothesizing after the results are known falls in the same bucket here. This is why pre-publishing kicked off, yet it's also hardly a savior. The point I make is that peer review can not be guaranteed to 'fix' science in any way we might like. The Sokal Affair [1] has now been replicated repeatedly, including in peer reviewed journals. The most recent one even got quite cheeky and published it under the names "Sage Owens, Kal Avers-Lynde III" - Sokal III. [2] It always preys on the same weakness - bias confirmation. [1] - https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair [2] - https://www.nationalreview.com/news/academic-journal-publishes-apparent-hoax-paper-alleging-right-wing-donor-influence-in-universities/ https://www.nationalreview.com/news/academic-journal-publish...
kjkjadksj 3y ago
What is honestly a lot more compelling than peer review is multiple supporting pieces of evidence from different sources. One result might be spurious, but if you can find a couple independent studies showing as much its probably a real phenomenon.

by kibibu 3y ago

jakear 3y ago

Industry doesn't need peer review. The proof of the pudding is in the sales.

ModernMech 3y ago
This is the final stage of capitalism, where market success is conflated with scientific rigor.
- jakear 3y ago
  Engineering, not scientific. High Sales = A large number of people can attest to the engineered good or service being of high enough quality that they will exchange hard earned money for the ability to use it. Peer Review = Some folks who derive self-worth from citations ask you to add a citation to their work and you do it because you'll probably need to ask them for the same some time later.
  - depereo 3y ago
    This doesn't really account for fraud or abuse. Plenty of profitable companies whose product is a scam, or has other factors that make it successful. Market success isn't the same thing as validity.
    - jakear 3y ago
      Peer review isn't the same thing as validity either. Both are proxies, sales are simply a much better proxy. Especially long term sales from an established company. Look at it like this: a Peer Reviewed (tm) article comes out saying "Foos cannot Bar, it is impossible". The same day, Apple releases "Bar for Foos, $8/month". Over the next year you see media outlets discussing how well Foo Barring works. Online reviewers talk about how they've incorporated Barring their Foos into everyday life and it has benefitted them in all these great ways. Your colleagues at work mention their fruitful Foo Barring adventure over the past weekend. Routine posts on HN come up where hackers describe how they've incorporated the Foo Barring API's into their own products in some novel way. Your mother then calls you up to ask if she should get involved with this new Foo Barring thing. What do you say, can Foos Bar?
  - tsimionescu 3y ago
    A quick trip around audiophile companies should quickly disabuse you of the notion that high sales implies any kind of worth to a product. There are several companies making lots of money on selling gold HDMI, ethernet etc cables "to improve sound quality", and plenty of rubes buying them and "hearing the difference".
    - memefrog 3y ago
      Real Hi-Fi companies like Sennheiser and Bose make far more money than people selling gold HDMI cables. And they've been making much more money for decades and will continue to make money for decades. Grifters don't win in the long-term.
      - Qwertious 3y ago
        >Grifters don't win in the long-term. Plenty of religions suggest otherwise. (Dear reader: I'm not referring to your religion - I'm referring to the other religions.)
        memefrog 3y ago
        Wow so edgy and cool. Are you a teenager?
  - optionalsquid 3y ago
    You frequently see this kind of reasoning in medical quackery: > Butt-candling[1] must work, just look all these happy customers! But history is replete with ineffective or downright harmful treatments being popular long after the evidence showed them to be ineffective or harmful. Homeopathy is a prime example of this, seeing as those concoctions contain either no active ingredients or (in cases of low dilutions still labeled "homeopathic") contain ingredients picked based on notions of sympathetic magic ("like cures like"). [1] A hopefully fictional example.
    - jakear 3y ago
      This is a perfect example. In the real world, some placebos work. You take them, believe that they will work, and they do. Fantastic! Your problem has been solved, you share your story with friend(s), the cure spreads based on how well it works for them. It's only in academia that they must instead work in a sterile environment that some person who has never stepped foot off a school campus thinks is "more authentic" for them to be seen as legitimate.
  - deleted 3y ago
    [deleted]
kibibu 3y ago
Then why does it dress its marketing up in academic paper format?
- jakear 3y ago
  In want of sales.
- kergonath 3y ago
  It gives a nice patina and some legitimacy. They have no need to publish quasi-scientific articles.

somenameforme 3y ago

You can often see a p-hacked study from a mile away because they measure large numbers of unnecessary variables. They then pick those 4 variables that yield a probably false signal and publish on it. One would think these would never pass peer review, but they're regularly peer reviewed, published, and then, shockingly enough, fail to replicate. Hypothesizing after the results are known falls in the same bucket here. This is why pre-publishing kicked off, yet it's also hardly a savior. The point I make is that peer review can not be guaranteed to 'fix' science in any way we might like. The Sokal Affair [1] has now been replicated repeatedly, including in peer reviewed journals. The most recent one even got quite cheeky and published it under the names "Sage Owens, Kal Avers-Lynde III" - Sokal III. [2] It always preys on the same weakness - bias confirmation. [1] - https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair [2] - https://www.nationalreview.com/news/academic-journal-publishes-apparent-hoax-paper-alleging-right-wing-donor-influence-in-universities/ https://www.nationalreview.com/news/academic-journal-publish...

kjkjadksj 3y ago

What is honestly a lot more compelling than peer review is multiple supporting pieces of evidence from different sources. One result might be spurious, but if you can find a couple independent studies showing as much its probably a real phenomenon.