6 ms·

The silver lining is that the media hype probably got a lot of people interested in physics and materials science. Some of them will undoubtedly go on to make g

by xeckr 3y ago

The silver lining is that the media hype probably got a lot of people interested in physics and materials science. Some of them will undoubtedly go on to make great discoveries and contributions to the field in the future.

Loughla 3y ago
I like that take and I appreciate you for putting forward optimism instead of (all too common) cynical sniping. Thank you. Listening to someone who plainly had no idea what they were talking about is what got me interested in the field of education.
tareqak 3y ago
I remembered the account of developers committing bugs so that reviewers were forced to find something instead of just giving the all clear. Maybe there should be a group of researchers that submit papers that seem believable with known issues to avoid scientific journals stagnating. Maybe the names and institutions of the group submitting papers should be hidden so that more reputable/famous people and institutions get the same level of scrutiny as everyone else.
- vkou 3y ago
  You want to solve reproducibility? Apply forensic scrutiny to every publication? That's a great idea, but who is going to pay for it? Every scientist knows that it's more than possible to publish deliberately fraudulent work. Every non-scientist thinks that detecting this fraud is easy.
  - _jal 3y ago
    > Every non-scientist thinks that detecting this fraud is easy. Eh, this working engineer doesn't think so. Freeform fraud detection is a hard problem in general, and when you're talking about (apparent) cutting-edge research, the pool of people who might spot specific details is tiny. A low probability of detection does suggest that the penalty should be relatively high to have a deterrent effect.
- armchairhacker 3y ago
  Great way to test peer-review: submit a bunch of BS articles, particularly with big words and notable authors, and see which ones get published where In fact I know for a fact this has been done at least once before (at least with the big words): https://en.m.wikipedia.org/wiki/SCIgen https://en.m.wikipedia.org/wiki/SCIgen But it should be done in every field, regularly
  - vouaobrasil 3y ago
    A better way would be to change the system so that publications in big-name journals are not important for career advancement.
    - fooker 3y ago
      What is your alternative solution?
      - MarkusQ 3y ago
        Seems clear to me: "change the system so that publications in big-name journals are not important for career advancement." There are many fields that don't have this requirement, or anything like it, and the suggestion is that academia join their ranks. The "alternative solution" to having a thing that wacks people on the head every Tuesday is...not having it.
        EVa5I7bHFq9mnYK 3y ago
        Businesses have money. Governments have votes. What measurable quantity do you propose for scientific work? Votes of colleagues? It's even more corrupt.
        fooker 3y ago
        Are you familiar with the concept of Chesterton’s fence?
      - vouaobrasil 3y ago
- jghn 3y ago
  > developers committing bugs so that reviewers were forced to find something This assumes code review's key purpose is to find bugs. I always wonder about this when I review something, but at the end of the day there are many other things I'm looking more at than "is this a bug"
  - iamthepieman 3y ago
    Agreed. I probably won't catch a bug that the tests wouldn't also catch unless it's obvious. I'm looking for things like: -Is this duplicating something else? -Should this be part of this object? -if I come back in a year, will it take me 10 minutes or an hour to understand it? -is this necessary? -should I approve this but add a task to the tracking board to figure out if this should be done in the service instead of the client? -etc
- shusaku 3y ago
  If my understanding of the timeline is right, Dias already had a retraction when this was sent in. I don’t think reputation is to blame here, the reviewers just weren’t up to the task.
  - Laforet 3y ago
    It might be worth pointing out that reviewers are only on board to offer their opinion with the material given at face value. After all they are probably not in a position to experimentally verify whether the results are reproducible. The editor always has the final word on whether a manuscript gets accepted.
- TehCorwiz 3y ago
  This reminded me of this apocryphal story. "Just one thing - get rid of the duck." https://bwiggs.com/notebook/queens-duck/ https://bwiggs.com/notebook/queens-duck/
  - peeters 3y ago
    Funny, I always heard this anecdote in the context of adding hair to Mickey Mouse's arms. Apocryphal indeed, I guess: https://www.npr.org/2014/11/17/364760847/whats-with-all-of-the-hairy-arms-in-graphic-design https://www.npr.org/2014/11/17/364760847/whats-with-all-of-t... In terms of function though, the "hairy arm" is the complete opposite though: it's not to make sure reviewers are paying attention, it's to distract meddling reviewers from the changes you don't want them to veto.
    - saghm 3y ago
      From the linked article on the duck: > A feature added for no other reason than to draw management attention and be removed, thus avoiding unnecessary changes in other aspects of the product. Sounds like it was for the same reason
      - peeters 3y ago
        Yes, sorry what I meant was opposite to the reason that started this thread (to ensure review participation)
  - xoxxala 3y ago
    I worked at Interplay at the time and can confirm the duck story.
- vsareto 3y ago
  > I remembered the account of developers committing bugs so that reviewers were forced to find something instead of just giving the all clear. You'd be better off pair programming everything and forgoing code reviews. Sometimes I think the people that come up with this shit are sociopaths.
- atrettel 3y ago
  Weird, I was peer reviewing a manuscript earlier today and actually thought of something similar. The manuscript needs some work to improve its English and I wondered if some of the broken English passages were a test to see what nonsense a referee like me would put up with. I don't think it is a test but the thought did occur to me.
- golergka 3y ago
  > Maybe there should be a group of researchers that submit papers that seem believable with known issues to avoid scientific journals stagnating. There was. People were not very pleased. https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair
- ttldr 3y ago
  depends how you do it? good idea: https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair bad idea: https://www.theverge.com/2021/4/22/22398156/university-minnesota-linux-kernal-ban-research https://www.theverge.com/2021/4/22/22398156/university-minne...
- WhitneyLand 3y ago
  “…submit papers that seem believable with known issues to avoid scientific journals stagnating…” It’s called a scholarly hoax, and it’s been done more than once: https://en.m.wikipedia.org/wiki/List_of_scholarly_publishing_stings https://en.m.wikipedia.org/wiki/List_of_scholarly_publishing...
- dotnet00 3y ago
  Years before this current trend of LLM generated papers, there was a tool IIRC by some MIT students which would generate plausible looking papers which when actually read were obviously garbage. The title and abstract would have the sort of words you would expect of a research paper, but the resulting sentences would be nonsense. The idea being that if the submission got published, the review process was obviously a total joke. I think they even had some examples that had actually been published. Edit, Link: https://pdos.csail.mit.edu/archive/scigen/ https://pdos.csail.mit.edu/archive/scigen/
  - agileAlligator 3y ago
    I'm having a hard time believing those papers weren't written by humans after a cursory glance
- m463 3y ago
  A more benign example was the college professor who started each lecture saying he was going to say one wrong thing. (it would keep the students alert) or maybe Van Halen's "CRC check", where their venue contract would ask for a bunch of things set up in advance, including a bowl of M&Ms with the brown ones removed. (it was a test of diligence, that they read the contract and knew the requirements)
- 3seashells 3y ago
  Maybe the review institution of a paper should always add one known flaw that peer review must find to ensure a quality control of the process?
Nican 3y ago
Careful not to repeat what the University of Minnesota did to the Linux kernel, when willing submitting security bugs to write out a paper about trust.
- tedunangst 3y ago
  The duality of HN. I want other people to be forced to filter out bullshit, but I only want to receive forthright submissions myself.
  - cobbal 3y ago
    It's really weird. I'm starting to suspect hacker news isn't a single person.
    - sltkr 3y ago
      I'm pretty sure I'm you.
      - dllthomas 3y ago
        Who am us, anyway?
    - Regnore 3y ago
      Then why is it called the singular “Hacker” news?
      - golergka 3y ago
        Because there's only one intended reader who is sufficiently educated and smart. All others of us are impostors who should not have pursued this career at all.
      - thomastjeffery 3y ago
        If only the Hackers Knew..
Mistletoe 3y ago
As a scientist I see it the other way, where people will just ignore future discoveries and science and this goes on the trash pile with other bad science that makes the public distrust science. We saw the endgame of that with Covid and it was really ugly. The public doesn't understand the minutiae of each study and just gets really tired with "Carbs are bad" and "Now carbs are good again" headlines.
whycome 3y ago
When it's "not even wrong" it's easy to dismiss something. But if it takes some work to disprove, there can be some cool insights and knowledge (and new avenues of exploration).
darklycan51 3y ago
The problem is when too much fake stuff comes out (example: health/climate) to the point where average people cannot see what is real or not anymore.
furyofantares 3y ago
Are you talking about the hype around LK-99, which this article is not about?
d3w4s9 3y ago
Nah. The reality is that very very few people will "make great discoveries and contributions" even for graduate students in those fields or after getting a PhD degree, and many people who are good enough to become professors do mostly mediocre research or publish papers that have very little impact. For this LK-99 thing, I doubt more than 5 people of all those that are interested will be making any significant contributions in 10 to 20 years.

by xeckr 3y ago

Loughla 3y ago

I like that take and I appreciate you for putting forward optimism instead of (all too common) cynical sniping. Thank you. Listening to someone who plainly had no idea what they were talking about is what got me interested in the field of education.

tareqak 3y ago

I remembered the account of developers committing bugs so that reviewers were forced to find something instead of just giving the all clear. Maybe there should be a group of researchers that submit papers that seem believable with known issues to avoid scientific journals stagnating. Maybe the names and institutions of the group submitting papers should be hidden so that more reputable/famous people and institutions get the same level of scrutiny as everyone else.

vkou 3y ago
You want to solve reproducibility? Apply forensic scrutiny to every publication? That's a great idea, but who is going to pay for it? Every scientist knows that it's more than possible to publish deliberately fraudulent work. Every non-scientist thinks that detecting this fraud is easy.
- _jal 3y ago
  > Every non-scientist thinks that detecting this fraud is easy. Eh, this working engineer doesn't think so. Freeform fraud detection is a hard problem in general, and when you're talking about (apparent) cutting-edge research, the pool of people who might spot specific details is tiny. A low probability of detection does suggest that the penalty should be relatively high to have a deterrent effect.
armchairhacker 3y ago
Great way to test peer-review: submit a bunch of BS articles, particularly with big words and notable authors, and see which ones get published where In fact I know for a fact this has been done at least once before (at least with the big words): https://en.m.wikipedia.org/wiki/SCIgen https://en.m.wikipedia.org/wiki/SCIgen But it should be done in every field, regularly
- vouaobrasil 3y ago
  A better way would be to change the system so that publications in big-name journals are not important for career advancement.
  - fooker 3y ago
    What is your alternative solution?
    - MarkusQ 3y ago
      Seems clear to me: "change the system so that publications in big-name journals are not important for career advancement." There are many fields that don't have this requirement, or anything like it, and the suggestion is that academia join their ranks. The "alternative solution" to having a thing that wacks people on the head every Tuesday is...not having it.
      - EVa5I7bHFq9mnYK 3y ago
        Businesses have money. Governments have votes. What measurable quantity do you propose for scientific work? Votes of colleagues? It's even more corrupt.
      - fooker 3y ago
        Are you familiar with the concept of Chesterton’s fence?
    - vouaobrasil 3y ago
jghn 3y ago
> developers committing bugs so that reviewers were forced to find something This assumes code review's key purpose is to find bugs. I always wonder about this when I review something, but at the end of the day there are many other things I'm looking more at than "is this a bug"
- iamthepieman 3y ago
  Agreed. I probably won't catch a bug that the tests wouldn't also catch unless it's obvious. I'm looking for things like: -Is this duplicating something else? -Should this be part of this object? -if I come back in a year, will it take me 10 minutes or an hour to understand it? -is this necessary? -should I approve this but add a task to the tracking board to figure out if this should be done in the service instead of the client? -etc
shusaku 3y ago
If my understanding of the timeline is right, Dias already had a retraction when this was sent in. I don’t think reputation is to blame here, the reviewers just weren’t up to the task.
- Laforet 3y ago
  It might be worth pointing out that reviewers are only on board to offer their opinion with the material given at face value. After all they are probably not in a position to experimentally verify whether the results are reproducible. The editor always has the final word on whether a manuscript gets accepted.
TehCorwiz 3y ago
This reminded me of this apocryphal story. "Just one thing - get rid of the duck." https://bwiggs.com/notebook/queens-duck/ https://bwiggs.com/notebook/queens-duck/
- peeters 3y ago
  Funny, I always heard this anecdote in the context of adding hair to Mickey Mouse's arms. Apocryphal indeed, I guess: https://www.npr.org/2014/11/17/364760847/whats-with-all-of-the-hairy-arms-in-graphic-design https://www.npr.org/2014/11/17/364760847/whats-with-all-of-t... In terms of function though, the "hairy arm" is the complete opposite though: it's not to make sure reviewers are paying attention, it's to distract meddling reviewers from the changes you don't want them to veto.
  - saghm 3y ago
    From the linked article on the duck: > A feature added for no other reason than to draw management attention and be removed, thus avoiding unnecessary changes in other aspects of the product. Sounds like it was for the same reason
    - peeters 3y ago
      Yes, sorry what I meant was opposite to the reason that started this thread (to ensure review participation)
- xoxxala 3y ago
  I worked at Interplay at the time and can confirm the duck story.
vsareto 3y ago
> I remembered the account of developers committing bugs so that reviewers were forced to find something instead of just giving the all clear. You'd be better off pair programming everything and forgoing code reviews. Sometimes I think the people that come up with this shit are sociopaths.
atrettel 3y ago
Weird, I was peer reviewing a manuscript earlier today and actually thought of something similar. The manuscript needs some work to improve its English and I wondered if some of the broken English passages were a test to see what nonsense a referee like me would put up with. I don't think it is a test but the thought did occur to me.
golergka 3y ago
> Maybe there should be a group of researchers that submit papers that seem believable with known issues to avoid scientific journals stagnating. There was. People were not very pleased. https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair
ttldr 3y ago
depends how you do it? good idea: https://en.wikipedia.org/wiki/Sokal_affair https://en.wikipedia.org/wiki/Sokal_affair bad idea: https://www.theverge.com/2021/4/22/22398156/university-minnesota-linux-kernal-ban-research https://www.theverge.com/2021/4/22/22398156/university-minne...
WhitneyLand 3y ago
“…submit papers that seem believable with known issues to avoid scientific journals stagnating…” It’s called a scholarly hoax, and it’s been done more than once: https://en.m.wikipedia.org/wiki/List_of_scholarly_publishing_stings https://en.m.wikipedia.org/wiki/List_of_scholarly_publishing...
dotnet00 3y ago
Years before this current trend of LLM generated papers, there was a tool IIRC by some MIT students which would generate plausible looking papers which when actually read were obviously garbage. The title and abstract would have the sort of words you would expect of a research paper, but the resulting sentences would be nonsense. The idea being that if the submission got published, the review process was obviously a total joke. I think they even had some examples that had actually been published. Edit, Link: https://pdos.csail.mit.edu/archive/scigen/ https://pdos.csail.mit.edu/archive/scigen/
- agileAlligator 3y ago
  I'm having a hard time believing those papers weren't written by humans after a cursory glance
m463 3y ago
A more benign example was the college professor who started each lecture saying he was going to say one wrong thing. (it would keep the students alert) or maybe Van Halen's "CRC check", where their venue contract would ask for a bunch of things set up in advance, including a bowl of M&Ms with the brown ones removed. (it was a test of diligence, that they read the contract and knew the requirements)
3seashells 3y ago
Maybe the review institution of a paper should always add one known flaw that peer review must find to ensure a quality control of the process?

Nican 3y ago

Careful not to repeat what the University of Minnesota did to the Linux kernel, when willing submitting security bugs to write out a paper about trust.

tedunangst 3y ago
The duality of HN. I want other people to be forced to filter out bullshit, but I only want to receive forthright submissions myself.
- cobbal 3y ago
  It's really weird. I'm starting to suspect hacker news isn't a single person.
  - sltkr 3y ago
    I'm pretty sure I'm you.
    - dllthomas 3y ago
      Who am us, anyway?
  - Regnore 3y ago
    Then why is it called the singular “Hacker” news?
    - golergka 3y ago
      Because there's only one intended reader who is sufficiently educated and smart. All others of us are impostors who should not have pursued this career at all.
    - thomastjeffery 3y ago
      If only the Hackers Knew..

Mistletoe 3y ago

As a scientist I see it the other way, where people will just ignore future discoveries and science and this goes on the trash pile with other bad science that makes the public distrust science. We saw the endgame of that with Covid and it was really ugly. The public doesn't understand the minutiae of each study and just gets really tired with "Carbs are bad" and "Now carbs are good again" headlines.

whycome 3y ago

When it's "not even wrong" it's easy to dismiss something. But if it takes some work to disprove, there can be some cool insights and knowledge (and new avenues of exploration).

darklycan51 3y ago

The problem is when too much fake stuff comes out (example: health/climate) to the point where average people cannot see what is real or not anymore.

furyofantares 3y ago

Are you talking about the hype around LK-99, which this article is not about?

d3w4s9 3y ago

Nah. The reality is that very very few people will "make great discoveries and contributions" even for graduate students in those fields or after getting a PhD degree, and many people who are good enough to become professors do mostly mediocre research or publish papers that have very little impact. For this LK-99 thing, I doubt more than 5 people of all those that are interested will be making any significant contributions in 10 to 20 years.