6 ms·

As I’ve transitioned to more exploratory and researchy roles in my career, I have started to understand the science fraudsters like Jan Hendrik Schön. When you

As I’ve transitioned to more exploratory and researchy roles in my career, I have started to understand the science fraudsters like Jan Hendrik Schön. When you spent an entire week working on a test or experiment that you know should work, at least if you give it enough time, but it isn’t for whatever reason, it can be extremely tempting to invent the numbers that you think it should be, especially if your employer is pressuring you for a result. Now, obviously, reason we run these tests is precisely because we don’t actually know what the results will be, but that’s sometimes more obvious in hindsight. Obviously it’s wrong, and I haven’t done it, but I would be lying if I said that the thought hadn’t crossed my mind.

bluefirebrand 2y ago
> When you spent an entire week working on a test or experiment that you know should work I thought the whole point of doing experiments was to challenge what we "know" so we can refine our understanding?
- MathMonkeyMan 2y ago
  Because of that, backing up a claim with research adds weight to the claim. If the claim is false, though, you can still sometimes get research to support it. If you or the researcher stands to profit from the false claim, then there is a conflict of interest.
- llamaimperative 2y ago
  Sure in la-la-land where science isn't conducted by humans. In reality, scientists are highly motivated (i.e. biased) individuals like anyone else. Therefore science cannot be done effectively by individuals. The system that derives truth from experiments - the actual scientific system - is the competitive dynamic between scientists who are trying to tarnish each others' legacies and bolster their own. The scientific method etc. primarily makes scientific claims scrutinizable in detail, but without scrutiny they are still highly liable to produce false information.
  - wredue 2y ago
    And yet, it is still the best we got for also producing highly reliable and correct information. Personally, I think the “highly” in your statement is quite over exaggerated. Humans can be convinced to produce bad science, for sure, and there are even journals set up by religious orgs that specifically exist to do just that. But at the same time, science landed humans on the moon.
    - kelnos 2y ago
      > Personally, I think the “highly” in your statement is quite over exaggerated. Except that the entire point of the article here is that it's not exaggerated. > But at the same time, science landed humans on the moon. Cherry-picking a highly successful, well-known example doesn't prove a point.
      - Tainnor 2y ago
        > Cherry-picking a highly successful, well-known example doesn't prove a point. There must be hundreds, if not thousands, of successful scientific discoveries that went into something as complicated as the moon landing, and if you still don't think that's convincing, just look at the world around you - which looks just radically different from the world of, say, just a couple hundred years ago.
      - consteval 2y ago
        > Cherry-picking As if our lifespans and quality of life haven't been drastically improved by modern medicine. I mean, we can cut people open and replace entire parts of them and they're fine. They don't even get sick anymore - thanks germ theory and aseptic technique! Do you not understand how much of a marvel that is? Before that, people used to get cuts and scratches and just... die. We can now fully rummage inside an arbitrary person's internal organs. And don't even get me started on long-term illnesses. High blood pressure and cholesterol has been killing humans since forever, and we have medicine that just fixes that. And now, we're getting medicine to rewire our brains to prevent addiction in the first place (semaglutide)
    - 2y ago
- renewiltord 2y ago
  Well, that depends. What are you paying the guy to do?
- SilasX 2y ago
  I think that’s what the parent is acknowledging in the end of the second paragraph.
- tombert 2y ago
  In theory, but it is extremely easy to get into the mindset that your hypothesis is absolutely true, and as such your goal is to prove that hypothesis. I’ve never fabricated numbers for anything I’ve done, but there certainly have been times where I thought about it, usually after the fourth or fifth broken multi-hour test, especially if the test breakage doesn’t directly contradict the hypothesis.
  - DiggyJohnson 2y ago
    Thanks for staying your point so clearly. I’m a bystander to this discussion, but agree with you about the reality of this.
  - thayne 2y ago
    Maybe it's different in other fields, but from my background in physics it seems like if your hypothesis is wrong that is usually way more interesting than it being right. As long as it isn't just because of some contamination in the data. Although, contrary to what I was taught in elementary school, most of the experiments in the physics department of my university didn't even really have a hypothesis. They were usually either of the form "we are going to do this thing, and see what happens", or "we're going to measure this thing more accurately than anyone before".
    - tombert 2y ago
      So I'm not a "scientist" really, I'm just a bit more of a theory-focused software engineer. An example has been times where I really want to use a certain concurrency style, and I'm convinced that it should be faster than the way we're doing things before, so I will write a few non-trivial tests to make sure that's right, and I'll get inconclusive numbers.
      - thayne 2y ago
        That's still an interesting result. If you thought it would have a significant impact, and it doesn't, that means something is wrong with your mental model that lead you to think it would be faster. And if you can figure out why, that can help you find a solution that is faster. Of course, that is still very frustrating if you are on a tight deadline and all you have is one thing you know doesn't work.
- ants_everywhere 2y ago
  There are externally motivated scientists who are in it for the prestige or awards. Some fields are more like this than others, but they show up in all fields. Plus these days there's a lot of pressure to run universities more like businesses. To eat, academics have to hit certain numbers, so you see behaviors common in business like faking the KPIs.
- kqr 2y ago
  That's a valid way to look at it, but Fisher (who all but invented hypothesis testing) took a different perspective. To him, most things we know because of informal experiences. Only when trying to find small effects or when we have insufficient experience do we conduct experiments, which are effectively experience meticulously planned in advance. A significant result in an experiment, according to Fisher, is just an experience to add to the mental pros-and-cons list. It is not defintive proof of anything.
- _fizz_buzz_ 2y ago
  Setting up real experiments in a lab is super hard e.g. is all equipment properly calibrated, is the way I am measuring actually right, are my reference measurements correct, are my samples "clean". A lot of things can go wrong that it is even sometimes challenging to replicate experiments that are 100% known to work. So, it takes some discipline to not cheat in the sense that e.g. one cleans up the data a bit too much.
pragmomm 2y ago
You should also understand that there are external forces here, like state sponsorships that monetarily rewards for scientists to simply file enough research findings. The startling rise in the publication of sham science papers has its roots in China, where young doctors and scientists seeking promotion were required to have published scientific papers. Shadow organisations – known as “paper mills” – began to supply fabricated work for publication in journals there. https://www.theguardian.com/science/2024/feb/03/the-situation-has-become-appalling-fake-scientific-papers-push-research-credibility-to-crisis-point https://www.theguardian.com/science/2024/feb/03/the-situatio... The number of retractions issued for research articles in 2023 has passed 10,000 — smashing annual records — as publishers struggle to clean up a slew of sham papers and peer-review fraud. Among large research-producing nations, Saudi Arabia, Pakistan, Russia and China have the highest retraction rates over the past two decades, a Nature analysis has found. https://www.nature.com/articles/d41586-023-03974-8 https://www.nature.com/articles/d41586-023-03974-8 That's why a recent article https://news.ycombinator.com/item?id=41607430 https://news.ycombinator.com/item?id=41607430, where the measurement of China leads world in 57 of 64 critical technologies was based on number of journal citations, was laughable.
- a_bonobo 2y ago
  Talking with some Chinese colleagues in the past, they were talking about having a 'base' salary which was not enough to have a family on. For every published paper they'd get a one-time payment. So you'd have to get a bunch of papers out every year just to survive; no wonder people start to invent papers. Of course the same thing is happening in the 'Western' world too, with a publication ratchet going on. New hire has 50 papers out? OK! The next pool of potential hires has 50, 55, 52 papers out, so obviously you take the 55 papers-person. You want outstanding people! Then the next hire needs 60 papers. And so on.
  - nativeit 2y ago
    ...an effect known as "wonkflation".
- resoluteteeth 2y ago
  I think there are maybe two separate issues here. Paper mills are bad but mostly from the perspective of academic institutions trying to verify people's credentials/resumes. Paper mills aren't really that much of a concern in the sense of published research results being false in the way the article is talking about because people aren't really reading the papers they publish. In that sense it doesn't really matter if there are places where non-scientists need to get one paper published to check some box to get a promotion, because nobody is really considering those papers part of established scientific knowledge. On the other hand, scientists intentionally (by actually falsifying data) or unintentionally (as a result of statistical effects of what is researched and what is published) publishing bogus results in journals that are considered legitimate which aren't paper mills actually causes real harm as a result of people believing the bogus results, and unfortunately the pressures that cause that (publishing papers quickly, getting publishable results, etc.) exist everywhere, and definitely not just in China, nor did they originate in China.
aaron695 2y ago
[dead]
highcountess 2y ago
I’ve been in a meeting with government research officials where a director of the primary global institution in that field described how when she writes or does research and writes papers she draws a graph she needs to support her research or a point she is trying to make and then goes to look for the data to create that graph. Maybe I’m missing something, but I do not believe that is the way it is stopped to go. Btw, she has a PhD and failed up into a global scale. I’ve been meaning to find out if there are any open tools to evaluate someone’s dissertation. It was equal part stunning and seemingly a bit traumatizing to me considering I still remember it as if it had happened earlier today. I think what surprised me too was her open admission of it, even with external parties present.
- randomdata 2y ago
  So she establishes a hypothesis (draws a graph or picks a point to make) and then tests it through experimentation (looks for data to support the hypothesis)? Isn't that just the scientific method worded another way?
  - elashri 2y ago
    Wait until the GP knows about how scientists generate Monte-Carlo (MC) simulation data to see what a positive results looks like and then do meta analysis for both real data and MC.
veunes 2y ago
Rapid outcomes should not be a priority
BeetleB 2y ago
Only a week? The stakes are higher my friend. It's usually months at a minimum.
- tombert 2y ago
  Heh, totally fair, I'm not a scientist, I'm just a more research-oriented software engineer, and generally I have to keep my tests smaller in scope. No doubt that in the case of physics and chemistry and the like, testing can be a lot longer.

bluefirebrand 2y ago

> When you spent an entire week working on a test or experiment that you know should work I thought the whole point of doing experiments was to challenge what we "know" so we can refine our understanding?

MathMonkeyMan 2y ago
Because of that, backing up a claim with research adds weight to the claim. If the claim is false, though, you can still sometimes get research to support it. If you or the researcher stands to profit from the false claim, then there is a conflict of interest.
llamaimperative 2y ago
Sure in la-la-land where science isn't conducted by humans. In reality, scientists are highly motivated (i.e. biased) individuals like anyone else. Therefore science cannot be done effectively by individuals. The system that derives truth from experiments - the actual scientific system - is the competitive dynamic between scientists who are trying to tarnish each others' legacies and bolster their own. The scientific method etc. primarily makes scientific claims scrutinizable in detail, but without scrutiny they are still highly liable to produce false information.
- wredue 2y ago
  And yet, it is still the best we got for also producing highly reliable and correct information. Personally, I think the “highly” in your statement is quite over exaggerated. Humans can be convinced to produce bad science, for sure, and there are even journals set up by religious orgs that specifically exist to do just that. But at the same time, science landed humans on the moon.
  - kelnos 2y ago
    > Personally, I think the “highly” in your statement is quite over exaggerated. Except that the entire point of the article here is that it's not exaggerated. > But at the same time, science landed humans on the moon. Cherry-picking a highly successful, well-known example doesn't prove a point.
    - Tainnor 2y ago
      > Cherry-picking a highly successful, well-known example doesn't prove a point. There must be hundreds, if not thousands, of successful scientific discoveries that went into something as complicated as the moon landing, and if you still don't think that's convincing, just look at the world around you - which looks just radically different from the world of, say, just a couple hundred years ago.
    - consteval 2y ago
      > Cherry-picking As if our lifespans and quality of life haven't been drastically improved by modern medicine. I mean, we can cut people open and replace entire parts of them and they're fine. They don't even get sick anymore - thanks germ theory and aseptic technique! Do you not understand how much of a marvel that is? Before that, people used to get cuts and scratches and just... die. We can now fully rummage inside an arbitrary person's internal organs. And don't even get me started on long-term illnesses. High blood pressure and cholesterol has been killing humans since forever, and we have medicine that just fixes that. And now, we're getting medicine to rewire our brains to prevent addiction in the first place (semaglutide)
  - 2y ago
renewiltord 2y ago
Well, that depends. What are you paying the guy to do?
SilasX 2y ago
I think that’s what the parent is acknowledging in the end of the second paragraph.
tombert 2y ago
In theory, but it is extremely easy to get into the mindset that your hypothesis is absolutely true, and as such your goal is to prove that hypothesis. I’ve never fabricated numbers for anything I’ve done, but there certainly have been times where I thought about it, usually after the fourth or fifth broken multi-hour test, especially if the test breakage doesn’t directly contradict the hypothesis.
- DiggyJohnson 2y ago
  Thanks for staying your point so clearly. I’m a bystander to this discussion, but agree with you about the reality of this.
- thayne 2y ago
  Maybe it's different in other fields, but from my background in physics it seems like if your hypothesis is wrong that is usually way more interesting than it being right. As long as it isn't just because of some contamination in the data. Although, contrary to what I was taught in elementary school, most of the experiments in the physics department of my university didn't even really have a hypothesis. They were usually either of the form "we are going to do this thing, and see what happens", or "we're going to measure this thing more accurately than anyone before".
  - tombert 2y ago
    So I'm not a "scientist" really, I'm just a bit more of a theory-focused software engineer. An example has been times where I really want to use a certain concurrency style, and I'm convinced that it should be faster than the way we're doing things before, so I will write a few non-trivial tests to make sure that's right, and I'll get inconclusive numbers.
    - thayne 2y ago
      That's still an interesting result. If you thought it would have a significant impact, and it doesn't, that means something is wrong with your mental model that lead you to think it would be faster. And if you can figure out why, that can help you find a solution that is faster. Of course, that is still very frustrating if you are on a tight deadline and all you have is one thing you know doesn't work.
ants_everywhere 2y ago
There are externally motivated scientists who are in it for the prestige or awards. Some fields are more like this than others, but they show up in all fields. Plus these days there's a lot of pressure to run universities more like businesses. To eat, academics have to hit certain numbers, so you see behaviors common in business like faking the KPIs.
kqr 2y ago
That's a valid way to look at it, but Fisher (who all but invented hypothesis testing) took a different perspective. To him, most things we know because of informal experiences. Only when trying to find small effects or when we have insufficient experience do we conduct experiments, which are effectively experience meticulously planned in advance. A significant result in an experiment, according to Fisher, is just an experience to add to the mental pros-and-cons list. It is not defintive proof of anything.
_fizz_buzz_ 2y ago
Setting up real experiments in a lab is super hard e.g. is all equipment properly calibrated, is the way I am measuring actually right, are my reference measurements correct, are my samples "clean". A lot of things can go wrong that it is even sometimes challenging to replicate experiments that are 100% known to work. So, it takes some discipline to not cheat in the sense that e.g. one cleans up the data a bit too much.

pragmomm 2y ago

You should also understand that there are external forces here, like state sponsorships that monetarily rewards for scientists to simply file enough research findings. The startling rise in the publication of sham science papers has its roots in China, where young doctors and scientists seeking promotion were required to have published scientific papers. Shadow organisations – known as “paper mills” – began to supply fabricated work for publication in journals there. https://www.theguardian.com/science/2024/feb/03/the-situation-has-become-appalling-fake-scientific-papers-push-research-credibility-to-crisis-point https://www.theguardian.com/science/2024/feb/03/the-situatio... The number of retractions issued for research articles in 2023 has passed 10,000 — smashing annual records — as publishers struggle to clean up a slew of sham papers and peer-review fraud. Among large research-producing nations, Saudi Arabia, Pakistan, Russia and China have the highest retraction rates over the past two decades, a Nature analysis has found. https://www.nature.com/articles/d41586-023-03974-8 https://www.nature.com/articles/d41586-023-03974-8 That's why a recent article https://news.ycombinator.com/item?id=41607430 https://news.ycombinator.com/item?id=41607430, where the measurement of China leads world in 57 of 64 critical technologies was based on number of journal citations, was laughable.

a_bonobo 2y ago
Talking with some Chinese colleagues in the past, they were talking about having a 'base' salary which was not enough to have a family on. For every published paper they'd get a one-time payment. So you'd have to get a bunch of papers out every year just to survive; no wonder people start to invent papers. Of course the same thing is happening in the 'Western' world too, with a publication ratchet going on. New hire has 50 papers out? OK! The next pool of potential hires has 50, 55, 52 papers out, so obviously you take the 55 papers-person. You want outstanding people! Then the next hire needs 60 papers. And so on.
- nativeit 2y ago
  ...an effect known as "wonkflation".
resoluteteeth 2y ago
I think there are maybe two separate issues here. Paper mills are bad but mostly from the perspective of academic institutions trying to verify people's credentials/resumes. Paper mills aren't really that much of a concern in the sense of published research results being false in the way the article is talking about because people aren't really reading the papers they publish. In that sense it doesn't really matter if there are places where non-scientists need to get one paper published to check some box to get a promotion, because nobody is really considering those papers part of established scientific knowledge. On the other hand, scientists intentionally (by actually falsifying data) or unintentionally (as a result of statistical effects of what is researched and what is published) publishing bogus results in journals that are considered legitimate which aren't paper mills actually causes real harm as a result of people believing the bogus results, and unfortunately the pressures that cause that (publishing papers quickly, getting publishable results, etc.) exist everywhere, and definitely not just in China, nor did they originate in China.

aaron695 2y ago

[dead]

highcountess 2y ago

I’ve been in a meeting with government research officials where a director of the primary global institution in that field described how when she writes or does research and writes papers she draws a graph she needs to support her research or a point she is trying to make and then goes to look for the data to create that graph. Maybe I’m missing something, but I do not believe that is the way it is stopped to go. Btw, she has a PhD and failed up into a global scale. I’ve been meaning to find out if there are any open tools to evaluate someone’s dissertation. It was equal part stunning and seemingly a bit traumatizing to me considering I still remember it as if it had happened earlier today. I think what surprised me too was her open admission of it, even with external parties present.

randomdata 2y ago
So she establishes a hypothesis (draws a graph or picks a point to make) and then tests it through experimentation (looks for data to support the hypothesis)? Isn't that just the scientific method worded another way?
- elashri 2y ago
  Wait until the GP knows about how scientists generate Monte-Carlo (MC) simulation data to see what a positive results looks like and then do meta analysis for both real data and MC.

veunes 2y ago

Rapid outcomes should not be a priority

BeetleB 2y ago

Only a week? The stakes are higher my friend. It's usually months at a minimum.

tombert 2y ago
Heh, totally fair, I'm not a scientist, I'm just a more research-oriented software engineer, and generally I have to keep my tests smaller in scope. No doubt that in the case of physics and chemistry and the like, testing can be a lot longer.