5 ms·

I'm reminded of the somewhat recent news of a line of Alzheimer's research being based on a fabricated paper that was only caught many years later [0]. Previou

I'm reminded of the somewhat recent news of a line of Alzheimer's research being based on a fabricated paper that was only caught many years later [0]. Previously, we've relied on a number of heuristics to determine if something is real or not, such as if an image has any signs of a poor photoshop job, or if a written work has proper grammar. These heuristics work somewhat, but a motivated adversary can still get through. As the quality of fakes gets better, we'll need to develop better tools to dealing with them. For science, this could, hopefully, result in better work replicating previous works. I'm quite likely being overly optimistic, but there's a chance for positive outcomes here. [0]: https://www.science.org/content/article/potential-fabrication-research-images-threatens-key-theory-alzheimers-disease https://www.science.org/content/article/potential-fabricatio...

xiphias2 4y ago
The requirement to detect something fake is quite easy, and we knew it for a long time: publish all data code and everything to make the expriments reproducible. Even if everything is fake, the code has value for further research. It would be nice to have that as a minimun standard at this point, as I would prefer to see much less publications that can be trusted more than the current situation.
- danuker 4y ago
  > Even if everything is fake, the code has value for further research. I'd call that human-computer science partnership. If it checks out, it's not fake. Nonhuman scientists are still scientists.
  - venv 4y ago
    Are pipettes or vials scientists? Computers are tools like hammers, and have equal agency. There are no nonhuman scientists.
    - kleer001 4y ago
      Thank you. All this airy fairy talk of Ai is fun, but at the end of the day it's an inert tool (or toy) without human interaction.
      - rhn_mk1 4y ago
        For now.
        bombolo 4y ago
        And for foreseeable future.
    - danuker 4y ago
      If a developer codes up an AI to scour the web, write an article, and submit it to a scientific journal without letting the developer see the article, is the developer doing science? If someone trains a translation model between languages they don't know, is that someone a translator? I guess the users of said model would be "translators" as they would be doing the translation (without necessarily knowing the languages either).
      - reillyse 4y ago
        unsure if anyone is "doing science". Doing science is applying the scientific method. Making conjectures, deriving predictions from the hypotheses as logical consequences, and then carrying out experiments or empirical observations based on those predictions. Not sure AI is up to that, and it's debatable if it'll ever be able to make and test conjectures. There is a difference between symbol manipulation (like outputting text) and actual conjecture.
        danuker 4y ago
        > and it's debatable if it'll ever be able to make and test conjectures https://en.wikipedia.org/wiki/Robot_Scientist https://en.wikipedia.org/wiki/Robot_Scientist Adam is capable of: - hypothesizing to explain observations
- tptacek 4y ago
  That's not easy: a reproduction is a scientific project in its own right. Some research is straightforward to reproduce, but a lot of it isn't. That's not to say scientists shouldn't publish their data; they should.
  - asah 4y ago
    I believe OP is suggesting that this effort be borne by the original authors making the new claims, not by society to reproduce them. In the limit, any claims one makes are rejected unless you also provide a way to reproduce the claim. That's a lot of work, but perhaps the world doesn't need more monkeys and typewriters creating faux Shakespeare.
    - tptacek 4y ago
      What work? The original authors should reproduce their own work?
- bombolo 4y ago
  Paper authors: my algorithm is this fast! Here's some cool benchmarks charts! No you can't have any code to run the benchmarks yourself!
- rcxdude 4y ago
  Not really (I mean it helps, but it doesn't make it easy). It can still be very difficult to reproduce legitimate results, even with help from the original researcher. And faked results can last a long time even without reproduction because it's very easy to assume you're doing something wrong.

xiphias2 4y ago

The requirement to detect something fake is quite easy, and we knew it for a long time: publish all data code and everything to make the expriments reproducible. Even if everything is fake, the code has value for further research. It would be nice to have that as a minimun standard at this point, as I would prefer to see much less publications that can be trusted more than the current situation.

danuker 4y ago
> Even if everything is fake, the code has value for further research. I'd call that human-computer science partnership. If it checks out, it's not fake. Nonhuman scientists are still scientists.
- venv 4y ago
  Are pipettes or vials scientists? Computers are tools like hammers, and have equal agency. There are no nonhuman scientists.
  - kleer001 4y ago
    Thank you. All this airy fairy talk of Ai is fun, but at the end of the day it's an inert tool (or toy) without human interaction.
    - rhn_mk1 4y ago
      For now.
      - bombolo 4y ago
        And for foreseeable future.
  - danuker 4y ago
    If a developer codes up an AI to scour the web, write an article, and submit it to a scientific journal without letting the developer see the article, is the developer doing science? If someone trains a translation model between languages they don't know, is that someone a translator? I guess the users of said model would be "translators" as they would be doing the translation (without necessarily knowing the languages either).
    - reillyse 4y ago
      unsure if anyone is "doing science". Doing science is applying the scientific method. Making conjectures, deriving predictions from the hypotheses as logical consequences, and then carrying out experiments or empirical observations based on those predictions. Not sure AI is up to that, and it's debatable if it'll ever be able to make and test conjectures. There is a difference between symbol manipulation (like outputting text) and actual conjecture.
      - danuker 4y ago
        > and it's debatable if it'll ever be able to make and test conjectures https://en.wikipedia.org/wiki/Robot_Scientist https://en.wikipedia.org/wiki/Robot_Scientist Adam is capable of: - hypothesizing to explain observations
tptacek 4y ago
That's not easy: a reproduction is a scientific project in its own right. Some research is straightforward to reproduce, but a lot of it isn't. That's not to say scientists shouldn't publish their data; they should.
- asah 4y ago
  I believe OP is suggesting that this effort be borne by the original authors making the new claims, not by society to reproduce them. In the limit, any claims one makes are rejected unless you also provide a way to reproduce the claim. That's a lot of work, but perhaps the world doesn't need more monkeys and typewriters creating faux Shakespeare.
  - tptacek 4y ago
    What work? The original authors should reproduce their own work?
bombolo 4y ago
Paper authors: my algorithm is this fast! Here's some cool benchmarks charts! No you can't have any code to run the benchmarks yourself!
rcxdude 4y ago
Not really (I mean it helps, but it doesn't make it easy). It can still be very difficult to reproduce legitimate results, even with help from the original researcher. And faked results can last a long time even without reproduction because it's very easy to assume you're doing something wrong.