7 ms·

The situation is out of control. I've also seen that during peer review a reviewer may give a list of publications they 'suggest' for the literature review, som

by bArray 3y ago

The situation is out of control. I've also seen that during peer review a reviewer may give a list of publications they 'suggest' for the literature review, sometimes completely unrelated fields. If you choose not to, you will likely see your paper rejected. The truth is that almost all papers published across almost all fields cannot be replicated, full stop. For those that can, the results rarely match up, indicating that the tests are chosen to boost the performance of the method. If you want to boost citations legitimately, purposefully leak your own paper to get past the publisher pay wall, make it colourful and accessible (if it's too dense they are unlikely to understand it), give examples on how to replicate your method, the abstract is likely the only part that will be read so make it count.

eimrine 3y ago
> The truth is that almost all papers published across almost all fields cannot be replicated, full stop. This is not true for Mathematics.
- sukruh 3y ago
  Sure, but most Maths papers are read by like 3 people in total globally, so there's that.
  - psychoslave 3y ago
    Citation needed. XD
    - lucioperca 3y ago
      The pareto effect!?
  - User3456335 3y ago
    If that's true it's only because most people don't understand these pure maths fields. But their results are generally very useful to the applied mathematicians (whose results are and should be used in most other fields) and in this way there is still an impact although it's not always reflected in the citations. The fact is that these pure mathematicians could almost surely write the more applied papers but don't because it would feel to repetitive. You don't need to read Euler's work in order to indirectly benefit from it. It's the same for a lot of pure math.
    - lupire 3y ago
      Applied math does not use much pure math. It hasn't for decades.
    - bsdpufferfish 3y ago
      This is not even close. Pure math is often complete clueless about how it applies. For example Developing algorithms to compute properties used in the most basic pure math is a massive pursuit.
    - itointegral 3y ago
      As an applied mathematician, I'd say this depends on the field. Pure maths more often than not comes with assumptions that make the theory more elegant, but by doing so they also drastically limit its application to real life scenarios.
- nxpnsv 3y ago
  It is not true for Physics either.
  - josephg 3y ago
    Or computer science, at least when the code is also published. (That said, performance measurements in most papers are complete junk. Most code written by academics is missing trivial, obvious optimisations which could completely change any comparative benchmarking results.)
    - PaulHoule 3y ago
      I'm not sure that the average CS paper really tested the algorithm they thought they were testing. That comes from a lot of commercial projects where people weren't using the algorithm they thought they were using (even simple algorithms) and a few cases of working with code from CS researchers. There was that time I got a C program from a famous ML researcher, tried to run it and it segfaulted, I pointed gdb at it and realized it was crashing before it got into main(). Understanding why was an educational experience (don't allocate a 8 GB array you never use on a 32-bit machine... It worked for the researcher because he had a 64-bit machine) but it did not increase my faith in the code quality of academic projects. Some CS researchers are brilliant programmers but the fact is that ordinary devs make a living writing code and CS researchers make a living writing papers.
      - eimrine 3y ago
        Allocating a big struct for everything and not using pointers is one of possible techniques of dealing with data in C programs. Anyway having a requirement of 8G RAM is not a good example of something "can not be replicated".
        PaulHoule 3y ago
        I know all about alternate ways to deal with memory, if I was just programming purely for fun I'd write assembly and not waste time moving the stack pointer around and other meaningless activities associated with calling conventions. It's a sign of poor quality code. He probably was using the array for something at one point, quit using it, and never bothered to delete it. And of course it is C so there are no unit tests despite it being the kind of code that is eminently unit testable (if it wasn't written in C, not like you couldn't write unit tests for C but who does?)
- PaulHoule 3y ago
  I dunno. When I was in grad school I regularly would find papers in Annals of Mathematical Physics that would go on for 50 pages and have critical errors on page 17 and 45. I did a calculation like that which I know what right at the equation-by-equation level because I made unit tests for everything, but the whole thing was still useless (published in my thesis but not Annals) because the coordinate system we were using converted a periodic orbit into something like a spiral so the we couldn't compute a topological factor because there was no topological invariant...
  - alafactor 3y ago
    I've published in some well-respected, but not highest-tier, journals (think impact factor ~ 5-7), where this was absolutely the case. You could very often identify the reviewer by who the corresponding author of the twenty papers they asked you to cite was. Perhaps it's field dependent, my field (chemistry) was not always as, shall we say, ethical as some of the other fields.
    - PaulHoule 3y ago
      … I thought people in physics were pretty sportsmanlike but my understanding was that biologists would stab you in the back for the slightest advantage. I would complain about how hard it was to make it in physics but in biology it seemed almost impossible to get established since young researchers just couldn’t get grants period.
  - Balgair 3y ago
    I had on old physics professor who said that by the second page of any math/physics/equation-heavy paper, assume all signs are randomized. I've discovered in the intervening years since graduation that he was wrong, you should start assuming by the bottom of the first page.
    - PaulHoule 3y ago
      I told my thesis advisor that!
- Verdex 3y ago
  I don't know about that. IIRC the creation of HoTT was because someone was notified that one of his important papers from earlier in his career had a non-trivial flaw in it. To be sure, reproduction is a bit easier in mathematics because you normally don't need exotic equipment, just time. However, that doesn't mean that people are necessarily taking the time to double check everything with sufficient rigor.
- nickpsecurity 3y ago
  In the article below, they claimed one study said a third of the papers had a false theorem. https://news.ycombinator.com/item?id=17430577 https://news.ycombinator.com/item?id=17430577 A number of papers on formal verification reported that restructuring their specifications for proof revealed problems they missed in the specification. Other works found through empirical testing that the specs didn’t match reality. Many theories in physics changed that way. So, we should assume they’re all false by default until the specs are reviewed, the proofs are checked, and there’s empirical validation. If theoretical, its component theories must be built on observed truths instead of imaginary constructs. Then, we might trust the paper (i.e. mathematical claims).
etrautmann 3y ago
this is overstating the case. Throwing out all publications is ridiculous. Yes, many fail to replicate, but not nearly all. A larger concern may be that many fail to generalize (e.g. mouse cancer cures fail to work in humans). It's true that it's hard to ignore all of the citations a reviewer suggests, but that's part of the role of an editor, and the response to reviewers (while being a frustrating exercise) can make it clear what is and isn't an appropriate citation, which serves as communication to the editor. Yes, this process is frustratingly low bandwidth
- stathibus 3y ago
  How do you know? It's rare that anyone even tries to replicate scientific results outside of clinical medicine
- patall 3y ago
  Exactly. What kind of editors does your field have? In my field, that kind of oversight would have been called out on twitter immediately.
eigenket 3y ago
> I've also seen that during peer review a reviewer may give a list of publications they 'suggest' for the literature review, sometimes completely unrelated fields. If you choose not to, you will likely see your paper rejected. You are severely misrepresenting how things work here. A reviewer doesn't accept or reject a paper. A reviewer gives their response to an editor and the editor makes a decision. The editor knows who the reviewer is, and has access to the reviews and the author's response to them. If the editor sees the reviewer has suggested adding bunch of references to their own work, and has suggested rejecting the paper after the authors refuse this then the editor is going to know what is going on. They aren't completely stupid. > purposefully leak your own paper to get past the publisher pay wall Many papers in maths, physics and computer science appear on arXiv before being submitted for publication. In the fields I have worked in you can replace the "most" with "pretty much all".
- jwsteigerwalt 3y ago
  I think you’re giving the editors too much credit.
  - eigenket 3y ago
    I guess it depends on the journal. If you're publishing in some low tier or even predatory journal then yeah there might be problems like this with the editor but in good, respected journals the editors usually do their jobs properly.
vladms 3y ago
> The truth is that almost all papers published across almost all fields cannot be replicated, full stop. Isn't it ironic that we can't replicate your assessment?... I personally have no idea how much papers can be replicated or not, but replicating an experiment would not guarantee that the idea/conclusion in the paper is sound (ex: paper extracting some data, observing a correlation and concluding it is an implication). I do agree that it should be made easier to replicate for the fields where this is possible, but there are many other problems.
asah 3y ago
Is replication a big problem in computer science? If anything, I'd think replication has gotten easier with the advent of open source and the expectation that top papers will include git repos which make it easy to reproduce. The advent of docker, locked-version-dependencies etc have all contributed as well. 10-20 years ago, we didn't have this - but I also didn't see * widespread * problems even then: if someone published a result saying that technique <x> led to <y> performance gain, I thought it usually did? (that said, computer science research doesn't have the same money on the line as medicine and the "perish" part of the publish-or-perish equation isn't the same when the fallback for failed academics is still six figure salaries at tech companies)
- nradov 3y ago
  What are the odds that in 2044 we will still be able to run the code published with CS papers today? If it's pure Java SE or ANSI C with no dependencies then it will probably still work, but anything more complex tends to decay pretty rapidly.
  - bsdpufferfish 3y ago
    Most CS papers are theoretical, and this is another reason why HN is wrong about requiring papers to publish code.
newzisforsukas 3y ago
> The truth is that almost all papers published across almost all fields cannot be replicated, full stop. And yet we still have advances in technology. I wonder how this academic nihilism matches reality?
- mcmoor 3y ago
  This is where I got cognitive dissonance. Can both actually be true at once? If not, which one is wrong?

by bArray 3y ago

eimrine 3y ago

> The truth is that almost all papers published across almost all fields cannot be replicated, full stop. This is not true for Mathematics.

sukruh 3y ago
Sure, but most Maths papers are read by like 3 people in total globally, so there's that.
- psychoslave 3y ago
  Citation needed. XD
  - lucioperca 3y ago
    The pareto effect!?
- User3456335 3y ago
  If that's true it's only because most people don't understand these pure maths fields. But their results are generally very useful to the applied mathematicians (whose results are and should be used in most other fields) and in this way there is still an impact although it's not always reflected in the citations. The fact is that these pure mathematicians could almost surely write the more applied papers but don't because it would feel to repetitive. You don't need to read Euler's work in order to indirectly benefit from it. It's the same for a lot of pure math.
  - lupire 3y ago
    Applied math does not use much pure math. It hasn't for decades.
  - bsdpufferfish 3y ago
    This is not even close. Pure math is often complete clueless about how it applies. For example Developing algorithms to compute properties used in the most basic pure math is a massive pursuit.
  - itointegral 3y ago
    As an applied mathematician, I'd say this depends on the field. Pure maths more often than not comes with assumptions that make the theory more elegant, but by doing so they also drastically limit its application to real life scenarios.
nxpnsv 3y ago
It is not true for Physics either.
- josephg 3y ago
  Or computer science, at least when the code is also published. (That said, performance measurements in most papers are complete junk. Most code written by academics is missing trivial, obvious optimisations which could completely change any comparative benchmarking results.)
  - PaulHoule 3y ago
    I'm not sure that the average CS paper really tested the algorithm they thought they were testing. That comes from a lot of commercial projects where people weren't using the algorithm they thought they were using (even simple algorithms) and a few cases of working with code from CS researchers. There was that time I got a C program from a famous ML researcher, tried to run it and it segfaulted, I pointed gdb at it and realized it was crashing before it got into main(). Understanding why was an educational experience (don't allocate a 8 GB array you never use on a 32-bit machine... It worked for the researcher because he had a 64-bit machine) but it did not increase my faith in the code quality of academic projects. Some CS researchers are brilliant programmers but the fact is that ordinary devs make a living writing code and CS researchers make a living writing papers.
    - eimrine 3y ago
      Allocating a big struct for everything and not using pointers is one of possible techniques of dealing with data in C programs. Anyway having a requirement of 8G RAM is not a good example of something "can not be replicated".
      - PaulHoule 3y ago
        I know all about alternate ways to deal with memory, if I was just programming purely for fun I'd write assembly and not waste time moving the stack pointer around and other meaningless activities associated with calling conventions. It's a sign of poor quality code. He probably was using the array for something at one point, quit using it, and never bothered to delete it. And of course it is C so there are no unit tests despite it being the kind of code that is eminently unit testable (if it wasn't written in C, not like you couldn't write unit tests for C but who does?)
PaulHoule 3y ago
I dunno. When I was in grad school I regularly would find papers in Annals of Mathematical Physics that would go on for 50 pages and have critical errors on page 17 and 45. I did a calculation like that which I know what right at the equation-by-equation level because I made unit tests for everything, but the whole thing was still useless (published in my thesis but not Annals) because the coordinate system we were using converted a periodic orbit into something like a spiral so the we couldn't compute a topological factor because there was no topological invariant...
- alafactor 3y ago
  I've published in some well-respected, but not highest-tier, journals (think impact factor ~ 5-7), where this was absolutely the case. You could very often identify the reviewer by who the corresponding author of the twenty papers they asked you to cite was. Perhaps it's field dependent, my field (chemistry) was not always as, shall we say, ethical as some of the other fields.
  - PaulHoule 3y ago
    … I thought people in physics were pretty sportsmanlike but my understanding was that biologists would stab you in the back for the slightest advantage. I would complain about how hard it was to make it in physics but in biology it seemed almost impossible to get established since young researchers just couldn’t get grants period.
- Balgair 3y ago
  I had on old physics professor who said that by the second page of any math/physics/equation-heavy paper, assume all signs are randomized. I've discovered in the intervening years since graduation that he was wrong, you should start assuming by the bottom of the first page.
  - PaulHoule 3y ago
    I told my thesis advisor that!
Verdex 3y ago
I don't know about that. IIRC the creation of HoTT was because someone was notified that one of his important papers from earlier in his career had a non-trivial flaw in it. To be sure, reproduction is a bit easier in mathematics because you normally don't need exotic equipment, just time. However, that doesn't mean that people are necessarily taking the time to double check everything with sufficient rigor.
nickpsecurity 3y ago
In the article below, they claimed one study said a third of the papers had a false theorem. https://news.ycombinator.com/item?id=17430577 https://news.ycombinator.com/item?id=17430577 A number of papers on formal verification reported that restructuring their specifications for proof revealed problems they missed in the specification. Other works found through empirical testing that the specs didn’t match reality. Many theories in physics changed that way. So, we should assume they’re all false by default until the specs are reviewed, the proofs are checked, and there’s empirical validation. If theoretical, its component theories must be built on observed truths instead of imaginary constructs. Then, we might trust the paper (i.e. mathematical claims).

etrautmann 3y ago

this is overstating the case. Throwing out all publications is ridiculous. Yes, many fail to replicate, but not nearly all. A larger concern may be that many fail to generalize (e.g. mouse cancer cures fail to work in humans). It's true that it's hard to ignore all of the citations a reviewer suggests, but that's part of the role of an editor, and the response to reviewers (while being a frustrating exercise) can make it clear what is and isn't an appropriate citation, which serves as communication to the editor. Yes, this process is frustratingly low bandwidth

stathibus 3y ago
How do you know? It's rare that anyone even tries to replicate scientific results outside of clinical medicine
patall 3y ago
Exactly. What kind of editors does your field have? In my field, that kind of oversight would have been called out on twitter immediately.

eigenket 3y ago

> I've also seen that during peer review a reviewer may give a list of publications they 'suggest' for the literature review, sometimes completely unrelated fields. If you choose not to, you will likely see your paper rejected. You are severely misrepresenting how things work here. A reviewer doesn't accept or reject a paper. A reviewer gives their response to an editor and the editor makes a decision. The editor knows who the reviewer is, and has access to the reviews and the author's response to them. If the editor sees the reviewer has suggested adding bunch of references to their own work, and has suggested rejecting the paper after the authors refuse this then the editor is going to know what is going on. They aren't completely stupid. > purposefully leak your own paper to get past the publisher pay wall Many papers in maths, physics and computer science appear on arXiv before being submitted for publication. In the fields I have worked in you can replace the "most" with "pretty much all".

jwsteigerwalt 3y ago
I think you’re giving the editors too much credit.
- eigenket 3y ago
  I guess it depends on the journal. If you're publishing in some low tier or even predatory journal then yeah there might be problems like this with the editor but in good, respected journals the editors usually do their jobs properly.

vladms 3y ago

> The truth is that almost all papers published across almost all fields cannot be replicated, full stop. Isn't it ironic that we can't replicate your assessment?... I personally have no idea how much papers can be replicated or not, but replicating an experiment would not guarantee that the idea/conclusion in the paper is sound (ex: paper extracting some data, observing a correlation and concluding it is an implication). I do agree that it should be made easier to replicate for the fields where this is possible, but there are many other problems.

asah 3y ago

Is replication a big problem in computer science? If anything, I'd think replication has gotten easier with the advent of open source and the expectation that top papers will include git repos which make it easy to reproduce. The advent of docker, locked-version-dependencies etc have all contributed as well. 10-20 years ago, we didn't have this - but I also didn't see * widespread * problems even then: if someone published a result saying that technique <x> led to <y> performance gain, I thought it usually did? (that said, computer science research doesn't have the same money on the line as medicine and the "perish" part of the publish-or-perish equation isn't the same when the fallback for failed academics is still six figure salaries at tech companies)

nradov 3y ago
What are the odds that in 2044 we will still be able to run the code published with CS papers today? If it's pure Java SE or ANSI C with no dependencies then it will probably still work, but anything more complex tends to decay pretty rapidly.
- bsdpufferfish 3y ago
  Most CS papers are theoretical, and this is another reason why HN is wrong about requiring papers to publish code.

newzisforsukas 3y ago

> The truth is that almost all papers published across almost all fields cannot be replicated, full stop. And yet we still have advances in technology. I wonder how this academic nihilism matches reality?

mcmoor 3y ago
This is where I got cognitive dissonance. Can both actually be true at once? If not, which one is wrong?