6 ms·

It is not true for Physics either.

by nxpnsv 3y ago

josephg 3y ago
Or computer science, at least when the code is also published. (That said, performance measurements in most papers are complete junk. Most code written by academics is missing trivial, obvious optimisations which could completely change any comparative benchmarking results.)
- PaulHoule 3y ago
  I'm not sure that the average CS paper really tested the algorithm they thought they were testing. That comes from a lot of commercial projects where people weren't using the algorithm they thought they were using (even simple algorithms) and a few cases of working with code from CS researchers. There was that time I got a C program from a famous ML researcher, tried to run it and it segfaulted, I pointed gdb at it and realized it was crashing before it got into main(). Understanding why was an educational experience (don't allocate a 8 GB array you never use on a 32-bit machine... It worked for the researcher because he had a 64-bit machine) but it did not increase my faith in the code quality of academic projects. Some CS researchers are brilliant programmers but the fact is that ordinary devs make a living writing code and CS researchers make a living writing papers.
  - eimrine 3y ago
    Allocating a big struct for everything and not using pointers is one of possible techniques of dealing with data in C programs. Anyway having a requirement of 8G RAM is not a good example of something "can not be replicated".
    - PaulHoule 3y ago
      I know all about alternate ways to deal with memory, if I was just programming purely for fun I'd write assembly and not waste time moving the stack pointer around and other meaningless activities associated with calling conventions. It's a sign of poor quality code. He probably was using the array for something at one point, quit using it, and never bothered to delete it. And of course it is C so there are no unit tests despite it being the kind of code that is eminently unit testable (if it wasn't written in C, not like you couldn't write unit tests for C but who does?)
- matthewdgreen 3y ago
  > Most code written by academics is missing trivial, obvious optimisations I agree with this, but I would pose it the opposite way. Academics shouldn't be prematurely optimizing their code: they don't have the resources for it, nor will their results be representative across all hardware. I routinely see (and publish) papers with novel cryptographic protocols where the reviewers will say things like "compare the running time of your algorithm to a state-of-the-art production system that Apple spent $50m developing and optimizing for their silicon, using hand-tuned assembly." We're not going to win in that benchmark, and we don't have the resources to even compete. Moreover, the resulting code wouldn't make for good reference code. But that doesn't mean we shouldn't publish our work, so that e.g., Apple (or other resources) can begin the process of optimizing it for production.
  - josephg 3y ago
    > Academics shouldn't be prematurely optimizing their code While I agree with that in principle, the question of whether or not a given algorithm is fast enough to be practically usable is very useful information for the reader of your paper. Do I need to read your paper, understand it, implement it myself, and only then figure out that its 3 orders of magnitude slower than existing algorithms? I know its a lot of work, but as the reader I don't want to do that work either. I've spent the last few years working on a novel collaborative editing algorithm for text editing. If the algorithm was orders of magnitude slower than existing CRDTs, it would never get adopted. And I want my work to be used. I want to use it, but again - only if its fast enough to be practically usable. I ended up writing a very highly optimised implementation of the algorithm - which took a massive amount of time - to answer that question. For the paper, I spent another couple of days writing a much slower reference implementation that people can read to understand how it works. (The optimised version is thousands of lines, and the reference code is a few hundred lines). I understand how much work and expertise is needed to do things like this, but I still want someone to do this work. Even if its "We wrote a straightforward implementation of <competing algorithm> and <our algorithm> in plain C code. They perform at roughly similar levels of performance <see chart>. We suspect highly optimised versions of both approaches would also perform at roughly similar levels of performance."
PaulHoule 3y ago
Who's going to replicate an experiment that takes a $5 billion particle accelerator? For that matter, who's going to replicate a condensed matter experiment for which a grad student took three years to build the apparatus at a cost of $250,000? Peer review is an absolute joke. In principle researchers could publish their data sets and reviewers could see if they could replicate the analysis but they don't do it because they know if they tried they'd get different answers, see https://www.nature.com/articles/d41586-023-03177-1 https://www.nature.com/articles/d41586-023-03177-1
- eigenket 3y ago
  > Who's going to replicate an experiment that takes a $5 billion particle accelerator? In the case of the LHC, which I assume is what you're talking about, this is why we have 4 pretty much independent detectors around the LHC looking at different but at least partially overlapping stuff. The Higgs boson was detected by the ATLAS detector, but wasn't considered to be "discovered" until it had also been detected at the CMS detector. > Peer review is an absolute joke That strongly depends on the field and the journal in question.
  - PaulHoule 3y ago
    If they flew out the peer reviewer to interview the researcher and take a look at their apparatus or if the peer reviewer loaded up a Jupyter notebook and rechecked the analysis there might be some value in peer review. If peer review is just somebody looking at the paper superficially than it’s of very limited worth. When I wrote my first paper somebody else had done a very similar experiment at another university and we wound method up peer reviewing each other’s papers. Turned out we were using the same bogus data analysis that was being used in 1000s of other papers. I was vaguely aware of this when I wrote the paper but did not feel psychologically safe at all in an environment where the American Physical Society claimed a PhD had a 3% chance of having a permanent career in the field. Someone who was later to become a titan in the field was at our lab and was crying some nights because he had no idea where he was going to get his next postdoc wrote a paper about this problem more than a decade later after he had gotten tenure in a statistics journal that physicists probably won’t even read.
BeetleB 3y ago
Sorry, what? It's extremely true in physics. I've published in (good) physics journals. Computational papers: Authors leave out details because they don't want to give up the goose that lays the golden eggs. They broadly describe the technique, ("solve this PDE computationally"), and anyone who wants to replicate has to craft his/her own solution. If they fail to replicate, the authors just say "your simulation was poor and probably numerically unstable". This was the norm. Experimental papers: Same thing. Experimentalists are inventors. They don't buy off the shelf equipment - they build their own. They don't provide details of how they built it in their papers. At conferences, PIs would openly discuss whether they "believed" a highly cited paper. When I was a new grad student, this troubled me. Why should belief play a role? Just replicate it! A year later: "Oh, of course these studies can't be replicated." It's one reason I left academia. I was doing theoretical/computational physics, but the whole game reminded me of English literature: The value of your work was always seen with subjective eyes. Your paper could be rejected simply because the reviewer doesn't "believe" it.
- abdullahkhalids 3y ago
  > Why should belief play a role? Just replicate it! If you are talking about theory or numerical simulation papers, just replicating the calculations in the paper are not enough to believe the conclusions. There are many assumptions that go into any calculations, in terms of how much of spherical cow you treat reality as. Changing those assumptions will change the conclusions. The assumptions are not always clearly stated, and even the authors might not realize all their assumptions. It can sometimes take many years or decades for people in the field to figure out how a paper was wrong. So yes, experts do have to depend on their subject intuition to determine how much they believe some surprising result. Not to mention, if you go through historical papers in your field you will inevitably come across decades long fights where one group is coming to one conclusion in their paper and other is coming to another, before the situation is resolved one way or the other.
  - BeetleB 3y ago
    Except it's the same problem for experimental papers. My thesis was based on a highly cited experimental study. My role was to "support" that paper with numerical simulations. And yet, at conferences, people would openly discuss if the original, highly cited paper, was fact or fiction. The guy built/fabricated a device, and reported interesting measurements. Want to know if it's true? Build it yourself! Except when you do and get differing results, there will be quibbling over whether you built it properly. There are always details left out of the original paper that can be appealed to. The original author is king of the field, and people will simply ignore your contrarian findings. For theoretical/numerical results, it is for sure valid to criticize assumptions. But in the longer time frame, the question shouldn't be "Are these assumptions valid?" but "Is there experimental evidence to support these calculations?" With a lot of the papers published (including mine), I can assure you that no one will ever be able to construct an experiment to (in)validate my results. And that was true for probably over half of the theoretical/numerical results in my field. We were all publishing things that most of us believed could never be connected meaningfully to the physical world. We're not talking string theory or high energy stuff - more like material properties. The goal was to publish, and convince others. Not to understand reality. Hence, more like literature.

It is not true for Physics either.

by nxpnsv 3y ago

It is not true for Physics either.

Or computer science, at least when the code is also published. (That said, performance measurements in most papers are complete junk. Most code written by academics is missing trivial, obvious optimisations which could completely change any comparative benchmarking results.)

PaulHoule 3y ago
I'm not sure that the average CS paper really tested the algorithm they thought they were testing. That comes from a lot of commercial projects where people weren't using the algorithm they thought they were using (even simple algorithms) and a few cases of working with code from CS researchers. There was that time I got a C program from a famous ML researcher, tried to run it and it segfaulted, I pointed gdb at it and realized it was crashing before it got into main(). Understanding why was an educational experience (don't allocate a 8 GB array you never use on a 32-bit machine... It worked for the researcher because he had a 64-bit machine) but it did not increase my faith in the code quality of academic projects. Some CS researchers are brilliant programmers but the fact is that ordinary devs make a living writing code and CS researchers make a living writing papers.
- eimrine 3y ago
  Allocating a big struct for everything and not using pointers is one of possible techniques of dealing with data in C programs. Anyway having a requirement of 8G RAM is not a good example of something "can not be replicated".
  - PaulHoule 3y ago
    I know all about alternate ways to deal with memory, if I was just programming purely for fun I'd write assembly and not waste time moving the stack pointer around and other meaningless activities associated with calling conventions. It's a sign of poor quality code. He probably was using the array for something at one point, quit using it, and never bothered to delete it. And of course it is C so there are no unit tests despite it being the kind of code that is eminently unit testable (if it wasn't written in C, not like you couldn't write unit tests for C but who does?)
matthewdgreen 3y ago
> Most code written by academics is missing trivial, obvious optimisations I agree with this, but I would pose it the opposite way. Academics shouldn't be prematurely optimizing their code: they don't have the resources for it, nor will their results be representative across all hardware. I routinely see (and publish) papers with novel cryptographic protocols where the reviewers will say things like "compare the running time of your algorithm to a state-of-the-art production system that Apple spent $50m developing and optimizing for their silicon, using hand-tuned assembly." We're not going to win in that benchmark, and we don't have the resources to even compete. Moreover, the resulting code wouldn't make for good reference code. But that doesn't mean we shouldn't publish our work, so that e.g., Apple (or other resources) can begin the process of optimizing it for production.
- josephg 3y ago
  > Academics shouldn't be prematurely optimizing their code While I agree with that in principle, the question of whether or not a given algorithm is fast enough to be practically usable is very useful information for the reader of your paper. Do I need to read your paper, understand it, implement it myself, and only then figure out that its 3 orders of magnitude slower than existing algorithms? I know its a lot of work, but as the reader I don't want to do that work either. I've spent the last few years working on a novel collaborative editing algorithm for text editing. If the algorithm was orders of magnitude slower than existing CRDTs, it would never get adopted. And I want my work to be used. I want to use it, but again - only if its fast enough to be practically usable. I ended up writing a very highly optimised implementation of the algorithm - which took a massive amount of time - to answer that question. For the paper, I spent another couple of days writing a much slower reference implementation that people can read to understand how it works. (The optimised version is thousands of lines, and the reference code is a few hundred lines). I understand how much work and expertise is needed to do things like this, but I still want someone to do this work. Even if its "We wrote a straightforward implementation of <competing algorithm> and <our algorithm> in plain C code. They perform at roughly similar levels of performance <see chart>. We suspect highly optimised versions of both approaches would also perform at roughly similar levels of performance."

PaulHoule 3y ago

Who's going to replicate an experiment that takes a $5 billion particle accelerator? For that matter, who's going to replicate a condensed matter experiment for which a grad student took three years to build the apparatus at a cost of $250,000? Peer review is an absolute joke. In principle researchers could publish their data sets and reviewers could see if they could replicate the analysis but they don't do it because they know if they tried they'd get different answers, see https://www.nature.com/articles/d41586-023-03177-1 https://www.nature.com/articles/d41586-023-03177-1

eigenket 3y ago
> Who's going to replicate an experiment that takes a $5 billion particle accelerator? In the case of the LHC, which I assume is what you're talking about, this is why we have 4 pretty much independent detectors around the LHC looking at different but at least partially overlapping stuff. The Higgs boson was detected by the ATLAS detector, but wasn't considered to be "discovered" until it had also been detected at the CMS detector. > Peer review is an absolute joke That strongly depends on the field and the journal in question.
- PaulHoule 3y ago
  If they flew out the peer reviewer to interview the researcher and take a look at their apparatus or if the peer reviewer loaded up a Jupyter notebook and rechecked the analysis there might be some value in peer review. If peer review is just somebody looking at the paper superficially than it’s of very limited worth. When I wrote my first paper somebody else had done a very similar experiment at another university and we wound method up peer reviewing each other’s papers. Turned out we were using the same bogus data analysis that was being used in 1000s of other papers. I was vaguely aware of this when I wrote the paper but did not feel psychologically safe at all in an environment where the American Physical Society claimed a PhD had a 3% chance of having a permanent career in the field. Someone who was later to become a titan in the field was at our lab and was crying some nights because he had no idea where he was going to get his next postdoc wrote a paper about this problem more than a decade later after he had gotten tenure in a statistics journal that physicists probably won’t even read.

BeetleB 3y ago

Sorry, what? It's extremely true in physics. I've published in (good) physics journals. Computational papers: Authors leave out details because they don't want to give up the goose that lays the golden eggs. They broadly describe the technique, ("solve this PDE computationally"), and anyone who wants to replicate has to craft his/her own solution. If they fail to replicate, the authors just say "your simulation was poor and probably numerically unstable". This was the norm. Experimental papers: Same thing. Experimentalists are inventors. They don't buy off the shelf equipment - they build their own. They don't provide details of how they built it in their papers. At conferences, PIs would openly discuss whether they "believed" a highly cited paper. When I was a new grad student, this troubled me. Why should belief play a role? Just replicate it! A year later: "Oh, of course these studies can't be replicated." It's one reason I left academia. I was doing theoretical/computational physics, but the whole game reminded me of English literature: The value of your work was always seen with subjective eyes. Your paper could be rejected simply because the reviewer doesn't "believe" it.

abdullahkhalids 3y ago
> Why should belief play a role? Just replicate it! If you are talking about theory or numerical simulation papers, just replicating the calculations in the paper are not enough to believe the conclusions. There are many assumptions that go into any calculations, in terms of how much of spherical cow you treat reality as. Changing those assumptions will change the conclusions. The assumptions are not always clearly stated, and even the authors might not realize all their assumptions. It can sometimes take many years or decades for people in the field to figure out how a paper was wrong. So yes, experts do have to depend on their subject intuition to determine how much they believe some surprising result. Not to mention, if you go through historical papers in your field you will inevitably come across decades long fights where one group is coming to one conclusion in their paper and other is coming to another, before the situation is resolved one way or the other.
- BeetleB 3y ago
  Except it's the same problem for experimental papers. My thesis was based on a highly cited experimental study. My role was to "support" that paper with numerical simulations. And yet, at conferences, people would openly discuss if the original, highly cited paper, was fact or fiction. The guy built/fabricated a device, and reported interesting measurements. Want to know if it's true? Build it yourself! Except when you do and get differing results, there will be quibbling over whether you built it properly. There are always details left out of the original paper that can be appealed to. The original author is king of the field, and people will simply ignore your contrarian findings. For theoretical/numerical results, it is for sure valid to criticize assumptions. But in the longer time frame, the question shouldn't be "Are these assumptions valid?" but "Is there experimental evidence to support these calculations?" With a lot of the papers published (including mine), I can assure you that no one will ever be able to construct an experiment to (in)validate my results. And that was true for probably over half of the theoretical/numerical results in my field. We were all publishing things that most of us believed could never be connected meaningfully to the physical world. We're not talking string theory or high energy stuff - more like material properties. The goal was to publish, and convince others. Not to understand reality. Hence, more like literature.