8 ms·
I don't want to sound too triumphant, but over here in fundamental physics we have decades-old safeguards in place against all these problems. The big experime
by knzhou 7y ago
I don't want to sound too triumphant, but over here in fundamental physics we have decades-old safeguards in place against all these problems.
The big experiments don't have publication bias: they proudly say exactly what they did, even if 90% of the time there are only negative results, because exclusions are important too. Experiments are inherently replicated, with multiple independent simultaneous experiments (LHC) or multiple independent analyses (EHT), data blinding throughout, and even occasionally a further layer of blinding using decoy signals (LIGO). The statistical standards for discovery are, in terms of p-values, about 10,000 times more stringent, and even still people are moving away from p-values entirely.
The resulting publications are put out for free, publicly, on the ArXiv. Later they are submitted to a relatively small family of low-cost journals, which everybody knows the reputations of.
Hopefully some of these lessons can be adapted to other fields.
- SiempreViernes 7y agocought 750 GeV cough... ahem. Let me sing you an OPERA about a how Andrei Linde was astonished by the biggest BICEP you've ever seen! It belonged to a LIBRA* that was faster than light and could fly trough walls! * Couldn't get DAMA to work :(
- knzhou 7y agoSure, but there's a difference between the accuracy of speculative theory and experiment. Experiment is very reliable -- they never called the 750 GeV bump a discovery of anything, because it didn't meet the statistical standards. Even things that do meet the statistical standards are often not called discoveries, because the experimentalists think they are systematic errors at play, as we see with MiniBooNE and DAMA. Speculative theory is not reliable, which is why all the papers on the 750 GeV bump were wrong. But it has never been reliable, because, well, it's speculative! For each new phenomenon there's probably only one right explanation, but far more than one paper. Being 99% wrong has been par for the course for almost a century.
- SiempreViernes 7y agoI mean, if CMS just hadn't played up their 2.3 sigma excess we probably wouldn't have had the 750 bonanza* so don't pretend there is no blood on the experimental hands :p OPERA is a bit of a special case, but BICEP just straight up declared discovery and wanted the Nobel, have you seen their video of when they tell Linde? * Any time I can, I link to the conclusion: http://resonaances.blogspot.com/2016/06/game-of-thrones-750-gev-edition.html http://resonaances.blogspot.com/2016/06/game-of-thrones-750-...
- oefrha 7y ago> BICEP just straight up declared discovery and wanted the Nobel, have you seen their video of when they tell Linde? That’s a bit exaggerated. You mean the champagne popping video? Well, it’s somewhat awkward in hind sight, but a bit of excitement was warranted at that time. I was at Stanford around that time, might even talked to Andrei right after, and don’t recall a festive atmosphere or anything.
- SiempreViernes 7y agoI feel bad for his wife every time I see it now :(
- oefrha 7y agoShe’s also a Stanford physicist. Great person.
- knzhou 7y agoI agree that BICEP was the worst debacle of this kind that’s occurred in recent memory. But also see how the safeguards worked: their results were properly questioned, the hype was over in months, and they did not get a Nobel.
- oefrha 7y agoThe high profile debacles were pretty quickly caught, so I’d say the system is working.
- SiempreViernes 7y agoThere are two aspects: how many errors are published and how they long before they are caught? The BICEP debacle is an example of a claim that should have gotten caught before it was published, doing an important calibration step using a photo of preliminary data?!
- knzhou 7y agoActually, the error was caught before publication. They sent it to ArXiv, the standard pre-publication space, so that other physicists could read and comment on it. It was there that the error was quickly caught.
- SiempreViernes 7y agoOh, I didn't know that. I though with all the announcements they did they had a paper ready to publish...
- knzhou 7y agoWell, it's all a little ambiguous. You can choose to pre-publish or not, and you can choose to announce whenever you want: before pre-publishing, after but before publishing, and at the moment of publication itself. It's certainly true that they announced early, but it's also true that the community at large regarded it with appropriate skepticism, causing the whole thing to be self-corrected in months.
- fungicide 7y ago"Stanford Professor Andrei Linde celebrates physics breakthrough" https://youtube.com/watch?v=ZlfIVEy_YOA https://youtube.com/watch?v=ZlfIVEy_YOA
- vadansky 7y agoWhile it's certainly more robust, it still has some issues. For example, I remember reading that because the initial results for the experiment measuring the size of the electron was too low, every experiment afterwards reported slightly bigger sizes until they reached the correct value. Since they trusted the original result, if their results varied too much they discarded them. Ultimately they redid the experiment until they got something closer, and the only values that ended up getting published were results that were slightly bigger. Correct me if I am wrong, as I read this story a while ago.
- knzhou 7y agoThat's an old Feynman story, but he was kind of bullshitting to make a point. If you actually look at the history of measurements, it doesn't look like that at all -- it instantly corrects to about the correct value right after Milliken and hovers around that value. https://hsm.stackexchange.com/questions/264/timeline-of-measurements-of-the-electrons-charge https://hsm.stackexchange.com/questions/264/timeline-of-meas...
- vadansky 7y agoActually if you look at the follow up answer, they added some more measurements that back up Faynman's claim.
- knzhou 7y agoHuh, I didn’t even know! The new answer really does change things.
- SiempreViernes 7y agoThere is similar example for the speed of light as well.
- astro123 7y agoThis is exactly the sort of thing that blinding is there to prevent. It is the same reason we in cosmology blind our data - to prevent us from biasing ourselves towards what other experiments have found. Also, the charge of the electron experiments you are talking about happening over 100 years ago, so not that relevant to today :)
- oefrha 7y ago> 90% of the time there are only negative results That’s an understatement.
- knzhou 7y agoWhat people forget is that this was true in the old days as well. Even back in the 60s and 70s there were over ten accelerators throughout the world constantly doing science, and you can bet they were booked up with new speculative searches every month. The vast majority found nothing, and were forgotten. That's because there's a very finite number of things to discover in fundamental physics, which is of course what makes the field so exciting in the first place.
- oefrha 7y agoOf course. It’s actually pretty awesome that every time the energy frontier was pushed, something new was discovered, so there was return on every round of investment.
- orbifold 7y agoCERN and the LHC are to me prime examples of bad science. You build huge machines at incredible cost, wasting hundreds of thousands of man hours of brilliant young people’s time. They are paid almost nothing (a checkout clerk at a Swiss supermarket makes more than a PhD student at CERN). Then you string along a subset of those for years, exploiting them for further cheap labor, somehow making them believe they are “lucky” to get that opportunity (a postdoc at CERN pays a fraction of what you can make at Google Zurich). In the time between experiments people only ever see simulated data, leading to a rude awakening when actual experimental data comes in (c.f. ALICE’s desaster of a analysis Pipeline). Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter). Defunding the super collider in the 90s in the US was probably one of the best science policy decisions they made.
- whymauri 7y agoI kinda wish this comment focused more on bad science rather than bad pay. If bad pay implied bad science then most of science is bad! I suppose I'd be much more interested in learning about the latter half of your comment (specific scientific failures).
- eganist 7y ago> Then there is literally decades of over promising on ground breaking discoveries right around the corner (super symmetry, extra dimensions, dark matter). Every statement in your diatribe would seem to be predicated by the assumption that discoveries in theoretical physics (what you're complaining about) will result in revolutionary changes to your day-to-day experience (kinda i.e. applied physics) in a short enough span of time that you can actually enjoy them with an able body. Probably not. Flight/space and semiconductors were notable exceptions, and they were largely driven by the century of hot and cold war that we know to be the 1900s. Bad science is science that fundamentally doesn't yield a new understanding of the world, either by falsifying theories or confirming them. What you described is: 1. failure to meet exaggerated lay expectations, and 2. poor working conditions in the nonprofit space. #2 (and possibly #1 depending on where the theoretical discoveries take us) can largely be resolved by pouring more money into science, not less. I'll point back to our century of war if you need past precedent. Edit: removed a line that was unnecessarily insulting. Sorry about that.
- glofish 7y agoYeah but then you have different problems. Hundreds of authors on each publication. Whose contributions are real and who is just cruising?
- cortesoft 7y agoMaybe this just shows that using authorship credit as a proxy for contribution is a bad system.
- knzhou 7y agoI mean, can't you say that for any large group of people? You can't easily evaluate who's pulling the most weight at Google either. But that's not a problem, because the people at Google deciding promotions do know, and that's what matters. CERN works like that, as does experimental physics at large.
- pfortuny 7y agoMost positions at CERN are fixed time. Afterwards you have a problem.
- knzhou 7y agoIf you leave CERN to apply for a postdoc or professorship somewhere else, they ask for lots of recommendation letters. That's how they learn what you actually did.
- glofish 7y agoDo you think a recommendation letter is a more objective and better evaluation of what someone actually did? That there is less embellishment, less subjectivity in it than a scientific paper that was meant to reflect on your work as a scientist. I would expect that recommendation letters heavily favor some individuals for subjective reasons.
- knzhou 7y ago
- itronitron 7y ago>> Hopefully some of these lessons can be adapted to other fields the primary lesson here is to get a commitment for decades of government funding
- jfengel 7y agoPreferably in the billions of dollars. Nothing generates five sigma confidence levels like ten-figure budgets.
- knzhou 7y agoSorry, do you think that fundamental physics gets more than other fields? Have you ever seen a breakdown of research budgets?
- jfengel 7y agoI don't have such a breakdown, but I do know that the budget of the LHC alone is a billion dollars a year plus $13B to build it. That wouldn't have been possible if it weren't for a long-term commitment. And I know that few fields get that kind of commitment. Maybe cancer, maybe something in semiconductors. Lots of other fields would produce things if you offered them many years of billion-dollar commitments. I'm not saying that it was wrong to give so much to fundamental physics. I'd just be very happy if we gave that level of funding to lots of other things.
- Schiphol 7y ago>The statistical standards for discovery are, in terms of p-values, about 10,000 times more stringent But this has mostly to do with your subject matter being better behaved than others, no? The most profound changes and the healthiest research culture could not make 5 sigma a reasonable goal in psychology.
- knzhou 7y agoOur statistical standards may be unrealistic for social science, but there was a petition a while ago to lower the cutoff from 0.05 to 0.001. I think that really would help.
- Fomite 7y agoThere's also a very healthy question of weather or not it's worth while. A p-value that's theoretical physics stringent will mean leaving a lot of cures on the shelf.