13 ms·
Quantum computing’s reproducibility crisis: Majorana fermions
- Greek0 6y agoBrief summary: Majorana fermions are particles can potentially be created under stringent laboratory conditions. Whether this has been achieved in practice is unclear: many scientific papers purport to show evidence of Majorana particles, but there are other explanations that could explain to the observed data as well. New research is frequently published that claims Majorana production, but most often doesn't even acknowledge potential problems or alternative explanations. These sloppy practices cast doubt over the whole field, despite the large impact Majorana particles could have for quantum computing applications. We need: * More stringent data reporting: raw data, full data (not only the small subset supporting the hypothesis) * More critical evaluation of other explanations for the observed data * Transparent publication processes, that prevent a paper that was rejected by one journal on scientific grounds appear in another journal unchanged
- nxpnsv 6y agoThanks, this is truly helpful
- scythmic_waves 6y agoYeah I would pay money to have one of these at the top of every article I read.
- raziel2701 6y agoI'd love to see the back and forth between the reviewers and the authors, there's tons of information and nuance there that goes unpublished.
- deleted 6y ago[deleted]
- dataflow 6y ago> Transparent publication processes, that prevent a paper that was rejected by one journal on scientific grounds appear in another journal unchanged This is great when reviewers are reviewing properly. But when you run into reviewers that literally don't read some parts of the paper and then object to things already addressed there, it starts backfiring. I don't know how to address this, but I'm thinking maybe making reviewer comments public without necessarily requiring a change to publish elsewhere would tackle both issues? It would seem to encourage both high quality reviews and the addressing of those reviews.
- spicybright 6y agoMaybe we need to start putting brown M&Ms into research papers /s https://www.insider.com/van-halen-brown-m-ms-contract-2016-9 https://www.insider.com/van-halen-brown-m-ms-contract-2016-9
- perl4ever 6y agoI haven't gotten any M&Ms in a while, but I feel like long ago there were two shades of brown and more recently only one. Also, whatever happened to blonde Oreos with chocolate filling? I know they existed once, but every time I'm the grocery store I check and they're not one of the dozens of flavors.
- spicybright 6y agoI THINK those were labeled Uh oh oreos. I'm almost 99% positive they're still in stores regularly. I live in north east US.
- d110af5ccf 6y agoI agree with the sentiment, but I think "branding" a paper that was rejected in any way whatsoever would make things worse on the whole. The ability to try again seems to be an important part of the (admittedly imperfect) peer review process. (Relevant caricature: http://matt.might.net/articles/peer-fortress http://matt.might.net/articles/peer-fortress)
- lamontcg 6y agoThere was also the problem that analyzing the experimental results are complex to analyze and beyond any one expert. The results need a team to analyze them, and probably a team to properly review the papers. One thing I'd have HN consider though is that the peer review process was never intended as a sufficiently strong filter to ensure that bad science was never published in the first place. The point is to get it such that it isn't wasting everyone's time to read it and try to figure out how to attack it and rebut it. The whole broader community of science is supposed to participate in the scientific process, which is what seems to be happening here. Journals and the peer review process also shouldn't be the sole gatekeeper of the truth either, they do make mistakes in the other direction, rejecting papers incorrectly: https://en.wikipedia.org/wiki/Fermi%27s_interaction https://en.wikipedia.org/wiki/Fermi%27s_interaction
- trhway 6y ago>peer review process was never intended as a sufficiently strong filter to ensure that bad science was never published in the first place. peer review is basically an artifact of and a business process solely for the purpose of commercial science publishing, ie. it is a commercial product quality control. Peer review pretty much killed scientific debate as it amplifies the dogma and makes questioning of it nearly impossible. Even during Dark Ages people were challenging dogma more deeply and freely (even though doing so carried the risk of being burnt at stake) than the scientists risk doing today. Back then there were public debates, and today we have anonymous peer review instead - how is that for the progress... >The point is to get it such that it isn't wasting everyone's time to read it this is what you have your students for (depending on the complexity of the work - seniors and/or PhD students). Like puppies they need something to work their teeth on :) Finding flaws/errors/etc. is a good training, and it makes the students a real, though minor, participants and partners in doing of the actual current science, and being such a lowly grunts they get to focus on finding real issues/errors, fact checking/etc. instead of higher level opining.
- kkylin 6y ago> One thing I'd have HN consider though is that the peer review process was never intended as a sufficiently strong filter to ensure that bad science was never published in the first place. In addition, in most fields of science, refereeing is also not meant to guarantee correctness -- even good research can turn out to be wrong, and conversely some mistakes are very instructive. I think it's generally more accurate to view published journal articles as part of an on-going conversation, rather than as a lasting record of scientific truth [0]. This is not to say that scientists should not do the best they can to ascertain correctness, nor that they should not look for alternate explanations. But one should look at published work as what it is -- the best one can conclude after X years of work (whatever X is). [0] Unfortunately, it's often hard for outsiders to jump into these conversations, i part because journal papers are almost invariably aimed at others who are already know the context. But that's a discussion for another time.
- pomian 6y agoVery good article. Clear and concise. It pertains not just to this field of study, but to most scientific research. We have seen many of these issues discussed (scientific reporting, publishing) on HN, but this writer summarises a few very good solutions, towards the end of the article. How to achieve reproducibility, establishing shared experimental techniques, editorial ownership, are examples.
- nightcracker 6y agoDamnit, after the mistake of calling the study of computation 'computer science', I had thought we'd avoided the issue with 'quantum computing'. But no, even this term seems to get muddied by things that aren't computation. There is no reproducibility crisis in quantum computing, there is in experimental quantum physics with quantum *computer* applications. To give a classical analogy, you could claim there would be a crisis in computer science because electrical engineers struggle with making a specific kind of transistor.
- de6u99er 6y agoThe problem with your analogy is, that we already have functioning hardware while quantum computers are nowhere near being practicable.
- bawolff 6y agoI think OP is being needlessly pedantic, but in fairness you can do all sorts without quantum hardware. Making quantum algorithms (shor did not have a quantum computer), quantum complexity theory, etc.
- andi999 6y agoYes, you can also practice swimming without water.
- Delk 6y agoI don't think that's an entirely fair comparison. You can design algorithms and prove things about them without hardware, and the algorithms and their properties will be valid in and of themselves. You don't get to "swim", but the algorithms will be there. One could argue that "computer science" is the study of the logical processes involved in computation.
- bawolff 6y ago"[Computer science] is not really about computers -- and it's not about computers in the same sense that physics is not really about particle accelerators, and biology is not about microscopes and Petri dishes...and geometry isn't really about using surveying instruments. Now the reason that we think computer science is about computers is pretty much the same reason that the Egyptians thought geometry was about surveying instruments: when some field is just getting started and you don't really understand it very well, it's very easy to confuse the essence of what you're doing with the tools that you use." -- Hal Abelson (1986)
- Wxc2jjJmST9XWWL 6y agoOff-topic: I submitted the exact same article only a few hours before https://news.ycombinator.com/item?id=26777030 https://news.ycombinator.com/item?id=26777030 ; I thought someone else submitting the same thing would end up being a simple upvote on my submission instead of a separate submission? I don't care about the points or credit, glad it's on front page. But apparently there's something I don't know about the workings of HN? Thankful for someone clearing that one up and sorry for taking up the space.
- amelius 6y agoPerhaps they submitted it before you and then changed the submission (e.g. title)? Just a guess.
- pseudolus 6y agoI submitted the article in question and didn't make any alterations to either the title or the link. It's possible (just a guess) that the link I submitted was slightly different from the link that the OP submitted as I tend to strip out extraneous parts of the URL. After that the submission was likely then just fortuitously upvoted.
- Wxc2jjJmST9XWWL 6y agohad the same idea, but I do so as well and link is exactly the same. edit: it doesn't really matter I guess. HN has some mechanisms which aren't public afaik. How downvoting works exactly, how flagging works, who can, what the algorithms are for sorting. This account I'm using hadn't a successful submission before, but it's not that I spam submissions either. Four submissions in total and the account is 8 months old. But I guess it's just some automatic behavior of the site, well... shrug.
- dang 6y agoThat's true for 8 hours, but this one was submitted 9 hours after yours.
- FabHK 6y agoA bit disheartening that there is a reproducibility crisis not only in psychology (and maybe social sciences in general?), but also physics... On the other hand, after the breakthroughs in physics in the first half (third?) of the 20th century and the stagnation in the latter half of that century, it seems to me (as a layperson) that the number of anomalies in physics seems to be increasing, so that maybe we'll transition from a period of "normal" science to a scientific revolution again soon (in Kuhnian terms). Exciting!
- Chris2048 6y agoAFAIK the crisis in psychology applies generally, but this is just QC? Are there other physics-based reproducibility issues?
- klyrs 6y agoOne big one comes to mind... nobody else has an LHC. It's not hard to imagine a systematic issue in an experiment producing a wrong "5 sigma" result that isn't caught until the world has an equivalent / higher energy beam to play with. This particular issue seems to be of a similar nature. You've got a research group who made and tested a device, and nobody else has duplicated that (not sure -- but if there's patents covering their fab, there could be Problems for anybody seeking to reproduce the result).
- andi999 6y agoI don't know about lhc, but at Hera they had two independent groups running similiar experiments at different sides of the ring without talking to each other. This was exactly to adress the problem you mentioned.
- pa7x1 6y agoIt's the same in LHC there are two experiments that run independently, ATLAS and CMS. For exactly this reason. The Higgs was discovered by both independently with higher than 5 sigma each. The combined sigma of the 2 experiments was 7-ish if my memory serves correct. EDIT: My memory serves correct because sqrt(5^2 + 5^2) ~ 7
- systemvoltage 6y agoRelated: Is Quantum Computing bullshit? https://www.wired.com/story/revolt-scientists-say-theyre-sick-of-quantum-computings-hype/ https://www.wired.com/story/revolt-scientists-say-theyre-sic... I thought this was pretty hilarious.
- nerdponx 6y ago"Sick of the hype" is not equivalent to "it is bullshit".
- systemvoltage 6y agoI didn’t make it clear, obviously the entire field isn’t bullshit. The wired article is talking about a Twitter account that judges if some QC news or paper is bullshit or not.
- tabtab 6y agoCutting edge does often cut.
- deleted 6y ago[deleted]
- hardtke 6y agoWhen I was in physics, I always read "a typical example of the data from an experimental run is shown in figure 1" as "we have carefully selected the best data, shown in figure 1." It sounds like that is the case here.
- mhh__ 6y agoPerhaps worryingly the same is true in CS papers also. I genuinely think there should be a "No code? No Data? No Paper" rule instated from the top.
- nick_kline 6y agoHa! Perhaps you have not published a theory paper. I'm on the systems side but theory is real too, even if there is no perf test. Systems papers can benefit from code and data though ;-)
- mhh__ 6y agoPut it this way, I have read a lot of papers that are basically pure theory (~5-10 pages of mathematics) but with some graphs of how effective the theory of at the end. Again, no code, no error bars, no data etc.
- inasio 6y agoSurprised I haven't seen any comments about Microsoft. This was the qubit type that Microsoft was betting on, I think the lead researcher behind the reproducibility issue worked for them too. Very curious if they are going to pivot to something else, they have also invested heavily in QC startups.