8 ms·
There's certainly a kernel of truth to this. I worked in a cancer lab as an undergrad. New PI had just come over to the school. He had us working on a protocol
by wittyreference 6y ago
There's certainly a kernel of truth to this.
I worked in a cancer lab as an undergrad. New PI had just come over to the school. He had us working on a protocol he'd developed in his last lab to culture a specific type of glandular tissue in a specific way.
I and two other students spent months trying to recreate his results, at his behest. When we couldn't get it to work, he'd shrug and say some shit like "keep plugging away at the variables," or "you didn't pipette it right." I don't even know what the fuck the second one means.
Then an experienced grad student joined the lab. He spent, I don't know, a week at it? and was successfully reproducing the culture.
I still don't know how he did it. I just know that he wasn't carrying a magic wand, and the PI certainly wasn't perpetrating a fraud against himself. It was just, I guess, experience and skill.
- savanaly 6y agoReminds me of the discussion of "tacit knowledge" HN had the other week [0]. [0] https://news.ycombinator.com/item?id=23465862 https://news.ycombinator.com/item?id=23465862
- shultays 6y agoReminds me this hacker koan: A novice was trying to fix a broken Lisp machine by turning the power off and on. Knight, seeing what the student was doing, spoke sternly: "You cannot fix a machine by just power-cycling it with no understanding of what is going wrong." Knight turned the machine off and on. The machine worked.
- steveklabnik 6y agoI have been trying to find a source for this story for a long time. Any pointers? That is, I don't mean that I expect it to be a true story, but if you were to cite this parable, how would you cite it?
- whatshisface 6y agoYou won't find any pointers to Lisp stories.
- m463 6y ago(setq stories (list "http://catb.org/~esr/jargon/html/koans.html")) (car stories)
- stewbrew 6y agoThe first cons pair is passed by value though, not by reference.
- m463 6y agoI had hoped, just for a moment, to have the Contents of the Address Register.
- stedaniels 6y agoLast-Modified Mon, 29 Dec 2003 18:34:39 GMT
- kranner 6y ago"http://catb.org/~esr/jargon/html/koans.html" http://catb.org/~esr/jargon/html/koans.html" is a valid SEXP.
- stan_rogers 6y agoThey live here at the Jargon File: http://www.catb.org/~esr/jargon/html/koans.html http://www.catb.org/~esr/jargon/html/koans.html
- steveklabnik 6y agoThanks; I always find the “more magic switch” story when I’m looking. I thought it was the jargon file...
- koanist 6y agoIt's plausible! I have a similar story involving myself and my sister. She was power cycling our family computer over and over, becoming more and more frustrated as it went into 'safe mode' equivalent. I allowed it to fully power on and then rebooted it, and it was 'fixed'. The story isn't unreasonable.
- steveklabnik 6y agoOh yeah I’m not saying it can’t happen, I just mean, the literal truth isn’t what I’m after here.
- jackhalford 6y agoWhen this situation occurs, I always pause to pet the computer and say something nice to it, as to make the other person believe the server is listening.
- 6nf 6y agoThe student was thus enlightened.
- bsder 6y ago> It was just, I guess, experience and skill. Lab technique is definitely a thing. Mostly it's about knowing what not to pay attention to so that you can give extra attention to the things that matter. It's also about recognizing the intermediate signals that say "Yep, still good." (ie. Are the bubbles the right size? Is the foam the right color? Does it smell right/wrong? Did a layer of glop form at the correct step?) However, I rarely see anyone taking good enough notes to reproduce an experiment. I had a Physics Lab taught by a professor who would randomly take someone's notebook in an assignment and reproduce the experiment using your notes--and fail you on that assignment if he couldn't. Very few students ever passed that. You really have to be excessively meticulous. However, when you are debugging a real experiment without a known correct answer, if you aren't that meticulous you will never find your own bugs.
- scott_s 6y agoI'm reminded of baking. The recipes for everything are all known, but there's still an enormous amount of skill and tacit knowledge required.
- asdff 6y agoFinding an old protocol that has been marked up by a former lab member is like finding the Half Blood Prince's potion book.
- amluto 6y agoHah. A lot of excellent bread recipes involve a step like “add flour until it looks right.”
- kwhitefoot 6y agoNot the ones intended for professional use. I have a textbook of cookery [1] written for students intending to become professional cooks, it emphasizes precision measurements and repeatable processes. Where it expects adjustments it explains what it means and what constitutes "it looks right". It stands head and shoulders above pretty much all other cookery books I have read. But of course it lacks glossy pictures and is not backed up by a video of Nigella Lawson in a low cut dress (not that I'm objecting to Nigella and her cleavage) [1] Professional Cookery: The Process Approach by Daniel R. Stevenson
- p1esk 6y agoSimilar thing sometimes happens in deep learning field. Just a couple of weeks ago I helped a coworker reproduce a result from a paper. He had been debugging it for several days. Took me about 5 minutes to make a correct guess that his batch size is too small for batchnorm to be effective. He trained the model on 8 GPUs, using a decent batch size, but he didn't realize that the default batchnorm op in Pytorch isn't synchronizing batch statistics across GPUs, so the effective batch size was in fact 8x smaller. How was I able to see it so quickly? Experience.
- michaelhoffman 6y agoThis is what Raymond Chen calls the "psychic debugger".
- taneq 6y agoAlso sometimes called 'spidey sense'. You don't know how you know, but your subconscious synthesizes the idea somehow.
- robocat 6y agoI think this type of non-rational intuition shows a combination of intelligence and experience. “It was obvious” gets said by a smart person for things that are opaque to someone that lacks a correctly functioning intuition.
- taneq 6y agoI'd say it's a product of unconscious competence. "I can't articulate why, but this bit here is the problem." I do maintain, though, that if you can't explain your reasoning for something then you don't understand it. Doesn't mean you're wrong, though!
- StandardFuture 6y agoBut, this is only one kind of reproduction ineptitude. The process description/definition was not explicit enough. Either a process can or it cannot be reproduced as specified. There is no middle ground there. This is literally what a proof is. A sequence of reproducible steps. You made an important point in considering that other causal factors can be present besides fraud in science's reproduction problems. But, it appears like this was made to suggest that fraud is not as serious of a problem as it is in the reproduction crisis. If more emphasis is placed in the scientific community on the necessity for proofs, then fraud would stand out like a sore thumb. Until then, it can easily be masked within the lax atmosphere of (a lack of) explicit reproduction specifications.
- asdff 6y agoTypically you don't take a single result with any face value, you hope to see validation in the literature from another source or you set up your own assay and see for yourself. If you were to ask my advisor whether the sky is blue, he would respond that he doesn't believe in anything, and that is a pretty reliable worldview to hold as a scientist. I've had to sometimes step up concentrations quite a bit or veer way off protocol to reproduce a result, and even then it won't look as pretty, but I don't blame the authors for things like that. Maybe their undergrad mixed a higher concentration than instructed that day. The results that get published are the prettiest pictures, your very ugly initial result before optimization is only shown in lab meetings. It does get laid bare eventually. I've heard of a well known PI being investigated by their funding agencies due to a post doc doctoring a western blot result. I heard a story about a grad student who tried to repeat a graduated grad student's protocol. They tried and tried for years without much success, initially blaming their own ignorance given they were a freshman when they started their efforts on the experiment. Eventually, after going back and forth with the former student, the PI, and collaborators, it was revealed the original protocol was outright wrong. In fact, more digging was done and that former students thesis research turned out to be fraudulent. The former student lost their degree and their current job, and the current student wasted two years of their life trying to make something impossible work. It's awful when fraud happens for everyone who is involved, whether intentionally or not. Fraud does tend to shake itself out, but it can take painful time.
- mirimir 6y agoI had a similar experience as a beginning grad student. A postdoc in the lab had developed a radioligand binding assay, which we were using to purify a small protein. And I could just not get it to work, no matter how careful I was. TL;DR - I was being too careful, and so too slow. Because the binding was transient. So once I focused on speed instead of precision, the assay worked fine.
- dekhn 6y agoI distinctly recall a situation where speed made a huge difference. I took an undergraduate lab at UCSC where you were given raw bacterial cells and had to use the original literature to purify restriction endonucleases with it. The course ran for a quarter, and on the first day the professor said: "Most of you are going to spend a few days planning, then running your purifications for a few weeks. If I were going to do this, I'd spend a couple weeks planning and then do the whole experiment in a day, because the moment you break the cells open, all the proteases will start eating up your restriction endonucleases and you will lose activity in a couple days. I've also found a few situations where I couldn't get somethign to work and it turned out the person who had it "working" wasn't running controls (positive or negative), so you didn't really know if it worked.
- pwdisswordfish2 6y agoHow did you get a job in a lab if you were not familiar with a pipette? "I still don't know how he did it." Practice. Learning through mistakes. Tips from others.
- t_von_doom 6y agoI don't think the user was unfamiliar with a pipette, but rather didn't know what was meant by pipetting 'right'.
- wittyreference 6y agoThat is it precisely. It was a matter of ejecting the pipette contents at the right moment while removing the pipette from the medium at the correct speed (I exaggerated my ignorance a little bit for rhetorical effect.) But there was no way to verbally teach the right "feel" for it - you just had to do it over and over until you learnt how pipetting "right" felt. You could (and I did) observe my PI do it the "right" way, but you can only get so close to hitting a baseball by watching a pro - at some point, you pick up the bat and learn by swinging.
- pwdisswordfish2 6y agoWhat if I had said "pipetting"? Semantics. Familiarirty with a piece of equipment versus familiarity proper technique when using it.
- kovvy 6y agoI had a protocol fail repeatedly, and expensively, because of a combination of a contaminated reagent and a tiny amount of sample. Took me ages to even approach being able to debug that one as the protocol worked for everyone else. Turned out everyone else using the same materials had so much input sample that the contamination didn't matter, or used the non-contaminated part of the output. Almost to the day I worked out the issue, I saw an immensely more experienced lab person running an almost identical experiment to my setup. They always added another reagent "in case there's contamination".
- kurthr 6y agoIt turns out for most complex analytical procedures there are a million more ways to do something wrong than right. That's why as much as I like the idea of publishing failures along with successes, it just doesn't make sense. However, as you mention, that doesn't mean that something is irreproducible... get the right grad student to show you how (hopefully they can actually comment on your procedure and help, but pharma/bio-chem is notoriously touchy). Like cooking, good experimental method and helpful documentation (recipes) take both temperament and training. http://pages.cs.wisc.edu/~kovar/hall.html http://pages.cs.wisc.edu/~kovar/hall.html "Electron Band Structure In Germanium, My Ass", is a great example of this... I'm pretty sure they had a bad contact (using indium solder?) when they ran the experiment. When you're getting garbage like that... stop and see if you can reproduce a single result, and until you can, something about your method/equipment/model sucks.
- nordsieck 6y ago> It turns out for most complex analytical procedures there are a million more ways to do something wrong than right. That's why as much as I like the idea of publishing failures along with successes, it just doesn't make sense. However, as you mention, that doesn't mean that something is irreproducible... get the right grad student to show you how (hopefully they can actually comment on your procedure and help, but pharma/bio-chem is notoriously touchy). Like cooking, good experimental method and helpful documentation (recipes) take both temperament and training. This sounds like a complicated way of saying the procedures for some scientific research are underspecified.
- kurthr 6y agoWell, yes... the problem is that if you're not reasonably proficient and observant then entire books worth of specification will be insufficient. Intel deals with this in their fabs by Copy Exactly! Not just the equipment and recipes, but the floor wax, the paint, the roof, soda machines, cafeteria... everything but the people who don't go inside except to fix stuff. So, if you're in software enjoy the fact that you have the wonder of a compiler, an OS, and a CPU you can (mostly) trust. The rest of the physical world dreams of that level of control and repeatability for anything like the tiny incremental cost. Other than math and theoretical physics that's as close as you'll get to fully specified.
- LatteLazy 6y agoI had the same experience in physics as an undergrad. Our lab got some early graphene transistors, they were meant to work around the 5 kelvin mark. No matter how slowly I cool them or how I set things up, I couldn't fet them going. The PhD had no issues with the exact same transistors, cryostat, electronics etc. Maddening.