9 ms·

Academic Urban Legends

129 points by dougmccune 12y ago

hackuser 12y ago
Be sure to read to the end, or, after you read enough to grasp the idea then skip down to the section "The narrow path has its temptations".
- peatmoss 12y ago
  Yeah, my first thought was, "wow, I always thought that about spinach!" But, that wasn't the point of the article at all. This was an engagingly written article that kind of read like a mystery whodoneit?
  - vitoreiji 12y ago
    And the most concerningly satisfying is that everybodydoneit, apparently. Including me, who didn't bother to read any one single paper cited.
- the_af 12y ago
  Even better, read the section "An avalanche of low-quality research?" near the end. Even this notion is put to the test, and found wanting :) Excellent article!
  - OBRekdal 12y ago
    Thanks a lot! :)
acbart 12y ago
It'd be interesting if we could break down academic papers into stand-alone facts, which could be cited more succinctly than the paragraphs of prose that we do now. Ideally, this would look more like academic proofs, and it'd be easier to argue and discuss components of a given paper. I'm sure that a good amount of beautiful subtext would be lost, but might it be worth it to have a more accurate way of linking knowledge?
- danieltillett 12y ago
  This could now be accomplished by providing an offset (say number of characters from the start) to where the information cited can be found. What I find more of an issue is when trying to track down the source of some “fact” is going three or four papers up the citation chain and finding the paper at the end doesn’t actually say what the starting paper claims.
  - judk 12y ago
    Thanks for summarizing the OP.
  - romaniv 12y ago
    This sounds like a problem that hyper-links should have solved.
- hackuser 12y ago
  > It'd be interesting if we could break down academic papers into stand-alone facts, which could be cited more succinctly than the paragraphs of prose that we do now. Perhaps there could be some kind of semantic syntax that would group together the factual claim and supporting information (data and references). For example, <"e=mc^2",cite1,cite2,data1,data2> The claims still could be included in prose, but also easily extracted, analyzed, manipulated, etc. With a little work the claims could form a semantic web, with all that entails, and the validity of cites could be checked automatically when the document is opened or on a regular basis, updating the claim appropriately. For example, the UI could display the claim differently based on whether the cited source has been withdrawn, partially invalidated, superseded, confirmed, rarely cited, heavily cited, based on impact factor, if the source's own citations have been invalidated (e.g., claimC cites claimB which cites claimA; if claimA is invalidated, there should be some sort of indication in claimC -- consider that the invalidation of a seminal claimA would set off a massive chain reaction, as it should), etc. But that all seems obvious and I suspect I'm not the first to think of it.
  - jasonpriem 12y ago
    It is indeed interesting, and as you suspect it has attracted some attention. Check out "nanopubs,"(http://nanopub.org http://nanopub.org), or Simon Buckingham Shum's very cool work on ScholOnto (http://projects.kmi.open.ac.uk/scholonto/ http://projects.kmi.open.ac.uk/scholonto/) among others.
danieltillett 12y ago
There are an amazing number of scientific urban legends in the lab too that are passed down orally from senior to junior members. My personal favorite is that drinking MilliQ (ultra pure water) will cause the cells in your gut to explode [1]. How such a stupid idea ever got taken seriously by scientists who should know better has always amazed me. [1] On the topic of drinking MilliQ water I have always encourage all the members of my lab to drink it. The reason why (apart from the taste which I quite like), is that it allows you to detect when you have bacterial contamination of your MilliQ carboy that will ruin your experiments.
- vitoreiji 12y ago
  Yup, I heard the gut-exploding-ddH2O in a lab and a girl in the class eventually did drink a little bit of it and was absolutely fine. A science champion, if you ask me.
  - danieltillett 12y ago
    I have had people try to physically stop me from drinking MilliQ water as they thought I was going to die in front of them! I have always thought it would be a good business idea to sell MilliQ branded water for human consumption. The complete lack of taste is an interesting taste in itself - the closest I can describe it as is it is like drinking liquid air.
- TrainedMonkey 12y ago
  I've heard that handling ultra pure water, can cause acid like burns due to water drawing out chemicals from the skin. I heard it from a bio PHD, and can't find any evidence googling it now.
  - danieltillett 12y ago
    This one is new to me - even more stupid than the drinking myth if that is even possible.
    - 12y ago
jannic 12y ago
I guess the original publication which listed the high iron content of spinach was: Bunge, G.. 'Weitere Untersuchungen über die Aufnahme des Eisens in den Organismus des Säuglings'. Zeitschrift für Physiologische Chemie, 16 (1892) http://echo.mpiwg-berlin.mpg.de/ECHOdocuView?url=/permanent/vlp/lit16848/index.meta&pn=2 http://echo.mpiwg-berlin.mpg.de/ECHOdocuView?url=/permanent/... If that's true, then indeed, it was not a misplaced decimal point. Instead, Bunge listed the iron contents of dried food.
- danieltillett 12y ago
  It is interesting that the paper does not actually cite the original paper. I guess it is not critical to the paper, but it does complete the story. This does bring up a important point in that is old scientific publications in languages other than english. I have many times run into references in German, French or Russian that are only available as low quality scanned pdfs. As a monolingual English speaker these are completely inaccessible to me. I would be great to have all of them OCRed so at least we can run them through Google Translate.
  - jannic 12y ago
    It seems more interesting to me that Rekdal doesn't cite this article: http://www.bestthinking.com/articles/science/chemistry/biochemistry/the-spinach-popeye-iron-decimal-error-myth-is-finally-busted http://www.bestthinking.com/articles/science/chemistry/bioch... It's a 2012 followup by Mike Sutton to the two publications from 2010. It contains many new details, including a reference to Bunge (1892). Perhaps, one should not only try to cite the original source, but also look for recent research on the same topic?
    - danieltillett 12y ago
      >Perhaps, one should not only try to cite the original source, but also look for recent research on the same topic? This is (or should be) done standardly by all scientists. It is actually the best way to learn about a field. Find the key paper (or at least the paper the field thinks is key) and then read all the papers that cite that key paper. I am sure this has been done by somebody, but it would be great to have this citation link structure as a graph where you could click on any node and read the paper.
b_emery 12y ago
> The explanation for this phenomenon is usually that authors have lazily, sloppily, or fraudulently employed sources, and peer reviewers and editors have not discovered these weaknesses in the manuscripts during evaluation. I've seen this sort of thing in the literature I'm familiar with (HF Radar). I don't think it's laziness or fraud, but a version of the telephone game played as a kid. A 'fact' will get stated, and then repeated over and over until some of the original meaning is lost and it becomes gospel. This can happen because the fact is not crucial to the science in the paper but always part of the introductions or statements about how the system works. The statement I'm thinking about in particular is "if you have N receive antennas, you can only find (simultaneously) N-1 signals". This is true if you are using covariance-based methods such as Multiple Signal Classification (MUSIC). It got repeated and repeated until the caveats (simultaneous, and MUSIC) are longer included. Of course, if you have two orthogonal antennas you can distinguish two directions, perhaps not in complex situations!
- deleted 12y ago
  [deleted]
- jimmaswell 12y ago
  So that meams you can't get anything useful out of one antenna?
  - b_emery 12y ago
    If it's directional, you can get one direction. Which disproves the N-1 meme that is in so many papers.
juliendorra 12y ago
This article made me think of The Leprechauns of Software Engineering[1], a good book about the myths that are being passed as facts because of bad citations in Computer Science papers. [1] https://leanpub.com/leprechauns https://leanpub.com/leprechauns

129 points by dougmccune 12y ago

Be sure to read to the end, or, after you read enough to grasp the idea then skip down to the section "The narrow path has its temptations".

peatmoss 12y ago
Yeah, my first thought was, "wow, I always thought that about spinach!" But, that wasn't the point of the article at all. This was an engagingly written article that kind of read like a mystery whodoneit?
- vitoreiji 12y ago
  And the most concerningly satisfying is that everybodydoneit, apparently. Including me, who didn't bother to read any one single paper cited.
the_af 12y ago
Even better, read the section "An avalanche of low-quality research?" near the end. Even this notion is put to the test, and found wanting :) Excellent article!
- OBRekdal 12y ago
  Thanks a lot! :)

acbart 12y ago

It'd be interesting if we could break down academic papers into stand-alone facts, which could be cited more succinctly than the paragraphs of prose that we do now. Ideally, this would look more like academic proofs, and it'd be easier to argue and discuss components of a given paper. I'm sure that a good amount of beautiful subtext would be lost, but might it be worth it to have a more accurate way of linking knowledge?

danieltillett 12y ago
This could now be accomplished by providing an offset (say number of characters from the start) to where the information cited can be found. What I find more of an issue is when trying to track down the source of some “fact” is going three or four papers up the citation chain and finding the paper at the end doesn’t actually say what the starting paper claims.
- judk 12y ago
  Thanks for summarizing the OP.
- romaniv 12y ago
  This sounds like a problem that hyper-links should have solved.
hackuser 12y ago
> It'd be interesting if we could break down academic papers into stand-alone facts, which could be cited more succinctly than the paragraphs of prose that we do now. Perhaps there could be some kind of semantic syntax that would group together the factual claim and supporting information (data and references). For example, <"e=mc^2",cite1,cite2,data1,data2> The claims still could be included in prose, but also easily extracted, analyzed, manipulated, etc. With a little work the claims could form a semantic web, with all that entails, and the validity of cites could be checked automatically when the document is opened or on a regular basis, updating the claim appropriately. For example, the UI could display the claim differently based on whether the cited source has been withdrawn, partially invalidated, superseded, confirmed, rarely cited, heavily cited, based on impact factor, if the source's own citations have been invalidated (e.g., claimC cites claimB which cites claimA; if claimA is invalidated, there should be some sort of indication in claimC -- consider that the invalidation of a seminal claimA would set off a massive chain reaction, as it should), etc. But that all seems obvious and I suspect I'm not the first to think of it.
- jasonpriem 12y ago
  It is indeed interesting, and as you suspect it has attracted some attention. Check out "nanopubs,"(http://nanopub.org http://nanopub.org), or Simon Buckingham Shum's very cool work on ScholOnto (http://projects.kmi.open.ac.uk/scholonto/ http://projects.kmi.open.ac.uk/scholonto/) among others.

danieltillett 12y ago

There are an amazing number of scientific urban legends in the lab too that are passed down orally from senior to junior members. My personal favorite is that drinking MilliQ (ultra pure water) will cause the cells in your gut to explode [1]. How such a stupid idea ever got taken seriously by scientists who should know better has always amazed me. [1] On the topic of drinking MilliQ water I have always encourage all the members of my lab to drink it. The reason why (apart from the taste which I quite like), is that it allows you to detect when you have bacterial contamination of your MilliQ carboy that will ruin your experiments.

vitoreiji 12y ago
Yup, I heard the gut-exploding-ddH2O in a lab and a girl in the class eventually did drink a little bit of it and was absolutely fine. A science champion, if you ask me.
- danieltillett 12y ago
  I have had people try to physically stop me from drinking MilliQ water as they thought I was going to die in front of them! I have always thought it would be a good business idea to sell MilliQ branded water for human consumption. The complete lack of taste is an interesting taste in itself - the closest I can describe it as is it is like drinking liquid air.
TrainedMonkey 12y ago
I've heard that handling ultra pure water, can cause acid like burns due to water drawing out chemicals from the skin. I heard it from a bio PHD, and can't find any evidence googling it now.
- danieltillett 12y ago
  This one is new to me - even more stupid than the drinking myth if that is even possible.
  - 12y ago

jannic 12y ago

I guess the original publication which listed the high iron content of spinach was: Bunge, G.. 'Weitere Untersuchungen über die Aufnahme des Eisens in den Organismus des Säuglings'. Zeitschrift für Physiologische Chemie, 16 (1892) http://echo.mpiwg-berlin.mpg.de/ECHOdocuView?url=/permanent/vlp/lit16848/index.meta&pn=2 http://echo.mpiwg-berlin.mpg.de/ECHOdocuView?url=/permanent/... If that's true, then indeed, it was not a misplaced decimal point. Instead, Bunge listed the iron contents of dried food.

danieltillett 12y ago
It is interesting that the paper does not actually cite the original paper. I guess it is not critical to the paper, but it does complete the story. This does bring up a important point in that is old scientific publications in languages other than english. I have many times run into references in German, French or Russian that are only available as low quality scanned pdfs. As a monolingual English speaker these are completely inaccessible to me. I would be great to have all of them OCRed so at least we can run them through Google Translate.
- jannic 12y ago
  It seems more interesting to me that Rekdal doesn't cite this article: http://www.bestthinking.com/articles/science/chemistry/biochemistry/the-spinach-popeye-iron-decimal-error-myth-is-finally-busted http://www.bestthinking.com/articles/science/chemistry/bioch... It's a 2012 followup by Mike Sutton to the two publications from 2010. It contains many new details, including a reference to Bunge (1892). Perhaps, one should not only try to cite the original source, but also look for recent research on the same topic?
  - danieltillett 12y ago
    >Perhaps, one should not only try to cite the original source, but also look for recent research on the same topic? This is (or should be) done standardly by all scientists. It is actually the best way to learn about a field. Find the key paper (or at least the paper the field thinks is key) and then read all the papers that cite that key paper. I am sure this has been done by somebody, but it would be great to have this citation link structure as a graph where you could click on any node and read the paper.

b_emery 12y ago

> The explanation for this phenomenon is usually that authors have lazily, sloppily, or fraudulently employed sources, and peer reviewers and editors have not discovered these weaknesses in the manuscripts during evaluation. I've seen this sort of thing in the literature I'm familiar with (HF Radar). I don't think it's laziness or fraud, but a version of the telephone game played as a kid. A 'fact' will get stated, and then repeated over and over until some of the original meaning is lost and it becomes gospel. This can happen because the fact is not crucial to the science in the paper but always part of the introductions or statements about how the system works. The statement I'm thinking about in particular is "if you have N receive antennas, you can only find (simultaneously) N-1 signals". This is true if you are using covariance-based methods such as Multiple Signal Classification (MUSIC). It got repeated and repeated until the caveats (simultaneous, and MUSIC) are longer included. Of course, if you have two orthogonal antennas you can distinguish two directions, perhaps not in complex situations!

deleted 12y ago
[deleted]
jimmaswell 12y ago
So that meams you can't get anything useful out of one antenna?
- b_emery 12y ago
  If it's directional, you can get one direction. Which disproves the N-1 meme that is in so many papers.

juliendorra 12y ago

This article made me think of The Leprechauns of Software Engineering[1], a good book about the myths that are being passed as facts because of bad citations in Computer Science papers. [1] https://leanpub.com/leprechauns https://leanpub.com/leprechauns