5 ms·

An "academically correct" solution (i.e., not to taint the prestige of a scholar who cites the solution) would be to cite a real, proxy scholar who checks the s

An "academically correct" solution (i.e., not to taint the prestige of a scholar who cites the solution) would be to cite a real, proxy scholar who checks the solution, but gives all credit to the anonymous 4chan user. Scientists often abhor citing from blogs, forums, and Wikipedia, as those publications "are not peer-reviewed by trustworthy professionals".

PopeDotNinja 8y ago
They are also prone to disappearing, post-citation editing, etc.
- Bartweiss 8y ago
  Definitely true, and worth responding to. (For instance, with a specific Internet Archive source or a link to the Wikipedia version history instead of main page.) But it's also uncomfortably true that academic works have an ugly tendency to disappear post-citation. Less so in mathematics, I'm sure, but even in the hard sciences far too many papers out there reference things that weren't digitized, rely on data that wasn't published with the paper, or otherwise hinge on something you can't actually get to. There's probably no great answer here, it just feels like authors and editors sometimes use "it was reputably published once" to shrug off disappearing sources.
mkirklions 8y ago
>Scientists often abhor citing from blogs, forums, and Wikipedia, as those publications "are not peer-reviewed by trustworthy professionals". Academia. And even that is error prone due to bias.
simonbyrne 8y ago
There is precedent for this solution: https://academic.oup.com/biomet/article-abstract/76/1/183/225021?redirectedFrom=PDF https://academic.oup.com/biomet/article-abstract/76/1/183/22...
rtkwe 8y ago
The person writing that review is going to need to cite it somehow though so the question of how best to cite an ephemeral anonymous post that's only available through archives or secondary summaries (like the linked wiki) still stands.
qubax 8y ago
> Scientists often abhor citing from blogs, forums, and Wikipedia, as those publications "are not peer-reviewed by trustworthy professionals". Which is sad because most peer-reviewed material are not trustworthy either. https://bigthink.com/neurobonkers/believe-it-or-not-most-published-research-findings-are-probably-false https://bigthink.com/neurobonkers/believe-it-or-not-most-pub... https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... It's worrying that academia is slowly turning into a rigid "appeal to authority" church rather than a space for truth and inquiry.
- Certhas 8y ago
  Peer review is a base level quality check, not a correctness check in most disciplines. It provides value, indeed most of my papers have been improved by going through peer review. But it's good to be conscious of what it can and can not do. One thing that it does do is signal intent. It signals that you are willing to engage with the expert community, and specifically their feedback and criticism. Finally, there is of course a fallacy in saying "not trustworthy either". If 99% of random blog posts are wrong, but 60% of published papers are wrong, then a random paper is 40 times more likely to be correct than a random blog post and it's perfectly sensible to be hesitant about citing the latter. Finally, scientists have always cited outside of traditional sources and continue to do so [1]. I see no evidence for your assertion that "appeal to authority" has become more prevalent or important than it used to be. With all the ongoing open science initiatives it's rather the opposite. [1] Personal communication with various scientists, 2018
  - neuromantik8086 8y ago
    > Peer review is a base level quality check Ideally this is true. At the worst though, you'll end up with a peer reviewer that gets pedantic or one who's a member of a citation cartel [0] that demands that you cite his/her friends papers unnecessarily as a quid-pro-quo leading to publication. It's difficult to say how common this is in practice since academia is generally terrible about self-reflection, but I observed concerning evidence of this when I was still in science. In some fields at least, there's ample evidence [1] that peer review provides inadequate QC controls for methods sections, making it literally impossible to replicate results due to entire processing steps being omitted. This is why post-publication services such as PubPeer have become more and more relevant, since it has become obvious as time has elapsed how woefully inadequate the initial screening process is. [0] https://academia.stackexchange.com/questions/81261/reviewer-recommending-citation-of-certain-papers-of-specific-group-of-authors-ma https://academia.stackexchange.com/questions/81261/reviewer-... [1] https://www.ncbi.nlm.nih.gov/pubmed/22796459 https://www.ncbi.nlm.nih.gov/pubmed/22796459
empath75 8y ago
Indeed, someone[1] has already done that: https://docs.google.com/viewer?a=v&pid=forums&srcid=MTUwMTUxMjExNDk4NTk5NjY5OTkBMDMxNDgwMTA5ODA5OTYyNzcyNDQBdlNFMnM3eTVCUUFKATAuMQEBdjI&authuser=0 https://docs.google.com/viewer?a=v&pid=forums&srcid=MTUwMTUx... [1] http://jaypantone.com/ http://jaypantone.com/
kop316 8y ago
I actually feel that the response here is a great one: "I feel like you should be able (forced even!) to cite things that aren't journal articles- even blog posts- if they contributed to your body of work. But citing things that aren't prestigious reduces the perceived prestige of your own, which is a perverse incentive not to cite." EDIT: I can't seem to figure out how to link to specific tweets, so this is the guy that said it: https://twitter.com/_delta_zero https://twitter.com/_delta_zero He has a great point. If the proof is correct, you should cite the raw document. If you have to cite it through a proxy scholar, it feels like a form of gate keeping.
- ahjones 8y ago
  Click the date/time on a tweet to get the permalink. Here's the tweet you referenced: https://twitter.com/_delta_zero/status/1054820799473836032 https://twitter.com/_delta_zero/status/1054820799473836032
  - kop316 8y ago
    Ahh, that was the one place I did not look, thank you!
- thatcat 8y ago
  Exactly: a correct source is a correct source. No need to reserve legitimacy exclusively for paid journals. Maybe the source could be cited through internet archive or other so that the document is definitely the same version as the one seen at the time of citation.
- UncleMeat 8y ago
  Is that actually true? I have no experience in grad level math but in my field (security) people frequently cite all sorts of non academic content. I've personally done it a lot of times and never heard anything about it from reviewers.
  - thrmsforbfast 8y ago
    No, it's not. People cite "informal" sources all the time. This only really matters on the receiving end of the citation. I.e., if lots of people are citing your blog or forum post, then it can be a good idea to take your content and put it in arxiv or similar. Just so that the citations "count" for you in the academic setting. And similarly, it's a bit rude (or at least odd) to cite someone's blog post when there's an accompanying published paper that describes the same work. But citing informal sources where no formal source exists is not (and afaik never has been) a problem.
ghaff 8y ago
It feels like there is, or should be, a distinction between "I'm basing my analysis on this other data/analysis/etc. that I'm working from the assumption it's correct" vs. "This idea/analysis/etc. gave me a good starting point. I'm not depending on it but I want to credit it." Thinking about what I do. There's a difference between me depending on the results of a survey and hearing a novel take on a market or whatever. In the former case, I'm dependent of the survey methodology being sound (and I'm going to trust some sources far more than others). In the latter case it's more "I never thought of things that way" but I'm going to do my own analysis.
RowBoat6123 8y ago
Realistically, the proof would probably just be rewritten in a paper in standard notation, with an indication that the original proof (or the idea) was due to an anonymous author. You see very similar things happen from time to time where there are "folklore" theorems in mathematics that someone eventually writes down 20 years later. Sometimes they're able to give an attribution, and sometimes not. But always they give a formal proof using modern notation and credit the idea with whatever fidelity they can.
dorfsmay 8y ago
The problems with blogs and wikipedia is that they can be changed, or in the case of blogs disappear. For wikipedia, you should reference the specific version of the article, which will never change. For blogs, or wikipedia articles you feel might get deleted, you can reference a specific version from archive.org.
- honestoHeminway 8y ago
  Drag it through the blockchain then.
- djsumdog 8y ago
  You can archive it through various services and save an exact screenshot/html snippet/text that can be cited. In the sources you add "Retrieved on.." and "Archived link... Secondary archived link.." .. and some journals are allowing artifacts these days too. Typically this is used for code or data sets, but it could also be used for copies of the anonymous source material.
  - gmueckl 8y ago
    You should be able to follow the references in a paper even in a hundred years or later. This obviously works quite well for print. Blogs, wikis and other websites are in general not permanent enough to be useful as references. Even when you archive them, the archive is likely to disappear much earlier than the paper for which it was made. In fact, I fear that the trend to publish supplemental material may lead to another crisis because a lot of that material is not managed and archived well and may disappear without warning. Some papers contain important details like long, involved and non-obvious calculations only in their supplements and are not really reproducible without them.
wanderfowl 8y ago
I agree that this is probably the cultural solution, provided the author can't be coerced to post on Arxiv, but it's an ugly one, which reeks of an ivory tower and gatekeeping ("nobody who isn't an established scholar in our tradition is truly capable of our work"). Perhaps this is better seen as a substantial defect in the fact that the academic web-of-trust model implemented in publication is based on quality of source (e.g. journal's prestige, or the submitting scholar's prestige) rather than quality of scholarship (e.g. whether a critical mass of researchers, rather than 1-3 anonymous reviewers, find it compelling and 'sign off' on it being good work). I hope (and suspect) that whatever emerges or is developed to replace the paid journal model of academic trust is designed to handle these situations gracefully.
Bartweiss 8y ago
The first reply from the thread's OP got me thinking about something I'd never really noticed: > I used to think it was because as a quasiscientific community we highly value peer review but ML people are totally fine with citing papers on arxiv that haven't been accepted anywhere as long as you can replicate the results - I think perceived prestige nails it[1] Peer reviewed citations are excellent, and unstable citations like Wikipedia are obviously concerning. (At bare minimum, you'd want to link to the page history instead of the page.) But the willingness to cite arXiv content without hesitation definitely suggests that peer review is only part of what's going on there. [1]https://twitter.com/adversariel/status/1054836640307392512 https://twitter.com/adversariel/status/1054836640307392512
SilasX 8y ago
That's a bad reason. Even if the source isn't at the same, high standard, you should still cite it to: a) Tell others were to go if they want more context. b) Give credit for where you got the idea or for doing the legwork of putting it together. c) Allow auditing of the ultimate origin of a claim and e.g. identify whether one data point is getting repeated like it's two data points. https://en.wikipedia.org/wiki/Information_cascade https://en.wikipedia.org/wiki/Information_cascade

PopeDotNinja 8y ago

They are also prone to disappearing, post-citation editing, etc.

Bartweiss 8y ago
Definitely true, and worth responding to. (For instance, with a specific Internet Archive source or a link to the Wikipedia version history instead of main page.) But it's also uncomfortably true that academic works have an ugly tendency to disappear post-citation. Less so in mathematics, I'm sure, but even in the hard sciences far too many papers out there reference things that weren't digitized, rely on data that wasn't published with the paper, or otherwise hinge on something you can't actually get to. There's probably no great answer here, it just feels like authors and editors sometimes use "it was reputably published once" to shrug off disappearing sources.

mkirklions 8y ago

>Scientists often abhor citing from blogs, forums, and Wikipedia, as those publications "are not peer-reviewed by trustworthy professionals". Academia. And even that is error prone due to bias.

simonbyrne 8y ago

There is precedent for this solution: https://academic.oup.com/biomet/article-abstract/76/1/183/225021?redirectedFrom=PDF https://academic.oup.com/biomet/article-abstract/76/1/183/22...

rtkwe 8y ago

The person writing that review is going to need to cite it somehow though so the question of how best to cite an ephemeral anonymous post that's only available through archives or secondary summaries (like the linked wiki) still stands.

qubax 8y ago

> Scientists often abhor citing from blogs, forums, and Wikipedia, as those publications "are not peer-reviewed by trustworthy professionals". Which is sad because most peer-reviewed material are not trustworthy either. https://bigthink.com/neurobonkers/believe-it-or-not-most-published-research-findings-are-probably-false https://bigthink.com/neurobonkers/believe-it-or-not-most-pub... https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 https://journals.plos.org/plosmedicine/article?id=10.1371/jo... It's worrying that academia is slowly turning into a rigid "appeal to authority" church rather than a space for truth and inquiry.

Certhas 8y ago
Peer review is a base level quality check, not a correctness check in most disciplines. It provides value, indeed most of my papers have been improved by going through peer review. But it's good to be conscious of what it can and can not do. One thing that it does do is signal intent. It signals that you are willing to engage with the expert community, and specifically their feedback and criticism. Finally, there is of course a fallacy in saying "not trustworthy either". If 99% of random blog posts are wrong, but 60% of published papers are wrong, then a random paper is 40 times more likely to be correct than a random blog post and it's perfectly sensible to be hesitant about citing the latter. Finally, scientists have always cited outside of traditional sources and continue to do so [1]. I see no evidence for your assertion that "appeal to authority" has become more prevalent or important than it used to be. With all the ongoing open science initiatives it's rather the opposite. [1] Personal communication with various scientists, 2018
- neuromantik8086 8y ago
  > Peer review is a base level quality check Ideally this is true. At the worst though, you'll end up with a peer reviewer that gets pedantic or one who's a member of a citation cartel [0] that demands that you cite his/her friends papers unnecessarily as a quid-pro-quo leading to publication. It's difficult to say how common this is in practice since academia is generally terrible about self-reflection, but I observed concerning evidence of this when I was still in science. In some fields at least, there's ample evidence [1] that peer review provides inadequate QC controls for methods sections, making it literally impossible to replicate results due to entire processing steps being omitted. This is why post-publication services such as PubPeer have become more and more relevant, since it has become obvious as time has elapsed how woefully inadequate the initial screening process is. [0] https://academia.stackexchange.com/questions/81261/reviewer-recommending-citation-of-certain-papers-of-specific-group-of-authors-ma https://academia.stackexchange.com/questions/81261/reviewer-... [1] https://www.ncbi.nlm.nih.gov/pubmed/22796459 https://www.ncbi.nlm.nih.gov/pubmed/22796459

empath75 8y ago

Indeed, someone[1] has already done that: https://docs.google.com/viewer?a=v&pid=forums&srcid=MTUwMTUxMjExNDk4NTk5NjY5OTkBMDMxNDgwMTA5ODA5OTYyNzcyNDQBdlNFMnM3eTVCUUFKATAuMQEBdjI&authuser=0 https://docs.google.com/viewer?a=v&pid=forums&srcid=MTUwMTUx... [1] http://jaypantone.com/ http://jaypantone.com/

kop316 8y ago

I actually feel that the response here is a great one: "I feel like you should be able (forced even!) to cite things that aren't journal articles- even blog posts- if they contributed to your body of work. But citing things that aren't prestigious reduces the perceived prestige of your own, which is a perverse incentive not to cite." EDIT: I can't seem to figure out how to link to specific tweets, so this is the guy that said it: https://twitter.com/_delta_zero https://twitter.com/_delta_zero He has a great point. If the proof is correct, you should cite the raw document. If you have to cite it through a proxy scholar, it feels like a form of gate keeping.

ahjones 8y ago
Click the date/time on a tweet to get the permalink. Here's the tweet you referenced: https://twitter.com/_delta_zero/status/1054820799473836032 https://twitter.com/_delta_zero/status/1054820799473836032
- kop316 8y ago
  Ahh, that was the one place I did not look, thank you!
thatcat 8y ago
Exactly: a correct source is a correct source. No need to reserve legitimacy exclusively for paid journals. Maybe the source could be cited through internet archive or other so that the document is definitely the same version as the one seen at the time of citation.
UncleMeat 8y ago
Is that actually true? I have no experience in grad level math but in my field (security) people frequently cite all sorts of non academic content. I've personally done it a lot of times and never heard anything about it from reviewers.
- thrmsforbfast 8y ago
  No, it's not. People cite "informal" sources all the time. This only really matters on the receiving end of the citation. I.e., if lots of people are citing your blog or forum post, then it can be a good idea to take your content and put it in arxiv or similar. Just so that the citations "count" for you in the academic setting. And similarly, it's a bit rude (or at least odd) to cite someone's blog post when there's an accompanying published paper that describes the same work. But citing informal sources where no formal source exists is not (and afaik never has been) a problem.

ghaff 8y ago

It feels like there is, or should be, a distinction between "I'm basing my analysis on this other data/analysis/etc. that I'm working from the assumption it's correct" vs. "This idea/analysis/etc. gave me a good starting point. I'm not depending on it but I want to credit it." Thinking about what I do. There's a difference between me depending on the results of a survey and hearing a novel take on a market or whatever. In the former case, I'm dependent of the survey methodology being sound (and I'm going to trust some sources far more than others). In the latter case it's more "I never thought of things that way" but I'm going to do my own analysis.

RowBoat6123 8y ago

Realistically, the proof would probably just be rewritten in a paper in standard notation, with an indication that the original proof (or the idea) was due to an anonymous author. You see very similar things happen from time to time where there are "folklore" theorems in mathematics that someone eventually writes down 20 years later. Sometimes they're able to give an attribution, and sometimes not. But always they give a formal proof using modern notation and credit the idea with whatever fidelity they can.

dorfsmay 8y ago

The problems with blogs and wikipedia is that they can be changed, or in the case of blogs disappear. For wikipedia, you should reference the specific version of the article, which will never change. For blogs, or wikipedia articles you feel might get deleted, you can reference a specific version from archive.org.

honestoHeminway 8y ago
Drag it through the blockchain then.
djsumdog 8y ago
You can archive it through various services and save an exact screenshot/html snippet/text that can be cited. In the sources you add "Retrieved on.." and "Archived link... Secondary archived link.." .. and some journals are allowing artifacts these days too. Typically this is used for code or data sets, but it could also be used for copies of the anonymous source material.
- gmueckl 8y ago
  You should be able to follow the references in a paper even in a hundred years or later. This obviously works quite well for print. Blogs, wikis and other websites are in general not permanent enough to be useful as references. Even when you archive them, the archive is likely to disappear much earlier than the paper for which it was made. In fact, I fear that the trend to publish supplemental material may lead to another crisis because a lot of that material is not managed and archived well and may disappear without warning. Some papers contain important details like long, involved and non-obvious calculations only in their supplements and are not really reproducible without them.

wanderfowl 8y ago

I agree that this is probably the cultural solution, provided the author can't be coerced to post on Arxiv, but it's an ugly one, which reeks of an ivory tower and gatekeeping ("nobody who isn't an established scholar in our tradition is truly capable of our work"). Perhaps this is better seen as a substantial defect in the fact that the academic web-of-trust model implemented in publication is based on quality of source (e.g. journal's prestige, or the submitting scholar's prestige) rather than quality of scholarship (e.g. whether a critical mass of researchers, rather than 1-3 anonymous reviewers, find it compelling and 'sign off' on it being good work). I hope (and suspect) that whatever emerges or is developed to replace the paid journal model of academic trust is designed to handle these situations gracefully.

Bartweiss 8y ago

The first reply from the thread's OP got me thinking about something I'd never really noticed: > I used to think it was because as a quasiscientific community we highly value peer review but ML people are totally fine with citing papers on arxiv that haven't been accepted anywhere as long as you can replicate the results - I think perceived prestige nails it[1] Peer reviewed citations are excellent, and unstable citations like Wikipedia are obviously concerning. (At bare minimum, you'd want to link to the page history instead of the page.) But the willingness to cite arXiv content without hesitation definitely suggests that peer review is only part of what's going on there. [1]https://twitter.com/adversariel/status/1054836640307392512 https://twitter.com/adversariel/status/1054836640307392512

SilasX 8y ago

That's a bad reason. Even if the source isn't at the same, high standard, you should still cite it to: a) Tell others were to go if they want more context. b) Give credit for where you got the idea or for doing the legwork of putting it together. c) Allow auditing of the ultimate origin of a claim and e.g. identify whether one data point is getting repeated like it's two data points. https://en.wikipedia.org/wiki/Information_cascade https://en.wikipedia.org/wiki/Information_cascade