8 ms·

It's not that nobody knows how to cite it, it's that nobody wants to cite things from outside of academia, perhaps because it lowers their own perceived prestig

by rspeer 8y ago

It's not that nobody knows how to cite it, it's that nobody wants to cite things from outside of academia, perhaps because it lowers their own perceived prestige. I have seen this in computational linguistics where people frequently don't cite their data sources, which often come from outside of the field or outside academia altogether. Instead they put a link to a webpage in a footnote, which does nothing for assigning credit. Reviewers don't take anyone to task for it, because they do that too. For a while, there was a collection of multilingual word counts over OpenSubtitles that everyone was using, and you would acquire it from the OneDrive of the pseudonymous blogger "Hermit Dave". Many people put the link to his blog in a footnote, but I think I was the only person to come up with a citation for (Dave 2011). People don't even cite Google Books Ngrams and that's got a real paper [1]. [1] Lin, Y., Michel, J.-B., Aiden, E. L., Orwant, J., Brockman, W., and Petrov, S. (2012). Syntactic annotations for the Google Books Ngram Corpus. Proceedings of the ACL 2012 system demonstrations, 169-174.

jessriedel 8y ago
> Instead they put a link to a webpage in a footnote, which does nothing for assigning credit. Why would a footnote be considered less credit than something in the list of references? It's pretty common in my field to use footnotes to give credit to other researchers for ideas that arose in an informal context, e.g., "I thank A. Zee for pointing this out". Indeed, the question of whether prose footnotes are at the bottom of the page rather than displayed intermixed with citations to published sources (as in Physical Review Letters) is just a stylistic choices that varies by journal. Obviously unpublished sources can't easily be counted into automated citation tracking systems for technical reasons, but if the author wanted that they could publish something (or, e.g., put it in a free repository like the arXiv).
- OscarCunningham 8y ago
  > It's pretty common in my field to use footnotes to give credit to other researchers for ideas that arose in an informal context, e.g., "I thank A. Zee for pointing this out". Is your field Quantum Field Theory? https://www.kitp.ucsb.edu/zee/ https://www.kitp.ucsb.edu/zee/
  - jessriedel 8y ago
    No, quantum information, but I always thought Tony Zee had the best whimsical name to use as an example in these situations :)
- rspeer 8y ago
  Crediting a web page in a footnote is almost no credit at all. The web page didn't compile the data. A person did. Sometimes the person is inconveniently pseudonymous, but other times they should be credited by name. I added the Google Books Ngrams example slightly later, but: People _do_ publish papers about their data, and their citation counts are often extremely low, because research users continue to use footnotes instead.
  - jessriedel 8y ago
    Are you talking about someone crediting a webpage without attempting to attach the name of the author? I have never seen this in academic research, ever. Can you link to an example? In the OP case, the 4chan author was anonymous. EDIT: > People _do_ publish papers about their data, and their citation counts are often extremely low, because research users continue to use footnotes instead. Are you talking about researchers not giving credit for pure data sets? That is a really very different topic.
    - rspeer 8y ago
      > Are you talking about someone crediting a webpage without attempting to attach the name of the author? I have never seen this in academic research, ever. Can you link to an example? Let's continue with the Google Ngrams example. All I had to do was search Google Scholar for "Google Ngrams" to find easy examples on the first page of results, like this one: http://www.aclweb.org/anthology/W14-1611 http://www.aclweb.org/anthology/W14-1611 The paper is written by famous Stanford researchers, and they do not cite a single author of their data. Just URLs in footnotes. This also happens to my research (ConceptNet) all the time. The fact that you haven't noticed it doesn't mean I'm making it up. I don't see why you think researchers not giving credit for data is a different topic from what I brought up. What I said was, researchers in my field are in the habit of not giving credit for data, and instead they have adopted the convention of using half-assed footnotes with URLs in them, which is not appropriate credit. I speculate that one reason for this habit is that data often comes from inconveniently non-academic sources and would look bad in the bibliography. The habit is so pervasive that authors remain in the habit even when a paper exists.
presscast 8y ago
>It's not that nobody knows how to cite it, it's that nobody wants to cite things from outside of academia, perhaps because it lowers their own perceived prestige. What makes you say this? Citing anybody does little to increase your personal prestige. I think the issue has more to do with 4chan not being peer-reviewed, and the proof being so recent that few papers have yet been written. On the contrary, citing a hot new thing is what academics do best.
- caconym_ 8y ago
  Somebody pointed out in the Twitter thread that academics often cite unreviewed papers on arXiv. If true (cannot confirm it myself, not in academia) then the peer-review-as-a-prerequisite-for-citation thing doesn't really hold water.
  - laretluval 8y ago
    It depends on the field. In fast moving fields like machine learning it is common to cite unreviewed stuff on arXiv. In slower fields it is less common/accepted.
  - ISL 8y ago
    Peer review is no prerequisite for citation. The first paper I looked at for an example (at the top of today's "Recent" papers in PRD), cites three arXiv entries, out of 91 references. https://journals.aps.org/prd/references/10.1103/PhysRevD.98.074023 https://journals.aps.org/prd/references/10.1103/PhysRevD.98.... Successful peer review in an upstanding journal is a signal of relative reliability and importance, but it is not the arbiter of usefulness.
    - sgc 8y ago
      Exactly. If I am sure of myself, I will cite anyone with information pertinent to my work. I am their peer and just reviewed it.
throwawaymath 8y ago
To give you a counterpoint, I've read plenty of computer science and mathematics papers where someone cited an arXiv paper by an obscure or unknown researcher. I've even seen blog posts and email correspondence cited. In fact, in one of my own papers I cited an unpublished manuscript which you can't even find on the internet - you have to personally contact the author, who hasn't touched the thing in the last 25 years. For all intents and purposes, this manuscript - while valuable - only exists in the tribal folklore of the research community. But it's still cited and most active people in the field probably have a personal copy of it on their hard drive.
- eindiran 8y ago
  I think the difference is that arXiv papers don't have the same perceptions of low prestige attached as a random blog or a 4chan thread does. The issue isn't that the work is (yet) unpublished or that its obscure: its that it is from outside academia or traditionally prestigious sources.
  - romwell 8y ago
    >academia or traditionally prestigious sources. You can't really put ArXiV and "traditionally prestigious sources" in the same sentence. Any crackpot can (and does) post on ArXiV. The reason why I'd be more comfortable citing a paper on ArXiV than a random web page (as a mathematician): 1. ArXiV is the archive for papers. You put something there, it's not going away. You can't even remove your own paper once you make it public there[1]. It's a reliable place I can refer people to. Personal websites? Well, my own very reliable academic web page has been removed by sysadmins without notice after graduation because "we don't maintain accounts". 2. Unrelated content. ArXiV won't have it - unlike, say, the 4chan thread in question. Do I really want to explain to people that I wasn't citing getting "14 * 14! derps out of the way"?. Not to mention ads, memes and NSFW content - which is also there[2][3] ---------- The other reasons are less important, but are still valid: 3. Presentation. Something on the ArXiV probably is formatted and structured like a readable paper; there are baseline expectations. Reading someone's scratchwork is borderline painful, and a proof on a webpage is usually just that. Compare [3] to a rewrite[4] - which one is easier to understand at a glance? 4. Reputation. This is different from "traditional prestige". The reputation of ArXiV is that most of the people who do serious work put it there. Most of the people produce nonsense (possibly including yours truly) also put it there, but the probability of any work having been done by someone who is willing to put effort into it goes down dramatically if the work is not on ArXiV. 5. Signaling. Since it's so easy to post there, any work not ArXiV is signaling "I don't care about my work being available to other people in the field". Why one read something that the author doesn't want them to? If they did, they'd put it on the ArXiV. 6. Attribution. A result on ArXiV makes it easy for me to attribute the result to someone. A thread on 4chan? Do I attribute it to everyone who participated? A single anonymous author? ---------- In short: people give credit to blackboard scribbles and napkin proofs that came out of discussions at conferences, but you won't see "4th beer coaster, such-and-such conference banquet, 2018" as something that people cite. A proof on 4chan is somewhere in the same category. ---------- The disclaimer here is that this applies mostly to new results in mathematics. Old results are not necessarily available online, and often we don't have a choice. Practices in other fields might be different. --------- [1] https://arxiv.org/help/withdraw https://arxiv.org/help/withdraw [2]https://warosu.org/sci/thread/S3751105#p3751197 https://warosu.org/sci/thread/S3751105#p3751197 [3]http://4watch.org/superstring/ http://4watch.org/superstring/ [4]https://docs.google.com/viewer?a=v&pid=forums&srcid=MTUwMTUxMjExNDk4NTk5NjY5OTkBMDMxNDgwMTA5ODA5OTYyNzcyNDQBdlNFMnM3eTVCUUFKATAuMQEBdjI&authuser=0 https://docs.google.com/viewer?a=v&pid=forums&srcid=MTUwMTUx...
coliveira 8y ago
I don't see a problem with a footnote entry, this is a form of credit. The issue with bibliographic entries is that they are each day more automatically processed, so publishers and editors try to avoid non-standard entries. On the other hand, if you're publishing something in a blog or webpage you're not looking for the kind of credit that is given on a bibliographic citation, so a footnote seems to be just fine.
timr 8y ago
"It's not that nobody knows how to cite it, it's that nobody wants to cite things from outside of academia, perhaps because it lowers their own perceived prestige." Unlikely. I can't speak for everybody, but I can speak for myself, and I would be extremely hesitant to cite a random webpage in a journal article for several reasons: 1) It can change. ArXiv has versioning, but even so, this is still a problem. You cite something, and months later you go back and find the thing you cited says something slightly different. 1b) It can disappear. Again, less of an issue for ArXiv, but a big problem for any other website. Linkrot is real. 2) There's probably a better source. Maybe in this example a 4chan thread is the original provenance of an idea, but 99 times out of 100, there's a better, original source for the idea. 3) There's no peer review. Peer-review is not a pre-requisite for publication, but if you don't have it, it just makes people more skeptical of your results. 4) It's perceived as lazy. This is closest to your point (i.e. prestige), but not the same. Like it or not, if you cite informal sources in a paper, people start to wonder about the rigor of everything else you do. It's an intangible, but when you're trying to get papers accepted, intangibles count. Basically, citing a webpage is risky. You avoid it if you can.
- incompatible 8y ago
  For a paper in mathematics, perhaps it would be best to quote the entire proof of interest, if it doesn't appear anywhere else in a peer-reviewed publication. That would avoid the link rot problem. Also make sure archive.org has a copy, if possible.
  - timr 8y ago
    Sure, you can quote a short proof. But again, that's a niche solution to a niche problem, and it defeats the purpose of having citations. It also doesn't convince anyone else that they should believe the proof you've cited. Archive.org is not a solution to this problem. Nobody is going to go fishing in archive.org for a citation that has disappeared. Even if you have the URL, what is the correct date to use? There are just practical problems with link citations, and you don't need to invent malevolent intent to understand why researchers are reluctant to use them.
mynewtb 8y ago
Why would you cite a paper on the technical details of your data source in linguistics?
- Zalastax 8y ago
  You want reproducibility, the results will be affected a lot by your data source, real citations enable tools like google scholar. Why not cite? I see only drawbacks in continuing with the footnote/no citation trend.
noobermin 8y ago
Unless I'm ignorant, no one does that in science. We cite where data or even a quote comes from. Seems perverse not to do so.
saagarjha 8y ago
> It's not that nobody knows how to cite it, it's that nobody wants to cite things from outside of academia, perhaps because it lowers their own perceived prestige. Such is academia, I guess. It’s more important to get “prestige” than to properly cite someone else’s work-where it was published should not matter.
rmbeard 8y ago
Not citing sources is plagiarism, are you actually saying that it is the norm in computational linguistics to plagiarize? If so this is extremely worrying.

by rspeer 8y ago

jessriedel 8y ago

> Instead they put a link to a webpage in a footnote, which does nothing for assigning credit. Why would a footnote be considered less credit than something in the list of references? It's pretty common in my field to use footnotes to give credit to other researchers for ideas that arose in an informal context, e.g., "I thank A. Zee for pointing this out". Indeed, the question of whether prose footnotes are at the bottom of the page rather than displayed intermixed with citations to published sources (as in Physical Review Letters) is just a stylistic choices that varies by journal. Obviously unpublished sources can't easily be counted into automated citation tracking systems for technical reasons, but if the author wanted that they could publish something (or, e.g., put it in a free repository like the arXiv).

OscarCunningham 8y ago
> It's pretty common in my field to use footnotes to give credit to other researchers for ideas that arose in an informal context, e.g., "I thank A. Zee for pointing this out". Is your field Quantum Field Theory? https://www.kitp.ucsb.edu/zee/ https://www.kitp.ucsb.edu/zee/
- jessriedel 8y ago
  No, quantum information, but I always thought Tony Zee had the best whimsical name to use as an example in these situations :)
rspeer 8y ago
Crediting a web page in a footnote is almost no credit at all. The web page didn't compile the data. A person did. Sometimes the person is inconveniently pseudonymous, but other times they should be credited by name. I added the Google Books Ngrams example slightly later, but: People _do_ publish papers about their data, and their citation counts are often extremely low, because research users continue to use footnotes instead.
- jessriedel 8y ago
  Are you talking about someone crediting a webpage without attempting to attach the name of the author? I have never seen this in academic research, ever. Can you link to an example? In the OP case, the 4chan author was anonymous. EDIT: > People _do_ publish papers about their data, and their citation counts are often extremely low, because research users continue to use footnotes instead. Are you talking about researchers not giving credit for pure data sets? That is a really very different topic.
  - rspeer 8y ago
    > Are you talking about someone crediting a webpage without attempting to attach the name of the author? I have never seen this in academic research, ever. Can you link to an example? Let's continue with the Google Ngrams example. All I had to do was search Google Scholar for "Google Ngrams" to find easy examples on the first page of results, like this one: http://www.aclweb.org/anthology/W14-1611 http://www.aclweb.org/anthology/W14-1611 The paper is written by famous Stanford researchers, and they do not cite a single author of their data. Just URLs in footnotes. This also happens to my research (ConceptNet) all the time. The fact that you haven't noticed it doesn't mean I'm making it up. I don't see why you think researchers not giving credit for data is a different topic from what I brought up. What I said was, researchers in my field are in the habit of not giving credit for data, and instead they have adopted the convention of using half-assed footnotes with URLs in them, which is not appropriate credit. I speculate that one reason for this habit is that data often comes from inconveniently non-academic sources and would look bad in the bibliography. The habit is so pervasive that authors remain in the habit even when a paper exists.

presscast 8y ago

>It's not that nobody knows how to cite it, it's that nobody wants to cite things from outside of academia, perhaps because it lowers their own perceived prestige. What makes you say this? Citing anybody does little to increase your personal prestige. I think the issue has more to do with 4chan not being peer-reviewed, and the proof being so recent that few papers have yet been written. On the contrary, citing a hot new thing is what academics do best.

caconym_ 8y ago
Somebody pointed out in the Twitter thread that academics often cite unreviewed papers on arXiv. If true (cannot confirm it myself, not in academia) then the peer-review-as-a-prerequisite-for-citation thing doesn't really hold water.
- laretluval 8y ago
  It depends on the field. In fast moving fields like machine learning it is common to cite unreviewed stuff on arXiv. In slower fields it is less common/accepted.
- ISL 8y ago
  Peer review is no prerequisite for citation. The first paper I looked at for an example (at the top of today's "Recent" papers in PRD), cites three arXiv entries, out of 91 references. https://journals.aps.org/prd/references/10.1103/PhysRevD.98.074023 https://journals.aps.org/prd/references/10.1103/PhysRevD.98.... Successful peer review in an upstanding journal is a signal of relative reliability and importance, but it is not the arbiter of usefulness.
  - sgc 8y ago
    Exactly. If I am sure of myself, I will cite anyone with information pertinent to my work. I am their peer and just reviewed it.

throwawaymath 8y ago

To give you a counterpoint, I've read plenty of computer science and mathematics papers where someone cited an arXiv paper by an obscure or unknown researcher. I've even seen blog posts and email correspondence cited. In fact, in one of my own papers I cited an unpublished manuscript which you can't even find on the internet - you have to personally contact the author, who hasn't touched the thing in the last 25 years. For all intents and purposes, this manuscript - while valuable - only exists in the tribal folklore of the research community. But it's still cited and most active people in the field probably have a personal copy of it on their hard drive.

eindiran 8y ago
I think the difference is that arXiv papers don't have the same perceptions of low prestige attached as a random blog or a 4chan thread does. The issue isn't that the work is (yet) unpublished or that its obscure: its that it is from outside academia or traditionally prestigious sources.
- romwell 8y ago
  >academia or traditionally prestigious sources. You can't really put ArXiV and "traditionally prestigious sources" in the same sentence. Any crackpot can (and does) post on ArXiV. The reason why I'd be more comfortable citing a paper on ArXiV than a random web page (as a mathematician): 1. ArXiV is the archive for papers. You put something there, it's not going away. You can't even remove your own paper once you make it public there[1]. It's a reliable place I can refer people to. Personal websites? Well, my own very reliable academic web page has been removed by sysadmins without notice after graduation because "we don't maintain accounts". 2. Unrelated content. ArXiV won't have it - unlike, say, the 4chan thread in question. Do I really want to explain to people that I wasn't citing getting "14 * 14! derps out of the way"?. Not to mention ads, memes and NSFW content - which is also there[2][3] ---------- The other reasons are less important, but are still valid: 3. Presentation. Something on the ArXiV probably is formatted and structured like a readable paper; there are baseline expectations. Reading someone's scratchwork is borderline painful, and a proof on a webpage is usually just that. Compare [3] to a rewrite[4] - which one is easier to understand at a glance? 4. Reputation. This is different from "traditional prestige". The reputation of ArXiV is that most of the people who do serious work put it there. Most of the people produce nonsense (possibly including yours truly) also put it there, but the probability of any work having been done by someone who is willing to put effort into it goes down dramatically if the work is not on ArXiV. 5. Signaling. Since it's so easy to post there, any work not ArXiV is signaling "I don't care about my work being available to other people in the field". Why one read something that the author doesn't want them to? If they did, they'd put it on the ArXiV. 6. Attribution. A result on ArXiV makes it easy for me to attribute the result to someone. A thread on 4chan? Do I attribute it to everyone who participated? A single anonymous author? ---------- In short: people give credit to blackboard scribbles and napkin proofs that came out of discussions at conferences, but you won't see "4th beer coaster, such-and-such conference banquet, 2018" as something that people cite. A proof on 4chan is somewhere in the same category. ---------- The disclaimer here is that this applies mostly to new results in mathematics. Old results are not necessarily available online, and often we don't have a choice. Practices in other fields might be different. --------- [1] https://arxiv.org/help/withdraw https://arxiv.org/help/withdraw [2]https://warosu.org/sci/thread/S3751105#p3751197 https://warosu.org/sci/thread/S3751105#p3751197 [3]http://4watch.org/superstring/ http://4watch.org/superstring/ [4]https://docs.google.com/viewer?a=v&pid=forums&srcid=MTUwMTUxMjExNDk4NTk5NjY5OTkBMDMxNDgwMTA5ODA5OTYyNzcyNDQBdlNFMnM3eTVCUUFKATAuMQEBdjI&authuser=0 https://docs.google.com/viewer?a=v&pid=forums&srcid=MTUwMTUx...

coliveira 8y ago

I don't see a problem with a footnote entry, this is a form of credit. The issue with bibliographic entries is that they are each day more automatically processed, so publishers and editors try to avoid non-standard entries. On the other hand, if you're publishing something in a blog or webpage you're not looking for the kind of credit that is given on a bibliographic citation, so a footnote seems to be just fine.

timr 8y ago

"It's not that nobody knows how to cite it, it's that nobody wants to cite things from outside of academia, perhaps because it lowers their own perceived prestige." Unlikely. I can't speak for everybody, but I can speak for myself, and I would be extremely hesitant to cite a random webpage in a journal article for several reasons: 1) It can change. ArXiv has versioning, but even so, this is still a problem. You cite something, and months later you go back and find the thing you cited says something slightly different. 1b) It can disappear. Again, less of an issue for ArXiv, but a big problem for any other website. Linkrot is real. 2) There's probably a better source. Maybe in this example a 4chan thread is the original provenance of an idea, but 99 times out of 100, there's a better, original source for the idea. 3) There's no peer review. Peer-review is not a pre-requisite for publication, but if you don't have it, it just makes people more skeptical of your results. 4) It's perceived as lazy. This is closest to your point (i.e. prestige), but not the same. Like it or not, if you cite informal sources in a paper, people start to wonder about the rigor of everything else you do. It's an intangible, but when you're trying to get papers accepted, intangibles count. Basically, citing a webpage is risky. You avoid it if you can.

incompatible 8y ago
For a paper in mathematics, perhaps it would be best to quote the entire proof of interest, if it doesn't appear anywhere else in a peer-reviewed publication. That would avoid the link rot problem. Also make sure archive.org has a copy, if possible.
- timr 8y ago
  Sure, you can quote a short proof. But again, that's a niche solution to a niche problem, and it defeats the purpose of having citations. It also doesn't convince anyone else that they should believe the proof you've cited. Archive.org is not a solution to this problem. Nobody is going to go fishing in archive.org for a citation that has disappeared. Even if you have the URL, what is the correct date to use? There are just practical problems with link citations, and you don't need to invent malevolent intent to understand why researchers are reluctant to use them.

mynewtb 8y ago

Why would you cite a paper on the technical details of your data source in linguistics?

Zalastax 8y ago
You want reproducibility, the results will be affected a lot by your data source, real citations enable tools like google scholar. Why not cite? I see only drawbacks in continuing with the footnote/no citation trend.

noobermin 8y ago

Unless I'm ignorant, no one does that in science. We cite where data or even a quote comes from. Seems perverse not to do so.

saagarjha 8y ago

> It's not that nobody knows how to cite it, it's that nobody wants to cite things from outside of academia, perhaps because it lowers their own perceived prestige. Such is academia, I guess. It’s more important to get “prestige” than to properly cite someone else’s work-where it was published should not matter.

rmbeard 8y ago

Not citing sources is plagiarism, are you actually saying that it is the norm in computational linguistics to plagiarize? If so this is extremely worrying.