4 ms·

> The end state of these models is to probably poison the public internet completely as a source of reliable knowledge. The Internet has made it easier than ev

> The end state of these models is to probably poison the public internet completely as a source of reliable knowledge. The Internet has made it easier than ever before to access academic journals – PubMed, JSTOR, Google Scholar, IEEE Xplore, etc. Of course, much of that content requires payment – but, an increasing percentage is open content, and there are often ways to find paywalled papers without paying for them (e.g. preprint archives; email the original author, who is often legally allowed to redistribute the paper, and happy to share their own work around; visit a library; ask friends/relatives who are university staff/students; ask a famous Kazakhstani computer programmer if she happens to have a copy) The content of academic journals isn't always guaranteed to be true, of course. But it is significantly less likely to be blatantly invented out of thin air, or automatically generated by some AI spambot. All academics are biased–especially so in fields which have high relevance to contemporary social/political/cultural controversies–but usually the bias of a paper's author is rather obvious, and you can take that into account when evaluating their work. One can generally infer the quality of a journal by signs such as its publisher (journals published by Springer or Elsevier are generally more respected than those published by MDPI), who is on its editorial board (respected names in the field, professors at prestigious universities, or a bunch of nobodies you've never heard of?), whether or not it gets included in databases such as PubMed, JSTOR, Science Citation Index, etc.

majormajor 4y ago
Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one). How far away do you think we are from some being able to auto-generate entire fake journals, with plausible-sounding and -looking fake text, fake images, etc, pushing pre-specified prompted conclusions. Maybe mix in some real plagiarized articles as well in tangential areas. And then you auto-generate a bunch of blogs, that occasionally have some posts talking about how JoesFakeJournal is just as reputable as JSTOR, etc, and actually better because [some random, probably fake, controversial claim]. Or how it's even better for certain topics. And then auto-generate a bunch of other blogs which occasionally have posts claiming that the legitimate ones are actually fraudulent. How big of a web of this sort would need to be faked before you, the outsider without special inside-baseball knowledge of academic publishing, would struggle to tell which journals you could trust from which you couldn't? EDIT: And could a clever enough spammer not just brute-force the creation of all that crap, but figure out how to use prompts to get the language models to create the necessary follow-up prompts, etc, to recursively build out the web of bullshit?
- skissane 4y ago
  > Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one). I'm not an academic, so I don't actually have much in the way of "institutional knowledge". My understanding of academic publishing is mostly just stuff I've worked out by observing it from the outside, not through actual membership of any of the communities it serves. I'm sure there are (discipline-specific) nuances and complexities that an actual academic would understand to which I am completely oblivious, but it is entirely possible to get a mostly-accurate high-level understanding of that space without having any personal links to any academic community. For PubMed at least, consider the fact it has a .GOV domain, and the "An official website of the United States government" banner at the top. Sure, anyone can fake that banner, but faking the .GOV is a fair bit harder. I really doubt the US government is going to start hosting databases filled with fake AI-generated research papers. Even if someone hacked a .GOV website and uploaded such a database to it, it surely wouldn't last long; meanwhile, PubMed has been at the exact same web address for the last several years. Nobody approaches the Internet as a tabula rasa – everyone comes to it with some pre-existing "real world" knowledge. How do I know ".GOV" is reserved for US governmental entities? I was probably first introduced to that fact by a book or magazine article I read in high school, but I've since seen it repeated too many times to individually recall. I remember having a conversation with my brother (a doctor) about PubMed – I didn't learn about it from him, I was the one who introduced the topic into the conversation, but his response confirmed that it is a real thing to which actual medical professionals refer, as opposed to some AI-generated phantasm.

majormajor 4y ago

Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one). How far away do you think we are from some being able to auto-generate entire fake journals, with plausible-sounding and -looking fake text, fake images, etc, pushing pre-specified prompted conclusions. Maybe mix in some real plagiarized articles as well in tangential areas. And then you auto-generate a bunch of blogs, that occasionally have some posts talking about how JoesFakeJournal is just as reputable as JSTOR, etc, and actually better because [some random, probably fake, controversial claim]. Or how it's even better for certain topics. And then auto-generate a bunch of other blogs which occasionally have posts claiming that the legitimate ones are actually fraudulent. How big of a web of this sort would need to be faked before you, the outsider without special inside-baseball knowledge of academic publishing, would struggle to tell which journals you could trust from which you couldn't? EDIT: And could a clever enough spammer not just brute-force the creation of all that crap, but figure out how to use prompts to get the language models to create the necessary follow-up prompts, etc, to recursively build out the web of bullshit?

skissane 4y ago
> Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one). I'm not an academic, so I don't actually have much in the way of "institutional knowledge". My understanding of academic publishing is mostly just stuff I've worked out by observing it from the outside, not through actual membership of any of the communities it serves. I'm sure there are (discipline-specific) nuances and complexities that an actual academic would understand to which I am completely oblivious, but it is entirely possible to get a mostly-accurate high-level understanding of that space without having any personal links to any academic community. For PubMed at least, consider the fact it has a .GOV domain, and the "An official website of the United States government" banner at the top. Sure, anyone can fake that banner, but faking the .GOV is a fair bit harder. I really doubt the US government is going to start hosting databases filled with fake AI-generated research papers. Even if someone hacked a .GOV website and uploaded such a database to it, it surely wouldn't last long; meanwhile, PubMed has been at the exact same web address for the last several years. Nobody approaches the Internet as a tabula rasa – everyone comes to it with some pre-existing "real world" knowledge. How do I know ".GOV" is reserved for US governmental entities? I was probably first introduced to that fact by a book or magazine article I read in high school, but I've since seen it repeated too many times to individually recall. I remember having a conversation with my brother (a doctor) about PubMed – I didn't learn about it from him, I was the one who introduced the topic into the conversation, but his response confirmed that it is a real thing to which actual medical professionals refer, as opposed to some AI-generated phantasm.