3 ms·

Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" com

Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one). How far away do you think we are from some being able to auto-generate entire fake journals, with plausible-sounding and -looking fake text, fake images, etc, pushing pre-specified prompted conclusions. Maybe mix in some real plagiarized articles as well in tangential areas. And then you auto-generate a bunch of blogs, that occasionally have some posts talking about how JoesFakeJournal is just as reputable as JSTOR, etc, and actually better because [some random, probably fake, controversial claim]. Or how it's even better for certain topics. And then auto-generate a bunch of other blogs which occasionally have posts claiming that the legitimate ones are actually fraudulent. How big of a web of this sort would need to be faked before you, the outsider without special inside-baseball knowledge of academic publishing, would struggle to tell which journals you could trust from which you couldn't? EDIT: And could a clever enough spammer not just brute-force the creation of all that crap, but figure out how to use prompts to get the language models to create the necessary follow-up prompts, etc, to recursively build out the web of bullshit?

skissane 4y ago
> Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one). I'm not an academic, so I don't actually have much in the way of "institutional knowledge". My understanding of academic publishing is mostly just stuff I've worked out by observing it from the outside, not through actual membership of any of the communities it serves. I'm sure there are (discipline-specific) nuances and complexities that an actual academic would understand to which I am completely oblivious, but it is entirely possible to get a mostly-accurate high-level understanding of that space without having any personal links to any academic community. For PubMed at least, consider the fact it has a .GOV domain, and the "An official website of the United States government" banner at the top. Sure, anyone can fake that banner, but faking the .GOV is a fair bit harder. I really doubt the US government is going to start hosting databases filled with fake AI-generated research papers. Even if someone hacked a .GOV website and uploaded such a database to it, it surely wouldn't last long; meanwhile, PubMed has been at the exact same web address for the last several years. Nobody approaches the Internet as a tabula rasa – everyone comes to it with some pre-existing "real world" knowledge. How do I know ".GOV" is reserved for US governmental entities? I was probably first introduced to that fact by a book or magazine article I read in high school, but I've since seen it repeated too many times to individually recall. I remember having a conversation with my brother (a doctor) about PubMed – I didn't learn about it from him, I was the one who introduced the topic into the conversation, but his response confirmed that it is a real thing to which actual medical professionals refer, as opposed to some AI-generated phantasm.

skissane 4y ago

> Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one). I'm not an academic, so I don't actually have much in the way of "institutional knowledge". My understanding of academic publishing is mostly just stuff I've worked out by observing it from the outside, not through actual membership of any of the communities it serves. I'm sure there are (discipline-specific) nuances and complexities that an actual academic would understand to which I am completely oblivious, but it is entirely possible to get a mostly-accurate high-level understanding of that space without having any personal links to any academic community. For PubMed at least, consider the fact it has a .GOV domain, and the "An official website of the United States government" banner at the top. Sure, anyone can fake that banner, but faking the .GOV is a fair bit harder. I really doubt the US government is going to start hosting databases filled with fake AI-generated research papers. Even if someone hacked a .GOV website and uploaded such a database to it, it surely wouldn't last long; meanwhile, PubMed has been at the exact same web address for the last several years. Nobody approaches the Internet as a tabula rasa – everyone comes to it with some pre-existing "real world" knowledge. How do I know ".GOV" is reserved for US governmental entities? I was probably first introduced to that fact by a book or magazine article I read in high school, but I've since seen it repeated too many times to individually recall. I remember having a conversation with my brother (a doctor) about PubMed – I didn't learn about it from him, I was the one who introduced the topic into the conversation, but his response confirmed that it is a real thing to which actual medical professionals refer, as opposed to some AI-generated phantasm.