4 ms·

The end state of these models is to probably poison the public internet completely as a source of reliable knowledge. Things that don't have citation chains tha

The end state of these models is to probably poison the public internet completely as a source of reliable knowledge. Things that don't have citation chains that go back to human-created offline content won't be trustworthy - is that a real newspaper article linked from that Wikipedia article, or a fake one for a fake paper generated from thin air? You were looking at Reddit because you think you're more likely to get a real testimonial vs SEO spam? Whoops, there goes that too. Etc. This sounds bad, but I agree that it's only the acceleration of an existing trend. The only bright side is that it's really only rolling back a couple of decades of status quo. It's not like humanity has never operated without Wikipedia or instant search for news articles (specifically scoping it down there from "google" in general since a lot of what a plain web search on many topics will turn up is already unreliable).

themitigating 4y ago
"The end state of these models is to probably poison the public internet completely" Based on what?
- majormajor 4y ago
  Start with the damage done by cheaply-paid lightly-researched copypasta SEO spam junk pages to the trustworthiness of info returned by the typical Google search result. There are no shortage of people out there - or on HN - who know better than to trust most random webpages. Now automate that to 10x, 100x, 1000x the volume. Then throw in fake images and video too.
- erybodyknows 4y ago
  I believe they are referring to an exponential increase in noise alongside a relatively linear progression of signal.
falcor84 4y ago
>Things that don't have citation chains that go back to human-created offline content won't be trustworthy - is that a real newspaper article linked from that Wikipedia article, or a fake one for a fake paper generated from thin air? Why stop at "human-created offline content"? Seeing how a lot of peer-reviewed science is based on improper methodology, bad analysis, false data and buggy code, maybe it would be better to question everything. Obviously I don't have a full solution here, but a possible approach might be something like a fuzzy "epistemic status" you can attach to each item, and have that somehow flow across a network of trust, that everyone could personally customize. In other words, seeing how we're all already living in some echo chamber or another, maybe we could at least codify this.
- majormajor 4y ago
  Yeah, in my heads, human creation or curation is a "necessary, but not sufficient" thing. A ton of junk is already out there pre-computer-generated-content. So being human created isn't enough to let us know it's high quality - but not being human created is enough for the immediate future to let us know that it can't be trusted for domains where you aren't expert enough to tell the difference between "plausible" and "correct." And then in the anonymous public internet, you simply won't be able to tell "human created" from "computer created" so you'll have to just treat it all as low-value spam. Another interesting question here: let's say someone created a future AI that could actually be relied on to give "correct" or at least "properly caveated and nuanced" answers to questions - how is the layman possibly to know if that engine is actually being used behind the scenes of any online text? E.g. does the bullshit ML-generated content poison the well for "better" AI too, not just for anonymous people?
- skissane 4y ago
  > Seeing how a lot of peer-reviewed science is based on improper methodology, bad analysis, false data and buggy code, maybe it would be better to question everything. Peer-review isn't perfect, but it is still a more reliable source of information than (e.g) Reddit At least retraction is a thing – it doesn't happen as often as it should, but sometimes bad papers do get retracted, and there is a process people can use to challenge work they think should not have been published. Also, many journal databases have a "Cited by" feature, and so you can always read papers/letters-to-the-editor/etc responding to it - often, if research is particularly bad, sooner or later someone else will publish a criticism of it
CWuestefeld 4y ago
Your last paragraph seems to indicate that you already recognize the giant hole in the argument - that it's no different than what we've got today, between people posting poorly-informed opinions, or even our media in general.
- haswell 4y ago
  I think there's a meaningful difference between what we have today and a future where those poorly informed opinions can be trivially generated at scale. If individuals posting poorly informed opinions are small weapons fire, the AI equivalent is a nuclear bomb.
- Jensson 4y ago
  The difference is the language, with AI anyone can churn out text that look like a scientific paper, that makes it way harder to filter out the bullshit.
skissane 4y ago
> The end state of these models is to probably poison the public internet completely as a source of reliable knowledge. The Internet has made it easier than ever before to access academic journals – PubMed, JSTOR, Google Scholar, IEEE Xplore, etc. Of course, much of that content requires payment – but, an increasing percentage is open content, and there are often ways to find paywalled papers without paying for them (e.g. preprint archives; email the original author, who is often legally allowed to redistribute the paper, and happy to share their own work around; visit a library; ask friends/relatives who are university staff/students; ask a famous Kazakhstani computer programmer if she happens to have a copy) The content of academic journals isn't always guaranteed to be true, of course. But it is significantly less likely to be blatantly invented out of thin air, or automatically generated by some AI spambot. All academics are biased–especially so in fields which have high relevance to contemporary social/political/cultural controversies–but usually the bias of a paper's author is rather obvious, and you can take that into account when evaluating their work. One can generally infer the quality of a journal by signs such as its publisher (journals published by Springer or Elsevier are generally more respected than those published by MDPI), who is on its editorial board (respected names in the field, professors at prestigious universities, or a bunch of nobodies you've never heard of?), whether or not it gets included in databases such as PubMed, JSTOR, Science Citation Index, etc.
- majormajor 4y ago
  Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one). How far away do you think we are from some being able to auto-generate entire fake journals, with plausible-sounding and -looking fake text, fake images, etc, pushing pre-specified prompted conclusions. Maybe mix in some real plagiarized articles as well in tangential areas. And then you auto-generate a bunch of blogs, that occasionally have some posts talking about how JoesFakeJournal is just as reputable as JSTOR, etc, and actually better because [some random, probably fake, controversial claim]. Or how it's even better for certain topics. And then auto-generate a bunch of other blogs which occasionally have posts claiming that the legitimate ones are actually fraudulent. How big of a web of this sort would need to be faked before you, the outsider without special inside-baseball knowledge of academic publishing, would struggle to tell which journals you could trust from which you couldn't? EDIT: And could a clever enough spammer not just brute-force the creation of all that crap, but figure out how to use prompts to get the language models to create the necessary follow-up prompts, etc, to recursively build out the web of bullshit?
jvm___ 4y ago
https://maggieappleton.com/ai-dark-forest https://maggieappleton.com/ai-dark-forest
krageon 4y ago
Newspapers are so filled with lies it is more fair to express which parts of them are true. The internet being a cesspit of misdirection and nonsense is not any different from every other part of human society. Of course the AI does these things too, because it is based on this input. It's a reflection of what was put in. Nothing happening today (on the internet) will be the cause of anything being "rolled back" to some sort of historical better moment, because that moment does not exist today and most likely has never existed.

themitigating 4y ago

"The end state of these models is to probably poison the public internet completely" Based on what?

majormajor 4y ago
Start with the damage done by cheaply-paid lightly-researched copypasta SEO spam junk pages to the trustworthiness of info returned by the typical Google search result. There are no shortage of people out there - or on HN - who know better than to trust most random webpages. Now automate that to 10x, 100x, 1000x the volume. Then throw in fake images and video too.
erybodyknows 4y ago
I believe they are referring to an exponential increase in noise alongside a relatively linear progression of signal.

falcor84 4y ago

>Things that don't have citation chains that go back to human-created offline content won't be trustworthy - is that a real newspaper article linked from that Wikipedia article, or a fake one for a fake paper generated from thin air? Why stop at "human-created offline content"? Seeing how a lot of peer-reviewed science is based on improper methodology, bad analysis, false data and buggy code, maybe it would be better to question everything. Obviously I don't have a full solution here, but a possible approach might be something like a fuzzy "epistemic status" you can attach to each item, and have that somehow flow across a network of trust, that everyone could personally customize. In other words, seeing how we're all already living in some echo chamber or another, maybe we could at least codify this.

majormajor 4y ago
Yeah, in my heads, human creation or curation is a "necessary, but not sufficient" thing. A ton of junk is already out there pre-computer-generated-content. So being human created isn't enough to let us know it's high quality - but not being human created is enough for the immediate future to let us know that it can't be trusted for domains where you aren't expert enough to tell the difference between "plausible" and "correct." And then in the anonymous public internet, you simply won't be able to tell "human created" from "computer created" so you'll have to just treat it all as low-value spam. Another interesting question here: let's say someone created a future AI that could actually be relied on to give "correct" or at least "properly caveated and nuanced" answers to questions - how is the layman possibly to know if that engine is actually being used behind the scenes of any online text? E.g. does the bullshit ML-generated content poison the well for "better" AI too, not just for anonymous people?
skissane 4y ago
> Seeing how a lot of peer-reviewed science is based on improper methodology, bad analysis, false data and buggy code, maybe it would be better to question everything. Peer-review isn't perfect, but it is still a more reliable source of information than (e.g) Reddit At least retraction is a thing – it doesn't happen as often as it should, but sometimes bad papers do get retracted, and there is a process people can use to challenge work they think should not have been published. Also, many journal databases have a "Cited by" feature, and so you can always read papers/letters-to-the-editor/etc responding to it - often, if research is particularly bad, sooner or later someone else will publish a criticism of it

CWuestefeld 4y ago

Your last paragraph seems to indicate that you already recognize the giant hole in the argument - that it's no different than what we've got today, between people posting poorly-informed opinions, or even our media in general.

haswell 4y ago
I think there's a meaningful difference between what we have today and a future where those poorly informed opinions can be trivially generated at scale. If individuals posting poorly informed opinions are small weapons fire, the AI equivalent is a nuclear bomb.
Jensson 4y ago
The difference is the language, with AI anyone can churn out text that look like a scientific paper, that makes it way harder to filter out the bullshit.

skissane 4y ago

> The end state of these models is to probably poison the public internet completely as a source of reliable knowledge. The Internet has made it easier than ever before to access academic journals – PubMed, JSTOR, Google Scholar, IEEE Xplore, etc. Of course, much of that content requires payment – but, an increasing percentage is open content, and there are often ways to find paywalled papers without paying for them (e.g. preprint archives; email the original author, who is often legally allowed to redistribute the paper, and happy to share their own work around; visit a library; ask friends/relatives who are university staff/students; ask a famous Kazakhstani computer programmer if she happens to have a copy) The content of academic journals isn't always guaranteed to be true, of course. But it is significantly less likely to be blatantly invented out of thin air, or automatically generated by some AI spambot. All academics are biased–especially so in fields which have high relevance to contemporary social/political/cultural controversies–but usually the bias of a paper's author is rather obvious, and you can take that into account when evaluating their work. One can generally infer the quality of a journal by signs such as its publisher (journals published by Springer or Elsevier are generally more respected than those published by MDPI), who is on its editorial board (respected names in the field, professors at prestigious universities, or a bunch of nobodies you've never heard of?), whether or not it gets included in databases such as PubMed, JSTOR, Science Citation Index, etc.

majormajor 4y ago
Let's assume you didn't have a pre-existing knowledge of "what are real, reputable journals for scientific research"? (As opposed to being in a "real world" community where that's institutional knowledge, where you would be looking to curate your internet usage instead of trusting the full public one). How far away do you think we are from some being able to auto-generate entire fake journals, with plausible-sounding and -looking fake text, fake images, etc, pushing pre-specified prompted conclusions. Maybe mix in some real plagiarized articles as well in tangential areas. And then you auto-generate a bunch of blogs, that occasionally have some posts talking about how JoesFakeJournal is just as reputable as JSTOR, etc, and actually better because [some random, probably fake, controversial claim]. Or how it's even better for certain topics. And then auto-generate a bunch of other blogs which occasionally have posts claiming that the legitimate ones are actually fraudulent. How big of a web of this sort would need to be faked before you, the outsider without special inside-baseball knowledge of academic publishing, would struggle to tell which journals you could trust from which you couldn't? EDIT: And could a clever enough spammer not just brute-force the creation of all that crap, but figure out how to use prompts to get the language models to create the necessary follow-up prompts, etc, to recursively build out the web of bullshit?

jvm___ 4y ago

https://maggieappleton.com/ai-dark-forest https://maggieappleton.com/ai-dark-forest

krageon 4y ago

Newspapers are so filled with lies it is more fair to express which parts of them are true. The internet being a cesspit of misdirection and nonsense is not any different from every other part of human society. Of course the AI does these things too, because it is based on this input. It's a reflection of what was put in. Nothing happening today (on the internet) will be the cause of anything being "rolled back" to some sort of historical better moment, because that moment does not exist today and most likely has never existed.