5 ms·

People already believe garbage at a pretty alarming rate. It's easy to guess at a number of possible outcomes here: - More junk text moves the public to doubt

People already believe garbage at a pretty alarming rate. It's easy to guess at a number of possible outcomes here: - More junk text moves the public to doubt legitimate information even further than they currently do. - There is so much human-generated junk text that adding more of it via AI actually doesn't have much of an effect. - People return to lean on experts, perhaps even more than before. (just as a number of tech-literate folks have now returned to relying on brand name.) Speculation is easy of course, so who knows what will actually happen.

ElFitz 6y ago
True. But a simple API to generate junk text? It can scale, cheaply, beyond measure. No need for a troll farm, hiring, managing and training tens or hundreds of people. A reasonable amount of cash, a bit of motivation, some moderate technical skills, and voilà! Anyone can compete with the Russian troll farms now and build their own networks of hundreds or hundreds of thousands sufficiently credible (as humans) fake accounts spewing garbage and patting each other on the back via likes, retweets and whatnots. All with the appropriate fake news blogs and sites happily churning out grammatically correct nonsense that makes (enough) sense. Basically, this kid’s dream: https://www.nbcnews.com/news/world/fake-news-how-partying-macedonian-teen-earns-thousands-publishing-lies-n692451 https://www.nbcnews.com/news/world/fake-news-how-partying-ma...
- jackTheMan 6y ago
  But even Russian/Chinese bots can step up as it is much easier now to flood forums (like reddit) etc, where e.g. a China critic article appears to kill any discussion.
  - ElFitz 6y ago
    Oh, definitely. Existing operations also absolutely can leverage that in order to amplify their reach and capabilities.
  - deleted 6y ago
    [deleted]
  - luckylion 6y ago
    I find it easier to identify humans that flood forums though. Especially non-native speakers usually are somewhat easy to spot, I assume that's true in any language. That's different for ML-generated texts. On the other hand, human texts are more "on message", but if all you want to do is create noise, I guess you don't need to have targeted communications.
    - TheAdamAndChe 6y ago
      > if all you want to do is create noise, I guess you don't need to have targeted communications. This is key in anti-extremist operations on anonymous boards. 4chan and other similar sites are absolutely nothing like they were a decade ago, I presume because of such bots flooding them with noise.
- TriNetra 6y ago
  And unfortunatley, the web will be forced to move toward verified human identities to fight with such junk and anonymous browsing will become a thing of the history.
  - fxtentacle 6y ago
    That wouldn't work. You can already rent humans and sim cards to do captchas and phone verification. There's nothing stopping someone from verifying their bots.
    - Jeff_Brown 6y ago
      It might be neither easy nor impossible. You could imagine s reputation system that depended on what others thought of you. Start the network with just people the developers know personally. Admit new people when enough existing members couch for them. Sometimes fake cancers will appear, but they'll sometimes get sniffed out, and the people who let them in punished somehow (temporarily blocked? Delete photos at random? Idk) strictly enough that people generally don't want to let in bad actors.
burlesona 6y ago
I think we may come to see the era of roughly 1990-2010 as the golden age of information: relative abundance creating new opportunity, before the noise drowned it all out. I suspect that in the future people will, ironically, return more strictly to tribal knowledge, as the media and the internet will be (already is) a vast ocean from which you can pull anything you want to believe. Thus nothing you see or hear from mass media or the internet can be trusted, there are no experts, and you go back to information scarcity as you have to rely on your immediate human network for trust. Actually I think we’re already seeing the return to tribal authority, the early waves are already here on Facebook and YouTube... they just haven’t devolved to strictly local circles of trust yet.
- mylons 6y ago
  neal stephenson has a cool prediction on how to handle this in one of his latest books: curators. people of means will have a curated view of the internet and information that costs $$$
  - burlesona 6y ago
    It’s already here. How many people are paying for newsletter subscriptions etc, and how quickly is that growing? :)
  - fapjacks 6y ago
    In many ways, predicting the future is just about looking at how to apply past trends to our modern world. People and organizations with the resources to do so are actively working to cram their "curation" into the public sphere, to try and sway the masses. That's advertising. It's the same thing it was in the 20th century, just more personal and using more sinister means of hogging your attention span. Peoples' intuition about "advertising" is as a commercial enterprise, but I'm certain all the textbooks that talk about the 2010s and 2020s will mention the role of mass bot/shill campaigns to sway the public by governments, companies, organizations, individual people.
isolli 6y ago
> just as a number of tech-literate folks have now returned to relying on brand name out of curiosity, what are you referring to?
- everdrive 6y ago
  In the early-ish days of the consumer internet, consumers had a new and huge information advantage over companies. People moved from relying on brand name, to reading online reviews. Often finding niche brands which they had otherwise not heard of. Now, in 2021, that experience is flipped on its head. Amazon reviews are gamed and cannot be trusted. Companies build niche brands like fly-by-night companies, and the lesser known brands have a very high chance of being both seriously inferior, and also short lived. At least, this has been my experience, and the experience of some others. [edit] And as further anecdotal proof that things have come full circle, my elderly mother in law keeps getting tricked by Amazon purchases. "The reviews were good," she'll say before returning something.
  - whimsicalism 6y ago
    I just rely on reviewers like NYT Wirecutter and then buy whatever the reviewers suggest (and is cheap) on Amazon.
    - deleted 6y ago
      [deleted]
    - fapjacks 6y ago
      Why do you imagine the NYT isn't taking a bigger paycheck for saying the same thing the fake Amazon reviews are saying? We live in a world where brands are trash. They're built by the lowest bidder with the absolute cheapest parts ("the Toyota Way", heavy airquotes), and packaged with lies (marketing). Where did this idea come from that it's only the foreign resellers of Chinese products that are posting fake Amazon reviews, when it is absolutely the same case with household brand name companies? Go to Fakespot and look up random brand name companies you would normally "trust" and see what how Fakespot rates the company. We are WAY past the point where the vast majority of companies (governments, organizations, etc) aren't using these types of mass propaganda campaigns. A brand name in the general case is no more trustworthy than some imitator.
deleted 6y ago
[deleted]
hanniabu 6y ago
> People return to lean on experts The problem with this is that people look at anybody confirming their bias as an expert. I can't tell you how many FB posts I've seen where some armchair poster claims that a researcher is wrong because of xyz and it's being reposted thousands of times.
api 6y ago
Concrete prediction: There will be a global cult similar in nature to Qanon driven by an AI spitting out generated bullshit within the next ten years. That's assuming some percentage of Qanon word salad isn't the output of Markov chain generators. A lot of it resembles low-order statistical text generator output after having been trained on a corpus of 1990s Usenet alt.conspiracy and the Protocols of the Elders of Zion.
cblconfederate 6y ago
People believe what's believable (even if backed up by garbage). GPTs dont make believable stuff, but they can be used to flower up some b.s. idea. Nothing that can't be done with a few hired trolls, and the proliferation of garbage will endanger the troll industry, as people will start becoming suspicious. So i doubt its impact can go beyond generating spam and noise.
UnFleshedOne 6y ago
There is a market (well, a need at least) for nonsense detectors that work similarly to the way ad blockers work. Detect internal inconsistencies, non-sequiturs, low information density and other similar reasons to avoid reading the text -- and visibly flag or block that. That should eliminate 80%+ of existing human generated text content and lead to text generators composing useful articles.

ElFitz 6y ago

True. But a simple API to generate junk text? It can scale, cheaply, beyond measure. No need for a troll farm, hiring, managing and training tens or hundreds of people. A reasonable amount of cash, a bit of motivation, some moderate technical skills, and voilà! Anyone can compete with the Russian troll farms now and build their own networks of hundreds or hundreds of thousands sufficiently credible (as humans) fake accounts spewing garbage and patting each other on the back via likes, retweets and whatnots. All with the appropriate fake news blogs and sites happily churning out grammatically correct nonsense that makes (enough) sense. Basically, this kid’s dream: https://www.nbcnews.com/news/world/fake-news-how-partying-macedonian-teen-earns-thousands-publishing-lies-n692451 https://www.nbcnews.com/news/world/fake-news-how-partying-ma...

jackTheMan 6y ago
But even Russian/Chinese bots can step up as it is much easier now to flood forums (like reddit) etc, where e.g. a China critic article appears to kill any discussion.
- ElFitz 6y ago
  Oh, definitely. Existing operations also absolutely can leverage that in order to amplify their reach and capabilities.
- deleted 6y ago
  [deleted]
- luckylion 6y ago
  I find it easier to identify humans that flood forums though. Especially non-native speakers usually are somewhat easy to spot, I assume that's true in any language. That's different for ML-generated texts. On the other hand, human texts are more "on message", but if all you want to do is create noise, I guess you don't need to have targeted communications.
  - TheAdamAndChe 6y ago
    > if all you want to do is create noise, I guess you don't need to have targeted communications. This is key in anti-extremist operations on anonymous boards. 4chan and other similar sites are absolutely nothing like they were a decade ago, I presume because of such bots flooding them with noise.
TriNetra 6y ago
And unfortunatley, the web will be forced to move toward verified human identities to fight with such junk and anonymous browsing will become a thing of the history.
- fxtentacle 6y ago
  That wouldn't work. You can already rent humans and sim cards to do captchas and phone verification. There's nothing stopping someone from verifying their bots.
  - Jeff_Brown 6y ago
    It might be neither easy nor impossible. You could imagine s reputation system that depended on what others thought of you. Start the network with just people the developers know personally. Admit new people when enough existing members couch for them. Sometimes fake cancers will appear, but they'll sometimes get sniffed out, and the people who let them in punished somehow (temporarily blocked? Delete photos at random? Idk) strictly enough that people generally don't want to let in bad actors.

burlesona 6y ago

I think we may come to see the era of roughly 1990-2010 as the golden age of information: relative abundance creating new opportunity, before the noise drowned it all out. I suspect that in the future people will, ironically, return more strictly to tribal knowledge, as the media and the internet will be (already is) a vast ocean from which you can pull anything you want to believe. Thus nothing you see or hear from mass media or the internet can be trusted, there are no experts, and you go back to information scarcity as you have to rely on your immediate human network for trust. Actually I think we’re already seeing the return to tribal authority, the early waves are already here on Facebook and YouTube... they just haven’t devolved to strictly local circles of trust yet.

mylons 6y ago
neal stephenson has a cool prediction on how to handle this in one of his latest books: curators. people of means will have a curated view of the internet and information that costs $$$
- burlesona 6y ago
  It’s already here. How many people are paying for newsletter subscriptions etc, and how quickly is that growing? :)
- fapjacks 6y ago
  In many ways, predicting the future is just about looking at how to apply past trends to our modern world. People and organizations with the resources to do so are actively working to cram their "curation" into the public sphere, to try and sway the masses. That's advertising. It's the same thing it was in the 20th century, just more personal and using more sinister means of hogging your attention span. Peoples' intuition about "advertising" is as a commercial enterprise, but I'm certain all the textbooks that talk about the 2010s and 2020s will mention the role of mass bot/shill campaigns to sway the public by governments, companies, organizations, individual people.

isolli 6y ago

> just as a number of tech-literate folks have now returned to relying on brand name out of curiosity, what are you referring to?

everdrive 6y ago
In the early-ish days of the consumer internet, consumers had a new and huge information advantage over companies. People moved from relying on brand name, to reading online reviews. Often finding niche brands which they had otherwise not heard of. Now, in 2021, that experience is flipped on its head. Amazon reviews are gamed and cannot be trusted. Companies build niche brands like fly-by-night companies, and the lesser known brands have a very high chance of being both seriously inferior, and also short lived. At least, this has been my experience, and the experience of some others. [edit] And as further anecdotal proof that things have come full circle, my elderly mother in law keeps getting tricked by Amazon purchases. "The reviews were good," she'll say before returning something.
- whimsicalism 6y ago
  I just rely on reviewers like NYT Wirecutter and then buy whatever the reviewers suggest (and is cheap) on Amazon.
  - deleted 6y ago
    [deleted]
  - fapjacks 6y ago
    Why do you imagine the NYT isn't taking a bigger paycheck for saying the same thing the fake Amazon reviews are saying? We live in a world where brands are trash. They're built by the lowest bidder with the absolute cheapest parts ("the Toyota Way", heavy airquotes), and packaged with lies (marketing). Where did this idea come from that it's only the foreign resellers of Chinese products that are posting fake Amazon reviews, when it is absolutely the same case with household brand name companies? Go to Fakespot and look up random brand name companies you would normally "trust" and see what how Fakespot rates the company. We are WAY past the point where the vast majority of companies (governments, organizations, etc) aren't using these types of mass propaganda campaigns. A brand name in the general case is no more trustworthy than some imitator.

deleted 6y ago

[deleted]

hanniabu 6y ago

> People return to lean on experts The problem with this is that people look at anybody confirming their bias as an expert. I can't tell you how many FB posts I've seen where some armchair poster claims that a researcher is wrong because of xyz and it's being reposted thousands of times.

api 6y ago

Concrete prediction: There will be a global cult similar in nature to Qanon driven by an AI spitting out generated bullshit within the next ten years. That's assuming some percentage of Qanon word salad isn't the output of Markov chain generators. A lot of it resembles low-order statistical text generator output after having been trained on a corpus of 1990s Usenet alt.conspiracy and the Protocols of the Elders of Zion.

cblconfederate 6y ago

People believe what's believable (even if backed up by garbage). GPTs dont make believable stuff, but they can be used to flower up some b.s. idea. Nothing that can't be done with a few hired trolls, and the proliferation of garbage will endanger the troll industry, as people will start becoming suspicious. So i doubt its impact can go beyond generating spam and noise.

UnFleshedOne 6y ago

There is a market (well, a need at least) for nonsense detectors that work similarly to the way ad blockers work. Detect internal inconsistencies, non-sequiturs, low information density and other similar reasons to avoid reading the text -- and visibly flag or block that. That should eliminate 80%+ of existing human generated text content and lead to text generators composing useful articles.