3 ms·

This reminds me of the legend of the Tower of Babel. God cursed humanity, rendering them unable to understand each other. Except in this case it's only "anonym

This reminds me of the legend of the Tower of Babel. God cursed humanity, rendering them unable to understand each other. Except in this case it's only "anonymous humanity" that's screwed. And it isn't that we can't understand but that we have lost all trust. Is it a human with something to say or is it a robot playing a game?

shoubidouwah 3y ago
Sam Kriss has a pretty wild post about this, with a running metaphor of LLMs as golems: https://samkriss.substack.com/p/the-cacophony https://samkriss.substack.com/p/the-cacophony
- swayvil 3y ago
  Thanks. I'll read that. I was thinking demons mself. Being, above all, liars. A quadrillion synthetic liars. Breeding and speaking at the speed of light.
EGreg 3y ago
People still think about it in quaint ways. Spammers are a minir annoyance. I have one word for you: swarms. Swarms of sleeper accounts can be deployed with existing technology. They can astroturf revolutions, rile people up against others, make reputational attacks at scale — and perhaps most destructively, they can overwhelm anyone with a different viewpoint, with 10-1 content ratio (eg I could deploy 20 bots on HN to downvote and argue with the people who disagree with me ad nauseum). As you can imagine, forums could use “grandfathered” accounts to sort of mitigate this. But when regular people start preferring bots, it is game over. You think it is hard for bots to endear themselves over humans? Karma / likes / upvotes is actually a measurable and gameable metric, and the bots can excel at it better than humans. Not to mention if you own a dating site you have access to all the messages that have ever worked to get a date or get a laugh. HN anyone can analyze the type of messages that got the most upvotes. We keep analyzing one bot among 8 humans at a poker table. No one seriously considers the exponential jump to 1000 bots surrounding every human. At best - we are building a zoo for ourselves. But we can’t stop!
- irln 3y ago
  What are your thoughts on requiring payment to participate as a way to reduce spam? I realize that in this particular context payment could prevent participation for some, however, are there other reasons why payment would be ineffective?
  - EGreg 3y ago
    That was always required for sybil attacks. But if the long-term benefit exceeds the cost of membership, it will be paid. Think of a conqueror renting an army for a small bit of time to go pillage and gain very long term control of a village. I am in a good position to speak about this since I designed decentralized social platforms like https://qbix.com https://qbix.com and smart contract platforms like https://intercoin.app https://intercoin.app that build coins for communities. If anyone can make an account, your community has unlimited cost dealing with it. It could be a sleeper account acting like a human, until one day the cost starts getting imposed in the form of, say, coordinated swarm vandalism: https://en.m.wikipedia.org/wiki/Wikipedia:Wikiality_and_Other_Tripling_Elephants https://en.m.wikipedia.org/wiki/Wikipedia:Wikiality_and_Othe... But it could be a LOT more subtle and plant the seeds to move opinion of a forum gradually towards anything you want, including simply accepting and upvoting articles from your domains where your bots create content. If you just mint a coin and give it out for basic tasks, people will take advantage of that by creating many accounts. You have to start asking people to appear in person to get a certificate at least — which is what Sam Altman’s Worldcoin is about. But you don’t need to go that far, every community can just have in-person get togethers once in a while to say hi to each other (“shocker!”) I foresee a lot of people starting to retreat to gated communities. The thing is that anyone can invite a bot in. If you give everyone N invites, and the people they invite N invites, they’ll eventually use them tk invite N to the power X people. And those people might be real, but eventually may run a bot once in a while. You may think that’s not likely but actually it has already has been happening, people loove to run automation on their own behalf, and have no idea what it does, exposing themselves to compromises and attacks at scale: packages in package managers NYC landlords pricing apps wall street trading bots I wrote about it here: https://magarshak.com/blog/?p=385 https://magarshak.com/blog/?p=385 But as I say, the real trouble begins when people start to prefer bots. They choose convenience over security and truth. Bots won’t have resistance to an agenda and a lie like humans will. I guess our only hope is to dilute the bot swarms with opposing bot swarms, like Bitcoin miners competing kind of. But at that point the entire internet will be a DARK FOREST.

shoubidouwah 3y ago

Sam Kriss has a pretty wild post about this, with a running metaphor of LLMs as golems: https://samkriss.substack.com/p/the-cacophony https://samkriss.substack.com/p/the-cacophony

swayvil 3y ago
Thanks. I'll read that. I was thinking demons mself. Being, above all, liars. A quadrillion synthetic liars. Breeding and speaking at the speed of light.

EGreg 3y ago

People still think about it in quaint ways. Spammers are a minir annoyance. I have one word for you: swarms. Swarms of sleeper accounts can be deployed with existing technology. They can astroturf revolutions, rile people up against others, make reputational attacks at scale — and perhaps most destructively, they can overwhelm anyone with a different viewpoint, with 10-1 content ratio (eg I could deploy 20 bots on HN to downvote and argue with the people who disagree with me ad nauseum). As you can imagine, forums could use “grandfathered” accounts to sort of mitigate this. But when regular people start preferring bots, it is game over. You think it is hard for bots to endear themselves over humans? Karma / likes / upvotes is actually a measurable and gameable metric, and the bots can excel at it better than humans. Not to mention if you own a dating site you have access to all the messages that have ever worked to get a date or get a laugh. HN anyone can analyze the type of messages that got the most upvotes. We keep analyzing one bot among 8 humans at a poker table. No one seriously considers the exponential jump to 1000 bots surrounding every human. At best - we are building a zoo for ourselves. But we can’t stop!

irln 3y ago
What are your thoughts on requiring payment to participate as a way to reduce spam? I realize that in this particular context payment could prevent participation for some, however, are there other reasons why payment would be ineffective?
- EGreg 3y ago
  That was always required for sybil attacks. But if the long-term benefit exceeds the cost of membership, it will be paid. Think of a conqueror renting an army for a small bit of time to go pillage and gain very long term control of a village. I am in a good position to speak about this since I designed decentralized social platforms like https://qbix.com https://qbix.com and smart contract platforms like https://intercoin.app https://intercoin.app that build coins for communities. If anyone can make an account, your community has unlimited cost dealing with it. It could be a sleeper account acting like a human, until one day the cost starts getting imposed in the form of, say, coordinated swarm vandalism: https://en.m.wikipedia.org/wiki/Wikipedia:Wikiality_and_Other_Tripling_Elephants https://en.m.wikipedia.org/wiki/Wikipedia:Wikiality_and_Othe... But it could be a LOT more subtle and plant the seeds to move opinion of a forum gradually towards anything you want, including simply accepting and upvoting articles from your domains where your bots create content. If you just mint a coin and give it out for basic tasks, people will take advantage of that by creating many accounts. You have to start asking people to appear in person to get a certificate at least — which is what Sam Altman’s Worldcoin is about. But you don’t need to go that far, every community can just have in-person get togethers once in a while to say hi to each other (“shocker!”) I foresee a lot of people starting to retreat to gated communities. The thing is that anyone can invite a bot in. If you give everyone N invites, and the people they invite N invites, they’ll eventually use them tk invite N to the power X people. And those people might be real, but eventually may run a bot once in a while. You may think that’s not likely but actually it has already has been happening, people loove to run automation on their own behalf, and have no idea what it does, exposing themselves to compromises and attacks at scale: packages in package managers NYC landlords pricing apps wall street trading bots I wrote about it here: https://magarshak.com/blog/?p=385 https://magarshak.com/blog/?p=385 But as I say, the real trouble begins when people start to prefer bots. They choose convenience over security and truth. Bots won’t have resistance to an agenda and a lie like humans will. I guess our only hope is to dilute the bot swarms with opposing bot swarms, like Bitcoin miners competing kind of. But at that point the entire internet will be a DARK FOREST.