3 ms·

That solves the anonymity requirement part and I agree. But even if we made sure all account holders are humans, still we don’t have a way to test if the conten

That solves the anonymity requirement part and I agree. But even if we made sure all account holders are humans, still we don’t have a way to test if the content was truly written by the account holder or created by a bot. Manual human labor requirement creates a rate limiting which is removed by automated content creation in the name of the human. Worse yet, as long as the bot content drives engagement and bot action on ads can be prevented, platforms would have little objection to this. Reddit would benefit from us thinking we are talking to humans while interacting with bot content if it keeps us on the site. Even worse yet, being controversial or scandalous tends to generate more engagement than being agreeable, which has been so far only exploited at the recommendation layer, now will poison the content layer itself. I wonder if we will look back and see that we have witnessed the creation of the nuclear weapons of information warfare.

endgame 6y ago
> I wonder if we will look back and see that we have witnessed the creation of the nuclear weapons of information warfare. I wouldn't be surprised. What's the point of reading anything on the internet, or arguing with someone on a forum? Gmail-style canned replies were the beginning, now the only way I can be sure that I'm talking to a real human will be by meeting him or her face-to-face (and we all know how easy that is right now.)
cosmojg 6y ago
> Manual human labor requirement creates a rate limiting which is removed by automated content creation in the name of the human. What if we impose artificial rate limiting with posting fees or some kind of proof-of-work requirement? > I wonder if we will look back and see that we have witnessed the creation of the nuclear weapons of information warfare. That's a terrifying thought.
- pixl97 6y ago
  Proof of work can easily be farmed off on botnets.
  - cosmojg 6y ago
    Oh, definitely, but that will get very expensive very quickly, especially with a self-updating difficulty mechanism. I don't see botnets being much of a problem outside of one-off posts (or maybe a few more for state actors).
- Reelin 6y ago
  > > Manual human labor requirement creates a rate limiting which is removed by automated content creation in the name of the human. > What if we impose artificial rate limiting with posting fees or some kind of proof-of-work requirement? These hypothetical anonymized but authenticated interactions are backed (via ZKP or otherwise) by a trusted identity (which itself might be pseudonymous, government verified, whatever). As such, why not track and publish the (approximate) rate at which attestation interactions are undertaken by a given identity (per some very coarse grained unit of time)? This rate could then be linked to the otherwise anonymized actions upon attestation. Then people or platform moderators would be free to make their own judgments about what constitutes a reasonable (ie trustworthy) rate of interaction within a given context. (This is a form of information leakage, so in certain scenarios it could allow deanonymization or at least linking anonymous interactions together. Extreme care would have to be taken when selecting the degree of approximation and the duration of unit time.)
- Sebb767 6y ago
  > What if we impose artificial rate limiting with posting fees or some kind of proof-of-work requirement? This is going to be a tough one. Someone with access to GPT-3 is going to have access to enough funds to post extensively. Also, muting poorer people is not a good idea. Proof of work seems more promising, but you'll have trouble finding something that can't be automated. Even with captchas - when you're solving them, you're effectively training Googles network to solve their own challenges.
Reelin 6y ago
It strikes me that none of these are really novel problems. ML reduces the cost but otherwise the issues already exist. The current solution seems to be largely reputational - either interacting with identities that you personally trust or using a platform whose moderation you have some amount of faith in.
- ben_w 6y ago
  Scale has a quality all of its own. While I disapprove of government and business propaganda, I’m a bit more worried by the idea that any nut who takes a personal dislike to me or any group I just happen to be in, could single-handedly wage a propaganda war against me as competently and effectively as any of the government-led campaigns which led to the disappointingly large list of post-WW2 genocides.
  - acituan 6y ago
    Not only the scale but also the multi-agent nature of the problem makes it hard to predict. As an example, humans killing each other is an ancient problem. But humans killing each other at scale with nuclear weapons is a much different problem. We are able to keep that threat at bay by the virtue of only nation states having the power of killing at this scale; number of agents that need to coordinate not defecting (that understand mutually assured destruction) is a handful. Now imagine if some new technology made it possible to make nuclear weapons with materials found in a kitchen. Then we have a coordination problem that requires planet level cooperation with billions of agents. It only takes one defecting to cause disproportionate harm. By the same token, information warfare is not a new problem. Lying is an ancient invention. News outlets being biased makes it easy to guess those biases and somewhat balance them out. We expect states to engage in some degree of propaganda. But human created but AI curated content (i.e. recommendation algorithms) is already driving us a bit nuts; we are having a hard time making a cohorent narrative of what’s going on. AI created and curated content intermixed with human content will completely destroy our sensemaking capacity for the benefit of a few who can deploy this technology. Imagine a per-voter propaganda model that drives AI created social media interactions with them, that can precisely tell what issues would you need to change your mind on to make you change your vote, which could be polar opposite to what the next voter is going to hear. Imagine this was deployed by multiple nations whenever there is an election in a country. This would still at the level of nation-states having nuclear weapons. I can’t imagine what the consequences of a more diffuse access to this tech would look like.

endgame 6y ago

> I wonder if we will look back and see that we have witnessed the creation of the nuclear weapons of information warfare. I wouldn't be surprised. What's the point of reading anything on the internet, or arguing with someone on a forum? Gmail-style canned replies were the beginning, now the only way I can be sure that I'm talking to a real human will be by meeting him or her face-to-face (and we all know how easy that is right now.)

cosmojg 6y ago

> Manual human labor requirement creates a rate limiting which is removed by automated content creation in the name of the human. What if we impose artificial rate limiting with posting fees or some kind of proof-of-work requirement? > I wonder if we will look back and see that we have witnessed the creation of the nuclear weapons of information warfare. That's a terrifying thought.

pixl97 6y ago
Proof of work can easily be farmed off on botnets.
- cosmojg 6y ago
  Oh, definitely, but that will get very expensive very quickly, especially with a self-updating difficulty mechanism. I don't see botnets being much of a problem outside of one-off posts (or maybe a few more for state actors).
Reelin 6y ago
> > Manual human labor requirement creates a rate limiting which is removed by automated content creation in the name of the human. > What if we impose artificial rate limiting with posting fees or some kind of proof-of-work requirement? These hypothetical anonymized but authenticated interactions are backed (via ZKP or otherwise) by a trusted identity (which itself might be pseudonymous, government verified, whatever). As such, why not track and publish the (approximate) rate at which attestation interactions are undertaken by a given identity (per some very coarse grained unit of time)? This rate could then be linked to the otherwise anonymized actions upon attestation. Then people or platform moderators would be free to make their own judgments about what constitutes a reasonable (ie trustworthy) rate of interaction within a given context. (This is a form of information leakage, so in certain scenarios it could allow deanonymization or at least linking anonymous interactions together. Extreme care would have to be taken when selecting the degree of approximation and the duration of unit time.)
Sebb767 6y ago
> What if we impose artificial rate limiting with posting fees or some kind of proof-of-work requirement? This is going to be a tough one. Someone with access to GPT-3 is going to have access to enough funds to post extensively. Also, muting poorer people is not a good idea. Proof of work seems more promising, but you'll have trouble finding something that can't be automated. Even with captchas - when you're solving them, you're effectively training Googles network to solve their own challenges.

Reelin 6y ago

It strikes me that none of these are really novel problems. ML reduces the cost but otherwise the issues already exist. The current solution seems to be largely reputational - either interacting with identities that you personally trust or using a platform whose moderation you have some amount of faith in.

ben_w 6y ago
Scale has a quality all of its own. While I disapprove of government and business propaganda, I’m a bit more worried by the idea that any nut who takes a personal dislike to me or any group I just happen to be in, could single-handedly wage a propaganda war against me as competently and effectively as any of the government-led campaigns which led to the disappointingly large list of post-WW2 genocides.
- acituan 6y ago
  Not only the scale but also the multi-agent nature of the problem makes it hard to predict. As an example, humans killing each other is an ancient problem. But humans killing each other at scale with nuclear weapons is a much different problem. We are able to keep that threat at bay by the virtue of only nation states having the power of killing at this scale; number of agents that need to coordinate not defecting (that understand mutually assured destruction) is a handful. Now imagine if some new technology made it possible to make nuclear weapons with materials found in a kitchen. Then we have a coordination problem that requires planet level cooperation with billions of agents. It only takes one defecting to cause disproportionate harm. By the same token, information warfare is not a new problem. Lying is an ancient invention. News outlets being biased makes it easy to guess those biases and somewhat balance them out. We expect states to engage in some degree of propaganda. But human created but AI curated content (i.e. recommendation algorithms) is already driving us a bit nuts; we are having a hard time making a cohorent narrative of what’s going on. AI created and curated content intermixed with human content will completely destroy our sensemaking capacity for the benefit of a few who can deploy this technology. Imagine a per-voter propaganda model that drives AI created social media interactions with them, that can precisely tell what issues would you need to change your mind on to make you change your vote, which could be polar opposite to what the next voter is going to hear. Imagine this was deployed by multiple nations whenever there is an election in a country. This would still at the level of nation-states having nuclear weapons. I can’t imagine what the consequences of a more diffuse access to this tech would look like.