4 ms·

There is a real security problem here and it is insidiously dangerous. Some prominent academics are stating that this type of thing is creating real civil and

There is a real security problem here and it is insidiously dangerous. Some prominent academics are stating that this type of thing is creating real civil and geopolitical implications that are generally responsible for the global rise of authoritarianism. In security, when a company has a vulnerability, this community generally considers it both ethical and appropriate to practice responsible disclosure where a company is warned of a vulnerability and given a period to fix it before their vulnerability is published with a strong implication that bad actors would then be free to abuse it after it is published. This creates a strong incentive for the company to spend resources that they otherwise have no desire to spend on security. I think there is potentially real value in an organization effectively using "force," in a very similar way to this to get these platforms to spend resources preventing abuse by posting AI generated content and then publishing the content they succeeded in posting 2 weeks later. Practically, what I think we will see is the end of anonymization for public discourse on the internet, I don't think there is any way to protect against AI generated content other than to use stronger forms of authentication/provenance. Perhaps vouching systems could be used to create social graphs that could turn any one account determined to be creating AI generated content into contagion for any others in it's circle of trust. That clearly weakens anonymity, but doesn't abandon it entirely.

ethersteeds 1y ago
> I don't think there is any way to protect against AI generated content other than to use stronger forms of authentication/provenance. Is that even enough though? Just like mobile apps today resell the the legitimacy of residential IP addresses, there's always going to be people willing to let bots post under their government-ID-validared internet persona for easy money. I really don't know what the fix is. It is Pandora's box.
- janalsncm 1y ago
  No system is foolproof. The purpose is to add enough friction that it’s pretty inconvenient to do. In the example in OP, these are university researchers who are probably unlikely to go to the measures you mention.
janalsncm 1y ago
There’s no way to prevent these things categorically but they can be made harder. A few ways (some more heavy handed than others and not always appropriate): Requiring a verified email address. Requiring a verified phone number. Requiring a verified credit card. Charging a nominal membership fee (e.g. $1/month) which makes scaling up operations expensive. Requiring a verified ID (not tied to the account, but can prevent duplicates). In small forums, reputation matters. But it’s not scalable. Limiting the size of groups to ~100 members might work, with memberships by invite only.
hashstring 1y ago
> This creates a strong incentive for the company to spend resources that they otherwise have no desire to spend on security Sometimes though, “Responsible Disclosure” or CVD is creating an incentive to silence security issues and long lead times for fixes. Going public fast is arguably more sustainable in the long run as it forces companies and clients to really get their shit together.

ethersteeds 1y ago

> I don't think there is any way to protect against AI generated content other than to use stronger forms of authentication/provenance. Is that even enough though? Just like mobile apps today resell the the legitimacy of residential IP addresses, there's always going to be people willing to let bots post under their government-ID-validared internet persona for easy money. I really don't know what the fix is. It is Pandora's box.

janalsncm 1y ago
No system is foolproof. The purpose is to add enough friction that it’s pretty inconvenient to do. In the example in OP, these are university researchers who are probably unlikely to go to the measures you mention.

janalsncm 1y ago

There’s no way to prevent these things categorically but they can be made harder. A few ways (some more heavy handed than others and not always appropriate): Requiring a verified email address. Requiring a verified phone number. Requiring a verified credit card. Charging a nominal membership fee (e.g. $1/month) which makes scaling up operations expensive. Requiring a verified ID (not tied to the account, but can prevent duplicates). In small forums, reputation matters. But it’s not scalable. Limiting the size of groups to ~100 members might work, with memberships by invite only.

hashstring 1y ago

> This creates a strong incentive for the company to spend resources that they otherwise have no desire to spend on security Sometimes though, “Responsible Disclosure” or CVD is creating an incentive to silence security issues and long lead times for fixes. Going public fast is arguably more sustainable in the long run as it forces companies and clients to really get their shit together.