5 ms·

Content based auto moderation has been shitty since it’s inception. I don’t like that GPT will cause the biggest flood of shit mankind has ever seen, but I am h

by klabb3 4y ago

Content based auto moderation has been shitty since it’s inception. I don’t like that GPT will cause the biggest flood of shit mankind has ever seen, but I am happy that it will kill these flawed ideas about policing. The obvious problem is we don’t have any great alternatives. We have captcha, and we can look at behavior and source data (IP), and of course everyone’s favorite fingerprinting. To make matters worse: abuse, spam and fraud prevention lives in the same security-by-obscurity paradigm that cyber security lived in for decades before “we” collectively gave up on it, and decided that openness is better. People would laugh at you to suggest abuse tech should be open (“you’d just help the spammers”). I tried to find whether academia has taken a stab at these problems but came up pretty much empty handed. Hopefully I’m just bad at searching. I truly don’t get why people aren’t looking at these issues seriously and systematically. In the medium term, I’m worried that we’ll not address the systemic threats, and continue to throw ID checks, heuristics and ML at the wall, enjoying the short lived successes when some classifier works for a month before it’s defeated. The reason this is concerning is that we will be neck deep in crap (think SEO blogspam and recipe sites but for everything) which will be disorienting for long enough to erode a lot of trust that we could really use right now.

coldtea 4y ago
>The obvious problem is we don’t have any great alternatives. There's always identity based network of trust. Several other members vouch for new people to be included.
- eternalban 4y ago
  Maybe even push that a level higher and have org to org vouching as well (so it can scale and reputation propagates social bubbles.) Bootstrapping remains somewhat an issue.
  - wongarsu 4y ago
    One somewhat popular solution for bootstrapping is to allow people to buy in, paired with quickly banning those members in cases of rule violation. It's by no means perfect, but it puts a real price on abuse and thus reduces it a lot
- groestl 4y ago
  I've mentioned a "market of lemons" elsewhere in this thread. One such market is the market for malware and stolen credit card details. One result of the market being broken: serious criminals restrict themselves to very small (company like) social circles and invite only forums. One signal of trust that remained very long: a very short ICQ number. You don't want to burn such a handle with a bad trade, so trust was given upfront.
- wpietri 4y ago
  How would you imagine that applying here? If fake accounts are at least as convincing as real ones, then it seems like trust networks would be quickly prone to corruption as the fake accounts gain enough of a foothold to start recommending each other.
  - coldtea 4y ago
    On a network started by 2-3-10 people, the first new members would need to be vouched by a percentage of those to get in - and so on. If someone down the line does some BS activity, the accounts that vouched for it have their reputation on the line. A whole tree of the person who did the BS and 1-2 layers of vouching above gets put on check, gets big red warning label in their UI presence (e.g. under their avatar/name), and loses privileges. It could even just get immediately deleted. And since I said "identity based", you would need to provide to real world id to get in, on top of others vouching for you. It can be made so you wouldn't be able to get a fake account any easier than you can get a fake passport.
    - wpietri 4y ago
      Are you talking about in-person verification and vouching? Or can it be digitally mediated? If the former, it looks quite impractical unless there are widely trusted bulk verifiers. E.g., state DMVs. If the latter, then it all looks quite prone to corruption once bots become as convincing correspondents as the median person.
      - coldtea 4y ago
        >Are you talking about in-person verification and vouching? Or can it be digitally mediated? Yes and yes. >If the former, it looks quite impractical unless there are widely trusted bulk verifiers. E.g., state DMVs. It's happened already in some cases, e.g.: https://en.wikipedia.org/wiki/Real-name_system https://en.wikipedia.org/wiki/Real-name_system >If the latter, then it all looks quite prone to corruption once bots become as convincing correspondents as the median person How about a requirement to personally know the other person in what hackers in the past called "meatspace"? Just brainstorming here, but for a cohesive forum, even of tens of thousands of people, it shouldn't be that difficult to achieve. For something Facebook / Tweeter scale it would take "bulk verifiers" that are trusted, and where you need to register in person.
lifeisstillgood 4y ago
I am unclear why a reasonable digital ID (probably government ID card style) plus rate limits is not going to be effective. I can see lots of reaosns people might oppose the idea but I am not sure why it's not a widely discussed option? (asking honestly and openly - please don't shout!)
- nprateem 4y ago
  Because the only way it'd work is if it was mandatory (because of point 2); it'd then be extended to porn sites to protect the children. That means politicians browsing history on pornhub would also be recorded and inevitably leaked when they get hacked.
- creakingstairs 4y ago
  Closest example I know of is Korean internet. It is almost nigh impossible to get an account in major websites without SSN and a phone number. Despite this, there are still countless bots and scammers that uses hacked or leaked personal data. So I’m not sure if it would be that effective
  - lifeisstillgood 4y ago
    I am thinking more like webauthn - but where I own a key pair, and I go to post office with my passport, they give me a nonce and prove that my it's my key pair then they post that public key is definitely me. I then can use that posting as my "username" and any challenge response includes the public key so they know that only I could be signing up I am very aware of "designing a security system they themselves cannot break" and the difficulties of key management etc. Would be interested in knowing more from smarter people (probably need to build a poc - one day :-( )
    - bombolo 4y ago
      > I own a key pair Right there… it won't work with the general population.
Andrew_nenakhov 4y ago
> The obvious problem is we don’t have any great alternatives. Of course we do. The rise of digital finance services has led to creation of a number of servives that offer identity verification necessary for KYC. All such services offer APIs, so adding an identity verification requirement to your forum is trivial. Of course, if it isn't obvious, I'm only half joking.

by klabb3 4y ago

coldtea 4y ago

>The obvious problem is we don’t have any great alternatives. There's always identity based network of trust. Several other members vouch for new people to be included.

eternalban 4y ago
Maybe even push that a level higher and have org to org vouching as well (so it can scale and reputation propagates social bubbles.) Bootstrapping remains somewhat an issue.
- wongarsu 4y ago
  One somewhat popular solution for bootstrapping is to allow people to buy in, paired with quickly banning those members in cases of rule violation. It's by no means perfect, but it puts a real price on abuse and thus reduces it a lot
groestl 4y ago
I've mentioned a "market of lemons" elsewhere in this thread. One such market is the market for malware and stolen credit card details. One result of the market being broken: serious criminals restrict themselves to very small (company like) social circles and invite only forums. One signal of trust that remained very long: a very short ICQ number. You don't want to burn such a handle with a bad trade, so trust was given upfront.
wpietri 4y ago
How would you imagine that applying here? If fake accounts are at least as convincing as real ones, then it seems like trust networks would be quickly prone to corruption as the fake accounts gain enough of a foothold to start recommending each other.
- coldtea 4y ago
  On a network started by 2-3-10 people, the first new members would need to be vouched by a percentage of those to get in - and so on. If someone down the line does some BS activity, the accounts that vouched for it have their reputation on the line. A whole tree of the person who did the BS and 1-2 layers of vouching above gets put on check, gets big red warning label in their UI presence (e.g. under their avatar/name), and loses privileges. It could even just get immediately deleted. And since I said "identity based", you would need to provide to real world id to get in, on top of others vouching for you. It can be made so you wouldn't be able to get a fake account any easier than you can get a fake passport.
  - wpietri 4y ago
    Are you talking about in-person verification and vouching? Or can it be digitally mediated? If the former, it looks quite impractical unless there are widely trusted bulk verifiers. E.g., state DMVs. If the latter, then it all looks quite prone to corruption once bots become as convincing correspondents as the median person.
    - coldtea 4y ago
      >Are you talking about in-person verification and vouching? Or can it be digitally mediated? Yes and yes. >If the former, it looks quite impractical unless there are widely trusted bulk verifiers. E.g., state DMVs. It's happened already in some cases, e.g.: https://en.wikipedia.org/wiki/Real-name_system https://en.wikipedia.org/wiki/Real-name_system >If the latter, then it all looks quite prone to corruption once bots become as convincing correspondents as the median person How about a requirement to personally know the other person in what hackers in the past called "meatspace"? Just brainstorming here, but for a cohesive forum, even of tens of thousands of people, it shouldn't be that difficult to achieve. For something Facebook / Tweeter scale it would take "bulk verifiers" that are trusted, and where you need to register in person.

lifeisstillgood 4y ago

I am unclear why a reasonable digital ID (probably government ID card style) plus rate limits is not going to be effective. I can see lots of reaosns people might oppose the idea but I am not sure why it's not a widely discussed option? (asking honestly and openly - please don't shout!)

nprateem 4y ago
Because the only way it'd work is if it was mandatory (because of point 2); it'd then be extended to porn sites to protect the children. That means politicians browsing history on pornhub would also be recorded and inevitably leaked when they get hacked.
creakingstairs 4y ago
Closest example I know of is Korean internet. It is almost nigh impossible to get an account in major websites without SSN and a phone number. Despite this, there are still countless bots and scammers that uses hacked or leaked personal data. So I’m not sure if it would be that effective
- lifeisstillgood 4y ago
  I am thinking more like webauthn - but where I own a key pair, and I go to post office with my passport, they give me a nonce and prove that my it's my key pair then they post that public key is definitely me. I then can use that posting as my "username" and any challenge response includes the public key so they know that only I could be signing up I am very aware of "designing a security system they themselves cannot break" and the difficulties of key management etc. Would be interested in knowing more from smarter people (probably need to build a poc - one day :-( )
  - bombolo 4y ago
    > I own a key pair Right there… it won't work with the general population.

Andrew_nenakhov 4y ago

> The obvious problem is we don’t have any great alternatives. Of course we do. The rise of digital finance services has led to creation of a number of servives that offer identity verification necessary for KYC. All such services offer APIs, so adding an identity verification requirement to your forum is trivial. Of course, if it isn't obvious, I'm only half joking.