5 ms·

I was just reading through the 'how the app keeps me safe' section. Not trying to be dour, but things like this tend to attract, trolls to put it nicely, if you

I was just reading through the 'how the app keeps me safe' section. Not trying to be dour, but things like this tend to attract, trolls to put it nicely, if your service expands, you're likely to have some fairly nasty people with plenty of time and ingenuity on their side, who see your service as an opportunity to fuck with people, to put it nicely. >We moderate the content and provide easy-to-use reporting tools. This requires exponentially larger numbers of staff as the service grows. >We use spam and bad word filters. Easy to get creative around. It's pretty easy to say fairly awful things without using bad words. >We timeout or ban users who repeatedly or flagrantly break our rules. Easy to circumvent by changing IP'S, or if you ban by IPS, by abusing multiple accounts. >Personally identifying information is banned from any messages. Falls under the above issues. >All new users agree to community expectations. Doesn't stop people who don't care. (Soon) If the app notices that somebody is writing about concerning topics like self-harm, it will privately and proactively offer resources from professionals. Not really applicable to my worries. I like the idea of your service, the internet can always use more things dedicated to just being nice, but I also know human nature and unfortunately, there's people who look at things like that as a challenge to be as horrible as possible cor no real reason other than they like being assholes. Just to make a suggestion, some kind of shadow banning system, similar to hn could be effective here. Having been on the dark side of the shadow bans, they really do keep undesirable things at bay and do tend to be hard to notice for a while. Allowing trolls and assholes to believe they're interacting, while having their posts be invisible to the rest of users could at least help curb repeated account creation and such.

andrewthebold 7y ago
I agree with all your points! One piece that will likely help minimize spammy users and multiple accounts is that at launch, it'll be a one-time purchase app. I do have uncertainties overall, but I'm doing my best to keep the experience clean and safe. If the experience starts to veer towards being a toxic and uncontrollable mess, I'll be quick to shut it down. Happy to talk more about this because I do see the abuse vectors, and don't take this lightly. Edit: on your last paragraph, shadow bans are definitely on the table
- jlokier 7y ago
  (I feel dirty asking this question, btw, but it's the first thing that came to mind when considering the GP's points.) How do you see the system handling someone's answer to "What are you worried about?" being something like "My dad's been raping me for years but I'm worried that if I tell someone he'll get into trouble and we won't be able to carry on paying for my mom's medicine and she will die"? That could either be someone in really difficult circumstances, or someone who is trolling others to upset them - there's no way to know from one message. If you shadow-ban the person in difficult circumstances it's not very kind, but if you let the troll through (with identical words), whoever reads it may have an awful day agonising over how to respond. I wouldn't call something like that toxicity, because it might be genuine distress.
  - andrewthebold 7y ago
    I think there's better scaffolding/explaining to do with content guidelines and expectations, but the general idea is that content that causes concern like this should be reported. A lot of explicit content would probably already be caught by a filter for manual review. It would be taken down and I would send a message to the writer with mental health or counseling resources (ex: crisis text line). I take all messages seriously by default, even if it appears to be trolling. If that person continued to write content like that, I'd do a bit more digging. This is where it'd be subjectively up to me as the moderator to take action. Banning or shadow-banning would be a last resort, generally. There's no strong accountability on my end (beyond people not using the app), but I'm interested in creating a system people can trust. If the app continues to gain traction, I'm thinking about writing formal moderation rules and expectations and seek input from the community. If you have any ideas, I'm eager to listen!
- wolco 7y ago
  Not sure a kindness app is a one time purchase app. If I buy the app I probably want some kindness my way. I would expect feedback.. and be less likely to offer kindness. If you made asking for kindness a paying feature but allowed replies from non-paying members. You have the same spam problem but at least it is limited to replies plus it provides a revenue stream. It creates another problem because people want to give and receive (not just give). The free open model is the best. But it has other problems. Anyone remember spiritweb.org from early the early 2000s? They had a feature where you would ask a question and a random person would get an email asking to answer a question (but the question was not given you had to psychically guess). Kind of a neat idea but in the end it falls apart because of spam and garbage questions. For a spammer being able to send kindness would be a great opportunity.
vegannet 7y ago
I think the key to understanding the scope for abuse is to understand the opportunity. There will always be abusive behaviour and tools are required to tackle it _but_ the majority of abusive behaviour on the internet comes from a desire for attention, and one-to-one abuse is much less common. Abusive people are typically looking for a platform to broadcast. I think that the methods outlined on the website will help but I think the concept is inherently anti-abuse. Context: I worked in community management for many years, dealing with some of the worst of abusive online behaviour.
- ALittleLight 7y ago
  Screenshot the abuse, post to forums, get attention and inspire copycat abusers. I like the idea of the service, and agree with the moderation approach, but I also think trolls are a potential problem.
- anticsapp 7y ago
  I think you are in the right here. If one is trolling an audience of one, it's not really the same dopamine hit as Tweeting a death threat to a Senator that thousands see. It gets boring quick. Also, troll management as a problem is sort of like the "What if I burn $100,000 on AWS in one month?" problem. It's a good problem to have, it means you have traction, and with patience and focus the trolls can be steamrolled.
  - def8cefe 7y ago
    >with patience and focus the trolls can be steamrolled. I agree with your observation that trolls are a problem that comes with scale, but I've never seen an online community solve it. At best you can only slow them down. If anybody has some examples of large online communities that did solve trolling I would be interested to see them.
    - CathedralBorrow 7y ago
      "Solve" is a tall order and I'm not sure it passes that bar, but I've always liked how the Something Awful forums have dealt with trolling and other shitty behaviour: An account costs $10 and bans for trolling are handed out quite aggressively.

andrewthebold 7y ago

I agree with all your points! One piece that will likely help minimize spammy users and multiple accounts is that at launch, it'll be a one-time purchase app. I do have uncertainties overall, but I'm doing my best to keep the experience clean and safe. If the experience starts to veer towards being a toxic and uncontrollable mess, I'll be quick to shut it down. Happy to talk more about this because I do see the abuse vectors, and don't take this lightly. Edit: on your last paragraph, shadow bans are definitely on the table

jlokier 7y ago
(I feel dirty asking this question, btw, but it's the first thing that came to mind when considering the GP's points.) How do you see the system handling someone's answer to "What are you worried about?" being something like "My dad's been raping me for years but I'm worried that if I tell someone he'll get into trouble and we won't be able to carry on paying for my mom's medicine and she will die"? That could either be someone in really difficult circumstances, or someone who is trolling others to upset them - there's no way to know from one message. If you shadow-ban the person in difficult circumstances it's not very kind, but if you let the troll through (with identical words), whoever reads it may have an awful day agonising over how to respond. I wouldn't call something like that toxicity, because it might be genuine distress.
- andrewthebold 7y ago
  I think there's better scaffolding/explaining to do with content guidelines and expectations, but the general idea is that content that causes concern like this should be reported. A lot of explicit content would probably already be caught by a filter for manual review. It would be taken down and I would send a message to the writer with mental health or counseling resources (ex: crisis text line). I take all messages seriously by default, even if it appears to be trolling. If that person continued to write content like that, I'd do a bit more digging. This is where it'd be subjectively up to me as the moderator to take action. Banning or shadow-banning would be a last resort, generally. There's no strong accountability on my end (beyond people not using the app), but I'm interested in creating a system people can trust. If the app continues to gain traction, I'm thinking about writing formal moderation rules and expectations and seek input from the community. If you have any ideas, I'm eager to listen!
wolco 7y ago
Not sure a kindness app is a one time purchase app. If I buy the app I probably want some kindness my way. I would expect feedback.. and be less likely to offer kindness. If you made asking for kindness a paying feature but allowed replies from non-paying members. You have the same spam problem but at least it is limited to replies plus it provides a revenue stream. It creates another problem because people want to give and receive (not just give). The free open model is the best. But it has other problems. Anyone remember spiritweb.org from early the early 2000s? They had a feature where you would ask a question and a random person would get an email asking to answer a question (but the question was not given you had to psychically guess). Kind of a neat idea but in the end it falls apart because of spam and garbage questions. For a spammer being able to send kindness would be a great opportunity.

vegannet 7y ago

I think the key to understanding the scope for abuse is to understand the opportunity. There will always be abusive behaviour and tools are required to tackle it _but_ the majority of abusive behaviour on the internet comes from a desire for attention, and one-to-one abuse is much less common. Abusive people are typically looking for a platform to broadcast. I think that the methods outlined on the website will help but I think the concept is inherently anti-abuse. Context: I worked in community management for many years, dealing with some of the worst of abusive online behaviour.

ALittleLight 7y ago
Screenshot the abuse, post to forums, get attention and inspire copycat abusers. I like the idea of the service, and agree with the moderation approach, but I also think trolls are a potential problem.
anticsapp 7y ago
I think you are in the right here. If one is trolling an audience of one, it's not really the same dopamine hit as Tweeting a death threat to a Senator that thousands see. It gets boring quick. Also, troll management as a problem is sort of like the "What if I burn $100,000 on AWS in one month?" problem. It's a good problem to have, it means you have traction, and with patience and focus the trolls can be steamrolled.
- def8cefe 7y ago
  >with patience and focus the trolls can be steamrolled. I agree with your observation that trolls are a problem that comes with scale, but I've never seen an online community solve it. At best you can only slow them down. If anybody has some examples of large online communities that did solve trolling I would be interested to see them.
  - CathedralBorrow 7y ago
    "Solve" is a tall order and I'm not sure it passes that bar, but I've always liked how the Something Awful forums have dealt with trolling and other shitty behaviour: An account costs $10 and bans for trolling are handed out quite aggressively.