6 ms·

This is an inevitable consequence from the push to make cloud companies responsible for the content they host. Analyzing all of it can only be automated; automa

This is an inevitable consequence from the push to make cloud companies responsible for the content they host. Analyzing all of it can only be automated; automation will deliver weird false positives.

slfnflctd 4y ago
I encountered a false positive where a pure text file made up of personal log entries was flagged by antivirus as malicious, and it started fighting me on trying to restore the file from a backup as well. Fortunately, I was eventually able to recover it, although I've forgotten how. Probably my most memorable lesson in unexpected failure modes for complex, automated software.
bjt2n3904 4y ago
Amazing that the Internet culture that vehemently opposed DRM and created things like bit torrent so quickly lined up behind algorithmic content moderation.
- bil7 4y ago
  fans of bittorrent and fans of services with auto content moderation share at least one common factor: they are free to use. Many people are not motivated much further than keeping their hard earned cash.
- benglish11 4y ago
  I think that internet (that opposed drm) and this one are very different. The anti drm movement was more of a popular movement and this one is a reaction by large corporations to new legal regulations (or threats of more regulations). It’s hard to even think of those as the same “culture”
- fluoridation 4y ago
  Yes, those two sets of people are one and the same.
- TheRealDunkirk 4y ago
  IMHO, the problem started when we all jumped on Gmail, knowing that there was no support, and if it broke, we got to keep both pieces. I watched my wife's account get taken over in real time a couple years ago. Ten years of personal and financial info, now in the hands of God-knows-who for God-knows-why. I simply deleted the account from her devices, and moved on. There was nothing I could do, and no one I could complain to about it. The WILD success of this model has led everyone else to do the same thing.
  - lostgame 4y ago
    >> IMHO, the problem started when we all jumped on Gmail, knowing that there was no support Back when Gmail was invite-only, most people used Hotmail or Yahoo Mail, which would offer a base level of (I think) about 50-100MB of storage at the time. If you wanted more storage; you'd have to pay for your email service - an idea I don't think anyone born past 2000 has even heard of. Gmail came along, and suddenly offers us 1 GB of free storage, for free. This was around a quarter the size of some people's hard drives at this point. Of course we didn't care if there was no support! Are you kidding? I don't think Hotmail had support, either - (maybe I'm wrong?) - even if you paid for it; but here Google was offering a substantial value for anyone using any other existing mainstream email service, immediately. They even had STMP and IMAP support, meaning you could use whatever client you preferred. Back in the day most people used computer-side email clients and had local backups of emails anyway - so if one went missing from Google's side it wouldn't be so bad. There's a reason Gmail is still one of Google's strongest and unusually longest-lasting product of Google's. There's less reason now - but my God, when it was introduced it was an honest-to-God mindfuck as to how they were offering such a large amount of space and features for nothing.
    - koolba 4y ago
      IIRC, Hotmail provided 2 MB when Gmail was released. The 1 GB amount was so large by contrast that people thought it was a joke.
      - renata 4y ago
- rakoo 4y ago
  The Internet culture fighting DRM and sharing of information is decidedly not the same group of people wishing for algorithmic content moderation
- rad_gruchalski 4y ago
  It’s a forced legislation in most places. Flagging is understandable, automatic removal with no way to recover is an awful practice.
Crontab 4y ago
Anything held on anyone else's computer is a candidate for scanning or pilfering. That's why I am a fan of encrypted backups.
temp8964 4y ago
But do they have to delete the flagged account directly? Also, can't they distinguish a spam account from a legit account based on the account history? I can't see why this is a difficult problem to solve.
- malux85 4y ago
  Actual LOL Have you ever tried to fight spam at scale? It’s trivial to block the 50% of obvious spam accounts, but we are at the stage of humanity where the activity of very dumb humans and the activity of very clever bots, EASILY overlap. A lot. Bots will use proxies that use residential IPs, they will maintain session info (Cookies, User-Agents, etc) they will move the mouse and introduce jitter, their access patterns aren’t random but are engineered to work in the day night hours that match the geo-data of the IP they are using. They solve captchas, at the same rate that humans do (time, error rate) Some human accounts look like bots. They sign up and upload a couple of files which immediately get high traffic. They use NordVPN so their public IP is shared by thousands of “known bots”, their access patterns are weird and unpredictable. Yes, you can use machine learning to try and identify theses but then you end up with false positives and real people having problems like the poster above. And on top of all that, bots are constantly subverting detection so whatever solution you have now won’t work next week.
  - temp8964 4y ago
    How many spammers invest months / years to upload and modify content to fake legitimate user accounts?
    - mr_czp 4y ago
      If spammers thought it'd make their spam look legit, all of them would do it!
    - mcguire 4y ago
      All it takes is one to write the code.
ashleyn 4y ago
This chilling effect is already going on. I'm writing a science-fiction thriller and took most of my notes off Google Drive out of fear that they would misconstrue them as political extremism. If AI is being used for content moderation then this only raises the likelihood that a false positive is identified, devoid of any context, and I lose my entire Google account to an uncontactable bureaucracy. I made a Nextcloud to host my references and storywriting notes. Nextcloud is horribly buggy, difficult to maintain, and has very poor user experience compared to Google Drive - but at least I don't need to worry about my entire digital life being auto-terminated by an overzealous robot, with no reasonable appeal process beyond "knowing enough people to make a stink".
- ByThyGrace 4y ago
  So why do you need all your writing notes to be on the cloud? Using multiple devices for work? An editor's request/requirement?
  - em-bee 4y ago
    always keep a copy outside of your home. my own writing is replicated on almost every device i have. it's the most valuable to me. more so even than photos which are just memories.
    - effingwewt 4y ago
      Still zero reason to have on the cloud. You can sync the files between your devices, or even better set up your own home server for close to $0. Why do people always forget the 'cloud' is just someone else's computer. Nothing important should ever exist solely on the 'cloud'. I wouldn't even trust a backup copy sent to email, yet I see that constantly as well.
      - cywick 4y ago
        If you keep your files only on your devices and a home server, you are making yourself vulnerable to scenarios like the one discussed here very recently: https://news.ycombinator.com/item?id=31652650 https://news.ycombinator.com/item?id=31652650 ("I've locked myself out of my digital life")
coolspot 4y ago
EncFS is great to encrypt data in DropBox-like services.
- aborsy 4y ago
  This software had security vulnerabilities. There have been since many apps for design.
raxxorraxor 4y ago
Well, European countries produce "studies" that say hate is effectively reduced and overblocking does not happen. Germany was leading here and the digital services act will copy the crap. No real relevant social networks in Europe and it only applies to platforms like Twitter and Facebook, but it is still silly legislation with negative effects.

slfnflctd 4y ago

I encountered a false positive where a pure text file made up of personal log entries was flagged by antivirus as malicious, and it started fighting me on trying to restore the file from a backup as well. Fortunately, I was eventually able to recover it, although I've forgotten how. Probably my most memorable lesson in unexpected failure modes for complex, automated software.

bjt2n3904 4y ago

Amazing that the Internet culture that vehemently opposed DRM and created things like bit torrent so quickly lined up behind algorithmic content moderation.

bil7 4y ago
fans of bittorrent and fans of services with auto content moderation share at least one common factor: they are free to use. Many people are not motivated much further than keeping their hard earned cash.
benglish11 4y ago
I think that internet (that opposed drm) and this one are very different. The anti drm movement was more of a popular movement and this one is a reaction by large corporations to new legal regulations (or threats of more regulations). It’s hard to even think of those as the same “culture”
fluoridation 4y ago
Yes, those two sets of people are one and the same.
TheRealDunkirk 4y ago
IMHO, the problem started when we all jumped on Gmail, knowing that there was no support, and if it broke, we got to keep both pieces. I watched my wife's account get taken over in real time a couple years ago. Ten years of personal and financial info, now in the hands of God-knows-who for God-knows-why. I simply deleted the account from her devices, and moved on. There was nothing I could do, and no one I could complain to about it. The WILD success of this model has led everyone else to do the same thing.
- lostgame 4y ago
  >> IMHO, the problem started when we all jumped on Gmail, knowing that there was no support Back when Gmail was invite-only, most people used Hotmail or Yahoo Mail, which would offer a base level of (I think) about 50-100MB of storage at the time. If you wanted more storage; you'd have to pay for your email service - an idea I don't think anyone born past 2000 has even heard of. Gmail came along, and suddenly offers us 1 GB of free storage, for free. This was around a quarter the size of some people's hard drives at this point. Of course we didn't care if there was no support! Are you kidding? I don't think Hotmail had support, either - (maybe I'm wrong?) - even if you paid for it; but here Google was offering a substantial value for anyone using any other existing mainstream email service, immediately. They even had STMP and IMAP support, meaning you could use whatever client you preferred. Back in the day most people used computer-side email clients and had local backups of emails anyway - so if one went missing from Google's side it wouldn't be so bad. There's a reason Gmail is still one of Google's strongest and unusually longest-lasting product of Google's. There's less reason now - but my God, when it was introduced it was an honest-to-God mindfuck as to how they were offering such a large amount of space and features for nothing.
  - koolba 4y ago
    IIRC, Hotmail provided 2 MB when Gmail was released. The 1 GB amount was so large by contrast that people thought it was a joke.
    - renata 4y ago
rakoo 4y ago
The Internet culture fighting DRM and sharing of information is decidedly not the same group of people wishing for algorithmic content moderation
rad_gruchalski 4y ago
It’s a forced legislation in most places. Flagging is understandable, automatic removal with no way to recover is an awful practice.

Crontab 4y ago

Anything held on anyone else's computer is a candidate for scanning or pilfering. That's why I am a fan of encrypted backups.

temp8964 4y ago

But do they have to delete the flagged account directly? Also, can't they distinguish a spam account from a legit account based on the account history? I can't see why this is a difficult problem to solve.

malux85 4y ago
Actual LOL Have you ever tried to fight spam at scale? It’s trivial to block the 50% of obvious spam accounts, but we are at the stage of humanity where the activity of very dumb humans and the activity of very clever bots, EASILY overlap. A lot. Bots will use proxies that use residential IPs, they will maintain session info (Cookies, User-Agents, etc) they will move the mouse and introduce jitter, their access patterns aren’t random but are engineered to work in the day night hours that match the geo-data of the IP they are using. They solve captchas, at the same rate that humans do (time, error rate) Some human accounts look like bots. They sign up and upload a couple of files which immediately get high traffic. They use NordVPN so their public IP is shared by thousands of “known bots”, their access patterns are weird and unpredictable. Yes, you can use machine learning to try and identify theses but then you end up with false positives and real people having problems like the poster above. And on top of all that, bots are constantly subverting detection so whatever solution you have now won’t work next week.
- temp8964 4y ago
  How many spammers invest months / years to upload and modify content to fake legitimate user accounts?
  - mr_czp 4y ago
    If spammers thought it'd make their spam look legit, all of them would do it!
  - mcguire 4y ago
    All it takes is one to write the code.

ashleyn 4y ago

This chilling effect is already going on. I'm writing a science-fiction thriller and took most of my notes off Google Drive out of fear that they would misconstrue them as political extremism. If AI is being used for content moderation then this only raises the likelihood that a false positive is identified, devoid of any context, and I lose my entire Google account to an uncontactable bureaucracy. I made a Nextcloud to host my references and storywriting notes. Nextcloud is horribly buggy, difficult to maintain, and has very poor user experience compared to Google Drive - but at least I don't need to worry about my entire digital life being auto-terminated by an overzealous robot, with no reasonable appeal process beyond "knowing enough people to make a stink".

ByThyGrace 4y ago
So why do you need all your writing notes to be on the cloud? Using multiple devices for work? An editor's request/requirement?
- em-bee 4y ago
  always keep a copy outside of your home. my own writing is replicated on almost every device i have. it's the most valuable to me. more so even than photos which are just memories.
  - effingwewt 4y ago
    Still zero reason to have on the cloud. You can sync the files between your devices, or even better set up your own home server for close to $0. Why do people always forget the 'cloud' is just someone else's computer. Nothing important should ever exist solely on the 'cloud'. I wouldn't even trust a backup copy sent to email, yet I see that constantly as well.
    - cywick 4y ago
      If you keep your files only on your devices and a home server, you are making yourself vulnerable to scenarios like the one discussed here very recently: https://news.ycombinator.com/item?id=31652650 https://news.ycombinator.com/item?id=31652650 ("I've locked myself out of my digital life")

coolspot 4y ago

EncFS is great to encrypt data in DropBox-like services.

aborsy 4y ago
This software had security vulnerabilities. There have been since many apps for design.

raxxorraxor 4y ago

Well, European countries produce "studies" that say hate is effectively reduced and overblocking does not happen. Germany was leading here and the digital services act will copy the crap. No real relevant social networks in Europe and it only applies to platforms like Twitter and Facebook, but it is still silly legislation with negative effects.