4 ms·

To clarify, these scores can be sanely used to decide what level of trust you have, and when you have none you get a capcha, a SMS check or something heavier to

To clarify, these scores can be sanely used to decide what level of trust you have, and when you have none you get a capcha, a SMS check or something heavier to authorize the access you are trying to get. In my book you’re never supposed to fully block a session because of the score, there needs to be a (potentially burdensome) way to prove the score wrong. Blocking a browser should be out of question.

klabb3 5y ago
Even so, we still need to have a debate about what levels of "papieren bitte" we are willing to accept for different functions of society. The currently ubiquitous corporate-centric trend is to "nudge" instead of outright ban, i.e. instead of a bool it's numeric, and I highly doubt that the difference in data type is as significant as people think it is -- "Well you can always create a new account/buy another device/ask your friend to do it/.." and so on. If these numeric scores are affecting search results, recommendations, when sharing stuff, etc etc, there's no question that it will affect societal discourse. These hidden "algorithms" (as in the popular term) and fraud prevention systems are so far from being understood that it may be too late to reverse it when we realize what's happened.
- eru 5y ago
  "Papiere, bitte." (Just fyi. Mangling the German [sic] doesn't detract from your post.) Not sure being corporate centric makes this problem any worse? If you had other kinds of organisation, you'd still have to deal with abuse and fraud? Any people doing weird, unusual stuff, look inherently more suspicious. That's a fact of life in meatspace, too. There might be some utopian, ideal way to organise activity so that non-mainstream stuff ain't suspicious. But I would count that as a great feature of that ideal way; not as a deficiency in the corporate way. Just because this deficiency of the corporate way seems to be a common deficiency of most means of organisation that have been tried. (Keep in mind, this isn't all or nothing. Softening the tendency might be enough of a relief, without having to eliminate it completely.)
  - klabb3 5y ago
    > "Papiere, bitte." (Just fyi. Mangling the German [sic] doesn't detract from your post.) Thanks! > Not sure being corporate centric makes this problem any worse? If you had other kinds of organisation, you'd still have to deal with abuse and fraud? I think it's definitely not unique to big corps, but an emergent property of a distributed and homogenous system of self interested agents, probably. It feels very game theoretical, at least. What's clear is these systems are becoming ubiquitous rapidly. Anyway, if my Google account gets locked today, for whatever reason, I am very seriously screwed. Google's human appeal process is best-effort at best, I know people personally who have been locked out for life. Now, I have the luxury to blame myself because I should have bought a domain and so on, but society at large doesn't have that foresight/insight.
    - eru 5y ago
      > I think it's definitely not unique to big corps, but an emergent property of a distributed and homogenous system of self interested agents, probably. It feels very game theoretical, at least. What's clear is these systems are becoming ubiquitous rapidly. Not sure the homogenity is necessary? To an extent, the market delivers what people are demanding. For most people, Google's package of cheap or even free services with minimal hassle in the common case, but almost no recourse in bad cases, is compelling. And for many it's a step up from having everything locally: I'd bet that more people lose their local data than get locked out of Google?
      - klabb3 5y ago
        > Not sure the homogenity is necessary? Oh, nice catch, I actually meant to write heterogeneous. > I'd bet that more people lose their local data than get locked out of Google? Probably. There's definitely some low hanging fruit/middle ground though. We desperately need to have ownership of the address itself, so we can transfer to different providers. Either with your own domain, or a domain provided by a truly neutral party, similar to phone numbers.
        eru 5y ago
- makeitdouble 5y ago
  I agree with you. I think this discussion started around the time shadowbanning was gradually used on forums to reduce moderation workload. It seemed somewhat legitimate for a small free to use site mods to try to not get their whole life sunk by dealing with trolls, but if the same behavior is applied to giant entities who have much more incentives to do it at scale, it becomes a different issue altogether. Hidden restrictions on search results or other functionalities would be distopian and something I hope doesn’t get accepted as standard.
Gigachad 5y ago
Problem is that having an account be breached is catastrophic while getting locked out temporarily ranges from annoying to very annoying but not nearly as bad as having the account be breached. So if the filters have determined that someone is absolutely almost certainly a bot/attacker, it might make sense to do a total block like a bank would lock your account. Unfortunately google doesn't have the support infrastructure like a bank to do recoveries.
tamrix 5y ago
I know cloudflare uses a trick with canvas element to fingerprint the browser. If you disable it, it can be actually impossible to bypass. It's not so much a social credit score to fear rather than privacy. Any site using cloudflare or Google knows where you're going and what your doing and if they don't then, access denied. They know so much more than your innocent mind will want to admit.

klabb3 5y ago

Even so, we still need to have a debate about what levels of "papieren bitte" we are willing to accept for different functions of society. The currently ubiquitous corporate-centric trend is to "nudge" instead of outright ban, i.e. instead of a bool it's numeric, and I highly doubt that the difference in data type is as significant as people think it is -- "Well you can always create a new account/buy another device/ask your friend to do it/.." and so on. If these numeric scores are affecting search results, recommendations, when sharing stuff, etc etc, there's no question that it will affect societal discourse. These hidden "algorithms" (as in the popular term) and fraud prevention systems are so far from being understood that it may be too late to reverse it when we realize what's happened.

eru 5y ago
"Papiere, bitte." (Just fyi. Mangling the German [sic] doesn't detract from your post.) Not sure being corporate centric makes this problem any worse? If you had other kinds of organisation, you'd still have to deal with abuse and fraud? Any people doing weird, unusual stuff, look inherently more suspicious. That's a fact of life in meatspace, too. There might be some utopian, ideal way to organise activity so that non-mainstream stuff ain't suspicious. But I would count that as a great feature of that ideal way; not as a deficiency in the corporate way. Just because this deficiency of the corporate way seems to be a common deficiency of most means of organisation that have been tried. (Keep in mind, this isn't all or nothing. Softening the tendency might be enough of a relief, without having to eliminate it completely.)
- klabb3 5y ago
  > "Papiere, bitte." (Just fyi. Mangling the German [sic] doesn't detract from your post.) Thanks! > Not sure being corporate centric makes this problem any worse? If you had other kinds of organisation, you'd still have to deal with abuse and fraud? I think it's definitely not unique to big corps, but an emergent property of a distributed and homogenous system of self interested agents, probably. It feels very game theoretical, at least. What's clear is these systems are becoming ubiquitous rapidly. Anyway, if my Google account gets locked today, for whatever reason, I am very seriously screwed. Google's human appeal process is best-effort at best, I know people personally who have been locked out for life. Now, I have the luxury to blame myself because I should have bought a domain and so on, but society at large doesn't have that foresight/insight.
  - eru 5y ago
    > I think it's definitely not unique to big corps, but an emergent property of a distributed and homogenous system of self interested agents, probably. It feels very game theoretical, at least. What's clear is these systems are becoming ubiquitous rapidly. Not sure the homogenity is necessary? To an extent, the market delivers what people are demanding. For most people, Google's package of cheap or even free services with minimal hassle in the common case, but almost no recourse in bad cases, is compelling. And for many it's a step up from having everything locally: I'd bet that more people lose their local data than get locked out of Google?
    - klabb3 5y ago
      > Not sure the homogenity is necessary? Oh, nice catch, I actually meant to write heterogeneous. > I'd bet that more people lose their local data than get locked out of Google? Probably. There's definitely some low hanging fruit/middle ground though. We desperately need to have ownership of the address itself, so we can transfer to different providers. Either with your own domain, or a domain provided by a truly neutral party, similar to phone numbers.
      - eru 5y ago
makeitdouble 5y ago
I agree with you. I think this discussion started around the time shadowbanning was gradually used on forums to reduce moderation workload. It seemed somewhat legitimate for a small free to use site mods to try to not get their whole life sunk by dealing with trolls, but if the same behavior is applied to giant entities who have much more incentives to do it at scale, it becomes a different issue altogether. Hidden restrictions on search results or other functionalities would be distopian and something I hope doesn’t get accepted as standard.

Gigachad 5y ago

Problem is that having an account be breached is catastrophic while getting locked out temporarily ranges from annoying to very annoying but not nearly as bad as having the account be breached. So if the filters have determined that someone is absolutely almost certainly a bot/attacker, it might make sense to do a total block like a bank would lock your account. Unfortunately google doesn't have the support infrastructure like a bank to do recoveries.

tamrix 5y ago

I know cloudflare uses a trick with canvas element to fingerprint the browser. If you disable it, it can be actually impossible to bypass. It's not so much a social credit score to fear rather than privacy. Any site using cloudflare or Google knows where you're going and what your doing and if they don't then, access denied. They know so much more than your innocent mind will want to admit.