5 ms·

There is a difference though. A system implemented by Equifax would presumably look at your social media habits or phone usage and say, "Sorry, we're denying yo

There is a difference though. A system implemented by Equifax would presumably look at your social media habits or phone usage and say, "Sorry, we're denying you a loan because our data indicates that you may default." The Chinese social credit system looks at your social media and phone usage and says, "Sorry, we're denying you a loan because you were critical of the government." It is fair to criticize the prospective Equifax system and the concerns around pervasive data collection are valid. But the Chinese system is in a whole different, far more evil league. > While English-language media describe China’s social credit system as a Black Mirror-like authoritarian implementation, Chinese social media users seem to focus more on the advantages than the burdens. This is a remarkably tone-deaf observation given that talking about the burdens will get them imposed on you.

sjg007 8y ago
US can do the same thing... using the patriot act.
throwaway2048 8y ago
What if politically active people are determined to be at high risk of default?
dingaling 8y ago
> Equifax would presumably look at your social media habits or phone usage and ... and exercise and driving telemetry data and family medical and criminal history and... At a certain threshold the distinction with the Chinese system becomes irrelevant given the impact on an individual.
adrianN 8y ago
Such a system, once in place, is really easy to change to filter for any kind of criteria. One must prevent the infrastructure for large scale surveillance. It's not okay just because you agree with its planned mode of use.
- sgt101 8y ago
  As per the case of the 1930's Dutch census data used by Nazi's to murder Jews. I doubt anyone in the Dutch government imagined such a thing for a moment, and yet - catastrophe and horror. We must be very careful.
- Someone 8y ago
  You may not even need to change it. Fed with enough data, a ML system should be able to derive “We’re denying you a loan because you’re likely to default on it because you’re likely to lose your job because you’re critical of the government”. It likely wouldn’t be able to express that to humans that way, though, so humans would say “We’re denying you a loan because the computer says we should”
Aunche 8y ago
> The Chinese social credit system looks at your social media and phone usage and says, "Sorry, we're denying you a loan because you were critical of the government." That's a common misconception of how Chinese social credit works. China doesn't really care if you're criticizing the government as long as if they don't think of you as a threat. If they do, they'll simply imprison you. If the government is only a little suspicious, they wouldn't want to send you on guard by lowering your credit score.
- jryle70 8y ago
  Other comments about Equifax say "Let's wait until sinister intents kick in, I'm telling you" This one says "Nope, the Chinese system wouldn't use the credit system for non benign reason. They don't care" Different standard?
- door4 8y ago
  > China doesn't really care if you're criticizing the government as long as if they don't think of you as a threat. If they do, they'll simply imprison you. Incidentally, this is also how it works in the United States. Or they'll just come to your apartment and shoot you. https://en.m.wikipedia.org/wiki/Fred_Hampton https://en.m.wikipedia.org/wiki/Fred_Hampton
deleted 8y ago
[deleted]
EGreg 8y ago
Actually China restricts people from boarding planes and trains, if they have a low score.
- deleted 8y ago
  [deleted]
hakfoo 8y ago
> It is fair to criticize the prospective Equifax system and the concerns around pervasive data collection are valid. But the Chinese system is in a whole different, far more evil league. I think calling it outright 'evil' is unfair. The Chinese model at least comes from a well-intentioned place. It's a ham-fisted version of the "nudge" programs you see in places like insurance plans, with a bolder underlying goal to create a better society. Their definition of "better" may not be yours, but the goal is not inherently malicious. The American take of this will have the exact same flaws-- the data will eventually be mishandled, the models will have holes you can drive a bus through-- but it's justified in the sake of "we'll get slightly more accurate pricing for credit and insurance".
- cinquemb 8y ago
  Personally, I find the push to allocating resources based on this kind of data interesting and could possibly be more efficient than the way it has been. That being said, since these systems are abstraction on debt, the failure mode at the margins means that companies that become reliant on funds from these loans will be increasingly sensitive to people at the margins who will be denied such loans. I also find it interesting that the public will also have increasingly more access to the data to be able to retroactively understand the factors that determine the go/no go (especially as these systems become increasingly automated). Combined with the above, it will be increasingly possible to create ways to dirty the waters in the signals for a variety of desired outcomes for 3rd party actors.
sgt101 8y ago
To understand the Chinese system you need to understand where legitimacy comes from in China. The Chinese government employs 50m people, it is a participatory state - everyone will know someone employed by the government. Personal obligations mesh people together to the state, migration and demographic change have disrupted connections and threaten this mechanism of legitimacy - the social credit system could be seen as a reaction and reinforcement of this. I think it is important that people try and see what is going on as it is rather than as a caricature because otherwise the lessons will be lost in the noise, in this case I think that the Equifax / capitalist system is remarkably similar to the Chinese system. Not only are people constrained by the predictions of loan repayment (and by the way that's a can of worms in terms of how & what) but also people are tracked and shamed by their opinions and habits online; and these are wilfully misquoted and misinterpreted to generate social constraints (the recent story about the estate agent who was falsely accused as a bigot and an adulterer : http://www.gizmodo.co.uk/2018/07/when-a-stranger-decides-to-destroy-your-life/ http://www.gizmodo.co.uk/2018/07/when-a-stranger-decides-to-...) My concern is that the truly evil have stealth and pretence on their side, while people who are developing the behaviours and norms that will be necessary for us to cope with the societal challenges we face - for example people who espouse radical anti-consumerism - may be driven out of debate by the norms and norm maintenance mechanisms that are accidentally and with good intention being developed in the west.
craftyguy 8y ago
Since equifax is a US company, the data is only one robed person away from being used as, for example, "you have been deemed a 'national security' risk and so now you cannot enter/leave the country and now require additional attention from some certain 3 letter acronym gov. organization"
lmm 8y ago
How do you imagine such an Equifax system would regard criticism of said Equifax system? Seems like the sort of thing that makes you a default risk, doesn't it.
- TravelAndFood 8y ago
  Maybe? The point of the system is to profit/ensure no default/etc. So if the system determined criticism actually did make you more likely to default, sure. If not, no.
  - Hello71 8y ago
    That may be the stated goal of the system. Any system, however, has an underlying goal of sustaining itself. If that's by accomplishing its stated goal, then great. If it's by suppressing criticism of itself, then that's just as good.
Sir_Cmpwn 8y ago
Equifax: "sorry, we're denying you a loan because an algorithm no one understands said to" They will feed your behavior into a machine learning algorithm, which will probably develop biases.
- Pica_soO 8y ago
  Want to optimize your credits score- we can do that for you- for a share!
- hakfoo 8y ago
  I always wondered: what if redlining worked from a financial perspective? If it turned out that Ethnic Group A is on average 5% more reliable about paying their debt, this is valuable information, but the banks are legally bound to ignore it. So they've probably spent a lot of time trying to produce similar metrics through proxy measurements. Seek out a profession or purchasing pattern that suggests a given ethnicity without outright saying it. I suspect AI is, from a legal CYA perspective, the greatest proxy of them all. "It's a black box, we don't/can't know how it produces its results... yet mysteriously it biases exactly how we would if we could overtly discriminate."

sjg007 8y ago

US can do the same thing... using the patriot act.

throwaway2048 8y ago

What if politically active people are determined to be at high risk of default?

dingaling 8y ago

> Equifax would presumably look at your social media habits or phone usage and ... and exercise and driving telemetry data and family medical and criminal history and... At a certain threshold the distinction with the Chinese system becomes irrelevant given the impact on an individual.

adrianN 8y ago

Such a system, once in place, is really easy to change to filter for any kind of criteria. One must prevent the infrastructure for large scale surveillance. It's not okay just because you agree with its planned mode of use.

sgt101 8y ago
As per the case of the 1930's Dutch census data used by Nazi's to murder Jews. I doubt anyone in the Dutch government imagined such a thing for a moment, and yet - catastrophe and horror. We must be very careful.
Someone 8y ago
You may not even need to change it. Fed with enough data, a ML system should be able to derive “We’re denying you a loan because you’re likely to default on it because you’re likely to lose your job because you’re critical of the government”. It likely wouldn’t be able to express that to humans that way, though, so humans would say “We’re denying you a loan because the computer says we should”

Aunche 8y ago

> The Chinese social credit system looks at your social media and phone usage and says, "Sorry, we're denying you a loan because you were critical of the government." That's a common misconception of how Chinese social credit works. China doesn't really care if you're criticizing the government as long as if they don't think of you as a threat. If they do, they'll simply imprison you. If the government is only a little suspicious, they wouldn't want to send you on guard by lowering your credit score.

jryle70 8y ago
Other comments about Equifax say "Let's wait until sinister intents kick in, I'm telling you" This one says "Nope, the Chinese system wouldn't use the credit system for non benign reason. They don't care" Different standard?
door4 8y ago
> China doesn't really care if you're criticizing the government as long as if they don't think of you as a threat. If they do, they'll simply imprison you. Incidentally, this is also how it works in the United States. Or they'll just come to your apartment and shoot you. https://en.m.wikipedia.org/wiki/Fred_Hampton https://en.m.wikipedia.org/wiki/Fred_Hampton

deleted 8y ago

[deleted]

EGreg 8y ago

Actually China restricts people from boarding planes and trains, if they have a low score.

deleted 8y ago
[deleted]

hakfoo 8y ago

> It is fair to criticize the prospective Equifax system and the concerns around pervasive data collection are valid. But the Chinese system is in a whole different, far more evil league. I think calling it outright 'evil' is unfair. The Chinese model at least comes from a well-intentioned place. It's a ham-fisted version of the "nudge" programs you see in places like insurance plans, with a bolder underlying goal to create a better society. Their definition of "better" may not be yours, but the goal is not inherently malicious. The American take of this will have the exact same flaws-- the data will eventually be mishandled, the models will have holes you can drive a bus through-- but it's justified in the sake of "we'll get slightly more accurate pricing for credit and insurance".

cinquemb 8y ago
Personally, I find the push to allocating resources based on this kind of data interesting and could possibly be more efficient than the way it has been. That being said, since these systems are abstraction on debt, the failure mode at the margins means that companies that become reliant on funds from these loans will be increasingly sensitive to people at the margins who will be denied such loans. I also find it interesting that the public will also have increasingly more access to the data to be able to retroactively understand the factors that determine the go/no go (especially as these systems become increasingly automated). Combined with the above, it will be increasingly possible to create ways to dirty the waters in the signals for a variety of desired outcomes for 3rd party actors.

sgt101 8y ago

To understand the Chinese system you need to understand where legitimacy comes from in China. The Chinese government employs 50m people, it is a participatory state - everyone will know someone employed by the government. Personal obligations mesh people together to the state, migration and demographic change have disrupted connections and threaten this mechanism of legitimacy - the social credit system could be seen as a reaction and reinforcement of this. I think it is important that people try and see what is going on as it is rather than as a caricature because otherwise the lessons will be lost in the noise, in this case I think that the Equifax / capitalist system is remarkably similar to the Chinese system. Not only are people constrained by the predictions of loan repayment (and by the way that's a can of worms in terms of how & what) but also people are tracked and shamed by their opinions and habits online; and these are wilfully misquoted and misinterpreted to generate social constraints (the recent story about the estate agent who was falsely accused as a bigot and an adulterer : http://www.gizmodo.co.uk/2018/07/when-a-stranger-decides-to-destroy-your-life/ http://www.gizmodo.co.uk/2018/07/when-a-stranger-decides-to-...) My concern is that the truly evil have stealth and pretence on their side, while people who are developing the behaviours and norms that will be necessary for us to cope with the societal challenges we face - for example people who espouse radical anti-consumerism - may be driven out of debate by the norms and norm maintenance mechanisms that are accidentally and with good intention being developed in the west.

craftyguy 8y ago

Since equifax is a US company, the data is only one robed person away from being used as, for example, "you have been deemed a 'national security' risk and so now you cannot enter/leave the country and now require additional attention from some certain 3 letter acronym gov. organization"

lmm 8y ago

How do you imagine such an Equifax system would regard criticism of said Equifax system? Seems like the sort of thing that makes you a default risk, doesn't it.

TravelAndFood 8y ago
Maybe? The point of the system is to profit/ensure no default/etc. So if the system determined criticism actually did make you more likely to default, sure. If not, no.
- Hello71 8y ago
  That may be the stated goal of the system. Any system, however, has an underlying goal of sustaining itself. If that's by accomplishing its stated goal, then great. If it's by suppressing criticism of itself, then that's just as good.

Sir_Cmpwn 8y ago

Equifax: "sorry, we're denying you a loan because an algorithm no one understands said to" They will feed your behavior into a machine learning algorithm, which will probably develop biases.

Pica_soO 8y ago
Want to optimize your credits score- we can do that for you- for a share!
hakfoo 8y ago
I always wondered: what if redlining worked from a financial perspective? If it turned out that Ethnic Group A is on average 5% more reliable about paying their debt, this is valuable information, but the banks are legally bound to ignore it. So they've probably spent a lot of time trying to produce similar metrics through proxy measurements. Seek out a profession or purchasing pattern that suggests a given ethnicity without outright saying it. I suspect AI is, from a legal CYA perspective, the greatest proxy of them all. "It's a black box, we don't/can't know how it produces its results... yet mysteriously it biases exactly how we would if we could overtly discriminate."