14 ms·
Viral Tweet About Apple Card Leads to Goldman Sachs Probe
- im3w1l 7y agoThis is a problem that needs to be solved, no question about it.
- minimaxir 7y agoThe original Twitter thread is still being updated, and it's a doozy. This is not an isolated incident. https://twitter.com/dhh/status/1193240508845510656 https://twitter.com/dhh/status/1193240508845510656 https://twitter.com/dhh/status/1193242909111398401 https://twitter.com/dhh/status/1193242909111398401
- DoreenMichele 7y agoI haven't read the whole thread, but I am seeing him bitch about her credit score being higher -- from a single agency. There are other agencies which may have different info. I know nothing about this particular credit card. I'm not as up on credit stuff as I once was (and the world has changed a lot since then). But when I was a homemaker, I had a credit card in my name with a much higher limit than my husband had on any of his cards. That's not exactly the norm. There are various factors that go into this. He's not wrong to suggest that there is a very big problem with employees having no idea what went wrong. I'm less confident that it is reasonable to infer gender is the entire explanation.
- dannyr 7y ago"I haven't read the whole thread," Seriously, please read before commenting! The thread is not that long.
- w4 7y ago> I haven't read the whole thread, but I am seeing him bitch about her credit score being higher -- from a single agency. There are other agencies which may have different info. If you read the thread, he was specifically told the Apple Card uses Transunion, which is why they checked with Transunion, and found her score was higher on that report: https://twitter.com/dhh/status/1192945415538106369?s=21 https://twitter.com/dhh/status/1192945415538106369?s=21
- DoreenMichele 7y agoHe was also repeatedly told "It's the algorithm, man! And I have no clue what's in it!" I worked for a Fortune 500 company at one time. Lots of entry level employees were not exactly reliable sources of info about how decisions got made there. You may or may not be talking with an entry level employee in their call center, but you probably aren't talking to a departmental head or member of the C suite.
- heartbreak 7y agoLet’s assume that the tweet author is reasonably well-versed in business operations, since he is himself a very successful businessman.
- DoreenMichele 7y agoSure. Let's assume I am very well versed in other pertinent domains of knowledge, like social psychology and the tendency for people to get mad as hell about social justice issues and leap to ugly conclusions that fit their SJW narrative about evil in the world that can be conveniently lumped under a one word heading, like sexism. Let's further assume that this is actually actively counterproductive, so it's reasonable to point out that correlation is not causation and it's unhelpful to insist on a particular conclusion you cannot prove. I already noted he's right to be outraged at the situation and critical of the black box nature of the decision. I'm just not comfortable with him ranting that it's clearly and obviously due to sexism.
- kennywinker 7y agoThere a many people chiming in that they have had the same experience, and nobody saying they had the opposite. If the credit limit difference was unrelated to sex there would be random distribution of couples with the inverse experience. So you sayyyy that he flew off the handle because he's an SJW or whatever, but because his initial assumption continues to be proven correct as more data is accumulated... it seems like you are wrong that it was an overreaction.
- sparkling 7y agoJikes. I appreciate DHH for his open source contributions, but this is straight up embarrassing. The sample size for his claims here is exactly 1. There are probably hundreds of data points that go into the algo that determines the initial credit limit. It could be literally anything, from the number of existing CCs in her name, her credit history, whatever. Stop crying, stop bitching, grow up.
- derision 7y agoExactly. Since it's a community property state, maybe they took into consideration the existing credit limit they already extended to the household.
- notadoc 7y agoSocial media anecdotes! Oh my!
- mc32 7y agoMaybe these CC issuers should just take gender out of the equation and solely base their credit extension on economic and behavioral data points.
- abcpassword 7y agoThis is bad science though. If an overwhelming percentage of people with attribute <X> have behavior <Y>, only a fool would ignore that data.
- matthewmacleod 7y agoThat would be the obvious first step and an absolute minimal requirement. The problem is that as these systems get more black-box-y there is little to stop them making predictions that are affected by inferred hidden characteristics. Like, “we don’t take race or ethnicity into account, but our insurance algorithm says that if you have black hair and brown eyes and a name commonly found in black communities and you live in an area with a large black population then you are more of a risk sorry but it’s nothing to do with race honest”.
- MBCook 7y agoThe chances there is a line in the algorithm that says ‘women get less’ is basically 0%. It’s got to be inferences like you said. And those are so much harder to spot.
- pacala 7y agoTo the extent there exists a correlation between some characteristic and the outcome, models that exhibit the correlation will be more accurate than models that don't, irrespective whether the characteristic is used as a model feature or not. In the context of risk assessment, "more accurate" means "more likely to survive in the market place".
- kyrieeschaton 7y agoImplying they don't.
- markdown 7y ago> David Heinemeier Hansson Is there a special reason this bloke gets his full name spelled out in the article? The same publication doesn't even use the full name of Donald Trump or Michael Bloomberg in its articles. EDIT: Received lots of great answers, thanks. I'm also getting punished for asking because apparently I'm supposed to be able to tell just by looking that John Trump is a middle and last name, but Heinemeier Hansson is just a last name.
- ciabattabread 7y agoBecause he chooses to go by his full name professionally? Edit: or that his last name is two names.
- raldi 7y agoHeinemeier Hansson is his two-word last name. It's customary to include a person's first name and full last time in an article. See also: Eddie van Halen
- krig 7y agoIt's a Scandinavian name. His last name has two parts. David is his first name, Heinemeier Hansson is his last name. That's just how it works.
- markdown 7y agoThanks, that's helpful.
- slg 7y ago>“Any algorithm, that intentionally or not results in discriminatory treatment of women or any other protected class of people violates New York [and federal] law.” I worry about this a lot with the growing importance of algorithms and machine learning. You can't just not actively program the thing to discriminate and assume that is enough. You have to specifically program it to not discriminate.
- deleted 7y ago[deleted]
- taway87 7y agoSo, "credit realism", to match "IQ realism"? I see this a lot, and the background assumption seems to be "in the real world, minority group A is actually riskier, dumber, or objectively worse in some other way, so in order to comply with anti-discrimination, we have to introduce special cases." Maybe instead we should start with the assumption that women are NOT riskier, dumber, or objectively worse, and fix the likely bug, instead?
- MaupitiBlue 7y agoBecause the goal was to be realistic, not to assuage white guilt.
- justapassenger 7y agoImplementation details of law and enforcement matter a lot, but in general, I agree with a spirit of it. As world depends more and more on algorithms, we need to have more security around them. Especially as algorithms are used a lot as cost cutting, so reaching competent human to appeal error is becoming harder and harder. If we don’t put more scrutiny around tech, we may end up living in kafkaesque world.
- deleted 7y ago[deleted]
- daenz 7y agoAll uneven distributions must be weighted accordingly to emulate a uniform distribution across classes. That's apparently what some people want.
- dghughes 7y agoHopefully anything linked to gender is looked into for example car insurance? Men typically signifigantly more than women for car insurance.
- zrail 7y agoCar insurers can point at actuarial data to back up their rating decisions.
- xenihn 7y agoAre you saying that there are insurers who don't base their rating decisions on actuarial data? Isn't that a requirement?
- zrail 7y agoParent comment said car insurer.
- ofcourseianal 7y agoMaybe GS can too?
- zrail 7y agoMaybe! Nobody knows!
- slg 7y agoCharging different rates on car insurance based on gender is illegal in some places, including California.
- sdinsn 7y agoBut there is still a discrepancy between genders in CA.
- function_seven 7y agoLet's assume that there is some valid reason for the difference in credit limits. (I'm not assuming one way or the other, but let's just grant it here.) This is still bad for Apple and for consumers. The "ALGORITHM" that can't be questioned, inspected, explained, or overruled is a massive failure. Whether its criminal justice, credit scores, or behavioral predictions, ceding authority to some "AI" overlord can't end in just or fair outcomes. (Scare quotes around "AI", because the black box may just be a chain of if/then statements that conveniently proxy in the worst of our institutional biases. Or it could be sophisticated ML... proxying those same things.) I don't think we're done hearing about this. I'll be very interested in what Apple has to say about it, or what—if anything—is discovered. And it'll doubleplusungood if gender is an explicit input into the "ALGORITHM". DHH is absolutely right to push back against all the respondents that offer plausible explanations. They're all missing the point. The point is that Apple should be providing a concrete explanation for their credit decisions. All credit providers should.
- matthewmacleod 7y agoYep. This is part of why GDPR is so important - the “right to explanation” for automated decision-making processes.
- rndgermandude 7y agoOther than him being very outraged and claiming gender-based discrimination again and again, is there actually anything to suggest the algorithm is biased on the gender axis, as opposed to any other axis? Just to be clear: I am not dismissing the possibility that the algorithm (meaning the training data, really) is gender biased, it just isn't clear from what I have seen in the tweet storm that this is necessarily the case. E.g. I had an ex girlfriend who had a slightly lower income and slightly worse credit score (well, German Schufa score) than me, and yet she got offered like twice as credit when she applied for a card. I am guessing (just guessing) that this was due to her having paid off e.g. a car loan in the past and being generally more consumerist than me, while I never had taken out any major loans.
- WA 7y agoThe whole thread is less about his wife having a lower credit limit, and more about Apple reps not understanding the fucking algorithm. The axis doesn’t really matter. Maybe it’s gender, maybe it’s not. In any case, it should be about money, credit score, trustworthiness. Apparently, it’s not. Being a women could be one indicator. It might be "being black" for other people. There seems to be something fishy going on and Apple doesn’t even know why. Hence, there is a bias.
- dkonofalski 7y ago>There seems to be something fishy going on and Apple doesn’t even know why That seems a little silly, though, since Goldman Sachs is the bank in charge of the card and approval process. Just like Amazon doesn't know anything about the processes that are used to determine credit worthiness for their cards (that falls to Chase and Synchrony), Apple is merely licensing services from GS. On top of that, GS representatives that work in a call center aren't going to know anything about the details or inner workings of the process either. With the amount of potential abuse for such information, it feels like it would be irresponsible for those people to have access to that info. The bigger issue is that this couple had a valid complaint and had to turn to Twitter to resolve it instead of GS having some formal method of requesting the info.
- jacquesm 7y agoGood for him. Of course if he had been a nobody this would never have gotten traction but typically DHH is found on the right side of issues like these and I commend him for speaking out. This is likely one of very many such instances and let's hope that insurance companies, banks and other financial service providers take note.
- rhexs 7y agoSpeaking out about what? The guy appears to be claiming that a company is discriminating against women with no evidence aside from a single anecdote. Seems like he's some sort of rich 1%er founder, probably not approved under any sort of standard model the typical applicant will fit into, and thus has a crazy high limit. Is there anything more to this than typical twitter mock indignation and outrage? Now, if his wife was enormously wealthy before they met and this is the outcome, I suppose that would raise some eyebrows!
- heartbreak 7y agoAs explained in the very first tweet in the thread, DHH’s wife is exactly as wealthy as DHH. That’s part of the frustration expressed.
- rhexs 7y agoSo, if that's the concern, Europeans are free to pass regulation that requires credit companies to extend the same credit line to husband and wife. Naturally, that will potentially have unforeseen consequences with regard to modeling risk, buyer beware. Seems like my initial comment was on point then.
- themgt 7y agoI would take DHH's interpretation of this with a large grain of salt. He seems to assume that living in a "community property state" will be taken into account (presumably then considering his wife to have parity with his income/score) by the credit limit algorithm, but we have no way to know that that's the case. He also adds "It gets even worse. Even when she pays off her ridiculously low limit in full, the card won’t approve any spending until the next billing period. Women apparently aren’t good credit risks even when they pay off the fucking balance in advance and in full." Are we really to imagine the Apple/Goldman algorithm has some "if(gender.female){ cc_payment_terms = :discrimination }" sort of code in it? FWIW I'm a male and have a mid-700s credit score and was denied Apple card approval. I am fairly certain there was no sex discrimination involved in the denial.
- rriepe 7y agoIf they're using machine learning, then yes it may literally have code like that in it.
- minimaxir 7y ago> Are we really to imagine the Apple/Goldman algorithm has some "if(gender.female){ cc_payment_terms = :discrimination }" sort of code in it? Not intentionally, but it's possible for black box AI algorithms to engineer such a model feature.
- jacquesm 7y agoIt won't have that line in their code. What it very likely will have is some Bayesian algo or maybe if they're really high tech some ML black box that will have gender as one of its inputs. And it probably shouldn't have that input. The whole idea behind decisions like these is that they have to be explainable and ML especially does not lend itself well for that.
- matthewmacleod 7y agoNote that an ML box doesn’t even need gender as an input. Name alone probably gets you 90% of the way there.
- tootie 7y agoOne data point really isn't enough to draw a conclusion and I actually doubt it's true. GS is extremely scrupulous when it comes to avoiding liability and they probably put a ton of diligence into their risk model. It's possible there's some emergent gender bias but we'd need a better show of proof.
- xenihn 7y agoWhat we really need is the ability to force organizations to expose their models.
- toomuchtodo 7y agoTo regulators, sure. Entirely reasonable. To your average Joe? No. This is already done to prevent redlining when originating mortgages. Disclaimer: Work in financial services in risk management, interface with regulators. Opinions are my own.
- xenihn 7y agoWhat's your argument for opposing full exposure, other than the potential impact on your personal livelihood?
- toomuchtodo 7y agoThe model is not your property. Do you plan on releasing all of your company’s code on Github gratis? Unlikely. As long as the model doesn’t violate the law, you have no right to it, unless you’re a regulator. Regulators and oversight bodies get full (audited and logged) access. And while it’s none of your business, I don’t need my job, or a job at all. I work for projects I enjoy.
- xenihn 7y agoIf you're really financially independent, that's even more incentive for you to support exposure and transparency. People are being harmed by these models. You would lose nothing, and others would benefit. >As long as the model doesn’t violate the law, you have no right to it, unless you’re a regulator. If you have no realistic alternative to being judged by a model, you absolutely should have a right to it. Maybe not the complete implementation, but a verified summary that could be understood by the average person.
- man-o-man 7y agoFirst world problems, who cares. Oh no, my credit limit is $50,000 but my wife's is only $5,000! Fuck off.
- deleted 7y ago[deleted]
- bobobob420 7y agoHighly doubt Goldman Sachs included gender discrimination in their risk model...do regulators get to see the risk models credit companies use to determine how much credit one can use?
- tempsy 7y agoIt's well known there are proxies are you can't use when building a credit risk model e.g. zip codes are off limits because it can be used to a proxy for ethnicity. Intentional or not, it's possible they could have used something that proxied for gender.
- Simulacra 7y agoThis guy sounds like he’s really making a huge tin foil hat.
- donjp 7y agoOk, let's try to put it all into context. Denmark. one of the most egalitarian countries in the world. 1% of the 1%. I live here and aren't a Dane, their vision of the world can be quite bubblelike and skewed. Im almost sure that he's jumping into conclusions.
- esotericn 7y agoIt strikes me that longer term, we're just going to end up playing a cat and mouse game with people continually inventing different categories that businesses aren't permitted to 'discriminate' on. Today it might be gender and race. That makes a lot of sense, because the alternative is to further entrench what are basically inheritances. But aren't we just going to rattle on through and have the algorithms discover (whether we actually realise it or not) that, say, someone diagnosed with X is less creditworthy than someone diagnosed with Y, or that someone bullied in school is less creditworthy, or whatever else? The whole point of ML is to extract this sort of information from a dataset. Is it even possible or meaningful to create an unbiased model? Doesn't a model's profit imply bias, whether we currently consider it morally correct or not? I'd be interested in an argument to convince me otherwise. My view at the moment is basically 'we spend all of this time building models, and then we have to stop using them because they're socially negative/immoral, but for a brief period shareholder value was maximised'?
- matthewmacleod 7y agoThat’s not a great argument though - essentially “we can’t prevent an ML model from being biased so we should embrace it.” Western society has mostly accepted the idea that—outside of some specific cases—we should avoid building systems and processes that systematically discriminate against people on the basis of a selection of characteristics which have historically attracted it. The exact application of this concept, the interpretation of it, and the boundaries of discrimination or protected characteristics will continue to be subject to gray areas and refinement. The rules are not perfect. But I do think a better solution to the problem (“we have implemented a whizz-bang new technology which is inherently subject to bias”) is to either fix or discard that technology, rather than discarding the concepts of equalities regulation and civil rights.
- esotericn 7y agoI think you've mistaken me. My argument is that if we take the standpoint that we don't just want the metric to be whatever is short-term economically optimal for the designer of the model, we should just stop/ban it now, because we already know that we're going to have to kill it once we actually understand it properly. It's only allowed now because we haven't figured out the bad things that are happening. Inventing more and more categories that businesses are not permitted to discriminate upon is precisely the wrong approach - if we're talking about huge companies and not the bakery down the road, we need something that's more like 'you need a very good reason to exclude someone', rather than 'you can exclude someone for whatever reason they like, unless they're a member of the set of continually extending list of protected categories'.
- kyrieeschaton 7y agoI guess it worked for him to throw a hissy fit, but there's absolutely no reason to actually believe they're conditioning on gender. It could be "it's a new product, we randomly assign credit limits to see how it affects behavior". It could be "it's a community property state and we're overexposed to this household if we give the second card the same limit as the first". It could, realistically, be almost anything except for evil bankers deciding to use illegal criteria to underwrite that has a side effect of limiting the amount that can be charged to the account each month (you know, how they actually make money). Oh, random CSRs don't get a pithy explanation of a multivariate nonlinear underwriting decision to poorly convey to customers? That's kind of precedented!
- tenpies 7y agoI mean you know the system is sexist when one gender out-lives the other by a considerable number of years yet the retirement age for both is the same.
- elvinyung 7y agoI mean, don't you think it's kind of a strawman to assume anyone thinks machines are explicitly sexist? It seems much more likely that whatever method they use to do feature selection on this obviously high-dimensional data just happens to end up picking something that was a proxy for gender. You can be implicitly biased without being aware of it. This is true for both humans and algorithms.
- kyrieeschaton 7y agoIt is absolutely not a strawman. It is incredibly easy to find people granted authority by the state that will claim that, eg, mathematics per se is an instance of exclusionist masculine thinking. It's even easier to find instances where people granted authority by the state will claim that any discrepancy in outcomes is ipso facto intentional.
- gwd 7y ago> It seems much more likely that whatever method they use to do feature selection on this obviously high-dimensional data just happens to end up picking something that was a proxy for gender. Yes, this is exactly what everyone thinks happened. Nobody thinks people at Goldman-Sachs wrote if (applicant.gender == "F") limit /= 20 somewhere in their algorithm. But regardless of how the thing happened, if millions of people are treated significantly differently for no reason other than their plumbing, that's a major problem. People have been talking about this "accidental proxy for gender" for years now; there's absolutely no excuse for no doing a basic sanity check to make sure that this kind of thing isn't happening. edit: typo
- kchoudhu 7y agoWhat I don't see anywhere in that thread is how much his wife put in as her income when applying.
- purple_ducks 7y agoThe response to codinghorror's "then don't use the Apple Card? Solution seems obvious" is on point: > This is such a shallow, disappointing take. If we relegate all responsibility for discrimination to the individuals discriminated against, nothing is going to change! Individual action against structural problems is INSUFFICIENT.
- scrollaway 7y agoThe difference between a solution and a workaround. "Not using the product" is a workaround. A solution addresses the root of the problem, and the root of the product's problem is seldom "The consumer bought the product".
- zamalek 7y agoIn theory it corrects the problem too, as Apple would lose customers. In practice, Apple has customers no matter how badly they screw up.
- deleted 7y ago[deleted]
- r99g7 7y agoAnd my wife's Apple Card limit is 20x higher than mine. Are they're discriminating against men too‽ If I was as rich as DHH, who makes millions per year, I would cancel my credit cards and refuse to do business with companies like Goldman Sachs entirely. I wouldn't give a crap about 2% cashback and I don't know why he does. And despite the fact that he's probably wrong about the entire complaint, the least he could do is cancel his cards in protest. Instead, he seems to have accepted the "bribe" (his word) quite willingly.
- tempsy 7y agoWhat is it about this story that has triggered so many people (men) on here? Dear lord.
- MaupitiBlue 7y agoThe whining and expectation that big brother swoop in and make everything right.
- joemi 7y agoIts probably DHH's jumping to the conclusion that gender bias (whether intentional or not) is the only possible explanation for this credit discrepancy. Or it could be the fact that he expects customer support representatives to understand the inner workings of GS's risk calculations.
- shuckles 7y agoCredit lines are given based on stated income, not on credit score. A common problem in this industry is that non-working partners state their personal income instead of household income. Even if an applicant is approved due to strength of credit, the bank won’t underwrite a large line for low stated income.
- paggle 7y agoThere could be a manual process here - someone knows that DHH has a >$1 million car and manually edits his credit limit to super high.
- judge2020 7y agoThere are two arguments here: A. "is the credit limit algorithm sexist" and B. "we should be able to know how an algorithm decides things and/or we should be able to see the algorithm". You can't determine A, simple as that. The only way would be if Apple/GS comes out and says something like "he capped out the household credit limit", which Apple/GS would probably only tell him anyways. B is a different issue, one with lots of room to actually converse over. But it's one which the author seems to pivot to after some reasonable arguments are presented as to why her credit limit was literally $57, making his argument for A very weak.
- szczepano 7y agoCan you get insurance from algorithm damage ? I see a niche here.
- mikestew 7y agoSince we are slinging anecdotes around, starting with TFA, my wife and I file jointly, have what I believe to be a good credit score. I filled out what little form there is for both of our cards. Put down the same income, etc. I think our credit limits are the same (need her phone to verify), but she got 13% and I got 18% APR. Now how the hell is there a five percent difference? Not that it matters because they both get paid off, but WTF? (And 18%; seriously, GS?) But on topic, she got the considerably better interest rate.
- deleted 7y ago[deleted]
- tareqak 7y agoSame story from the Associated Press: “NY regulator vows to investigate Apple Card for sex bias” https://apnews.com/8754cf30526b4b94a3ba6e1cfc1d5054 https://apnews.com/8754cf30526b4b94a3ba6e1cfc1d5054
- neonate 7y agohttp://archive.is/fo7My http://archive.is/fo7My
- Simulacra 7y agoI think he’s making a lot out of nothing, there’s no way really to prove this, and we only have he and his wife as a sample size.
- brandonmenc 7y agoCredit card company extends known rich guy tons of credit - news at 11! Every credit card company probably maintains a list of "big fish" - i.e. famous people - and grants them all a huge credit line. It's a non-story.