9 ms·

I'm somewhat surprised no one here views this as an automated decisionmaking problem It's increasingly clear that automating important decisions like this is c

• nine_zeros 3y ago
  But the productivity gainz from AI!!! Jokes aside, we are entering a world of AI-decision hell. If it is any similar to the algo-decision hell from the past decade, humans are fucked.
  • advael 3y ago
    Not if we successfully neutralize this paradigm's ability to take over crucial institutions, such as by enforcing lines of responsibility as I've suggested
    • lazide 3y ago
      The hell is guaranteed because no one in the decision making loop cares - yet.
      • o11c 3y ago
        And they will continue to not care, because they will be able to avoid being affected. The racism mention in the article is notable - this isn't prima facie racism (and certainly affects more than minorities), but it is derivative racism since "people who can actually be heard to go through the system in the right way" is race-skewed.
• ToucanLoucan 3y ago
  Wholeheartedly agree. If you don't want to pay customer service people, do not enter or build a business that requires serving customers at scale.
  • BLKNSLVR 3y ago
    The problem is that the likes of Google, Meta, and Amazon are normalising (have normalised?) "scale via automation" and are making bank on an unprecedented scale as a result. If they can do it, so can we. Just like offshoring manufacturing and service centres to places that pay the lowest wages (even though us proles aren't allowed to buy entertainment media from those same places, but that's a separate topic) Race you to the bottom!
    • bruce511 3y ago
      Google, meta and Amazon all have good customer service. In the case of Google and Meta though don't confuse Customer with User. User service is not a thing. Clearly this thread -is- about Customer (not User) Service, so I'm not suggesting that banks are the same. Financial systems have a bot / scam / fraud problem. As others have pointed out, that is the cause of most of the human pain in that space.
      • altfredd 3y ago
        Must be nice when your sole customers are nation-states and 3-letter agencies. Better B2B than B2B (or should it be called something else... N2B — nation-to-business?)
• Sylamore 3y ago
  I don't have Cash App, Venmo, Zello or any of those other micro payment apps because every single time that I sign up for one, I get automatically blocked on my 1st transaction. Zello is even offered directly by my bank so it's automatically linked to my banking info. Calling customer service leads nowhere. It all works out for me in the end because if I can't Apple Pay or Paypal someone I probably don't need their product or service anyway.
  • bandrami 3y ago
    OTOH Paypal is an absolute crapshoot. Something like fifteen years ago they just decided I shouldn't be able to use Paypal anymore. No indication of why. It took them six months to cut me a hard check for the $100 I had in my account at the time. Complete BS system.
    • Sylamore 3y ago
      Yeah, how they can claim to "not be a bank" bu act like one all day long baffles me.
  • deleted 3y ago
    [deleted]
• lamontcg 3y ago
  Yeah we need strong anti-Kafka laws so that individuals can always find out what they've been "charged" with and that there always need to be an "appeal" process and businesses are strictly denied the ability to claim that they can't reveal their methods.
  • jimmydddd 3y ago
    I had my Venmo account frozen. I'd only made about three payments. Two to a food bank and one to a school organization. Via email, they said the decision was permanent and their policy was to not discuss why the decision was made. Luckily for me it was only Venmo, which I don't really need to use.
    • oblio 3y ago
      Similar story with Uber UK. Account banned but just in the UK... After tens of rides across the world.
  • nine_k 3y ago
    Listing the specific problem that led to a service denial could be helpful, but could be not. I bet that it's very rarely something like "You did <this one forbidden thing>, so we ban you according to the rule 3.6.4.2 of ToS". It could be something like: "The pattern of your activity is very similar to patterns of activity of 9823 known scam attacks". Or even more detailed: "8 out of 11 statistical models analyzing 982 parameters of your activity during last 48 days showed the probability of your account being used for illicit fund transfers above 0.92, and the remaining 3, above 0.57. In particular, our 100M-paremeter AI model (link to a PDF with technical details) has classified your activity thusly: (a 20-row table follows). According to the law, we use a publicly available model, widely accepted as a standard in the industry. You can inspect the model's weights at huggingface (link follows)". Interpreting such results is a job. Disputing such results in a court, or with an arbiter, would be really hard, unless you're a practitioner of the pattern-recognition craft yourself.
    • oblio 3y ago
• yterdy 3y ago
  Cory Doctorow has talked about how "they" test these kinds of things on captive "audiences" - mainly children and prisoners - before expanding to the rest of us. I tend to add marginalized groups to that list. Even when a person can nominally be held accountable, it doesn't matter if you can't reach them, protected by an administrative or institutional barrier of some kind. Standardized testing and remote learning monitoring, automated sentencing software, and any number of social programs are examples, if you were wondering. So, part of the problem is the signalling that the tolerating of these practices when they happen to those less able to defend themselves presents to companies. "Go ahead, do it, you'll get away with it." I tend to think: because the people with the power to stop it benefit from it.
  • vorpalhex 3y ago
    I want to dive into the "how to politely resist this in an effective way". 1. Opt out of these things as much as you can. Often times you can't opt out of the underlying actual thing (school, banking, etc) but you can opt out of the terrible new thing. You don't need to explain to people a deep "why". Instead give the people polite, reasonable (and generally true) excuses. "Oh, it doesn't work on my computer". "Oh, I don't understand how to use it." "I don't really like technology." 2. Help other people, particularly children, the elderly, the vulnerable, etc, in opting out of these things and bypassing them when needed. Help them fill out the paperwork, call the right people, cause the right fuss. Frequently, you'll find by helping one person they'll go on to help other people do the same thing. 3. Tell other people, in appropriate times and circumstances, that you don't care for these things and opt out of them. Don't be weirdly obnoxious about it, but when someone gripes about it or complains about it not working for them.. just agree with them. Most people don't like these things. These things don't generally work (for anyone). They suck. Let other people know you agree with them that these things suck. There are cases where this doesn't work - automated account closures, automated sentancing, rent shenanigans. You can of course do the usual things of bugging your congress critters (and you should) but you should also use the full range of pressure available to you. Don't do business with shitty companies - and when they inevitably try to sell to you, tell them why you don't do business with them. When a politician supports automated sentancing tools, explain to them why you aren't voting for them. Surprisingly a little bit of pressure goes quite far.
• gruez 3y ago
  >Coupled with the way automated decisions are used to perform and then launder fraud on a massive scale, maybe we should target laws at the automation itself [...] how about we stop at the actual source by not having the government launder the task of law enforcement to third parties? The article makes it pretty clear that banks aren't exiting customers for fun. They're doing it because they have to, on pain of facing regulatory action.
• deleted 3y ago
  [deleted]
• IG_Semmelweiss 3y ago
  Every app should have a signature that includes the devs that worked on it, the business manager that requested it, and the IT leadership that oversaw its use. This is already done for financial statements. If you put out software that is actively causing harm, you should know your fingerprints are on it Employees like devs would be protected by professional insurance. That would cause bad management to shape up, or otherwise lose inaurance coverage.
  • onionisafruit 3y ago
    I don’t think these decisions are happening in apps.
  • djbusby 3y ago
    Not the devs, they have no agency in the decision. Management, for sure, post their name and LinkedIn profile.
    • ncallaway 3y ago
      > they have no agency in the decision Everyone has agency. “Just following orders” is not an excuse. If someone asks you to implement something that crosses an ethical line for you, you ALWAYS have the agency to refuse. It’s hard, but it’s critical that we never give away our own agency and control of our own actions. You may not be able to stop an implementation from going forward, but you ALWAYS have the ability to not contribute to such an implementation.
      • djbusby 3y ago
        I'm not going the "following orders" mindset. I'm going with many times a junior contributor can't even see the (bad) outcome.
        • ncallaway 3y ago
          That’s totally fair. If someone doesn’t have the visibility into the decision, and isn’t aware of harm coming from their actions, then they don’t have a moral responsibility.
• freddie_mercury 3y ago
  > I'm somewhat surprised no one here views this as an automated decisionmaking problem It isn't an automated decision-making problem. Every instance involves humans in the loop making the final decision. And that article makes clear -- via spokesperson statements -- that it is auditable and the banks know exactly why accounts were terminated.
  • avs733 3y ago
    I know what you are saying but I want to counter the interpretation a bit. What the spokespeople say is a claim…it’s not truth, we don’t have independent evidence of it, of course they would claim that. Even if they did have human in the loop - what does that meaningfully look like? Are those humans given the authority to actually overrule the decision? Are they given the evidence to do so? Are they punished or perceive that they will be punished if they do? I don’t think taking the banks answer to any of those questions at face value is responsible citizenship. Yes it’s what they said - no I don’t think that supports calling it reality. I’ve mentioned this in other threads related to medical billing. My wife and I have had crazy medical bills for bizarre things the last few years. They are never correct, often they are basically fiction. The people you talk to just point to the bill as if it is inherently correct because it’s the bill. I’ve shown pictures of me on vacation on another continent that same day they charged me for a service…well the bills the bill. That’s not human in the loop, it’s a human shield around the automated decision making loop. It’s a fixable problem, make incorrect billing an actual crime. It already is to mid bill the government, just not people.
    • hfx21 3y ago
      You are right and I am speaking as someone who has worked on these systems. It's hard to convey to people, as these systems scale up, the bugs scale up too. And the resources to handle them shrink. Even if the false positives are less than 1% tens of thousands of people can get effected. And things are swept under the carpet, cause the costs to corporations are rounding errors.
    • freddie_mercury 3y ago
      > I don’t think taking the banks answer to any of those questions at face value is responsible citizenship I agree but I wanted to push back on the idea that this was somehow due to automated decisions being made. That isn't really the problem here. The same exact thing would happen if the government mandated "no computers are allowed to be involved". The problem is opaque decision making, no clear escalation path, a disregard for the short term pains, and so on. The fundamental problem isn't how the decision gets made. Even if a human made the decision you'd have the same issues of the front line consumer support saying "well the other department made their decision, I don't know why and I don't have the authority to change it and they don't accept calls". By talking about automated decisions, the OP is making the conversation about the wrong problem.
• A4ET8a8uTh0 3y ago
  I think that is mostly, because no good bank will rely on an automated process for this. Not one bank I have been to had 'derisking'/'divestiture'/'whatever they call it there, because, surprise, this idiocy is not standardized; government just gave companies general idea of what it wants without actual mechanisms' that was ultimately dependent on an automated process. The decision was always human with bankers ( especially commercial ones ) arguing for 'their' customers. Now, I do not know how bad fintech scene is, but I do know that they generally suck when it comes to actual BSA compliance. Then again, I do not really consider fintech banks. Neither does treasury for that matter[1]. Still, this is not THE problem even if you raised a valid and relevant concern here. The issue is and always has been BSA legislation that put this requirement on banks without specifics. More than that, the core issue is that the stupid population that was so scared post-9/11 that it let this thing pass through with minimal whimper. Clearly, it got bad enough that it got some important people and Treasury had to issue a guidance[2]. No one cared for 2 decades. I am sorry for the tone. I am glad this is getting a wider audience, but I do have a bee in my bonnet that it took this long. [1]https://home.treasury.gov/news/press-releases/jy1105 https://home.treasury.gov/news/press-releases/jy1105 [2]https://home.treasury.gov/news/press-releases/jy1438 https://home.treasury.gov/news/press-releases/jy1438
• throw3823423 3y ago
  I've been a part of a system like this (a financial system that happens to not be technically a bank). The system was constantly under attack by fraudsters: You could find subreddits with guides on how to try to outright rob us. So we had teams building detection systems that tied to detect said fraud as early as possible: Hopefully before we handed the fraudsters any money. Depending on how bad the score was, there might not even be a manual review step before we closed the account, because the numbers were that blatant. As with any classification system though, 100% accuracy isn't going to happen. But there's always some customer service rep that can look at the details of the account, and see why in the world the system said what it did. But a detailed explanation of why we thought something was fraudulent could (and sometimes would!) just lead to another fun reddit post where someone describes how to hide the fraud a little better. For any given system like this, how much harm is actually being done, vs how much is being prevented (as fraud just leads to raising prices to cover for it: financial companies are not charities)? I've read way too many CSR conversations where a blatant fraudster with world-class chutzpah would claim that we were destroying their family for no reason, when the data was damning. But this doesn't mean that everyone who isn't a fraudster really reaches out to the CSRs, and has the energy to prove there was no fraud. The actual levels of damage are just hard to measure. We should have sensible, mandatory, available customer service access, which costs just enough to access to not be hammered by bots, but that is completely refunded in case of error. But what is really causing this is that many companies have lowered the barrier of interaction so much that we are letting a lot of fraud through the door. Remember how getting a merchant account in a real bank is a multi-day affair? How getting hired to become a delivery driver needed an interview, with a real person, and a manager checking between deliveries? The price of not having to interact with a human to sign in is fraud detection that isn't a boss you interact with every day, makes sure you are working, and is paid from the work you do. Companies with billions of customers and probably hundreds of millions of suppliers aren't exactly workable without automating a lot of those intermediate jobs away. Maybe we made the wrong call across the board, and lower-productivity, but far higher trust commerce is the way to go... but a lot of that commerce is losing in the market, right now. So if we like it, we have to be willing to pay extra for it.
  • eru 3y ago
    > Maybe we made the wrong call across the board, and lower-productivity, but far higher trust commerce is the way to go... but a lot of that commerce is losing in the market, right now. Plenty of people in the comments here want to enforce that approach via laws and regulation.. > So if we like it, we have to be willing to pay extra for it. .. and I wonder if they are taking this into account.
• lotsofpulp 3y ago
  > I'm somewhat surprised no one here views this as an automated decisionmaking problem Because it is a legislative problem of not ensuring sufficient rights, such as the inalienable right to have access to an electronic money account with which you can receive, send, and keep money.
• deleted 3y ago
  [deleted]
• peterlk 3y ago
  Sounds like a kind of software engineering license. If you have one, you can make big money, but you’re also on the hook for failures. Really increases the incentive to not fuck up these systems. If there was a business decision that caused this, it is up to the engineer(ing team) to be able to audit the system and directly point blame; otherwise, they are liable
• unhammer 3y ago
  See also: - https://aisnakeoil.substack.com/p/the-bait-and-switch-behind-ai-risk https://aisnakeoil.substack.com/p/the-bait-and-switch-behind... - https://predictive-optimization.cs.princeton.edu/ https://predictive-optimization.cs.princeton.edu/ But see https://www.sciencedirect.com/science/article/pii/S0267364922000292 https://www.sciencedirect.com/science/article/pii/S026736492... > human oversight policies provide a false sense of security in > adopting algorithms and enable vendors and agencies to shirk > accountability for algorithmic harms. > propose a shift from human oversight to institutional oversight as > the central mechanism for regulating government algorithms. > First, agencies must justify that it is appropriate to incorporate > an algorithm into decision-making and that any proposed forms of > human oversight are supported by empirical evidence. Second, these > justifications must receive democratic review and approval before > the agency can adopt the algorithm. Also https://predictive-optimization.cs.princeton.edu/ https://predictive-optimization.cs.princeton.edu/