11 ms·

We need legislation forcing companies to manually review algorithmic decisions that impact people's lives - there has to be a proper appeals process. It's simp

• Symbiote 5y ago
  GDPR Art. 22: https://gdpr-info.eu/art-22-gdpr/ https://gdpr-info.eu/art-22-gdpr/ But that's only part of it. A business account might not fall within these rules, and Google can enforce them anyway (after human review) with very little recourse available. Breaking the App store duopoly is the solution.
  • Sebb767 5y ago
    > Breaking the App store duopoly is the solution. Not really. We a comparable situation in a lot of different industries; if a large companies serves tens of thousands of customers and get hundreds of thousands of malicious requests, they're going to do some filtering and it will lead to some people getting stuck in the cracks, even if it's just 0.01%. More app stores won't solve this; if enough developers would be burned, the problem would solve itself on its own. That's not to say that I like the current situation, but from what I see it's likely that the actual number of incidents is low (compared to the number of Google's customers) and a third or fourth store won't change this.
  • sschueller 5y ago
    It's coming but from the EU.
  • davidkuennen 5y ago
    Every day I'm more thankful to live in the EU.
  • merrywhether 5y ago
    There are alt stores on Android though, it just seemingly doesn’t matter. The root problem is that consumers don’t care about this problem and voted with their wallets for the current situation. That’s not necessarily surprising: most people didn’t care that their clothes/shoes were made by suffering child laborers, so a few relatively comfortable devs having to essentially find a new job is barely going to register. I think that forcing human recourse _is_ the solution. This problem is much bigger than just the “App store duopoly” axe. A law that required the ability to perform all account actions/appeals/etc realtime with another human (phone, chat, etc) would’ve also minimized the need for the recent unsubscription laws as well since the hassle would’ve been much less in the first place. These are the kind of foundational human-centric business laws we need instead of the reactionary hyper-focused ones that don’t address root problems and usually just wind up further cementing incumbents.
• leros 5y ago
  I agree. These walled gardens wouldn't be the end of the world if you could actually talk to a human customer service person to resolve these issues. The problem is that companies are willing to use algorithms that are 95% correct because it's good enough for them, but that ignores that it's screwing over a percentage of their customers.
  • Forgeties79 5y ago
    It’s kind of baffling to me that YouTube/google, Facebook, etc. have been able to get away with “well there’s just too much to manage so we deploy algorithms and pray to the code god it meets a minimum threshold to keep the government off our backs.” Imagine I’m a trillionaire real estate mogul. I have 1,000,000 properties under my domain across the world. Some will, inevitably, not get the upkeep they need and a situation like the apartment building in Miami occurs. Let’s say even that no one dies, to be generous. Another building the electrical is jacked up and held together by paper clips, regularly leaving residents without power. Another one has grey water coming out of every faucet. The government comes in and goes “hey, a bunch of stuff isn’t up to code. You can’t have buildings like this.” I then proceed to go, “well, it’s unreasonable to expect me to address every problem. I’m just too large of a company with too many assets. I’d need to hire a small army to manage it all.” The government goes, “huh, that’s reasonable. Well I guess just automate what you can and do your best!” That’s absolutely ridiculous, right?
• toomuchtodo 5y ago
  Call your congressional rep [1]. That's what it takes; to get policymakers to write and pass the law. I imagine there's bipartisan support at the moment for reining in Big Tech [2]. Google is never going to improve the situation out of the goodness of their corporate heart (if such a thing were even to exist). [1] https://www.commoncause.org/find-your-representative/ https://www.commoncause.org/find-your-representative/ [2] https://nypost.com/2022/03/29/biden-doj-backs-senate-antitrust-bill-targeting-big-tech-firms/ https://nypost.com/2022/03/29/biden-doj-backs-senate-antitru...
  • mnd999 5y ago
    • toomuchtodo 5y ago
      Sitting out means those who do reach out have a louder voice with policymakers. You can't not participate and also be upset by the outcomes due to your lack of participation. https://en.wikipedia.org/wiki/Brandolini%27s_law https://en.wikipedia.org/wiki/Brandolini%27s_law
  • reaperducer 5y ago
    I don't know if it's still true, but pre-pandemic I used to have occasional contact with legislative types, and they regularly told me that physical letters carry more weight than phone calls, and email was a distant third. Pretty much, the amount of effort you put into your communication is a reflection of how important the issue is, and how important it is to you. Which kind of goes hand-in-hand with lobbyists. Companies don't pay them millions of dollars to send emails or rant on web forums.
    • bitxbitxbitcoin 5y ago
      Can confirm, worked as a Congressional intern. Order of importance was (at least in 2011): 1. In person meeting 2. Physical letter 3. Phone call
• epolanski 5y ago
  > We need legislation forcing companies to manually review algorithmic decisions that impact people's lives, and have a proper appeals process. I don't think this is enforceable nor I think it's even possible to have such legislature. At the end of the day Google has every right to decide what they put and what they don't put on their store, with whom they do business and with whom they don't. > With how much our modern lives are dependent on services like Google's, they effectively become utilities and should be regulated as such. Calling an online application store an utility seems quite a stretch. De-googling and de-duopoling is the answer to these situations. At the end of the day you can't force Google or Apple to have your products on their shop.
  • nightski 5y ago
    You are right, just don't do business with this evil company. Simple as that.
    • matsemann 5y ago
      Except they are so big it's hard to avoid. So perhaps the real solution is to split up these giants..?
      • epolanski 5y ago
        Is that their fault? There's other operating systems one can install on mobile, they can even argue Android is open source and no one is stopping you from installing a different OS on your phone.
        • reaperducer 5y ago
          There's a lot more to it than just that. A couple of years ago, the New York Times ran an interesting article where someone tried to live an ordinary day without interacting in any way with Google. The result was that it simply wasn't possible.
      • deleted 5y ago
        [deleted]
• Nextgrid 5y ago
  I believe the GDPR already has that. The problem is that the GDPR is not enforced enough and not enough resources/willingness are allocated to enforce it. The problem in account termination cases would also be that: 1) they can claim they've reviewed the ban, that it's legitimate and they're refusing to disclose the reasons behind it to avoid helping people circumvent the ban, while in reality they didn't do any investigation at all. Proper enforcement should be able to pierce through this veil (by forcing them to disclose the reasons and data behind the bank to the regulator, a neutral third-party), but it's missing and nothing suggests it's going to get any better. 2) given that businesses are still allowed to essentially "fire" customers at will, and changing that is impossible due to wide-ranging repercussions, nothing prevents them from "firing" you anyway. Proper antitrust enforcement is needed here (so that you're not allowed to "fire" customers this way) but that's missing as well.
  • hnbad 5y ago
    The GDPR does require algorithmic decisions to be non-final. However it provides no mechanism beyond allowing the company to just claim they manually reviewed the case and came to the same conclusion as the algorithm. The purpose of that requirement is really more to limit the effect of automated decisions by insurance companies and such.
  • frankfrankfrank 5y ago
    That’s the problem with all these abstracted types of legislation. All it would take is to reestablish that you are the sole owner of your information that is your property, as has been established by the courts, and it cannot be sold without a formal contract, e.g., the way real estate is transferred; and that any tracking is illegal stalking and wire fraud (because it is) just like tapping someone’s phone would be since it is using the public internet. Alternately, these companies can stop relying on the public internet for illegal criminal actions and fraud, and build their own internet if they want to track and stalk people. The law exists, it doesn’t matter what other laws you make when none, even the fundamental Constitutional law, is not enforced and simply ignored. We have too many people who have these narcissistic perceptions that the real problem is that their pet legislation hasn’t been added to the mountain of legislation; when the real problem is people trying to control others, some in business, some through legislation. If the general public does not recognize this soon, the clutter of legalization will become a prison, if it isn’t already.
• notreallyserio 5y ago
  Google doesn't have billions of developers publishing Android apps, not even close. They shouldn't need to make these irreversible, algorithmic decisions on the Play Store. They choose to, and there are engineers and product managers and etc that individually choose to support those efforts.
  • headmelted 5y ago
    It's true that there aren't so many developer accounts, but there was a thread on here not long ago that discussed how someone's life was turned upside down by getting locked out of a cloud account without recourse. Not even the account being hacked, just being unable to access it. I would imagine any broad-based regulation around recourse for account locks is likely to start with the individual B2C user due to how many more of them there are (essentially everyone, even if you only count Apple and Google - legislation would of course cover every provider). The bigger issue is that I don't even know where or how you would start with this. Ironically, there's a certain amount of comfort in knowing that your data is behind an unresponsive brick wall these days as it makes it harder/impossible for someone to socially engineer their way in. The downsides to that are many and varied, and what the post references.
• yunohn 5y ago
  > It's simply not economical for many companies to deal with the long tail of false positives, so they don't. This happens everywhere, not just Big Tech. Even as humans, we try to handle the 99% and ignore the 1%. I still don’t understand what anyone is proposing - force companies to provide support against their will?
  • deleted 5y ago
    [deleted]
  • xboxnolifes 5y ago
    > I still don’t understand what anyone is proposing - force companies to provide support against their will? That's what laws are: forcing people to do things they would otherwise not do.
    • Aeolun 5y ago
      And kind of the point of laws. If people did them without the law then there wouldn’t be a need for the law in the first place.
  • browningstreet 5y ago
    You'd be amazed at how a little regulation right-sizes things in corporations. I work in banking -- there's no grumbling about our regulatory landscape. We just have teams of people who make sure we're compliant, and we get through our audits. So, to answer your question: yes.
    • yunohn 5y ago
      Why are you comparing banking regulations to tech support? The former is crucial to trust in money, governance, and societal functioning. Tech support is easily overrun and exploited by bad actors, and the upside for the company is very minimal.
      • freedomben 5y ago
• mtgx 5y ago
• rileymat2 5y ago
  > We need legislation forcing companies to manually review algorithmic decisions that impact people's lives, and have a proper appeals process. What you want is human judgement, but it will be hard to legislate that. What can easily happen is the human parroting back the underlying reasons for the decision. In this case "We have reviewed your case, and according to our records the account is associated with the problematic account" I have had this happen to me in bureaucratic situations with no computers involved.
  • lima 5y ago
    We have courts and the ability to appeal to courts when bureaucracies make mistakes.
    • buscoquadnary 5y ago
      That seems like the same problem we have hear just now you get the headache of the courts on top of the decision, how many people avoid court now for much more impactful and legitimate situations simply because they can't afford the time and money? What we want isn't review of automated decisions, what we want is openness, transperancy and clarity in the process. The problem people have isn't so much the appeals process it is the opaqueness and seeming arbitrary nature of the whole thing.
      • lima 5y ago
        > openness, transperancy and clarity in the process But that's what a public court gets you.
      • anaccountexists 5y ago
        There are problems with how transparent you make things though (i.e., giving away the underlying signals). There’s a moving target between fraudsters and risk teams at companies where the fraudsters will try to run just up to the edge of alerting systems without passing over, then scale and repeat it. If the signals used are made public, fraudsters will win every time. It’s the same with search engines- if they publish how a score is calculated, people will game it immediately. Maybe the signals should be required to go through a review with authorities? Idk.
• jahnu 5y ago
  Microsoft have a better solution, or at least one of their departments does. You pay for support questions and if it turns out it’s actually Microsoft’s fault they refund you the fee. It’s great and most of the time I never had to pay. When I did it was still money well spent as I got good support. There have been a few occasions where I would gladly have paid Google/Twitter/Apple to answer my questions.
  • boredumb 5y ago
    Honest question, why on earth would you ever have a question for twitter so important that you'd consider paying them to answer it?
    • e9 5y ago
      If they ban your account and you don’t know why
      • silon42 5y ago
        Also, to prevent an automated ban (or at least immediate manual review).
    • boplicity 5y ago
      There's a Twitter account that has our company name, but it hasn't been used for a decade, and has only one Tweet. I'd love to pay Twitter to talk to them, so we can take over the account. Maybe they have a process for this anyways? I don't know. All I know is this account gets tagged all the time by people trying to tag our company.
      • snowwrestler 5y ago
        Find your way to your local Twitter ad rep. They may be able to facilitate transfer of an inactive handle. It obviously helps if you are buying or intend to buy ads on Twitter.
    • lima 5y ago
• anaccountexists 5y ago
  In my experience, the algorithms are much more forgiving than humans. At least, when I worked with the risk team at my FinTech company, algorithms flagged people for review and then humans decided to unflag or terminate their accounts. The only time we’d do a freeze is if it looked like an account take over (since that could super badly affect the account owner).
  • freedomben 5y ago
    Very true, I've seen similar things. Humans can be just as cold-hearted and unempathetic as a computer is, especially when they deal with problem "people" all day every day.
• micromacrofoot 5y ago
  > We need legislation forcing companies to manually review algorithmic decisions that impact people's lives It's not hard to argue that all algorithmic decisions impact people's lives though, Facebook for example, prioritizes certain friends' posts over others... it doesn't seem like a stretch to say this could actually impact who you're close to.
• frankfrankfrank 5y ago
  I find it would be far more effective and faster to simply demand enforcing the Constitution (at least in the USA), while also enforcing competition and shatter all the cartels and de facto monopolies. If there were more competition, there would also be alternatives that companies like in the subject case could choose from and which would deter companies from making mistakes in order to prevent loss of market share. A good measure should be that anything should have more than 3 equal competitors, is 4+ search engine companies of equal scale, 4+ App marketplaces for 4+ phone OS/hardware makers, that can operate on 4+ telecom services, etc. And that transfer of data and services between each must be effortless. We don’t need new legislation that will not be enforced or is flawed because it is too specific and myopic, we need enforcement of basic and fundamental law and concepts that expand freedom and choice and our fundamental human rights. Submit to our every tech company dictate whim or we will destroy everything you have worked for and not be able to work in your industry ever again or feed yourself is not freedom and is a crime against humanity.
  • cortesoft 5y ago
    What aspect of the Constitution would apply to this situation?
    • frankfrankfrank 5y ago
      Freedom of speech and assembly, security in one’s person and affects, freedom of movement/transportation, even prohibition against slavery (even though that will be a bit too abstract for most), etc.
      • cortesoft 5y ago
        I really can’t see how any of those would apply to Google shutting down an account.
      • whoknew1122 5y ago
        Google isn't the government, so the first amendment doesn't apply. Security in one's person and affects? The OP still has possession of their apps. Google is just deciding it won't sell them for the user. And prohibition against slavery? It's both laughable and deeply offensive to consider this situation even remotely similar to chattel slavery.
• throwaway82652 5y ago
  >We need legislation forcing companies to manually review algorithmic decisions No, I'm sorry, this is a horrible, horrible idea. As long as these companies are targeted by automated spambots, they need automated systems to counteract and remove spam accounts. That's just the reality of the internet arms race in the last 20 years. If you make them manually review everything, you hand a victory over to the spammers and degrade service quality for everyone. >and have a proper appeals process. This is a much better idea. Do this and require them to have a functioning customer service department.
  • lima 5y ago
    Fully agreed - I wasn't suggesting they review every manual decision, but do it as part of a proper appeals process. Google, for instance, is likely seeing abuse on a scale where they have to have automated bans - for everyone's benefit! - and there's nothing fundamentally wrong with that as long as you can escalate to a real human. Edited the comment to reflect this - thanks! I worked on an abuse prevention system in the past and know the challenges very well, except my company actually put in the effort to respond to every appeal and compensate affected customers for their troubles. Yes, humans are expensive, and spammers will try to game the appeals process, too - but it's simply a cost of doing business.
    • Anthony-G 5y ago
      All good points. > Yes, humans are expensive, and spammers will try to game the appeals process, too - but it's simply a cost of doing business. Another Hacker News commentator¹ had a good suggestion for this problem: > Microsoft have a better solution, or at least one of their departments does. You pay for support questions and if it turns out it’s actually Microsoft’s fault they refund you the fee. If users paid some up-front fee for the appeal, similar to the above, regular users would have the opportunity to appeal an automated ban (if the appeal was done properly). On the other hand, spammers and other malicious actors would have to pay money for the opportunity to attempt to game the system. ¹ https://news.ycombinator.com/item?id=30855836 https://news.ycombinator.com/item?id=30855836
  • phaistra 5y ago
• antattack 5y ago
  There is a (state level) legislation proposed to 'open source' algorithms used for hiring[1]. Makes sense to expand it to all decision making processes. [1]https://www.brookings.edu/blog/techtank/2021/12/20/why-new-york-city-is-cracking-down-on-ai-in-hiring/ https://www.brookings.edu/blog/techtank/2021/12/20/why-new-y...
• brailsafe 5y ago
  Agreed. I'm facing this right now with my Amazon account, and posted a question related to it. Basically they detected something suspicious, locked my account a while ago, and now the account is irrecoverable for an opaque set of reasons that they won't elaborate on. I feel totally fucked.
• jsmith45 5y ago
  The EU's proposed Digital Services act mandates a proper appears process for account suspension/termination related to transmitting "illegal content", is "or incompatible with its terms and conditions". This appeal is required to be real, actually reviewing the content or conduct in question to determine if it violates the law or terms and conditions. If there is enough information to conclude that no violation occur, the company is required to reinstate the account. If after the appeal, the user is still not satisfied they can appeal to one of a set of government approved arbitrators, who will listen to the dispute and decide the case. It is the user who gets to decide which of the arbitrators will be used (among those certified for the relevant category of platform). The company always handles its own expenses associated with this process, and if the company loses, they must reimburse reasonable fees and expenses incurred by the user. These appeal processes also apply to removed content, not just account suspension/termination. There is an exception for "small and micro enterprises" though. The main downside of this law is that its primary purpose is to create a DCMA++ framework over in Europe, but it still looks to have a much better balance of concerns that the laws here in the US have.
• Seattle3503 5y ago
  In banking and financial services we have regulations that require "model validation". Any automated decision needs to be tracked and compliance needs to check the decisions to see that they are sound. They were required after the financial crash in 2008 in order to get a better handling on financial risk modeling, but the laws were written broadly enough that they apply to other automated decisions in financial institutions. We could take lessons from these regulations and apply them elsewhere.