3 ms·

I wonder if the devs behind these kind of automations ever stop and think about the ethical sides of it.

by matsemann 5y ago

inafewwords 5y ago
Nope. You can tell by how it is implemented.
- rexreed 5y ago
  Exactly. You think that developers are paid to stop and think about the outcomes of what they do? They're all too busy running sprints and meeting OKRs and performance metrics and hustling design and code reviews and playing internal politics to worry about the effect of what they are doing on their users and ordinary people. Besides, even if they did have an end user in mind, it's not the "users". They're motivated by the customers - advertisers. I really found this video to be enlightening on our current techno-dystopian state: https://yewtu.be/watch?v=GWvFZ99s558 https://yewtu.be/watch?v=GWvFZ99s558
bogwog 5y ago
Even if they do, it’s not their responsibility to fix those problems. It’s government’s responsibility to step in and fix bad behavior that market forces can’t fix (like when it’s more economically viable to destroy thousands of livelihoods per year than to invest in human review processes)
- matsemann 5y ago
  I do feel it's my responsibility to think of the side-effects of what I do and the consequences that bear on others. But as a sibling to you say, maybe I therefore self-select away from those kind of companies, and those not thinking or caring about those things end up taking those jobs.
  - bogwog 5y ago
    That’s because you probably have a moral compass, but that still isn’t enough. If you were a decision maker at Google, those morals would probably get you fired at some point. The people running a company have a legal responsibility to act in the interests of their shareholders. Even if making a morals>economics decision doesn’t get them into legal trouble, it can get them into trouble with shareholders, and they may be fired and replaced with someone who doesn’t have a moral compass. So if the infrastructure doesn’t have room for morally correct decisions, no amount of martyrdom from executives is going to eliminate bad behavior in the long run.
gambler 5y ago
People who consider this an issue either wouldn't work for Google to begin with or end up butting heads with management and leave the company. The benefits of self-selection. There is also a lot of internal corporate training and team-building designed to redefine "ethics" to make this sort of stuff acceptable.
leros 5y ago
It's probably something they're not really aware of. My company does something kinda similar. We're an aggregator of sorts. We have all sorts of quality and safety rules and we de-list things all the time for violations. As long as we aggregate 95% of what's out there, our users are happy, so we don't have massive incentive to spend lots of time manually working with rule violators, especially when a good chunk of them are actually scammers manipulating the system and harming our users.
hbn 5y ago
I'd be willing to bet that there was no one developer or team who was given a task such as, "run our audit AI on accounts and permanently blacklist them forever if that's what the algorithm wants" It was more likely years worth of minor changes that got it into this state. Teams adding different algorithmic checks for various things, where the output is just setting a flag on the account. And other teams adding account termination logic for certain flags or a certain number/combination of flags, not knowing exactly how the flags are set. Though maybe that would be the bad task. I'm just spitballing.
jrockway 5y ago
I've spent an inordinate amount of time banning fraudsters from my own cloud platform. The bots that companies write to do this are designed to fight off other bots. If you have an appeals process, the humans that control the malicious bots will show up to abuse the appeals process. One of my favorite cases was a user that signed up with hundreds of stolen credit card numbers (all from the same IP address, as they are apt to do), got banned, and then opened an issue in our open source repo complaining of trouble logging in. Yeah, you're having trouble because I banned you and your entire network of bots. There is a balance, of course, but if you haven't seen how much automated abuse there is on the Internet, be careful of what you wish for. Even the tiniest of the tiny services suffer from massive amount of automated abuse. At that tiny scale, it was nearly impossible for me to keep up with the abuse without the help of automation, very broad bans, and deleting related accounts by walking a reputation graph (like Google is being criticized for doing here). At Google's scale, I don't think there are enough humans on Earth to deal with the abuse. As a result, there are going to be some innocent casualties. I don't think it's an ethics thing, it's simply not possible to run a business without some sort of process like this. There are laws, rules, and processes that could cut all this down to levels that could be managed by humans, but the cure might be worse than the disease. (For example, it would be great if there was a 1:1 mapping between your national ID card and your IP address. All of these stolen-credit-card users could then be imprisoned. But, you know that that's a terrible thing, because it will also be used against anyone criticizing governments or large corporations.)
- nkrisc 5y ago
  > At Google's scale, I don't think there are enough humans on Earth to deal with the abuse. As a result, there are going to be some innocent casualties. Perhaps they've scaled too large then. I don't think "we're too big to be held accountable for screwing innocent people" is a valid excuse.
- matsemann 5y ago
  While I see your point, when it's accounts with years of history, millions of downloads, thousands of dollars paid in sales etc., it should be easy to discern those from spammers creating thousands of new accounts with no standing.
hwers 5y ago
They select for sociopaths coding these things. If someone is able to be bothered by these actions, they won't last long or will quit voluntarily and someone more sociopathic will take their place. It's a problem higher up the chain of management.

I wonder if the devs behind these kind of automations ever stop and think about the ethical sides of it.

inafewwords 5y ago

Nope. You can tell by how it is implemented.

rexreed 5y ago
Exactly. You think that developers are paid to stop and think about the outcomes of what they do? They're all too busy running sprints and meeting OKRs and performance metrics and hustling design and code reviews and playing internal politics to worry about the effect of what they are doing on their users and ordinary people. Besides, even if they did have an end user in mind, it's not the "users". They're motivated by the customers - advertisers. I really found this video to be enlightening on our current techno-dystopian state: https://yewtu.be/watch?v=GWvFZ99s558 https://yewtu.be/watch?v=GWvFZ99s558

bogwog 5y ago

Even if they do, it’s not their responsibility to fix those problems. It’s government’s responsibility to step in and fix bad behavior that market forces can’t fix (like when it’s more economically viable to destroy thousands of livelihoods per year than to invest in human review processes)

matsemann 5y ago
I do feel it's my responsibility to think of the side-effects of what I do and the consequences that bear on others. But as a sibling to you say, maybe I therefore self-select away from those kind of companies, and those not thinking or caring about those things end up taking those jobs.
- bogwog 5y ago
  That’s because you probably have a moral compass, but that still isn’t enough. If you were a decision maker at Google, those morals would probably get you fired at some point. The people running a company have a legal responsibility to act in the interests of their shareholders. Even if making a morals>economics decision doesn’t get them into legal trouble, it can get them into trouble with shareholders, and they may be fired and replaced with someone who doesn’t have a moral compass. So if the infrastructure doesn’t have room for morally correct decisions, no amount of martyrdom from executives is going to eliminate bad behavior in the long run.

gambler 5y ago

People who consider this an issue either wouldn't work for Google to begin with or end up butting heads with management and leave the company. The benefits of self-selection. There is also a lot of internal corporate training and team-building designed to redefine "ethics" to make this sort of stuff acceptable.

leros 5y ago

It's probably something they're not really aware of. My company does something kinda similar. We're an aggregator of sorts. We have all sorts of quality and safety rules and we de-list things all the time for violations. As long as we aggregate 95% of what's out there, our users are happy, so we don't have massive incentive to spend lots of time manually working with rule violators, especially when a good chunk of them are actually scammers manipulating the system and harming our users.

hbn 5y ago

I'd be willing to bet that there was no one developer or team who was given a task such as, "run our audit AI on accounts and permanently blacklist them forever if that's what the algorithm wants" It was more likely years worth of minor changes that got it into this state. Teams adding different algorithmic checks for various things, where the output is just setting a flag on the account. And other teams adding account termination logic for certain flags or a certain number/combination of flags, not knowing exactly how the flags are set. Though maybe that would be the bad task. I'm just spitballing.

jrockway 5y ago

I've spent an inordinate amount of time banning fraudsters from my own cloud platform. The bots that companies write to do this are designed to fight off other bots. If you have an appeals process, the humans that control the malicious bots will show up to abuse the appeals process. One of my favorite cases was a user that signed up with hundreds of stolen credit card numbers (all from the same IP address, as they are apt to do), got banned, and then opened an issue in our open source repo complaining of trouble logging in. Yeah, you're having trouble because I banned you and your entire network of bots. There is a balance, of course, but if you haven't seen how much automated abuse there is on the Internet, be careful of what you wish for. Even the tiniest of the tiny services suffer from massive amount of automated abuse. At that tiny scale, it was nearly impossible for me to keep up with the abuse without the help of automation, very broad bans, and deleting related accounts by walking a reputation graph (like Google is being criticized for doing here). At Google's scale, I don't think there are enough humans on Earth to deal with the abuse. As a result, there are going to be some innocent casualties. I don't think it's an ethics thing, it's simply not possible to run a business without some sort of process like this. There are laws, rules, and processes that could cut all this down to levels that could be managed by humans, but the cure might be worse than the disease. (For example, it would be great if there was a 1:1 mapping between your national ID card and your IP address. All of these stolen-credit-card users could then be imprisoned. But, you know that that's a terrible thing, because it will also be used against anyone criticizing governments or large corporations.)

nkrisc 5y ago
> At Google's scale, I don't think there are enough humans on Earth to deal with the abuse. As a result, there are going to be some innocent casualties. Perhaps they've scaled too large then. I don't think "we're too big to be held accountable for screwing innocent people" is a valid excuse.
matsemann 5y ago
While I see your point, when it's accounts with years of history, millions of downloads, thousands of dollars paid in sales etc., it should be easy to discern those from spammers creating thousands of new accounts with no standing.

hwers 5y ago

They select for sociopaths coding these things. If someone is able to be bothered by these actions, they won't last long or will quit voluntarily and someone more sociopathic will take their place. It's a problem higher up the chain of management.