7 ms·

I have to ask, why did you choose to take the contract? Financial pressures? Or did you not feel that such a system is morally reprehensible? Personally while

by fao_ 8y ago

I have to ask, why did you choose to take the contract? Financial pressures? Or did you not feel that such a system is morally reprehensible? Personally while I'm opposed to programmer certification, systems like these certainly make me wish such a thing existed.

fucking_tragedy 8y ago
There will always be someone who can find a moral reason that such systems need to exist. Engineers build weapons, after all.
- fao_ 8y ago
  But see, I can understand weapons building. The motivation is protective towards one's society, and that is the intended result, even if that might not be the actual case. Whereas the intention of building an oppressive system such as the one described above is, what, selfishness? Laziness? Programmers, because of the high-demand nature of our role, do not tend to be subject to the usual financial pressures that other communities are. I'm not sure what would motivate someone to build such a system as opposed to walking and finding a better job offer.
  - fucking_tragedy 8y ago
    > But see, I can understand weapons building. This is my point. Eventually someone will come in and say the system in the OP was necessary for the company to stay afloat, pay its employees or retain value for its retired shareholders.
    - chrismcb 8y ago
      There is a difference between defending your nation and defending your country. But in this particular case, some of the things described are actually illegal. While building a weapon is not necessarily illegal. Ignoring the ethical implications.
      - krageon 8y ago
        There is not really all that much difference. Weapons are used to terrorise and murder people. What this person wrote will give some humans stress but it will never cause physical harm. That you think one is justified and the other is not has very little to do with their relative harm, which is by no means a solved problem or a given. It has to do with your view of it, and those views will differ between different people.
  - joewee 8y ago
    I’ve always looked at these systems as “if someone else built this, it would be worse.” Doesn’t the same moral conundrum exist anytime you build a system that dehumanizes people for profit. Ad networks? Drug trials?
    - brokenmachine 8y ago
      Ad networks - people can just disconnect or look away if they like. Drug trials have benefit to society. I would argue that creating a system to fire people in an automated fashion when they take too many bathroom breaks like this is morally worse than both of those systems you mentioned.
  - derefr 8y ago
    Not saying this was likely, but one potential reason could be a kind of selfless penny-pinching. Say you have a call center run as a co-op. You’ve got workers and HR people. Both the workers and HR people are shareholders. If you can eliminate the HR people, then the workers can each have a larger proportionate share for the same work. Automating HR eliminates the HR people.
    - hndamien 8y ago
      Is Uber so different?
    - chrismcb 8y ago
      Automating a job is one thing. Some people may consider it unethical. But it is going to happen. But writing coffee to terrorize people is something else.
  - s3krit 8y ago
    >But see, I can understand weapons building. The motivation is protective towards one's society, and that is the intended result, even if that might not be the actual case. Not really. If you work for Raytheon, you are supporting a company that sells arms to the following countries: https://www.raytheon.com/ourcompany/global https://www.raytheon.com/ourcompany/global Northrop Grumman have a list of their worldwide presence here: https://www.northropgrumman.com/AboutUs/OurGlobalPresence/Pages/default.aspx https://www.northropgrumman.com/AboutUs/OurGlobalPresence/Pa... If you work for a defense contractor, you are not 'protecting your society' - that's not even the intent. The intent is to make money by selling tools designed to kill other human beings.
  - Noumenon72 8y ago
    It enables cooperation. Human nature is such that it's hard to keep working hard when others slack, you feel taken advantage of and a fool. This way, you know everyone's doing their part. It creates jobs. A lot of people don't have the self-control to keep from taking longer and longer breaks, either costing the company money or getting fired. Some jobs pay more to hire people that do have this self-control, but there are only so many of those people. This creates a business model that works when supplied only with the lazier employees who are left. Another system like this is the timeclock. It's a tyrant and getting out of bed on time every morning is the hardest thing I've ever had to do, but there's just no way to run a factory without it.
- King-Aaron 8y ago
  There will also always be someone else to do the work if you don't. Some people just like to eat.
  - fao_ 8y ago
    Just because someone is willing to, say, torture a rat, I should be the one to do it? That's a non-sequitur.
    - King-Aaron 8y ago
      I feel this is akin to strawman logic. However using that as an example, there are people who work as vermin exterminators, or people who work in labs that have to euthanise lab mice. They probably don't enjoy it, but when there's a need for something you'll find someone to do it.
      - fao_ 8y ago
        It's not a straw-man, because it's using your literal point. I can't get any closer to your argument than that. > However using that as an example, there are people who work as vermin exterminators, or people who work in labs that have to euthanise lab mice. They probably don't enjoy it, but when there's a need for something you'll find someone to do it. Yes, but that does not serve as a reason why I should be the one to do it (instead of them), that only postulates that there are other people out there to do it, which is an irrelevant point.
derefr 8y ago
Personally, as both a writer and a programmer, I often consider implementing the systems that I’ve written dystopian fiction about. Not to create a dystopia, but for the same reason I write: to point one out, such that people can get incensed, laws can be made, etc. To put this another way: the most efficient way we could have possibly found to get the use of nuclear arms in war globally banned, was to have someone use one. The Cold War would have been far riskier if the world hadn’t seen Hiroshima and Nagasaki—it would have been a stand-off to the use of weapons we would have as-yet had no understanding of the consequences of using. It probably would have ended with the use of hundreds of bombs, rather than just two. It’s sort of the moral equivalent of a “work to rule” strike: the best way to get through the lesson that something is bad, is to stop pushing back against it and just let it happen for once.
- andy_ppp 8y ago
  You’re actually suggesting being a sociopath is a good thing because it’ll make all the bad ideas eventually die by us trying them. This is simply not true, we have to make bad ideas die by reason and logic, before we do too much harm. I suppose doubling down of fossil fuels and burning of rainforest is great, we’ll just have to adapt quicker to the super extreme weather. But we can adapt with more aircon! Yay! Maybe we are just monkeys who can’t actually learn from things that we haven’t experienced but then we’ll be replaced by nature any day now. I’d rather learn things the easy way, if at all possible.
  - mindslight 8y ago
    This is an extremely uncharitable reading. Infosec is a field rife with kneejerk dismissals until people actually see the exploitation in action. How long has the (ethical) tech community been warning people about the NSA and Faceboot, until Snowden and Cambridge Analytica actually happened?
    - andy_ppp 8y ago
      Infosec is totally different because it’s exposing problems people have already created in a standard way. This is using technology for maliciousness in the first place. Try telling to those people who were abused by that system they were just collateral damage in a bigger plan.
hnzix 8y ago
Ironically, there's currently a post on HN frontpage about malevolent personality traits: https://news.ycombinator.com/item?id=17685733 https://news.ycombinator.com/item?id=17685733 I would file this under 3. Moral Disengagement - A generalized cognitive orientation to the world that differentiates individuals' thinking in a way that powerfully affects unethical behavior.
iamleppert 8y ago
I was just starting out in my career and was more pre-occupied with the task at hand to have enough mental headspace to contemplate the full picture of what was happening. The system itself was designed by traditional software means: trial and error, trying to see what worked and didn’t, usually trying to maximize some KPI at the company. There was also the thought that punishment and reward should be applied fairly and be “data driven”; take the bias and human factor out of decisions and implement performance management in a predicable, deterministic and transparent way. The problems arose when people realized that the same controls and abilities to instill this equity and “fairness” provided the platform to enable wide scale exploitation in the other direction. Want more profit? Change a variable that was a single number whose action would cause great stress for many people, but would produce a desired result. The goals and metrics would always start out resonable but eventually would cross a certain point, and once they did, there was no going back to the way things were. Draw your own conclusions to the similarities in this system to our modern day technical web triumphs.
- rrcaptain 8y ago
  It's really depressing how capitalism has taken automation and data and bastardized it into a toxic weapon.

by fao_ 8y ago

fucking_tragedy 8y ago

There will always be someone who can find a moral reason that such systems need to exist. Engineers build weapons, after all.

fao_ 8y ago
But see, I can understand weapons building. The motivation is protective towards one's society, and that is the intended result, even if that might not be the actual case. Whereas the intention of building an oppressive system such as the one described above is, what, selfishness? Laziness? Programmers, because of the high-demand nature of our role, do not tend to be subject to the usual financial pressures that other communities are. I'm not sure what would motivate someone to build such a system as opposed to walking and finding a better job offer.
- fucking_tragedy 8y ago
  > But see, I can understand weapons building. This is my point. Eventually someone will come in and say the system in the OP was necessary for the company to stay afloat, pay its employees or retain value for its retired shareholders.
  - chrismcb 8y ago
    There is a difference between defending your nation and defending your country. But in this particular case, some of the things described are actually illegal. While building a weapon is not necessarily illegal. Ignoring the ethical implications.
    - krageon 8y ago
      There is not really all that much difference. Weapons are used to terrorise and murder people. What this person wrote will give some humans stress but it will never cause physical harm. That you think one is justified and the other is not has very little to do with their relative harm, which is by no means a solved problem or a given. It has to do with your view of it, and those views will differ between different people.
- joewee 8y ago
  I’ve always looked at these systems as “if someone else built this, it would be worse.” Doesn’t the same moral conundrum exist anytime you build a system that dehumanizes people for profit. Ad networks? Drug trials?
  - brokenmachine 8y ago
    Ad networks - people can just disconnect or look away if they like. Drug trials have benefit to society. I would argue that creating a system to fire people in an automated fashion when they take too many bathroom breaks like this is morally worse than both of those systems you mentioned.
- derefr 8y ago
  Not saying this was likely, but one potential reason could be a kind of selfless penny-pinching. Say you have a call center run as a co-op. You’ve got workers and HR people. Both the workers and HR people are shareholders. If you can eliminate the HR people, then the workers can each have a larger proportionate share for the same work. Automating HR eliminates the HR people.
  - hndamien 8y ago
    Is Uber so different?
  - chrismcb 8y ago
    Automating a job is one thing. Some people may consider it unethical. But it is going to happen. But writing coffee to terrorize people is something else.
- s3krit 8y ago
  >But see, I can understand weapons building. The motivation is protective towards one's society, and that is the intended result, even if that might not be the actual case. Not really. If you work for Raytheon, you are supporting a company that sells arms to the following countries: https://www.raytheon.com/ourcompany/global https://www.raytheon.com/ourcompany/global Northrop Grumman have a list of their worldwide presence here: https://www.northropgrumman.com/AboutUs/OurGlobalPresence/Pages/default.aspx https://www.northropgrumman.com/AboutUs/OurGlobalPresence/Pa... If you work for a defense contractor, you are not 'protecting your society' - that's not even the intent. The intent is to make money by selling tools designed to kill other human beings.
- Noumenon72 8y ago
  It enables cooperation. Human nature is such that it's hard to keep working hard when others slack, you feel taken advantage of and a fool. This way, you know everyone's doing their part. It creates jobs. A lot of people don't have the self-control to keep from taking longer and longer breaks, either costing the company money or getting fired. Some jobs pay more to hire people that do have this self-control, but there are only so many of those people. This creates a business model that works when supplied only with the lazier employees who are left. Another system like this is the timeclock. It's a tyrant and getting out of bed on time every morning is the hardest thing I've ever had to do, but there's just no way to run a factory without it.
King-Aaron 8y ago
There will also always be someone else to do the work if you don't. Some people just like to eat.
- fao_ 8y ago
  Just because someone is willing to, say, torture a rat, I should be the one to do it? That's a non-sequitur.
  - King-Aaron 8y ago
    I feel this is akin to strawman logic. However using that as an example, there are people who work as vermin exterminators, or people who work in labs that have to euthanise lab mice. They probably don't enjoy it, but when there's a need for something you'll find someone to do it.
    - fao_ 8y ago
      It's not a straw-man, because it's using your literal point. I can't get any closer to your argument than that. > However using that as an example, there are people who work as vermin exterminators, or people who work in labs that have to euthanise lab mice. They probably don't enjoy it, but when there's a need for something you'll find someone to do it. Yes, but that does not serve as a reason why I should be the one to do it (instead of them), that only postulates that there are other people out there to do it, which is an irrelevant point.

derefr 8y ago

Personally, as both a writer and a programmer, I often consider implementing the systems that I’ve written dystopian fiction about. Not to create a dystopia, but for the same reason I write: to point one out, such that people can get incensed, laws can be made, etc. To put this another way: the most efficient way we could have possibly found to get the use of nuclear arms in war globally banned, was to have someone use one. The Cold War would have been far riskier if the world hadn’t seen Hiroshima and Nagasaki—it would have been a stand-off to the use of weapons we would have as-yet had no understanding of the consequences of using. It probably would have ended with the use of hundreds of bombs, rather than just two. It’s sort of the moral equivalent of a “work to rule” strike: the best way to get through the lesson that something is bad, is to stop pushing back against it and just let it happen for once.

andy_ppp 8y ago
You’re actually suggesting being a sociopath is a good thing because it’ll make all the bad ideas eventually die by us trying them. This is simply not true, we have to make bad ideas die by reason and logic, before we do too much harm. I suppose doubling down of fossil fuels and burning of rainforest is great, we’ll just have to adapt quicker to the super extreme weather. But we can adapt with more aircon! Yay! Maybe we are just monkeys who can’t actually learn from things that we haven’t experienced but then we’ll be replaced by nature any day now. I’d rather learn things the easy way, if at all possible.
- mindslight 8y ago
  This is an extremely uncharitable reading. Infosec is a field rife with kneejerk dismissals until people actually see the exploitation in action. How long has the (ethical) tech community been warning people about the NSA and Faceboot, until Snowden and Cambridge Analytica actually happened?
  - andy_ppp 8y ago
    Infosec is totally different because it’s exposing problems people have already created in a standard way. This is using technology for maliciousness in the first place. Try telling to those people who were abused by that system they were just collateral damage in a bigger plan.

hnzix 8y ago

Ironically, there's currently a post on HN frontpage about malevolent personality traits: https://news.ycombinator.com/item?id=17685733 https://news.ycombinator.com/item?id=17685733 I would file this under 3. Moral Disengagement - A generalized cognitive orientation to the world that differentiates individuals' thinking in a way that powerfully affects unethical behavior.

iamleppert 8y ago

I was just starting out in my career and was more pre-occupied with the task at hand to have enough mental headspace to contemplate the full picture of what was happening. The system itself was designed by traditional software means: trial and error, trying to see what worked and didn’t, usually trying to maximize some KPI at the company. There was also the thought that punishment and reward should be applied fairly and be “data driven”; take the bias and human factor out of decisions and implement performance management in a predicable, deterministic and transparent way. The problems arose when people realized that the same controls and abilities to instill this equity and “fairness” provided the platform to enable wide scale exploitation in the other direction. Want more profit? Change a variable that was a single number whose action would cause great stress for many people, but would produce a desired result. The goals and metrics would always start out resonable but eventually would cross a certain point, and once they did, there was no going back to the way things were. Draw your own conclusions to the similarities in this system to our modern day technical web triumphs.

rrcaptain 8y ago
It's really depressing how capitalism has taken automation and data and bastardized it into a toxic weapon.