7 ms·
> And that, folks, is why we as engineers and hackers have a moral duty to be very selective in the types of work we take on and for whom. Computer yechnology
by marvin 13y ago
> And that, folks, is why we as engineers and hackers have a moral duty to be very selective in the types of work we take on and for whom.
Computer yechnology work is still a wild west. Other fields which have similar ethical considerations - medicine, chemistry, engineering - have ethics courses which make it clear to the graduates that their professional decisions have consequences.
There is no such standard for computer people. It still baffles me that people would voluntarily work at implementing something like PRISM - I would literally quit my job over something like this; similar considerations have popped up in the research I do for my Masters thesis - but ethics is definitely something we would talk more about.
- agwa 13y agoYet there is no shortage of doctors willing to assist in torture of prisoners at Guantanamo, assist in executions of prisoners, or conduct non-consensual and medically unnecessary procedures (inserting catheters, rectal examinations, etc.) upon demand of the police. The problem isn't lack of ethics courses but a fundamental flaw in human nature, which is that people easily forget all their personal ethics when someone in authority (whether their boss or the police) tells them to do something.
- onosendai 13y ago> people easily forget all their personal ethics when someone in authority (whether their boss or the police) tells them to do something. I don't know whether you're aware of this or not, but you just described a famous experiment which essentially arrived at the same conclusion. It's worth a read for those not familiar with it. https://en.wikipedia.org/wiki/Milgram_experiment https://en.wikipedia.org/wiki/Milgram_experiment
- Crito 13y agoI haven't looked into it myself, but I did hear that there is an alternative interpretation of that experiment. Apparently the test subjects did as they were asked, up until the very last prompt used, which was the strongest. Less a request and more a demand. As soon as the prompts stopped appealing to the test subjects desire to further science or not fuck up the experiment, but began making demands, the test subjects almost universally balked. This interpretation isn't really any less disturbing, since it suggests that people will commit atrocities and simultaneously believe that they are doing the right thing. Perhaps it also says something about the general public's trust of, not necessarily authority figures, but rather scientists. Reading up on this has been on my back-burner for a while, this interpretation might be well considered and discarded, but I thought I'd throw it out there. I believe this is related: http://www.is.wayne.edu/MNISSANI/PAGEPUB/Milgram.htm http://www.is.wayne.edu/MNISSANI/PAGEPUB/Milgram.htm
- etiam 13y agoI know of no better or more accessible review of the psychology of evil than "The Lucifer effect" (2007) by Philip Zimbardo, a psychologist who, in spite of a long and illustrious career, probably is still best known for the infamous Stanford prison study. The companion website to the book is still up on http://www.lucifereffect.com/ http://www.lucifereffect.com/ Zimbardo has stated that he is still pained at the memories of the Stanford prison study, which had to be aborted prematurely because the vile characteristics it exposed in normal, healthy participants was going out of control. He reluctantly revisited the study in detail because he saw striking parallels to the torture and humiliation in the Abu Ghraib prison under US occupation. While he's at it he also reviews much other research into the psychology of authoritarianism, obedience, dehumanization and other facets of social psychology that add up to what may very reasonably be called Evil. I might add that organizations like the CIA are well aware of these principles, and appears to have been working on ways of weaponizing them. It is my opinion that they are using them to corrupt individuals and whole social environments when that suits their ends. In fact, in spite of "The Lucifer Effect" being a fascinating read from beginning to end, one of the parts I found most interesting was the evidence indicating that US intelligence actively created the psychological conditions for the abuses at Abu Ghraib. Highly recommended reading, and very relevant for much of the potential consequences of mass surveillance.
- lazyjones 13y ago> people easily forget all their personal ethics when someone in authority (whether their boss or the police) tells them to do something. Now if we could get some prominent figures in tech with some authority to get more involved in these matters, but no - they're all a bit too comfortable in their well-paid, US-based jobs to risk anything.
- Crito 13y agoI believe various states in the US have found it difficult to find staff and drugs for lethal injection. Perhaps the difference is if the doctors/nurses/vendors think that they can get away with it, not attracting the public's attention.
- QuasarLogic 13y agohttp://en.wikipedia.org/wiki/Eichmann_in_Jerusalem#The_banality_of_evil http://en.wikipedia.org/wiki/Eichmann_in_Jerusalem#The_banal...
- runn1ng 13y agoIt's way more ambiguous. PRISM people thought they were doing right. All NSA - based on all the official and unofficial responses - think they are doing right. That they help building a safer America. That they have to sacrifice privacy for safety. Etc, etc. I don't think NSA people are evil. They believe that the things they do are for good.
- stackcollision 13y ago"Your enemy is never a villain in his own eyes. Keep this in mind; it may offer a way to make him your friend. If not, you can kill him without hate — and quickly." -Heinlein
- marvin 13y agoI get a very icky feeling when I read academic papers that explore how to mine network traffic for information on how to discover terrorists. And sure enough, when reading the funding information, the money for the projects came straight from the DHS. This icky feeling was there long before I heard about Prism. You don't need an ethical guidebook to consider the implications of the work you're doing. I don't think lack of explicit instructions is an excuse. I don't want to invoke Godwin just yet...but suffice to say, a lot of bad things have been done in the name of good. And there were people in the loop who had the ability to see what was going on.
- Amadou 13y agoI don't think NSA people are evil. They believe that the things they do are for good. Evil like in the movies doesn't exist. In the real-world evil is simply a lack of perspective (some would call it empathy instead). No one ever wakes up and decides they want to be a villain. They always have some sort of logic that rationalizes their actions as being reasonable if not outright good. The more they act on that lack of perspective the greater the evil they perpetrate.
- infectoid 13y agoIndeed. But also imagine that as an engineer on any secret NSA project, being shown real information regarding atrocities that have never been publicised and knowing that you can help to make the world a better place. You would have the passion and motivations to do these things and have meaning in your life and work. Evil in this case is an aggregation of actions by many parties, not individuals and is rarely committed by the tool makers. However, it's always justified by the perpetrators.
- qq66 13y agoPrism may have been built by people holding their noses for the paycheck, but it could also have been built by people who genuinely believe that its social benefits outweigh its costs. Even if only 0.1% of software professionals believe this, it's enough to staff the project.
- latj 13y agoI agree. Just look at who Palantir hires. I imagine its a pretty small number of nose-holders and more often people who think they are doing good and people who think spying is a necessary evil so lets put fine grained controls on it.
- toyg 13y agoI'm pretty sure there's also a good number of people who actually enjoy the craft. How many geeks do you know who wouldn't enjoy the power-trip? From LOVEINT to immature Googlers, there's an abundance of literature on the joys of doing evil.
- lwhalen 13y agoSlightly OT, but I remember feeling QUITE shocked when I found that one of my favorite bloggers, Rands (aka Michael Lopp), was some sort of honcho over at Palantir. He's written some good stuff on management of 'geeks' and high-performance people in general, it was a sad day when I found out about his employer and removed him from my RSS feed.
- adamnemecek 13y agoIs it possible that the development of these technologies is compartmentalized? And the single engineers only know the part that they are supposed to work on but don't really understand what is the larger project that their slice will be a part of? It would probably be hard to pull this development methodology off.
- lhc- 13y agoWhile in school for Computer Engineering (2002-2007 or so), I actually had two ethics classes, one specifically for Engineering and one for Information ethics. I dont know how standard that is, but all the engineering disciplines had to take the Engineering one and computer scienc-y students had to take Info ethics.
- SheepSlapper 13y agoSame here (2006-2011), and it was actually pretty fun to talk about the ethical issues that can arise in our profession(s).
- cbrauchli 13y agoInteresting. I'd never heard of this, but it makes sense. Whereabouts did you go to school?
- ujeezy 13y agoI took a couple engineering ethics classes at the University of Virginia (class of '06), and found them valuable and thought-provoking. The curriculum was called "Science, Technology, and Society". More here: http://en.wikipedia.org/wiki/Science,_technology_and_society http://en.wikipedia.org/wiki/Science,_technology_and_society
- lhc- 13y agoDayton, Ohio. We talked about a lot of interesting topics, like Whistleblowing (particularly relevant nowadays) and what sort of ethical obligations you have, in addition to stuff like "dont cut corners, dont be lazy, etc", the normal engineering things.
- Crito 13y agoThe extent of computer ethics to me was "here is therac-25, you have an ethical responsibility to do your job well. here are various laws, don't break them." There was never talk of "consider the ramifications of your project, assuming it works".
- paulhauggis 13y ago
- SheepSlapper 13y agoIEEE Code of Ethics: http://www.ieee.org/about/corporate/governance/p7-8.html http://www.ieee.org/about/corporate/governance/p7-8.html ACM Code of Ethics: http://www.acm.org/about/code-of-ethics http://www.acm.org/about/code-of-ethics So, the standards exist but people choose to ignore them. Which, I guess since you're not forced to be a member of either professional society as a "computer person", isn't all that unreasonable. But personal ethics should win out in these situations without someone else having to tell you that it's wrong.
- nl 13y agoI took a look at the ACM Code of Ethics, and I don't see anything that would be violated by working on PRISM. Specifically: 1.7 Respect the privacy of others. .. User data observed during the normal duties of system operation and maintenance must be treated with strictest confidentiality, except in cases where it is evidence for the violation of law, organizational regulations, or this Code. In these cases, the nature or contents of that information must be disclosed only to proper authorities. That specifically allows the kind of work done by PRISM.
- SheepSlapper 13y agoMaybe 2.7 or 3.1 of the ACM code? 2.7 Improve public understanding of computing and its consequences. Computing professionals have a responsibility to share technical knowledge with the public by encouraging understanding of computing, including the impacts of computer systems and their limitations. This imperative implies an obligation to counter any false views related to computing. 3.1 Articulate social responsibilities of members of an organizational unit and encourage full acceptance of those responsibilities. Because organizations of all kinds have impacts on the public, they must accept responsibilities to society. Organizational procedures and attitudes oriented toward quality and the welfare of society will reduce harm to members of the public, thereby serving public interest and fulfilling social responsibility. Therefore,organizational leaders must encourage full participation in meeting social responsibilities as well as quality performance. Or number one on IEEE? 1. to accept responsibility in making decisions consistent with the safety, health, and welfare of the public, and to disclose promptly factors that might endanger the public or the environment; Given, what's "best for the public" is up for debate (at least according to the people doing the collecting).
- rickdale 13y agoI'll never forget when I was taking CS in high school and we were forced out of the lab for a week and had class in the library. Our teacher spent the week teaching computer programming ethics. We talked about a lot of stuff, as you can probably imagine, but I was glad to have someone show me what he believed a good path to take with development was. In the end we all had to write a paper that he graded pretty damn aggressively for a cs class. Thats the only time in all my cs education that computer programming ethics in this context came up. Definitely agree it deserves more conversation.
- ars_technician 13y ago>It still baffles me that people would voluntarily work at implementing something like PRISM It's almost like not everyone in the world shares your political views on privacy rights... Seriously, you are baffled by this? The majority of the population doesn't value privacy at all.
- rbanffy 13y ago> There is no such standard for computer people. That does not make our decisions inconsequential. And yes, we should talk a lot more about ethics in compsci courses. As for people who work on things like PRISM, it often happen in small steps, one ethical shade of gray at a time. When you get to a PRISM, your worldview is so completely distorted, you think it's a natural thing to do.
- googlemployee1 13y agoTo be completely honest, I love doing work on systems like PRISM. I don't want to be spied on, but working with big data (and some of the most interesting, at that) is a ton of fun. Yes, it's hypocritical, but I know most people have never had the opportunity to do so, making it hard for them to see both sides of it. The people who work on these systems like doing it, don't kid yourself.