9 ms·

Don't Get Distracted (2017)

206 points by Kerrick 2y ago

glitchc 2y ago
So when can we start a guild of Software Engineers?
- jerf 2y ago
  That won't solve this problem. The military and government will do what is necessary to make sure the Guild of Software Engineers is officially completely fine with military applications. This is something that only individuals can do. And there will always be individuals willing to write this code, and that's not anything that can be solved by any number of Guilds either. The military will pay anything it takes to make it worth working outside of the Guild's rules.
  - AlotOfReading 2y ago
    Being able to say no with legal force is helpful even outside defense. I've discussed one of my previous roles unintentionally building an employee surveillance tool, and another codebase I contributed to is currently on the front page as a distant result of hurting people.
  - glitchc 2y ago
    We can start with private enterprise, with ad tech, consumer data harvesting/surveillance and dark patterns targeting engagement and addiction. The military can be the last bastion to tackle once the guild has strength and can bargain to enforce guardrails. I'm looking at how SAG is so effective at navigating a multi-billion dollar entertainment industry filled with behemoths like Disney. If they can do it, we can too. At this point, the actors are looking smarter than software devs.
- weberer 2y ago
  Software engineers are free to join IEEE.
  - cmrdporcupine 2y ago
    Groups like IEEE and the ACM don't do what professional associations / unions / "guilds" do... help regulate / moderate employer behaviour in a given industry. Of course those associations are always a mixed bag, often fostering some mediocrity or being themselves corrupt. But definitely feel like there's some aspect of that missing in our sector.
  - glitchc 2y ago
    Those organizations don't have teeth, neither does the state-based PE except to regulate members. Where is the organization that regulates employers?
- duxup 2y ago
  I really don't want to get into the meta of what that guild "should" be doing / telling other folks who they should work for. My local teacher's union has some affinity groups and one group decided to invite a speaker who in the past has had some very strong words for people of a particular religion and point of view. Needless to say it is a mess of a problem. I'm not sure I'd work on a military project, but I also wouldn't be comfortable with a guild telling me that I or anyone else can't / shouldn't.
  - glitchc 2y ago
    Engineering already has a professional association that tells its members what can and cannot be done. A guild will have the power to also provide limits on what an employer can or cannot do, set minimum/floor rates for standard services and generally improve the labour conditions for its employees. That's exactly what SAG and AFTRA achieve, and it's worth looking into how they work.
- bee_rider 2y ago
  There are already professional societies for other engineers (IEEE for example), but I think there are engineers that work for military applications. It is a complicated topic I think; on one hand lots of people don’t want to work in those types of applications. On the other, we have a decision-making process for how violence is applied. If we are going to start inserting more veto points in there (which I think is a good idea), I don’t know that “has some rare technical skill” would the first qualifying characteristic I’d look at, right?
nemomarx 2y ago
This is a really interesting and thoughtful piece, although I think the repetition comes a little too often. Still the example of getting focused on engineering problems and not what the code will be used for lands well.
BiteCode_dev 2y ago
That's pretty much how all people working on tracking features at GAFAMS, all people working on algo manipulating people's behavior on social networks and shopping sites, and incidentally how people working on the manhattan project, moved. They get into the interesting problems and miss the forest for the tree. Plus it's fun to be among smart peers, and well-paid. The author woke up from this, but many do it for their whole life and see no problem with it. That's why I rejected interviews from Google and Facebook. Because I knew I would do the same. It's too tempting.
- cmrdporcupine 2y ago
  Yep, I -- an always very political, lefty, very critical person -- went down this trap. Nerd-sniped myself. Ended up working in ad-tech for some years and making pretty good money at it, and eventually ending (through acquisition) at Google because of exactly this. Because of intellectual interesting things... because of opportunity to do those things... Copious quantities of impression data? Get to do big-data-ish things? Play with things like Cassandra etc (back then kinda interesting), k-means clustering and fun stats stuff and eventually ML?, high throughput & low latency transaction processing... shaving milliseconds off here and there... and lots of money in the sector... to hire devs, to do these things... which were at least back then somewhat intellectually challenging... Especially for people not in the Bay Area, opportunities like this were/are hard to come by... But then once you take a step back and look at how the sausage is really made, you start to feel icky. Granted, that was ad-tech in 2010ish time-frame. It became much much worse. Luckily within Google I was able to transfer out of ads, and into things that seemed on the surface much less icky (consumer hardware, etc.) But the thing is... no matter what you're doing at Google, that project is funded by ad-tech... So...
  - AnimalMuppet 2y ago
    If you're using ad-tech money to do something actually good, does it matter that the money was made from ad tech?
    - cmrdporcupine 2y ago
iamwil 2y ago
I don't know. On one hand, it is true we are the last line of defense. I do wish more of us said no to putting tracking on websites, among other things. But on the other hand, a random software engineer's sense of ethics might be different than you. Do we want activist software engineers? I mean, field of study changes perspective. Recently, there was that backlash against a release of a 1-mil post dataset of bluesky users by some people on bluesky. https://eugeneyan.com/writing/anti/ https://eugeneyan.com/writing/anti/ I personally didn't think it was a big deal because the API and dataset is already public. And also if your text isn't available to LLMs, then your values are effectively invisible to the future, as society depends more and more on LLMs. To me, I think it's better that you get your thoughts out there into LLMs. As more of society relies on LLMs, culture wars will be fought not just on the internet, but in the embedding space of LLMs. So, given that, would these people want me to decide for them? I dunno. Just as journalism lost the trust of the public by deliberately taking an activist stance, so too would engineering lose public trust by deliberately taking an activist stance. (You may say tech already lost public trust, but I think of tech as the industry and engineering as the profession).
- vunderba 2y ago
  Maybe we do need a Engineer's Hippocratic Oath, and a means to police our own. Go look at the latest "Ask HN" thread - there's two projects proudly trumpeting their ability to allow applicants to cheat on their interviews. On the FAQ of one of the apps, a common question apparently is if it will work during proctored exams. https://news.ycombinator.com/item?id=42375440 https://news.ycombinator.com/item?id=42375440 https://news.ycombinator.com/item?id=42383928 https://news.ycombinator.com/item?id=42383928 With this kind of loose approach to morality, is it really any surprise that the world is filled with devs who build involuntarily telemetry, spyware, etc.?
  - golol 2y ago
    It just doesn't make sense. So it is immoral for engineers to design weapons? That's like saying it's immoral to butcher an animal. Sure go ahead, ban butchering animals in your society so everyone can feel good about themselves while they import their meat and thereby outsource the butchering instead. Or you say that everyone just stops eating meat? Fine that works but here the analogy breaks, because everyone won't just stop needing weapons. What I can agree with is that placing a moral cost on engineering designing weapons in some way increases the actual labor cost of weapon R&D, I guess.
eykanal 2y ago
It's worth mentioning that this really isn't programming-specific, or even engineering-specific, at all. The exact same story applies to the guy doing finance, or marketing, or project management, or legal work, or whatever. This is basically a reminder that your role as a cog in the wheel is to make the wheel move, and you shouldn't forget what the wheel is doing. That said, working for a US military contractor while not being supportive of the US military mission is kind of a silly thing to do.
- nemomarx 2y ago
  I think it's fair that you might think mostly about the military work overseas and not about it being licensed to local police or used to track citizens or so on. Wasn't a lot of the Snowden stuff that tools intended to be used on foreign enemies would be used at home?
  - ForHackernews 2y ago
    From my understanding (with no special access or insider info), it's unclear how much it was used "at home" (against US citizens). Certainly the collection infrastructure was in place, but there were technical and procedural safeguards that were intended to filter out US citizens' data, and analysts were not supposed to access US citizen data without a search warrant from a judge. It's interesting to me that many people who work for giant tech companies that carry out indiscriminate data collection feel confident in their employer's internal processes and controls, but don't extend that same optimism when it's the government that pinky-swears they aren't going to peek at anything they shouldn't.
    - wutwutwat 2y ago
      > there were technical and procedural safeguards that were intended to filter out US citizens' data, and analysts were not supposed to access US citizen data without a search warrant from a judge. Nope > The NSA has built a surveillance network that has the capacity to reach roughly 75% of all U.S. Internet traffic. > An internal NSA audit from May 2012 identified 2776 incidents i.e. violations of the rules or court orders for surveillance of Americans and foreign targets in the U.S. in the period from April 2011 through March 2012, while U.S. officials stressed that any mistakes are not intentional. > The FISA Court that is supposed to provide critical oversight of the U.S. government's vast spying programs has limited ability to do so and it must trust the government to report when it improperly spies on Americans. > A legal opinion declassified on August 21, 2013, revealed that the NSA intercepted for three years as many as 56,000 electronic communications a year of Americans not suspected of having links to terrorism, before FISA court that oversees surveillance found the operation unconstitutional in 2011. https://en.m.wikipedia.org/wiki/2010s_global_surveillance_di https://en.m.wikipedia.org/wiki/2010s_global_surveillance_di... The kicker to me: > it must trust the government to report when it improperly spies on Americans The courts told them to watch themselves and to self incriminate if they do crimez. An honor system. Which naturally, they did not snitch on themselves to the courts, because why would they. Also let's not forget that they can make all the illegal surveillance legal in an instant just by labeling any person a "terrorist"
hprotagonist 2y ago
Chris: So, I talked to him. Mitch: You did? Chris: Yeah, and he used to be the number one stud around here in the 70’s. (whispers) Smarter than you and me put together. Mitch: So what happened? Did he crack? Chris: Yes, Mitch. He cracked, severely. Mitch: Why? Chris: He loved his work. Mitch: Well what’s wrong with that. Chris: There’s nothing wrong with that, but that’s all he did. He loved solving problems, he loved coming up with the answers. But, he thought that the answers were the answer for everything. Wrong. All Science no Philosophy. So then one day someone tells him that the stuff he’s making was killing people. Mitch: So what’s your point? Are you saying I’m going to end up in a steam tunnel? Chris: Yeah, I am.
gregw2 2y ago
Pardon me another story. In the early 2000s, I wrote software for a startup where the founders and initial equity-holders had pledged 10% of the founding stock to a certain charity. This was part of the marketing approach for the company- help us succeed and it benefits others (without affecting the competitiveness of our cost/pricing structure of our services in any particular tangible way.) There were some weaknesses in the model of mixing charity with startup-mindset (the model later changed to 10% of profits and we actually brought the charity work in-house eventually), but as a lead engineer, I actually found it quite helpful in pushing back on questionable (in my eyes) things the CEO was otherwise tempted to pursue and asking me to build (the CEO not being initially a technical guy and having any developed sense of technical ethics around spam, dark web practices, etc). If we did something really unethical (and were caught), it would actually hurt the charity we were also all trying to help and undermine part of our company's core differentiation. If you are a technical founder and bake this sort of charity commitment into your startup and its marketing on day 1, you can help people at all levels of the firm develop a more ethical culture than the average startup which faces all sorts of (often stupid) temptations.
- ebiester 2y ago
  On the other hand, I've also seen companies that start saying that it's okay to cut corners and drive people into the ground because it's for the "greater good." I think SBF took advantage of this in many ways as well.
  - gregw2 2y ago
    Fair point. There is no substitute for keeping your eyes open.
uludag 2y ago
Great article! This was similar to my experience in ad-tech, though obviously more mundane. It's almost like people had to create artificial problems to avoid thinking about what they're actually building. Even language filled with euphemisms and acronyms was employed, I suppose to put the person at a distance from what they're building. Even if there's nothing morally objectionable, like other places I've worked at, the product is so worthless that people would enter the same patterns of behavior.
- troyvit 2y ago
  Exactly the same, down to the industry! My job was as a simple force multiplyer for account managers and sales folks, automating tasks to allow them to scale their projects and help more clients. It was all cool helping them get more ads out the door for Toyota and cool authors, weird when it started being for "One weird trick" ads, downright wtf when I started helping boost SERP pages and finally once we started promoting gambling apps I just couldn't take it anymore. It was great money. I miss that. And helping all these very kind, friendly people make numbers felt useful. But man, once I finally got un-distracted (after ten years?!) I had to get out of there.
mettamage 2y ago
I think a course would be a helpful good step. I remember watching Yale’s introduction to ethics class. Something like that but then applied to software engineers specifically
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
lmeyerov 2y ago
Half our work is on bringing GPUs/graphs/AI/etc to Serious Problems like cyber, misinfo, fraud, crime, etc, and a few things I've come to: * If you're unsure, quit * It's fine to distrust the system, and as with most of the population, do something else * If you do work on a Serious Problem, decide what your line in the sand is, and ensure that aligns with the organizations you work for & with * Constantly reevaluate yourself, the people, and the work Extremes like "don't do things if they can ever go wrong" would suggest the only answer is to do nothing. That doesn't work for most things. Likewise, I don't like painting people as unethical just because they have evolved different belief systems or, quite often, a more nuanced & evolved view of something difficult. Thankfully, I find most people in the US, and in many other countries, to try to do the right thing on average. This is even more true of people grinding on Serious Problems. So it often comes down to personal responsibility and awareness for whatever you care about, keeping the alignment with those around you... and leaving when that stops working. The AI & robotics booms have made this stuff even more important. We view it as a chance to do good, and with more accountability & quality than when those with less scruples & ability do so. But agreed, we pick where we work, and ensure new staff agree on where we drew our own lines in the sand.
- cmrdporcupine 2y ago
  Agree with your points overall, but one of the specifically worrying things about right now is the massive downswing in the tech job market means that -- especially for those of us outside of the Bay Area -- your first recommendation: "quit" has become a lot lot harder than it was 2-3 years ago. e.g. I work fulltime in Rust, and love it. I look at the job postings that emphasize Rust ... and 80% of them are crypto/web3 companies. I won't work for those companies because of my personal beliefs about "crypto"... But if I were to lose my pretty decent job right now... and the bills started to pile up and my family was in financial danger... The argument becomes more difficult.
  - lmeyerov 2y ago
    Agreed!
rlt 2y ago
Everyone's got their own ethical standards. This guy won't touch anything related to the military. Palmer Luckey is part of society's "warrior class that is enthused and excited about enacting violence on others in pursuit of good aims" [1]. I'm somewhere in between. 1. https://techcrunch.com/2024/10/01/palmer-luckey-every-country-needs-a-warrior-class-excited-to-enact-violence-on-others-in-pursuit-of-good-aims/ https://techcrunch.com/2024/10/01/palmer-luckey-every-countr...
amoss 2y ago
In a different place a long time ago we were approached by a company that wanted a collaboration. They had a very specific project in mind, with requirements already set. It was a humanitarian project to "find earthquake survivors". If you read the requirements in a very positive way it was an ethically clean project. But the company is a major defense contractor and it was very unusual for them to be involved in humanitarian work. So it made sense to question the requirements after a less generous reading. Detecting mostly occluded humans in a highly cluttered and destroyed urban environment has other use cases. They got as far as "highlighting the survivors with a laser designator" before I started to ask some very pointed questions and start discussing the ethical considerations of us helping to build a system to kill people. I was asked to leave. After that meeting the potential collaboration was rejected and it never came up again. The company continued shopping around other places to get help building their "earthquake rescue system". Never get distracted. Always question the purpose of work. Consider ethics. Getting another job can be a pain. Living with knowing what you helped to build would be much harder.
- duxup 2y ago
  I'm a little surprised that a defense contractor would bother with that much awkward subterfuge or even farm out a core part of their product to some random other company. The handful of US based defense contractors I've had contact with were VERY up front about what they develop. I think they more so wanted you to know so everyone was on the same page.
  - cl0ckt0wer 2y ago
    They probably already have a complete system. They're just looking for new revenue sources.
  - pc86 2y ago
    I'll bet my next paycheck the above scenario never happened, or this company was less "major defense contractor" and more "VC-funded SaaS-app defense contractor wannabe." Lockheed Martin doesn't need to trick anyone into working for them, and that includes software developers. If this was a real thing that actually happened, everyone would need a security clearance anyway.
Liquix 2y ago
FWIW the Fasten link (goes to https://fasten.com/us/cities/austin https://fasten.com/us/cities/austin) is broken
- caleb_thompson 2y ago
  Thanks, they went out of business in 2018. I've added a note to that effect and removed the link.
m_ke 2y ago
I used to work at one of the first deep learning image recognition startups, it was all going great, we raised a ton of money, hired a lot of people and built some amazing technology. The founder was on the covers of fancy business magazines and it looked like we were the next big thing. Then revenue growth stalled and we shifted to enterprise consulting, which caused a lot of the early employees to quit (including myself). A few months after I quit they took on the Maven project that Google and Amazon employees protested against and in no time after that they had an office near DC with a bunch of ex generals on their staff. Now they work on secret projects that nobody still working there can tell you anything about. Moral of the story, you can be a well meaning person and work on cool tech but you have no control over how it will get used in the future. The same thing that happened in CV will happen with all of these LLM startups once growth stalls and they're unable to raise another up round to cover their huge expenses.
raincole 2y ago
I feel we should attempt to hold decision-makers accountable instead of blaming the field workers for unethical practices.
0898 2y ago
It's amazing how many tech people, like this author in this piece, still believe that their phone is constantly listening to you to target ads.
- ddulaney 2y ago
  I don’t think it’s that surprising. Here’s two true statements: I don’t trust my phone on a fundamental level. I also have mentioned things out loud and gotten ads afterward about them. On the one hand, those two statements absolutely don’t prove that my phone is listening. Just because I don’t trust my phone doesn’t mean that it’s being used maliciously, and the ads I’ve seen might just be a coincidence. On the other hand, those two statements together are definitely suggestive. They feel suspicious. It would be so easy for Apple/Google/Samsung to secretly listen and make a boatload of money by telling me a lie. Lord knows they’ve lied about other things. So yeah, balance of evidence is that phones probably don’t do that. But don’t be surprised when people suspect that they do.
  - nuancebydefault 2y ago
    Your phone doesn't need to listen to you to know your interests. The ad algos are smart enough to figure it out from your browsing behavior. Also, it happened a few times I talked about some thing and an hour later it was talked about on broadcast TV. People over-fit on finding patterns.
- spencerflem 2y ago
  They're bragging about it at least: https://www.documentcloud.org/documents/24224884-how-voice-data-works-and-how-you-can-use-it-in-your-business-cmg-local-solutions/?ref=404media.co https://www.documentcloud.org/documents/24224884-how-voice-d... https://www.techdirt.com/2024/08/29/cox-caught-again-bragging-it-spies-on-users-with-embedded-device-microphones-to-sell-ads/ https://www.techdirt.com/2024/08/29/cox-caught-again-braggin...
shireboy 2y ago
I remember an anecdote about this topic where someone (I think it was Douglas Engelbart?) was giving another a hard time about their military software being used to kill people. At which point the other person mentioned their software used a mouse (which Engelbart invented). Obviously a complex topic with lots of considerations that I venture won't be solved in this HN thread, but good to discuss and think about.
- AlotOfReading 2y ago
  There's a pretty meaningful distinction between something (relatively) neutral that's occasionally used to support unethical purposes than something that can only be used for unethical purposes, plus shades of gray in-between. I don't think there's a reasonable argument that e.g. Keurig engineers need to grapple with the ethical dilemmas of murderers and terrorists using their products for example.
sjbremer 2y ago
[snark removed]
- draw_down 2y ago
  [dead]
- tines 2y ago
  Please keep to the HN guidelines: > Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. > Comments should get more thoughtful and substantive, not less, as a topic gets more divisive. > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.
caleb_thompson 2y ago
Author here. I gave this as a talk at several conferences through 2017. I didn't realize that the video embed was broken until someone pointed out that this was making the rounds again, so I've just fixed that. If you've already looked, the video is now available. The content is more or less the same as the written version so don't feel like you need to watch it if you've already read the post.
- vaylian 2y ago
  Thanks. I really appreciate the transcript! I'm still reading it. This is a really good and thoughtful talk. Thanks for sharing it!
- Teever 2y ago
  Do you believe that there is sufficient moral hazard for Ukrainians working in their domestic drone industry to dissuade them from doing that work? Why or why not? What is different about that situation from the situation you described in your talk? What factors would have to change in the global balance of power for you to consider building systems that kill for American companies?
  - caleb_thompson 2y ago
    Morality is personal. In any instance, we should think through first-, second-, and subsequent-order consequences of our actions and consider whether we're comfortable with those potential outcomes and whether those consequences balance one another. I won't take the bait on defining the differences between wartime and peacetime work.
    - myk9001 2y ago
      I don't know if the gp's question is intended as a bait or not. However, it describes a real scenario playing out literally as we speak. While I absolutely don't mean this in a confrontational manner, if your ethical framework doesn't provide you with means to address a real-world ongoing situation, what's it even useful for?
    - 2y ago
geetee 2y ago
It's nice that the author can hold opinions and write a blog post like this and not be disappeared. I wonder why that's possible...
- sakjur 2y ago
  Though oddly Boeing whistleblowers keep dying. Hamilton made a much better case than I could for a way to view the military as a necessary evil neither to be loved nor feared in Federalist no. 8.
josephcsible 2y ago
Where do you draw the line? Is it wrong to work on food preservation technology, because it gets used to make MREs that get fed to soldiers who kill people?
- tines 2y ago
  I'm not sure whether you're asking this question in good faith, but I think as with any moral spectrum, you draw the line where your best reflective judgment tells you to?
- wutwutwat 2y ago
  No. One of them is feeding people, all people. One of them is actively working on systems designed to kill people and only kill people
duxup 2y ago
I'm not sure if I would be comfortable working for the military / a contractor directly, but I'm more ambivalent than unsure. Certainly each individual has to make their call. I do wonder about this whole premise. Let's say your country is under attack by an oppressive neighbor or known bad terrorist organization? Would people feel differently? I get the same weapon could be used differently / against less bad targets, but it doesn't change that they can also be used to defend / attack people who genuinely want to do you harm. I do not see this as a cut and dry call.
- bitbasher 2y ago
  Yes-- and if your enemies are making the same tech you don't want to fall behind. That's also why we created the atomic bomb, so I'm not sure how to feel about it.
delta_p_delta_x 2y ago
All the software engineers in places with mandatory conscription like South Korea, Singapore, Israel, Finland, Switzerland are in shambles. Someone is complaining about writing software that could be used in implementations which kill people, whereas men in these countries straight-up don camouflage uniforms, helmets, and armour. They are given rifles, and are taught how to shoot and kill people, and sometimes even get to kill people. Cut out the middlemen and all the abstraction, I suppose; program in C and assembly rather than in Python.
- tines 2y ago
  I don't really see the incongruity here; somebody doesn't like helping people kill people, while that is mandated in other countries. What's your point?
- numpad0 2y ago
  The moral of the story is not "always be cynical and ridicule defense guys".
mediumsmart 2y ago
absolutely, you need focus when you work for the department of war
tristor 2y ago
Great article, and very important things to consider. As a practice, I take my holidays at the end of each year to consider my career, my life, and my worldview. A long holiday is an opportunity to refocus from work to introspection, and allows you to find comfort with your place in the world. The primary things I ask myself near the end of this introspection period are: 1. Is the work that I am doing advancing my personal goals? 2. Is the way I spend my time good for my mental health, my physical health, and my family? 3. Is the work that I do aligned with my ethical worldview? 4. Am I being renumerated appropriately aligned to the value that I am bringing? 5. Am I still learning something each day? 6. Are my family happy with the outcome of this year and the way I contributed to it through my work and otherwise? If I get a no to any one of these questions, I start looking for a new job. I've moved across country and across the world, and I've been at companies for a long time and a short time throughout my career (that's nearing 20 years at this point). I've switched industries and areas of technical focus many times, and I've even switched entire career paths by simply asking myself these questions after a relatively short period of introspection. I encourage everyone to consider the ethics of their work, but also to consider regularly whether or not your work is advancing your goals, making you happy, making you healthy, and supporting your family in the way they need to be happy and healthy. Ethics is a core part of this. If you are doing work you don't agree with for purposes you would not agree with, it can be traumatic to your own psyche and deeply affect both your health and happiness, including how you treat other people and especially those closest to you. Take some time, the holidays are coming.
martinjacobd 2y ago
Needs a [2017]. More substantively, my company is a contractor and has publicly committed to neither working on 'weapons' nor 'destructive financial systems.' Both terms are nebulous but we generally just decide en masse as a company (there are only 20 people) what does and doesn't count. I don't necessarily care for these particular lines in the sand but it's important to have some kind of process for determining and thinking about things, which I'm happy we do.
vhantz 2y ago
The reality is nobody is hired to decide why things are done. Under capitalism, the why is to make profits, to increase shareholder value, if you will. That's it. You don't plan what you don't control, you don't control what you don't own. For the vast majority (yes, including engineers), the only thing we own that is a part of the whole process of capitalist production is our ability to do some kind of work. When that ability is sold (for a wage/salary) to whoever possesses the resources to decide the why, we forego the right to decide how it's used. Just like when you sell someone a chair for example, you don't get to tell them where and how they can use that chair. You can refuse to get or do this or that job, but you need to have a job. At the end of the day, we all need food and shelter to stay alive and the bills aren't gonna pay themselves. So ultimately, someone—among the vast majority of us who don't have a choice—will also have to do the job you refuse. So we shouldn't delude ourselves into thinking we have some sort of "superpower". We don't. That is, not unless we can collectively withold our ability to work and force the hand of the minority that actually decides why things are done. A society where those who do the work decide why and how it's done, is a society where the working people own the resources. That is communism. That world is possible, but it won't build itself, you have to tirelessly fight for it. Look over at marxist dot com if you want to help out [1]. That doesn't mean that in the meantime we shouldn't strive to find meaningful work that fit with our values. We should, and whoever manages to find that good for you! You've been lucky (for now) but you won't be forever, and even if you are to be, the vast majority of people with pretty much the same skills and conditions as you, actually won't. We actually have to transform society if we want to secure that for ourselves and for everyone else. [1] https://marxist.com/about-us.htm https://marxist.com/about-us.htm
- vhantz 2y ago
  Don't get distracted by the word communism!
  - zer8k 2y ago
    Attributing "capitalism" to the increase of value to "shareholders" is a mischaracterization of capitalism. It's the only system that has worked for one singular reason: People want more value for more work. If I work 12 hours picking potatoes and have to give 3/4 of them "according to my need" I will refuse to work. If you create fake jobs like "bolt counter" to insure 100% employment so the means can justify the ends (as communism believes) you end up with a collapsing economic system. The current iteration of late-stage capitalism is not capitalism _at large_. It's feudalism clever disguised as an egalitarian economic system. The solution to this is not communism, it's undoing 100 years of stock-market oriented business design. We don't know what "late-stage" communism looks like simply because a revolution occurs several hundred years before it reaches that point. Not even the premiere implementers of communism, the Russians, could keep their system. It doesn't work. In any country, in any system, with any group of people larger than 4.
zahlman 2y ago
I skimmed through this because it was really obvious where it was going. > The unifying factor in all of the stories I’ve told is that a developer wrote the code that did these unethical or immoral things. As a profession, we have a superpower: we can make computers do things. We build tools, and ultimately some responsibility lies with us to think through how those tools will be used. No, it does not. It still boggles my mind that people actually seem to have this model of morality. If everyone thought that way, the world would not have nuclear power, to name just the most obvious example that comes to mind. Nobody is "responsible for" making it possible for others to do evil. That's their choice. (To expand a bit, because clearly not everyone shares my intuition: if this sort of moral failing can attach at one degree of separation, logically nothing prevents it from attaching at arbitrarily many degrees of separation. If I become culpable because my software "is used to kill people", then any person or company who enables me to write the software faces the same judgment. Vim is used to write software that is used to kill people. Do you really want to put that on the guy[0] who tried to get you to donate to humanitarian aid in Uganda?) (The same logic is used by activists to demand participation in various seemingly arbitrary corporate boycotts - and people I care about have been harassed because of it. The thing about "following the money" is that it goes literally everywhere.) [0]: https://en.wikipedia.org/wiki/Bram_Moolenaar https://en.wikipedia.org/wiki/Bram_Moolenaar
- wavemode 2y ago
  I think both things can be true. A person can take the position that developing something which will be used to directly kill someone is wrong, and thus refuse such jobs. A person can also take the position that software itself is morally neutral, and how it is used is the choice of the user. Both of these are perfectly valid lines of moral reasoning. Which one you choose is going to be a personal decision. Debating about which one is more "right" devolves into a philosophical discussion. Does it "boggle your mind" that some people choose to be vegetarians?
  - zahlman 2y ago
    The point is that "which will be" is load-bearing. The idea that someone would feel moral qualms about how software is used by the military is incongruous with signing up for the job in the first place. The military does in fact do a lot of things that are not killing people, and presumably could also find a use for the technology described that does not involve killing people. Like, say, locating allies for a rescue effort.
nuancebydefault 2y ago
This was 2017. Meanwhile the geo political situation has changed quite a bit in the world. Unfortunately, the ethics already feel outdated.
joemazerino 2y ago
China, Iran, Russia, ISIS and others would be happy to get their hands on this tech and kill you.
joelesler 2y ago
TL;DR: The author recounts their experience as an intern building software for the Department of Defense, unaware of its intended use. The software, designed to locate WiFi signals, utilized algorithms like R^2, Gaussian estimation, and Kalman Filters to improve accuracy and tracking capabilities. Despite the technical intrigue, the author acknowledges the software’s purpose was to aid in killing people, highlighting the ethical implications of their work. The author recounts a past experience working on a project for a Department of Defense contractor, where they were tasked with creating a tool to locate phones, ultimately realizing it was intended for targeting and potentially killing individuals. This experience, along with examples like a deceptive quiz and Uber’s “greyball” feature, highlights the potential for code to be used unethically or even illegally. The author emphasizes the importance of developers considering the ethical implications of their work, as software increasingly impacts various aspects of society. Developers have a responsibility to consider the potential misuse of their work, as unethical applications can have serious consequences. While there is no universal solution, developers should critically evaluate project requests, consider worst-case scenarios, and prioritize ethical considerations over deadlines. Ultimately, developers must decide whether to build a product, even if it has the potential for misuse, or to prioritize ethical concerns and potentially face consequences. Code can have serious consequences, even death.
myk9001 2y ago
I find it ironic that a US citizen gets to enjoy the privilege of making an ethical choice of whether to support their military thanks pricesely to the fact the US possesses the most advanced military force out there.
1970-01-01 2y ago
The undiscussed counterpoint is how a new weapon will also result in saving lives if used wisely. Nuclear stockpiling is a darn effective war deterrent, and has prevented an uncountable number of hot wars.
- wat10000 2y ago
  It’s too early to conclude that nuclear weapons save more lives than they take.
  - 1970-01-01 2y ago
    The boiled-down argument being 'Pax Americana' must end before nuclear stockpiling can be called an overall success?
    - wat10000 2y ago
      I’d say we need to reach the point where nuclear Armageddon is no longer possible. Even then, it’s possible we got lucky and nukes were actually more dangerous than not having them, but we’d at least be able to definitely say that their total kill count is only X, for whatever X turns out to be.
jjfrady31 2y ago
Long story short, the author lives peacefully and comfortably in a country defended by others and has ethical issues helping those who protect him. I don't see eye-to-eye with such people. I joined the military (the infantry) straight out of high school out of a sense of duty due to my upbringing. I knew what the job entailed but never thought about it much. It was peacetime, too, so that made not thinking about it easy. Then, the war on terror (9/11) happened, and the reality of the job blew up in my face (almost literally). It was a challenging experience to say the least, but I would do it all over again if given the chance.
- tines 2y ago
  To steelman the author's argument and as someone who's debated about this a lot with myself, what's your perspective on the argument that goes like this: "helping those who protect us" would be fine if that's all that those-who-protect-us did; but in addition, they go start unnecessary wars in foreign lands, kill innocent people, surveil their own population etc. Because they far overstep the reasonable bounds of keeping peace and defending the country, and because the "help" I'm giving them will go directly to that extra stuff that I abhor, I can't in good conscience help them. Would that argument help you to see eye-to-eye?
_zagj 2y ago
I wonder how many of the people back-patting the author and each other work for companies whose business models entail egregiously violating users' privacy or destroying the economic security of low-wage workers? How many work on exploitative AI projects--especially AI art--that rely on training data used without permission or compensation? Yet suddenly, when it comes to bettering the United States' and her allies' war-making capabilities, ethics become an issue? Thank god during the Cold War we had real tech companies, like Digital and HP, staffed by real engineers whose heads were screwed on straight and whose values weren't perverted.
NoGravitas 2y ago
> So how can we “carefully consider potential impacts”? Honestly, I don’t have any answers to this. I don’t think that there really is a universal answer yet, because if we had it I have to believe we’d not be building these dangerous pieces of software. So, there is no general answer to the problem, but you can always approximate it by asking: are you working for the US defense/intelligence industry or one of its contractors? If so, you are almost certainly building something harmful.

206 points by Kerrick 2y ago

glitchc 2y ago

So when can we start a guild of Software Engineers?

jerf 2y ago
That won't solve this problem. The military and government will do what is necessary to make sure the Guild of Software Engineers is officially completely fine with military applications. This is something that only individuals can do. And there will always be individuals willing to write this code, and that's not anything that can be solved by any number of Guilds either. The military will pay anything it takes to make it worth working outside of the Guild's rules.
- AlotOfReading 2y ago
  Being able to say no with legal force is helpful even outside defense. I've discussed one of my previous roles unintentionally building an employee surveillance tool, and another codebase I contributed to is currently on the front page as a distant result of hurting people.
- glitchc 2y ago
  We can start with private enterprise, with ad tech, consumer data harvesting/surveillance and dark patterns targeting engagement and addiction. The military can be the last bastion to tackle once the guild has strength and can bargain to enforce guardrails. I'm looking at how SAG is so effective at navigating a multi-billion dollar entertainment industry filled with behemoths like Disney. If they can do it, we can too. At this point, the actors are looking smarter than software devs.
weberer 2y ago
Software engineers are free to join IEEE.
- cmrdporcupine 2y ago
  Groups like IEEE and the ACM don't do what professional associations / unions / "guilds" do... help regulate / moderate employer behaviour in a given industry. Of course those associations are always a mixed bag, often fostering some mediocrity or being themselves corrupt. But definitely feel like there's some aspect of that missing in our sector.
- glitchc 2y ago
  Those organizations don't have teeth, neither does the state-based PE except to regulate members. Where is the organization that regulates employers?
duxup 2y ago
I really don't want to get into the meta of what that guild "should" be doing / telling other folks who they should work for. My local teacher's union has some affinity groups and one group decided to invite a speaker who in the past has had some very strong words for people of a particular religion and point of view. Needless to say it is a mess of a problem. I'm not sure I'd work on a military project, but I also wouldn't be comfortable with a guild telling me that I or anyone else can't / shouldn't.
- glitchc 2y ago
  Engineering already has a professional association that tells its members what can and cannot be done. A guild will have the power to also provide limits on what an employer can or cannot do, set minimum/floor rates for standard services and generally improve the labour conditions for its employees. That's exactly what SAG and AFTRA achieve, and it's worth looking into how they work.
bee_rider 2y ago
There are already professional societies for other engineers (IEEE for example), but I think there are engineers that work for military applications. It is a complicated topic I think; on one hand lots of people don’t want to work in those types of applications. On the other, we have a decision-making process for how violence is applied. If we are going to start inserting more veto points in there (which I think is a good idea), I don’t know that “has some rare technical skill” would the first qualifying characteristic I’d look at, right?

nemomarx 2y ago

This is a really interesting and thoughtful piece, although I think the repetition comes a little too often. Still the example of getting focused on engineering problems and not what the code will be used for lands well.

BiteCode_dev 2y ago

That's pretty much how all people working on tracking features at GAFAMS, all people working on algo manipulating people's behavior on social networks and shopping sites, and incidentally how people working on the manhattan project, moved. They get into the interesting problems and miss the forest for the tree. Plus it's fun to be among smart peers, and well-paid. The author woke up from this, but many do it for their whole life and see no problem with it. That's why I rejected interviews from Google and Facebook. Because I knew I would do the same. It's too tempting.

cmrdporcupine 2y ago
Yep, I -- an always very political, lefty, very critical person -- went down this trap. Nerd-sniped myself. Ended up working in ad-tech for some years and making pretty good money at it, and eventually ending (through acquisition) at Google because of exactly this. Because of intellectual interesting things... because of opportunity to do those things... Copious quantities of impression data? Get to do big-data-ish things? Play with things like Cassandra etc (back then kinda interesting), k-means clustering and fun stats stuff and eventually ML?, high throughput & low latency transaction processing... shaving milliseconds off here and there... and lots of money in the sector... to hire devs, to do these things... which were at least back then somewhat intellectually challenging... Especially for people not in the Bay Area, opportunities like this were/are hard to come by... But then once you take a step back and look at how the sausage is really made, you start to feel icky. Granted, that was ad-tech in 2010ish time-frame. It became much much worse. Luckily within Google I was able to transfer out of ads, and into things that seemed on the surface much less icky (consumer hardware, etc.) But the thing is... no matter what you're doing at Google, that project is funded by ad-tech... So...
- AnimalMuppet 2y ago
  If you're using ad-tech money to do something actually good, does it matter that the money was made from ad tech?
  - cmrdporcupine 2y ago

iamwil 2y ago

I don't know. On one hand, it is true we are the last line of defense. I do wish more of us said no to putting tracking on websites, among other things. But on the other hand, a random software engineer's sense of ethics might be different than you. Do we want activist software engineers? I mean, field of study changes perspective. Recently, there was that backlash against a release of a 1-mil post dataset of bluesky users by some people on bluesky. https://eugeneyan.com/writing/anti/ https://eugeneyan.com/writing/anti/ I personally didn't think it was a big deal because the API and dataset is already public. And also if your text isn't available to LLMs, then your values are effectively invisible to the future, as society depends more and more on LLMs. To me, I think it's better that you get your thoughts out there into LLMs. As more of society relies on LLMs, culture wars will be fought not just on the internet, but in the embedding space of LLMs. So, given that, would these people want me to decide for them? I dunno. Just as journalism lost the trust of the public by deliberately taking an activist stance, so too would engineering lose public trust by deliberately taking an activist stance. (You may say tech already lost public trust, but I think of tech as the industry and engineering as the profession).

vunderba 2y ago
Maybe we do need a Engineer's Hippocratic Oath, and a means to police our own. Go look at the latest "Ask HN" thread - there's two projects proudly trumpeting their ability to allow applicants to cheat on their interviews. On the FAQ of one of the apps, a common question apparently is if it will work during proctored exams. https://news.ycombinator.com/item?id=42375440 https://news.ycombinator.com/item?id=42375440 https://news.ycombinator.com/item?id=42383928 https://news.ycombinator.com/item?id=42383928 With this kind of loose approach to morality, is it really any surprise that the world is filled with devs who build involuntarily telemetry, spyware, etc.?
- golol 2y ago
  It just doesn't make sense. So it is immoral for engineers to design weapons? That's like saying it's immoral to butcher an animal. Sure go ahead, ban butchering animals in your society so everyone can feel good about themselves while they import their meat and thereby outsource the butchering instead. Or you say that everyone just stops eating meat? Fine that works but here the analogy breaks, because everyone won't just stop needing weapons. What I can agree with is that placing a moral cost on engineering designing weapons in some way increases the actual labor cost of weapon R&D, I guess.

eykanal 2y ago

It's worth mentioning that this really isn't programming-specific, or even engineering-specific, at all. The exact same story applies to the guy doing finance, or marketing, or project management, or legal work, or whatever. This is basically a reminder that your role as a cog in the wheel is to make the wheel move, and you shouldn't forget what the wheel is doing. That said, working for a US military contractor while not being supportive of the US military mission is kind of a silly thing to do.

nemomarx 2y ago
I think it's fair that you might think mostly about the military work overseas and not about it being licensed to local police or used to track citizens or so on. Wasn't a lot of the Snowden stuff that tools intended to be used on foreign enemies would be used at home?
- ForHackernews 2y ago
  From my understanding (with no special access or insider info), it's unclear how much it was used "at home" (against US citizens). Certainly the collection infrastructure was in place, but there were technical and procedural safeguards that were intended to filter out US citizens' data, and analysts were not supposed to access US citizen data without a search warrant from a judge. It's interesting to me that many people who work for giant tech companies that carry out indiscriminate data collection feel confident in their employer's internal processes and controls, but don't extend that same optimism when it's the government that pinky-swears they aren't going to peek at anything they shouldn't.
  - wutwutwat 2y ago
    > there were technical and procedural safeguards that were intended to filter out US citizens' data, and analysts were not supposed to access US citizen data without a search warrant from a judge. Nope > The NSA has built a surveillance network that has the capacity to reach roughly 75% of all U.S. Internet traffic. > An internal NSA audit from May 2012 identified 2776 incidents i.e. violations of the rules or court orders for surveillance of Americans and foreign targets in the U.S. in the period from April 2011 through March 2012, while U.S. officials stressed that any mistakes are not intentional. > The FISA Court that is supposed to provide critical oversight of the U.S. government's vast spying programs has limited ability to do so and it must trust the government to report when it improperly spies on Americans. > A legal opinion declassified on August 21, 2013, revealed that the NSA intercepted for three years as many as 56,000 electronic communications a year of Americans not suspected of having links to terrorism, before FISA court that oversees surveillance found the operation unconstitutional in 2011. https://en.m.wikipedia.org/wiki/2010s_global_surveillance_di https://en.m.wikipedia.org/wiki/2010s_global_surveillance_di... The kicker to me: > it must trust the government to report when it improperly spies on Americans The courts told them to watch themselves and to self incriminate if they do crimez. An honor system. Which naturally, they did not snitch on themselves to the courts, because why would they. Also let's not forget that they can make all the illegal surveillance legal in an instant just by labeling any person a "terrorist"

hprotagonist 2y ago

Chris: So, I talked to him. Mitch: You did? Chris: Yeah, and he used to be the number one stud around here in the 70’s. (whispers) Smarter than you and me put together. Mitch: So what happened? Did he crack? Chris: Yes, Mitch. He cracked, severely. Mitch: Why? Chris: He loved his work. Mitch: Well what’s wrong with that. Chris: There’s nothing wrong with that, but that’s all he did. He loved solving problems, he loved coming up with the answers. But, he thought that the answers were the answer for everything. Wrong. All Science no Philosophy. So then one day someone tells him that the stuff he’s making was killing people. Mitch: So what’s your point? Are you saying I’m going to end up in a steam tunnel? Chris: Yeah, I am.

gregw2 2y ago

Pardon me another story. In the early 2000s, I wrote software for a startup where the founders and initial equity-holders had pledged 10% of the founding stock to a certain charity. This was part of the marketing approach for the company- help us succeed and it benefits others (without affecting the competitiveness of our cost/pricing structure of our services in any particular tangible way.) There were some weaknesses in the model of mixing charity with startup-mindset (the model later changed to 10% of profits and we actually brought the charity work in-house eventually), but as a lead engineer, I actually found it quite helpful in pushing back on questionable (in my eyes) things the CEO was otherwise tempted to pursue and asking me to build (the CEO not being initially a technical guy and having any developed sense of technical ethics around spam, dark web practices, etc). If we did something really unethical (and were caught), it would actually hurt the charity we were also all trying to help and undermine part of our company's core differentiation. If you are a technical founder and bake this sort of charity commitment into your startup and its marketing on day 1, you can help people at all levels of the firm develop a more ethical culture than the average startup which faces all sorts of (often stupid) temptations.

ebiester 2y ago
On the other hand, I've also seen companies that start saying that it's okay to cut corners and drive people into the ground because it's for the "greater good." I think SBF took advantage of this in many ways as well.
- gregw2 2y ago
  Fair point. There is no substitute for keeping your eyes open.

uludag 2y ago

Great article! This was similar to my experience in ad-tech, though obviously more mundane. It's almost like people had to create artificial problems to avoid thinking about what they're actually building. Even language filled with euphemisms and acronyms was employed, I suppose to put the person at a distance from what they're building. Even if there's nothing morally objectionable, like other places I've worked at, the product is so worthless that people would enter the same patterns of behavior.

troyvit 2y ago
Exactly the same, down to the industry! My job was as a simple force multiplyer for account managers and sales folks, automating tasks to allow them to scale their projects and help more clients. It was all cool helping them get more ads out the door for Toyota and cool authors, weird when it started being for "One weird trick" ads, downright wtf when I started helping boost SERP pages and finally once we started promoting gambling apps I just couldn't take it anymore. It was great money. I miss that. And helping all these very kind, friendly people make numbers felt useful. But man, once I finally got un-distracted (after ten years?!) I had to get out of there.

mettamage 2y ago

I think a course would be a helpful good step. I remember watching Yale’s introduction to ethics class. Something like that but then applied to software engineers specifically

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

lmeyerov 2y ago

Half our work is on bringing GPUs/graphs/AI/etc to Serious Problems like cyber, misinfo, fraud, crime, etc, and a few things I've come to: * If you're unsure, quit * It's fine to distrust the system, and as with most of the population, do something else * If you do work on a Serious Problem, decide what your line in the sand is, and ensure that aligns with the organizations you work for & with * Constantly reevaluate yourself, the people, and the work Extremes like "don't do things if they can ever go wrong" would suggest the only answer is to do nothing. That doesn't work for most things. Likewise, I don't like painting people as unethical just because they have evolved different belief systems or, quite often, a more nuanced & evolved view of something difficult. Thankfully, I find most people in the US, and in many other countries, to try to do the right thing on average. This is even more true of people grinding on Serious Problems. So it often comes down to personal responsibility and awareness for whatever you care about, keeping the alignment with those around you... and leaving when that stops working. The AI & robotics booms have made this stuff even more important. We view it as a chance to do good, and with more accountability & quality than when those with less scruples & ability do so. But agreed, we pick where we work, and ensure new staff agree on where we drew our own lines in the sand.

cmrdporcupine 2y ago
Agree with your points overall, but one of the specifically worrying things about right now is the massive downswing in the tech job market means that -- especially for those of us outside of the Bay Area -- your first recommendation: "quit" has become a lot lot harder than it was 2-3 years ago. e.g. I work fulltime in Rust, and love it. I look at the job postings that emphasize Rust ... and 80% of them are crypto/web3 companies. I won't work for those companies because of my personal beliefs about "crypto"... But if I were to lose my pretty decent job right now... and the bills started to pile up and my family was in financial danger... The argument becomes more difficult.
- lmeyerov 2y ago
  Agreed!

rlt 2y ago

Everyone's got their own ethical standards. This guy won't touch anything related to the military. Palmer Luckey is part of society's "warrior class that is enthused and excited about enacting violence on others in pursuit of good aims" [1]. I'm somewhere in between. 1. https://techcrunch.com/2024/10/01/palmer-luckey-every-country-needs-a-warrior-class-excited-to-enact-violence-on-others-in-pursuit-of-good-aims/ https://techcrunch.com/2024/10/01/palmer-luckey-every-countr...

amoss 2y ago

In a different place a long time ago we were approached by a company that wanted a collaboration. They had a very specific project in mind, with requirements already set. It was a humanitarian project to "find earthquake survivors". If you read the requirements in a very positive way it was an ethically clean project. But the company is a major defense contractor and it was very unusual for them to be involved in humanitarian work. So it made sense to question the requirements after a less generous reading. Detecting mostly occluded humans in a highly cluttered and destroyed urban environment has other use cases. They got as far as "highlighting the survivors with a laser designator" before I started to ask some very pointed questions and start discussing the ethical considerations of us helping to build a system to kill people. I was asked to leave. After that meeting the potential collaboration was rejected and it never came up again. The company continued shopping around other places to get help building their "earthquake rescue system". Never get distracted. Always question the purpose of work. Consider ethics. Getting another job can be a pain. Living with knowing what you helped to build would be much harder.

duxup 2y ago
I'm a little surprised that a defense contractor would bother with that much awkward subterfuge or even farm out a core part of their product to some random other company. The handful of US based defense contractors I've had contact with were VERY up front about what they develop. I think they more so wanted you to know so everyone was on the same page.
- cl0ckt0wer 2y ago
  They probably already have a complete system. They're just looking for new revenue sources.
- pc86 2y ago
  I'll bet my next paycheck the above scenario never happened, or this company was less "major defense contractor" and more "VC-funded SaaS-app defense contractor wannabe." Lockheed Martin doesn't need to trick anyone into working for them, and that includes software developers. If this was a real thing that actually happened, everyone would need a security clearance anyway.

Liquix 2y ago

FWIW the Fasten link (goes to https://fasten.com/us/cities/austin https://fasten.com/us/cities/austin) is broken

caleb_thompson 2y ago
Thanks, they went out of business in 2018. I've added a note to that effect and removed the link.

m_ke 2y ago

I used to work at one of the first deep learning image recognition startups, it was all going great, we raised a ton of money, hired a lot of people and built some amazing technology. The founder was on the covers of fancy business magazines and it looked like we were the next big thing. Then revenue growth stalled and we shifted to enterprise consulting, which caused a lot of the early employees to quit (including myself). A few months after I quit they took on the Maven project that Google and Amazon employees protested against and in no time after that they had an office near DC with a bunch of ex generals on their staff. Now they work on secret projects that nobody still working there can tell you anything about. Moral of the story, you can be a well meaning person and work on cool tech but you have no control over how it will get used in the future. The same thing that happened in CV will happen with all of these LLM startups once growth stalls and they're unable to raise another up round to cover their huge expenses.

raincole 2y ago

I feel we should attempt to hold decision-makers accountable instead of blaming the field workers for unethical practices.

0898 2y ago

It's amazing how many tech people, like this author in this piece, still believe that their phone is constantly listening to you to target ads.

ddulaney 2y ago
I don’t think it’s that surprising. Here’s two true statements: I don’t trust my phone on a fundamental level. I also have mentioned things out loud and gotten ads afterward about them. On the one hand, those two statements absolutely don’t prove that my phone is listening. Just because I don’t trust my phone doesn’t mean that it’s being used maliciously, and the ads I’ve seen might just be a coincidence. On the other hand, those two statements together are definitely suggestive. They feel suspicious. It would be so easy for Apple/Google/Samsung to secretly listen and make a boatload of money by telling me a lie. Lord knows they’ve lied about other things. So yeah, balance of evidence is that phones probably don’t do that. But don’t be surprised when people suspect that they do.
- nuancebydefault 2y ago
  Your phone doesn't need to listen to you to know your interests. The ad algos are smart enough to figure it out from your browsing behavior. Also, it happened a few times I talked about some thing and an hour later it was talked about on broadcast TV. People over-fit on finding patterns.
spencerflem 2y ago
They're bragging about it at least: https://www.documentcloud.org/documents/24224884-how-voice-data-works-and-how-you-can-use-it-in-your-business-cmg-local-solutions/?ref=404media.co https://www.documentcloud.org/documents/24224884-how-voice-d... https://www.techdirt.com/2024/08/29/cox-caught-again-bragging-it-spies-on-users-with-embedded-device-microphones-to-sell-ads/ https://www.techdirt.com/2024/08/29/cox-caught-again-braggin...

shireboy 2y ago

I remember an anecdote about this topic where someone (I think it was Douglas Engelbart?) was giving another a hard time about their military software being used to kill people. At which point the other person mentioned their software used a mouse (which Engelbart invented). Obviously a complex topic with lots of considerations that I venture won't be solved in this HN thread, but good to discuss and think about.

AlotOfReading 2y ago
There's a pretty meaningful distinction between something (relatively) neutral that's occasionally used to support unethical purposes than something that can only be used for unethical purposes, plus shades of gray in-between. I don't think there's a reasonable argument that e.g. Keurig engineers need to grapple with the ethical dilemmas of murderers and terrorists using their products for example.

sjbremer 2y ago

[snark removed]

draw_down 2y ago
[dead]
tines 2y ago
Please keep to the HN guidelines: > Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes. > Comments should get more thoughtful and substantive, not less, as a topic gets more divisive. > Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith.

caleb_thompson 2y ago

Author here. I gave this as a talk at several conferences through 2017. I didn't realize that the video embed was broken until someone pointed out that this was making the rounds again, so I've just fixed that. If you've already looked, the video is now available. The content is more or less the same as the written version so don't feel like you need to watch it if you've already read the post.

vaylian 2y ago
Thanks. I really appreciate the transcript! I'm still reading it. This is a really good and thoughtful talk. Thanks for sharing it!
Teever 2y ago
Do you believe that there is sufficient moral hazard for Ukrainians working in their domestic drone industry to dissuade them from doing that work? Why or why not? What is different about that situation from the situation you described in your talk? What factors would have to change in the global balance of power for you to consider building systems that kill for American companies?
- caleb_thompson 2y ago
  Morality is personal. In any instance, we should think through first-, second-, and subsequent-order consequences of our actions and consider whether we're comfortable with those potential outcomes and whether those consequences balance one another. I won't take the bait on defining the differences between wartime and peacetime work.
  - myk9001 2y ago
    I don't know if the gp's question is intended as a bait or not. However, it describes a real scenario playing out literally as we speak. While I absolutely don't mean this in a confrontational manner, if your ethical framework doesn't provide you with means to address a real-world ongoing situation, what's it even useful for?
  - 2y ago

geetee 2y ago

It's nice that the author can hold opinions and write a blog post like this and not be disappeared. I wonder why that's possible...

sakjur 2y ago
Though oddly Boeing whistleblowers keep dying. Hamilton made a much better case than I could for a way to view the military as a necessary evil neither to be loved nor feared in Federalist no. 8.

josephcsible 2y ago

Where do you draw the line? Is it wrong to work on food preservation technology, because it gets used to make MREs that get fed to soldiers who kill people?

tines 2y ago
I'm not sure whether you're asking this question in good faith, but I think as with any moral spectrum, you draw the line where your best reflective judgment tells you to?
wutwutwat 2y ago
No. One of them is feeding people, all people. One of them is actively working on systems designed to kill people and only kill people

duxup 2y ago

I'm not sure if I would be comfortable working for the military / a contractor directly, but I'm more ambivalent than unsure. Certainly each individual has to make their call. I do wonder about this whole premise. Let's say your country is under attack by an oppressive neighbor or known bad terrorist organization? Would people feel differently? I get the same weapon could be used differently / against less bad targets, but it doesn't change that they can also be used to defend / attack people who genuinely want to do you harm. I do not see this as a cut and dry call.

bitbasher 2y ago
Yes-- and if your enemies are making the same tech you don't want to fall behind. That's also why we created the atomic bomb, so I'm not sure how to feel about it.

delta_p_delta_x 2y ago

All the software engineers in places with mandatory conscription like South Korea, Singapore, Israel, Finland, Switzerland are in shambles. Someone is complaining about writing software that could be used in implementations which kill people, whereas men in these countries straight-up don camouflage uniforms, helmets, and armour. They are given rifles, and are taught how to shoot and kill people, and sometimes even get to kill people. Cut out the middlemen and all the abstraction, I suppose; program in C and assembly rather than in Python.

tines 2y ago
I don't really see the incongruity here; somebody doesn't like helping people kill people, while that is mandated in other countries. What's your point?
numpad0 2y ago
The moral of the story is not "always be cynical and ridicule defense guys".

mediumsmart 2y ago

absolutely, you need focus when you work for the department of war

tristor 2y ago

Great article, and very important things to consider. As a practice, I take my holidays at the end of each year to consider my career, my life, and my worldview. A long holiday is an opportunity to refocus from work to introspection, and allows you to find comfort with your place in the world. The primary things I ask myself near the end of this introspection period are: 1. Is the work that I am doing advancing my personal goals? 2. Is the way I spend my time good for my mental health, my physical health, and my family? 3. Is the work that I do aligned with my ethical worldview? 4. Am I being renumerated appropriately aligned to the value that I am bringing? 5. Am I still learning something each day? 6. Are my family happy with the outcome of this year and the way I contributed to it through my work and otherwise? If I get a no to any one of these questions, I start looking for a new job. I've moved across country and across the world, and I've been at companies for a long time and a short time throughout my career (that's nearing 20 years at this point). I've switched industries and areas of technical focus many times, and I've even switched entire career paths by simply asking myself these questions after a relatively short period of introspection. I encourage everyone to consider the ethics of their work, but also to consider regularly whether or not your work is advancing your goals, making you happy, making you healthy, and supporting your family in the way they need to be happy and healthy. Ethics is a core part of this. If you are doing work you don't agree with for purposes you would not agree with, it can be traumatic to your own psyche and deeply affect both your health and happiness, including how you treat other people and especially those closest to you. Take some time, the holidays are coming.

martinjacobd 2y ago

Needs a [2017]. More substantively, my company is a contractor and has publicly committed to neither working on 'weapons' nor 'destructive financial systems.' Both terms are nebulous but we generally just decide en masse as a company (there are only 20 people) what does and doesn't count. I don't necessarily care for these particular lines in the sand but it's important to have some kind of process for determining and thinking about things, which I'm happy we do.

vhantz 2y ago

The reality is nobody is hired to decide why things are done. Under capitalism, the why is to make profits, to increase shareholder value, if you will. That's it. You don't plan what you don't control, you don't control what you don't own. For the vast majority (yes, including engineers), the only thing we own that is a part of the whole process of capitalist production is our ability to do some kind of work. When that ability is sold (for a wage/salary) to whoever possesses the resources to decide the why, we forego the right to decide how it's used. Just like when you sell someone a chair for example, you don't get to tell them where and how they can use that chair. You can refuse to get or do this or that job, but you need to have a job. At the end of the day, we all need food and shelter to stay alive and the bills aren't gonna pay themselves. So ultimately, someone—among the vast majority of us who don't have a choice—will also have to do the job you refuse. So we shouldn't delude ourselves into thinking we have some sort of "superpower". We don't. That is, not unless we can collectively withold our ability to work and force the hand of the minority that actually decides why things are done. A society where those who do the work decide why and how it's done, is a society where the working people own the resources. That is communism. That world is possible, but it won't build itself, you have to tirelessly fight for it. Look over at marxist dot com if you want to help out [1]. That doesn't mean that in the meantime we shouldn't strive to find meaningful work that fit with our values. We should, and whoever manages to find that good for you! You've been lucky (for now) but you won't be forever, and even if you are to be, the vast majority of people with pretty much the same skills and conditions as you, actually won't. We actually have to transform society if we want to secure that for ourselves and for everyone else. [1] https://marxist.com/about-us.htm https://marxist.com/about-us.htm

vhantz 2y ago
Don't get distracted by the word communism!
- zer8k 2y ago
  Attributing "capitalism" to the increase of value to "shareholders" is a mischaracterization of capitalism. It's the only system that has worked for one singular reason: People want more value for more work. If I work 12 hours picking potatoes and have to give 3/4 of them "according to my need" I will refuse to work. If you create fake jobs like "bolt counter" to insure 100% employment so the means can justify the ends (as communism believes) you end up with a collapsing economic system. The current iteration of late-stage capitalism is not capitalism _at large_. It's feudalism clever disguised as an egalitarian economic system. The solution to this is not communism, it's undoing 100 years of stock-market oriented business design. We don't know what "late-stage" communism looks like simply because a revolution occurs several hundred years before it reaches that point. Not even the premiere implementers of communism, the Russians, could keep their system. It doesn't work. In any country, in any system, with any group of people larger than 4.

zahlman 2y ago

I skimmed through this because it was really obvious where it was going. > The unifying factor in all of the stories I’ve told is that a developer wrote the code that did these unethical or immoral things. As a profession, we have a superpower: we can make computers do things. We build tools, and ultimately some responsibility lies with us to think through how those tools will be used. No, it does not. It still boggles my mind that people actually seem to have this model of morality. If everyone thought that way, the world would not have nuclear power, to name just the most obvious example that comes to mind. Nobody is "responsible for" making it possible for others to do evil. That's their choice. (To expand a bit, because clearly not everyone shares my intuition: if this sort of moral failing can attach at one degree of separation, logically nothing prevents it from attaching at arbitrarily many degrees of separation. If I become culpable because my software "is used to kill people", then any person or company who enables me to write the software faces the same judgment. Vim is used to write software that is used to kill people. Do you really want to put that on the guy[0] who tried to get you to donate to humanitarian aid in Uganda?) (The same logic is used by activists to demand participation in various seemingly arbitrary corporate boycotts - and people I care about have been harassed because of it. The thing about "following the money" is that it goes literally everywhere.) [0]: https://en.wikipedia.org/wiki/Bram_Moolenaar https://en.wikipedia.org/wiki/Bram_Moolenaar

wavemode 2y ago
I think both things can be true. A person can take the position that developing something which will be used to directly kill someone is wrong, and thus refuse such jobs. A person can also take the position that software itself is morally neutral, and how it is used is the choice of the user. Both of these are perfectly valid lines of moral reasoning. Which one you choose is going to be a personal decision. Debating about which one is more "right" devolves into a philosophical discussion. Does it "boggle your mind" that some people choose to be vegetarians?
- zahlman 2y ago
  The point is that "which will be" is load-bearing. The idea that someone would feel moral qualms about how software is used by the military is incongruous with signing up for the job in the first place. The military does in fact do a lot of things that are not killing people, and presumably could also find a use for the technology described that does not involve killing people. Like, say, locating allies for a rescue effort.

nuancebydefault 2y ago

This was 2017. Meanwhile the geo political situation has changed quite a bit in the world. Unfortunately, the ethics already feel outdated.

joemazerino 2y ago

China, Iran, Russia, ISIS and others would be happy to get their hands on this tech and kill you.

joelesler 2y ago

TL;DR: The author recounts their experience as an intern building software for the Department of Defense, unaware of its intended use. The software, designed to locate WiFi signals, utilized algorithms like R^2, Gaussian estimation, and Kalman Filters to improve accuracy and tracking capabilities. Despite the technical intrigue, the author acknowledges the software’s purpose was to aid in killing people, highlighting the ethical implications of their work. The author recounts a past experience working on a project for a Department of Defense contractor, where they were tasked with creating a tool to locate phones, ultimately realizing it was intended for targeting and potentially killing individuals. This experience, along with examples like a deceptive quiz and Uber’s “greyball” feature, highlights the potential for code to be used unethically or even illegally. The author emphasizes the importance of developers considering the ethical implications of their work, as software increasingly impacts various aspects of society. Developers have a responsibility to consider the potential misuse of their work, as unethical applications can have serious consequences. While there is no universal solution, developers should critically evaluate project requests, consider worst-case scenarios, and prioritize ethical considerations over deadlines. Ultimately, developers must decide whether to build a product, even if it has the potential for misuse, or to prioritize ethical concerns and potentially face consequences. Code can have serious consequences, even death.

myk9001 2y ago

I find it ironic that a US citizen gets to enjoy the privilege of making an ethical choice of whether to support their military thanks pricesely to the fact the US possesses the most advanced military force out there.

1970-01-01 2y ago

The undiscussed counterpoint is how a new weapon will also result in saving lives if used wisely. Nuclear stockpiling is a darn effective war deterrent, and has prevented an uncountable number of hot wars.

wat10000 2y ago
It’s too early to conclude that nuclear weapons save more lives than they take.
- 1970-01-01 2y ago
  The boiled-down argument being 'Pax Americana' must end before nuclear stockpiling can be called an overall success?
  - wat10000 2y ago
    I’d say we need to reach the point where nuclear Armageddon is no longer possible. Even then, it’s possible we got lucky and nukes were actually more dangerous than not having them, but we’d at least be able to definitely say that their total kill count is only X, for whatever X turns out to be.

jjfrady31 2y ago

Long story short, the author lives peacefully and comfortably in a country defended by others and has ethical issues helping those who protect him. I don't see eye-to-eye with such people. I joined the military (the infantry) straight out of high school out of a sense of duty due to my upbringing. I knew what the job entailed but never thought about it much. It was peacetime, too, so that made not thinking about it easy. Then, the war on terror (9/11) happened, and the reality of the job blew up in my face (almost literally). It was a challenging experience to say the least, but I would do it all over again if given the chance.

tines 2y ago
To steelman the author's argument and as someone who's debated about this a lot with myself, what's your perspective on the argument that goes like this: "helping those who protect us" would be fine if that's all that those-who-protect-us did; but in addition, they go start unnecessary wars in foreign lands, kill innocent people, surveil their own population etc. Because they far overstep the reasonable bounds of keeping peace and defending the country, and because the "help" I'm giving them will go directly to that extra stuff that I abhor, I can't in good conscience help them. Would that argument help you to see eye-to-eye?

_zagj 2y ago

I wonder how many of the people back-patting the author and each other work for companies whose business models entail egregiously violating users' privacy or destroying the economic security of low-wage workers? How many work on exploitative AI projects--especially AI art--that rely on training data used without permission or compensation? Yet suddenly, when it comes to bettering the United States' and her allies' war-making capabilities, ethics become an issue? Thank god during the Cold War we had real tech companies, like Digital and HP, staffed by real engineers whose heads were screwed on straight and whose values weren't perverted.

NoGravitas 2y ago

> So how can we “carefully consider potential impacts”? Honestly, I don’t have any answers to this. I don’t think that there really is a universal answer yet, because if we had it I have to believe we’d not be building these dangerous pieces of software. So, there is no general answer to the problem, but you can always approximate it by asking: are you working for the US defense/intelligence industry or one of its contractors? If so, you are almost certainly building something harmful.