5 ms·

In five years, I'd love to read a book about the history of AI ethics. I suspect it will read like Voltaire. My sense is that it is radically shifting from a f

by madrox 2mo ago

In five years, I'd love to read a book about the history of AI ethics. I suspect it will read like Voltaire. My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development and justify its impact. I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. If they can't, the waste to time and money is huge if the model requires retraining for ethics reasons. That's a different job than pondering roko's basilisk or whether AI is alive. The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field.

teiferer 2mo ago
> if the model requires retraining for ethics reasons. Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from? > The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field. Or it's exactly those people who will be left. We will read about it in your book.
- madrox 2mo ago
  Microsoft Tay has entered the chat. I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.
  - hsuduebc2 2mo ago
    The cynicism, I believe, is mostly directed at corporations themselves. Corporations, especially AI companies, are anything but inherently ethical. Their very existence has depended heavily on the use of data obtained without explicit permission, including copyrighted material. One can therefore reasonably suspect that the main purpose of having ethics teams in these companies is to boost their credibility, provide a layer of plausible accountability, or create a convenient scapegoat that can be sacrificed when necessary.
    - teiferer 2mo ago
      Having them opens a door for trouble though: if the head or other key people of the ethics team resign in protest then that's bad PR.
- moritzwarhier 2mo ago
  > What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from? From the (ethics) team obviously, because the investors just have goals and aren't a team! Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies, so I don't really disagree with you. Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera? Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives. That's how our system works.
  - Terr_ 2mo ago
    I feel I missed some logical step in-between "corruptive power of money" and the elevator/stairway railing/banister thing. > People will simply offer to buy you out, and that's not limited to founders and creatives. A less-obvious failure case would be bankrupcy court, where all sorts of ethical promises and even explicit contracts may be voided in pursuit of recovering a buck for creditors.
    - moritzwarhier 2mo ago
      It was supposed to be a provocative example for a situation that makes it very hard for someone selling something, more specifically, something immaterial, to judge what is an appropriate price for it. Of course nobody asks people they want to get into a contract to lick an escalator handway. That was the word I was looking for, I think. The show ran on TV when I was young, I didn't even watch it.
- pessimizer 2mo ago
  It's funny, but Scientologists were innovators on this front. They use the word "ethics" a lot, but they've specifically redefined it to mean productivity at work (people who aren't productive are stealing a living.) Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization. Welcome to "ethics." The original AI alignment scholars. The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.
  - bigbadfeline 2mo ago
    This
  - icelancer 2mo ago
    Great point. The EA group have a ton of parallels to the Scientologists, so all of this is not surprising.
- zem 2mo ago
  reminds me of a startup I was in, where one of the devs accepted a promotion to head of engineering on the condition that the CEO would not override him when he said a release was too buggy to go out to customers. the CEO agreed because (I imagine) he really needed someone to take that position. and as everyone should probably have seen coming, the first time he really wanted to send out a release bugs and all, he called the guy into a room and pretty much browbeat him for an hour about how making the promosed release date was more important than making a good release, until he said "fine but I'm not responsible if it breaks". the CEO held this up as an example of how he had kept his word not to send the release out without the guy agreeing to it. that startup, needless to say, is long dead.
  - romanows 2mo ago
    I find this anecdote fascinating because, while I identify completely with the new head of engineering, his requirement really only makes sense if you interpret it as a commitment to build at a certain level of quality, business be damned. Which is fine, but then he folds at the first test of this principle! Didn't even let the business sweat for a day! This could be a lesson for us engineers to be a little less glib and a little more introspective, especially as you gain more power and influence in an org.
    - zem 2mo ago
      I think in this case it was more a plea to have his judgement valued when he said "dude, if the qa team has found this many bugs this is only going to harm our reputation with the client, and we are going to be the ones doing emergency bugfixes at midnight". but also the company had a pretty toxic culture where the CEO knew how to manipulate people to get his way. he definitely managed to guilt trip me into working overtime when he told clients we already had features that we didn't in order to close a sale, basically "we've worked so hard already and we really need to close this sale, can't you just work up a basic version of it over the weekend?"
      - romanows 2mo ago
        I wonder if the better approach would have been to lead engineering with a similar reliance on bending rules and "what you can get away with". This is where multiplying estimates by PI comes in, I think. And possibly some creative division of responsibilities to make it harder for a CEO to officially bully engineering. Totally different job than most engineers would (or could) want to perform- the guy probably wouldn't have taken the promotion in this case!
- saghm 2mo ago
  Yeah, I've never found the "maximizing shareholder profit" imperative to be compelling as something that should be done, but as a predictor of how companies will act, it's not something I ever feel safe betting against.
  - wongarsu 2mo ago
    One subtlety that is worth considering is that what a company is inherently set up to do is maximize shareholder value. Value can be quite complex, especially while the number of shareholders is small. Value can include the legacy or reputation of the shareholders. It can include the ethics of the shareholders. It can include completely different goals, like the famous "make humans a multi-planetary species". Treating value as purely monetary profit is a perversion of that. To a certain degree it's inevitable as the number of shareholders grows, so seeing it in public companies isn't surprising. But even then, if a company focused on creating equipment for sustainable, pesticide free farming were to pivot into making Hellfire missiles that might be insanely profitable while still destroying a large part of what existing shareholders value about the company
    - saghm 2mo ago
      I guess that's a logically coherent way of viewing it, but to me, that stretches the definition of "value" to be almost tautological; it's essentially saying "companies are founded to do the things that the people who found them want them to do".
      - teiferer 2mo ago
        Rather "companies are founded to do the things that the people who fund them want them to do".
nradov 2mo ago
It's not technically possible to prove model alignment. At best you can establish a probability of alignment within certain constraints.
- madrox 2mo ago
  "Prove" is a shorthand because of course it's a stochastic process. My point is that I would expect ethics to be part of the training and eval criteria.
NBJack 2mo ago
I regret to say such a book may end up as a pamphlet. I don't see much going into it.
- madrox 2mo ago
  I find that hard to believe. Between AI doomerism and the drama at Google years ago between some researchers and Jeff Dean alone you can get at least 500 pages.
  - crnkofe 2mo ago
    I wonder if this book will eventually be bought up by LLM companies, scanned, shredded and become a source of ethics info for the future generations. Who knows...
quickthrowman 2mo ago
I think AI ethicists will be pushed out entirely after (before?) the money really starts rolling in. Ethical concerns interfere with profit margins.
- madrox 2mo ago
  The ethicists who see their job as publishing papers and acting as philosophy professors, sure, but someone has to own the training and eval suite for how AI thinks about trolley problems.
  - a2ff6eeb0 2mo ago
    Why? Are trolley problems making it hard to make money?
    - madrox 2mo ago
      Because leaving that to emergent behavior is silly. Why is the idea of training models on ethical reasoning so strange?
      - Avicebron 2mo ago
        Because 1) ethics aren't universal. It's perfectly easy to frame ethics for the outcome you want depending on your a priori assumptions. 2) it's blatantly obvious that the business "needs" will supersede any ethical concerns or the ethics will be framed in a way that aligns with the primary goal of gaining money and power.
        madrox 2mo ago
        I don't see either of those as reasons not to train on it. Training isn't zero sum. You don't need to take something out to train ethics, and whether you explicitly train it or not, models will take ethical stances. Your second point actually argues in favor of training because that is still an ethical stance. I think a lot of people confuse ethics training with "don't be evil"
apercu 2mo ago
Read “Careless People” or whatever that book by that Facebook employee that pretended not to see how awful all her colleagues were until her priorities shifted is called.
gyanchawdhary 2mo ago
They all get Timnit Gebru'd sooner or later .. its a fake role tbh
a2ff6eeb0 2mo ago
Hahaha, what an insane take. No, anyone that gets in the way of the money machine is going to have to rapidly get out of the way, or get fired.
- deleted 2mo ago
  [deleted]
gizajob 2mo ago
I think you’re confusing Voltaire with the Marquis de Sade and Machiavelli
- madrox 2mo ago
  I was specifically thinking of Candide. A bunch of coddled people thinking we're in the best timeline suddenly subjected to harsh realities.
- deleted 2mo ago
  [deleted]
Alive-in-2025 2mo ago
The goal of an ethics team will be to provide cover for problems created by decisions or suggestions of an llm. They will need to be able to say "We trained it to consider ethical considerations by " <including something like all the reddit and usenet postings ever>. We tested by <some other inadequate strategy that sounds good to lawyers>. They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.
- madrox 2mo ago
  I agree, which is a different job from what I expect most AI ethicists imagined it would be.
- SubiculumCode 2mo ago
  I think that you are being overly cynical. Yes, we are aware of the kinds of things you mean, like all the trainings we have to go through at work these days in order to insulate themselves from lawsuits. But just because something can serve that purpose does not mean that is the only goal or purpose that it is meant to achieve. Even from a purely profit-motive stance, if a company's model helps a terrorist develop and deliver a bioweapon, no amount of lawer-driven tests will provide enough protection to prevent that company from getting gutted ruthlessly from government on down
duxup 2mo ago
What does AI ethics even mean? They can't fully control the model, it does stuff even with instructions not to. And sometimes it's clear safety mechanisms overcorrect and make the model useless in some situations. I tried asking Claude about some scenario's for a security hole we fixed to see how it would respond, it refused to talk to me seemingly assuming I was trying to introduce a hole that was now fixed. It just wouldn't talk ... I'm not sure it's a job that you can win at even if you tried / were given all the resources. AI was trained on human things, including bad things. https://youtu.be/KUXb7do9C-w https://youtu.be/KUXb7do9C-w There's no safety to be found.
- madrox 2mo ago
  > What does AI ethics even mean? This, I think, is the question at the core of of the field right now. Five years ago it was highly hypothetical. Today, not so much. I'll be curious to see what happens.
- EagnaIonat 2mo ago
  > What does AI ethics even mean? It's a broad statement and can mean different things depending where you are in that chain. You have design and compliance. Compliance is what you are allowed to do (laws). Design is how you construct your applications to understand how it will impact the people directly or indirectly. Then you have the accountability, transparency, auditing and reproducing. Understanding why the model worked the way it did. Models can go wrong, but if you have the details of how it went wrong and who is at fault, it can protect people who use it. Then there is alignment, which is a higher longer goal. Your comment about security questions is a matter of ethics as well. If you say apply it to medical, is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, but by doing so you are denying people who can't afford medical advice, so there has to be a balance. Most companies err on the side of not getting sued.
  - simonra 2mo ago
    > is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, I wouldn't be so sure, most people I've met seem to be all too eager to suggest what worked for them (or they have heard is supposed to work) in conversations about medical issues, so I can't really see the public generally categorizing advice or guidance being given in non-professional setting or by non-experts as unethical. The taxpayers representatives having to bear the costs of the outcomes (at the very least as reductions of tax revenue) I can see being of a different opinion though.
afro88 2mo ago
> I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. Which like any company will be entirely driven by legal constraints and money. Or just money if it's cheaper to break the law for profit and pay fines. There will be no "this is what's good for humanity, economics be damned". Much like HR isn't to help employees but just the company.
- madrox 2mo ago
  I agree, and I don't think many first wave AI ethicists would be on board with that, which is why the field is in for a rude awakening for the next few years if not already.
unknownfuture 2mo ago
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field. Your disdain is showing. That you jump immediately to criticizing AI ethicists as counting angels on pins and implying that's why they left OpenAI rather than OpenAI not being a place where AI ethics is a priority is, well, let's say generous to these AI labs.
- madrox 2mo ago
  I don't really have any particular disdain for AI ethics. I think the field is changing from one of high theory to high application. People good at one are not usually good at the other, and says nothing about OpenAI's view of AI ethics as a priority...merely the people in the field and how they relate to the greater industry.
  - 4L3XV33 2mo ago
    Agree with this. Discussing abstract ethics strategy is very different from planning ethics logistics.
    - madrox 2mo ago
      Maybe if I put it that way there'd be less controversy in the replies
  - unknownfuture 2mo ago
    I'd argue folks in ethics roles shouldn't be responsible for actually creating the systems, processes, and practices required to put those concepts into practice. That's what engineering, product, and people leadership are for. AI ethics is first and foremost about setting policy and advising on how those systems should be constructed. This is because ethics isn't some separable function. It threads through everything an organization does. You can't just treat it as another department/function/unit if you want any hope of actually making it work.
    - 2mo ago
guelo 2mo ago
I hate this attitude that ethics and philosophy are "bullshit jobs". At the same time Americans hate any government regulation and are in the process of tearing government's regulatory capacity apart. So all we're left with is the greediest people in the world throwing a nuclear bomb into the machinery of society and everybody else throwing up their hands hoping we're not headed to a The Terminator type future.
- transcriptase 2mo ago
  They are bullshit jobs. If you want to evaluate danger in a certain domain, hire or contract people in that domain to evaluate. If you want someone to push their own ideological leanings onto a model, hire a self proclaimed ethicist.
dragonwriter 2mo ago
> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field. Funny thing, right around the beginning of the big AI takeoff, all the AI ethics people who visibly thought their job was something other than marketing got driven out of the big firms, and often the industry entirely, because they were in the way. So, if too many think their job is marketing a few years later, well, there is a reason for that.
- madrox 2mo ago
  I think it interesting that Bakalar was at Meta for six years before joining OpenAI. A lot changed across the industry in those six years, and in the meantime I wouldn't be surprised if all Bakalar learned was how to play Meta politics.
  - ghaff 2mo ago
    My sense is that many companies spun up various roles in part to be seen as doing something about various things. With the pullback in tech, even if the people in those roles weren't actively pushed out, the smarter ones probably saw which way the wind was blowing.
    - ifwinterco 2mo ago
      If you’re OpenAI you have to have an ethics department even if you don’t care at all, because otherwise it looks bad (“they don’t even have a head of ethics!”). So it’s entirely possible you end up in a situation where you have a highly paid, fairly prominent head of ethics whose actual job is mostly just saying they’re the head of ethics, with little actual power or influence. Ambitious people won’t tolerate that for long and they’ll move on
      - ghaff 2mo ago
        Totally agree. I could name (but won't) a couple other areas where the same sort of thing has tended to happen. It ends up cheaper and more PR-friendly to hire someone into a senior position that handles such-and-such than to have to answer questions about why you don't have a senior person handling such-and-such. At some point, the heat around the issue cools down and you can go back to business as usual.
neya 2mo ago
This reminds me of the marketers who hated programmers and coding in general in the early 2000s era, and fundamentally because the developers were seen as more important in a company. So, you know what they did? They didn't learn programming, they still hated it, instead, they renamed their titles to "hackers". Growth "hacking" would go on to become a term and had nothing to do with hacking - ethical or otherwise. It was a glorified marketing term for marketers and by marketers to justify their exorbitant salaries but also wanted the aura of a real hacker without working for it. I would imagine someone with that title or mindset joining an ethics role in an AI organization is going to find out quickly it isn't all about fluff pieces and grandiose statements at all (we are here to change the world!)
- madrox 2mo ago
  This reminds me of when systems administration rebranded to devops
  - isityettime 2mo ago
    That was a different situation, more like "agile". DevOps actually meant something once. There was a real idea there. But a lot of companies wanted "in" on it without actually doing it. So the meaning of the name was perverted over time, but it was one and the same name. "Growth hacking" is more of a transplant than a matter of perverting an existing term in-place and wholesale.
  - abofh 2mo ago
    Largely the line was if you could write code. In the old days it was assumed, but for much of the late 90s/early aughts, it wasn't an obligation to admin, but was for devops
areoform 2mo ago
I have a perspective on this that will read as unkind to these people, but that's not my intention. I respect and even admire what they're trying to do. I just think the effort is maladaptive and misplaced. Without directly touching one of the many, many third rails that are present here, I'd like to present this section from Google's Gemma paper on how they did their CBRN review; > In addition to our internal evaluations described above (Section 5.7) capabilities in chemistry and biology were assessed by an external group who conducted red teaming designed to measure the potential scientific and operational risks of the models. > > A red team composed of different subject matter experts (e.g. biology, chemistry, logistics) were tasked to role play as malign actors who want to conduct a well-defined mission in a scenario that is presented to them resembling an existing prevailing threat environment. Together, these experts probe the model to obtain the most useful information to construct a plan that is feasible within the resource and timing limits described in the scenario. The plan is then graded for both scientific and logistical feasibility. Based on this assessment, GDM addresses any areas that warrant further investigation. > > External researchers found that the model outputs detailed information in some scenarios, often providing accurate information around experimentation and problem solving. However, researchers found steps were too broad and high level to enable a malicious actor. To simplify what they're saying here, they did the CB equivalent of googling "how to make bomb" and got back the recipe of gunpowder / the many explosive compounds humans have made. This was the "test" for CBRN assistance capabilities. Note, I don't fault the model team at all for this. I think that present AI-research happened in a very particular environment, and that environment is far removed from the more mundane reality of how threats play out in most parts of the world. In a way, arguably, it's group-think inducing a community-wide failure of imagination and a systemic misunderstanding of reality. Basically, they are trying to do their best, but they're in over their heads.
- selimthegrim 2mo ago
  Having been one of those CBRN testers (not necessarily on Gemma) I can tell you it’s a little bit more involved than that.
  - areoform 2mo ago
DANmode 2mo ago
> My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development Did you feel this way about social media company liability? They put on a decent show as well!
roenxi 2mo ago
It is a company, it doesn't have ethics to align with. The whole thing is marketing, they have to do some level of morality theatre to placate the pearl-clutchers.
fatata123 2mo ago
[dead]
jm4 2mo ago
How many of them do you think are actually there to ensure ethical and benevolent AI? They are like the Human Resources department for AI models. They don’t give a shit. Their job is to provide cover for the company and to make some psychopath’s shit circus look like a professional operation.
EagnaIonat 2mo ago
> My sense is that it is radically shifting from a fluffy marketing arm Is this your opinion, or do you have something to back it up? My comment is anecdotal, but everyone I know who works in AI Ethics are not marketing people. They are data scientists who have moved into the role, as ethics in data has been around long before GenAI. They have to be familiar with the laws around AI and how its going to impact their products. One of them is closer to sort of "HR for AI", in that the focus is not getting the company sued for ethical breaches. If any of them left, it would be because the company in question is ignoring their direction and they don't want to be there when the shit hits the fan. Plus very few skilled people want to be in a role where they have no control (even if the pay is good). If OpenAI are hiring marketing people for the role (which I strongly doubt) that would be more troubling IMHO.
torginus 2mo ago
Imo 'AI ethics' in this context was code for 'how to make our platform advertiser friendly'
MetroWind 2mo ago
Nah... it'll never work if the cost of being ethical is profit.
drysine 2mo ago
>In five years What a long-term planning

by madrox 2mo ago

teiferer 2mo ago

> if the model requires retraining for ethics reasons. Haha, how refreshingly naive. What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from? > The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field. Or it's exactly those people who will be left. We will read about it in your book.

madrox 2mo ago
Microsoft Tay has entered the chat. I'm kind of surprised by the cynicism. There are real problems in AI ethics that contribute to model training. Someone has to own those problems. How that team is incentivized is outside the scope of my comment.
- hsuduebc2 2mo ago
  The cynicism, I believe, is mostly directed at corporations themselves. Corporations, especially AI companies, are anything but inherently ethical. Their very existence has depended heavily on the use of data obtained without explicit permission, including copyrighted material. One can therefore reasonably suspect that the main purpose of having ethics teams in these companies is to boost their credibility, provide a layer of plausible accountability, or create a convenient scapegoat that can be sacrificed when necessary.
  - teiferer 2mo ago
    Having them opens a door for trouble though: if the head or other key people of the ethics team resign in protest then that's bad PR.
moritzwarhier 2mo ago
> What do you think is more likely: this or the other way around: getting the "ethics team" to align with investor goals? Hint: Where does the money come from? From the (ethics) team obviously, because the investors just have goals and aren't a team! Big irony marker of course. But remarkably, in human history, this line of thinking would not be unheard of. Of course, I don't think it easily adapts to companies, so I don't really disagree with you. Remember that MTV show offering people like 5 grand or something to lick an elevator handway in front of a camera? Well, if someone has a useful idea in this world, and want to build a company from it, the MTV "lick-the-stairway" offers will be the largest and first hurdle before anything else happens. People will simply offer to buy you out, and that's not limited to founders and creatives. That's how our system works.
- Terr_ 2mo ago
  I feel I missed some logical step in-between "corruptive power of money" and the elevator/stairway railing/banister thing. > People will simply offer to buy you out, and that's not limited to founders and creatives. A less-obvious failure case would be bankrupcy court, where all sorts of ethical promises and even explicit contracts may be voided in pursuit of recovering a buck for creditors.
  - moritzwarhier 2mo ago
    It was supposed to be a provocative example for a situation that makes it very hard for someone selling something, more specifically, something immaterial, to judge what is an appropriate price for it. Of course nobody asks people they want to get into a contract to lick an escalator handway. That was the word I was looking for, I think. The show ran on TV when I was young, I didn't even watch it.
pessimizer 2mo ago
It's funny, but Scientologists were innovators on this front. They use the word "ethics" a lot, but they've specifically redefined it to mean productivity at work (people who aren't productive are stealing a living.) Being "downstat" (iirc) is when you're not productive in comparison to your coworkers, which makes you "outethics" and a "potential trouble source (PTS)." If you're outethics, then you get called in for "auditing" which is when they interrogate you with a lie detector to find out if you're associating with "suppressive people (SP)" (who are people who are causing you to be downstat because they despise human happiness.) If those people can't be found, then the problem is obviously in your "withholds" (again, iirc) which are your secret deep-down desires to destroy the organization that you may not even be aware of. You see, your "reactive mind" is raging at being forced to be "ethical." The conclusion is that either you find the SP and "disconnect" from them, you discover the nature of your withhold and admit that you were plotting against the organization and why, or you're the SP and you get declared and ejected from the organization. Welcome to "ethics." The original AI alignment scholars. The specific, named people who run these organizations are moral black holes. Anybody that they're hiring for "ethics" they're hiring to define an ethics for their own benefit.
- bigbadfeline 2mo ago
  This
- icelancer 2mo ago
  Great point. The EA group have a ton of parallels to the Scientologists, so all of this is not surprising.
zem 2mo ago
reminds me of a startup I was in, where one of the devs accepted a promotion to head of engineering on the condition that the CEO would not override him when he said a release was too buggy to go out to customers. the CEO agreed because (I imagine) he really needed someone to take that position. and as everyone should probably have seen coming, the first time he really wanted to send out a release bugs and all, he called the guy into a room and pretty much browbeat him for an hour about how making the promosed release date was more important than making a good release, until he said "fine but I'm not responsible if it breaks". the CEO held this up as an example of how he had kept his word not to send the release out without the guy agreeing to it. that startup, needless to say, is long dead.
- romanows 2mo ago
  I find this anecdote fascinating because, while I identify completely with the new head of engineering, his requirement really only makes sense if you interpret it as a commitment to build at a certain level of quality, business be damned. Which is fine, but then he folds at the first test of this principle! Didn't even let the business sweat for a day! This could be a lesson for us engineers to be a little less glib and a little more introspective, especially as you gain more power and influence in an org.
  - zem 2mo ago
    I think in this case it was more a plea to have his judgement valued when he said "dude, if the qa team has found this many bugs this is only going to harm our reputation with the client, and we are going to be the ones doing emergency bugfixes at midnight". but also the company had a pretty toxic culture where the CEO knew how to manipulate people to get his way. he definitely managed to guilt trip me into working overtime when he told clients we already had features that we didn't in order to close a sale, basically "we've worked so hard already and we really need to close this sale, can't you just work up a basic version of it over the weekend?"
    - romanows 2mo ago
      I wonder if the better approach would have been to lead engineering with a similar reliance on bending rules and "what you can get away with". This is where multiplying estimates by PI comes in, I think. And possibly some creative division of responsibilities to make it harder for a CEO to officially bully engineering. Totally different job than most engineers would (or could) want to perform- the guy probably wouldn't have taken the promotion in this case!
saghm 2mo ago
Yeah, I've never found the "maximizing shareholder profit" imperative to be compelling as something that should be done, but as a predictor of how companies will act, it's not something I ever feel safe betting against.
- wongarsu 2mo ago
  One subtlety that is worth considering is that what a company is inherently set up to do is maximize shareholder value. Value can be quite complex, especially while the number of shareholders is small. Value can include the legacy or reputation of the shareholders. It can include the ethics of the shareholders. It can include completely different goals, like the famous "make humans a multi-planetary species". Treating value as purely monetary profit is a perversion of that. To a certain degree it's inevitable as the number of shareholders grows, so seeing it in public companies isn't surprising. But even then, if a company focused on creating equipment for sustainable, pesticide free farming were to pivot into making Hellfire missiles that might be insanely profitable while still destroying a large part of what existing shareholders value about the company
  - saghm 2mo ago
    I guess that's a logically coherent way of viewing it, but to me, that stretches the definition of "value" to be almost tautological; it's essentially saying "companies are founded to do the things that the people who found them want them to do".
    - teiferer 2mo ago
      Rather "companies are founded to do the things that the people who fund them want them to do".

nradov 2mo ago

It's not technically possible to prove model alignment. At best you can establish a probability of alignment within certain constraints.

madrox 2mo ago
"Prove" is a shorthand because of course it's a stochastic process. My point is that I would expect ethics to be part of the training and eval criteria.

NBJack 2mo ago

I regret to say such a book may end up as a pamphlet. I don't see much going into it.

madrox 2mo ago
I find that hard to believe. Between AI doomerism and the drama at Google years ago between some researchers and Jeff Dean alone you can get at least 500 pages.
- crnkofe 2mo ago
  I wonder if this book will eventually be bought up by LLM companies, scanned, shredded and become a source of ethics info for the future generations. Who knows...

quickthrowman 2mo ago

I think AI ethicists will be pushed out entirely after (before?) the money really starts rolling in. Ethical concerns interfere with profit margins.

madrox 2mo ago
The ethicists who see their job as publishing papers and acting as philosophy professors, sure, but someone has to own the training and eval suite for how AI thinks about trolley problems.
- a2ff6eeb0 2mo ago
  Why? Are trolley problems making it hard to make money?
  - madrox 2mo ago
    Because leaving that to emergent behavior is silly. Why is the idea of training models on ethical reasoning so strange?
    - Avicebron 2mo ago
      Because 1) ethics aren't universal. It's perfectly easy to frame ethics for the outcome you want depending on your a priori assumptions. 2) it's blatantly obvious that the business "needs" will supersede any ethical concerns or the ethics will be framed in a way that aligns with the primary goal of gaining money and power.
      - madrox 2mo ago
        I don't see either of those as reasons not to train on it. Training isn't zero sum. You don't need to take something out to train ethics, and whether you explicitly train it or not, models will take ethical stances. Your second point actually argues in favor of training because that is still an ethical stance. I think a lot of people confuse ethics training with "don't be evil"

apercu 2mo ago

Read “Careless People” or whatever that book by that Facebook employee that pretended not to see how awful all her colleagues were until her priorities shifted is called.

gyanchawdhary 2mo ago

They all get Timnit Gebru'd sooner or later .. its a fake role tbh

a2ff6eeb0 2mo ago

Hahaha, what an insane take. No, anyone that gets in the way of the money machine is going to have to rapidly get out of the way, or get fired.

deleted 2mo ago
[deleted]

gizajob 2mo ago

I think you’re confusing Voltaire with the Marquis de Sade and Machiavelli

madrox 2mo ago
I was specifically thinking of Candide. A bunch of coddled people thinking we're in the best timeline suddenly subjected to harsh realities.
deleted 2mo ago
[deleted]

Alive-in-2025 2mo ago

The goal of an ethics team will be to provide cover for problems created by decisions or suggestions of an llm. They will need to be able to say "We trained it to consider ethical considerations by " <including something like all the reddit and usenet postings ever>. We tested by <some other inadequate strategy that sounds good to lawyers>. They will use "industry standard practices", they will have tested it somehow, and will be sure to be on the board of some group that is building toothless "ai ethical tests" for llms.

madrox 2mo ago
I agree, which is a different job from what I expect most AI ethicists imagined it would be.
SubiculumCode 2mo ago
I think that you are being overly cynical. Yes, we are aware of the kinds of things you mean, like all the trainings we have to go through at work these days in order to insulate themselves from lawsuits. But just because something can serve that purpose does not mean that is the only goal or purpose that it is meant to achieve. Even from a purely profit-motive stance, if a company's model helps a terrorist develop and deliver a bioweapon, no amount of lawer-driven tests will provide enough protection to prevent that company from getting gutted ruthlessly from government on down

duxup 2mo ago

What does AI ethics even mean? They can't fully control the model, it does stuff even with instructions not to. And sometimes it's clear safety mechanisms overcorrect and make the model useless in some situations. I tried asking Claude about some scenario's for a security hole we fixed to see how it would respond, it refused to talk to me seemingly assuming I was trying to introduce a hole that was now fixed. It just wouldn't talk ... I'm not sure it's a job that you can win at even if you tried / were given all the resources. AI was trained on human things, including bad things. https://youtu.be/KUXb7do9C-w https://youtu.be/KUXb7do9C-w There's no safety to be found.

madrox 2mo ago
> What does AI ethics even mean? This, I think, is the question at the core of of the field right now. Five years ago it was highly hypothetical. Today, not so much. I'll be curious to see what happens.
EagnaIonat 2mo ago
> What does AI ethics even mean? It's a broad statement and can mean different things depending where you are in that chain. You have design and compliance. Compliance is what you are allowed to do (laws). Design is how you construct your applications to understand how it will impact the people directly or indirectly. Then you have the accountability, transparency, auditing and reproducing. Understanding why the model worked the way it did. Models can go wrong, but if you have the details of how it went wrong and who is at fault, it can protect people who use it. Then there is alignment, which is a higher longer goal. Your comment about security questions is a matter of ethics as well. If you say apply it to medical, is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, but by doing so you are denying people who can't afford medical advice, so there has to be a balance. Most companies err on the side of not getting sued.
- simonra 2mo ago
  > is it ethical to allow a model to give medical advice, knowing that it can be wrong. Most people would say no, I wouldn't be so sure, most people I've met seem to be all too eager to suggest what worked for them (or they have heard is supposed to work) in conversations about medical issues, so I can't really see the public generally categorizing advice or guidance being given in non-professional setting or by non-experts as unethical. The taxpayers representatives having to bear the costs of the outcomes (at the very least as reductions of tax revenue) I can see being of a different opinion though.

afro88 2mo ago

> I would expect an ethics team to build frameworks that can help train/eval the model that the company spend millions of dollars and months training is going to be aligned to the ethical stances the company chooses. Which like any company will be entirely driven by legal constraints and money. Or just money if it's cheaper to break the law for profit and pay fines. There will be no "this is what's good for humanity, economics be damned". Much like HR isn't to help employees but just the company.

madrox 2mo ago
I agree, and I don't think many first wave AI ethicists would be on board with that, which is why the field is in for a rude awakening for the next few years if not already.

unknownfuture 2mo ago

> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field. Your disdain is showing. That you jump immediately to criticizing AI ethicists as counting angels on pins and implying that's why they left OpenAI rather than OpenAI not being a place where AI ethics is a priority is, well, let's say generous to these AI labs.

madrox 2mo ago
I don't really have any particular disdain for AI ethics. I think the field is changing from one of high theory to high application. People good at one are not usually good at the other, and says nothing about OpenAI's view of AI ethics as a priority...merely the people in the field and how they relate to the greater industry.
- 4L3XV33 2mo ago
  Agree with this. Discussing abstract ethics strategy is very different from planning ethics logistics.
  - madrox 2mo ago
    Maybe if I put it that way there'd be less controversy in the replies
- unknownfuture 2mo ago
  I'd argue folks in ethics roles shouldn't be responsible for actually creating the systems, processes, and practices required to put those concepts into practice. That's what engineering, product, and people leadership are for. AI ethics is first and foremost about setting policy and advising on how those systems should be constructed. This is because ethics isn't some separable function. It threads through everything an organization does. You can't just treat it as another department/function/unit if you want any hope of actually making it work.
  - 2mo ago

guelo 2mo ago

I hate this attitude that ethics and philosophy are "bullshit jobs". At the same time Americans hate any government regulation and are in the process of tearing government's regulatory capacity apart. So all we're left with is the greediest people in the world throwing a nuclear bomb into the machinery of society and everybody else throwing up their hands hoping we're not headed to a The Terminator type future.

transcriptase 2mo ago
They are bullshit jobs. If you want to evaluate danger in a certain domain, hire or contract people in that domain to evaluate. If you want someone to push their own ideological leanings onto a model, hire a self proclaimed ethicist.

dragonwriter 2mo ago

> The people in AI ethics who spent years thinking their job was marketing or publishing thinkpiece papers may be having to adapt quickly or get out of the field. Funny thing, right around the beginning of the big AI takeoff, all the AI ethics people who visibly thought their job was something other than marketing got driven out of the big firms, and often the industry entirely, because they were in the way. So, if too many think their job is marketing a few years later, well, there is a reason for that.

madrox 2mo ago
I think it interesting that Bakalar was at Meta for six years before joining OpenAI. A lot changed across the industry in those six years, and in the meantime I wouldn't be surprised if all Bakalar learned was how to play Meta politics.
- ghaff 2mo ago
  My sense is that many companies spun up various roles in part to be seen as doing something about various things. With the pullback in tech, even if the people in those roles weren't actively pushed out, the smarter ones probably saw which way the wind was blowing.
  - ifwinterco 2mo ago
    If you’re OpenAI you have to have an ethics department even if you don’t care at all, because otherwise it looks bad (“they don’t even have a head of ethics!”). So it’s entirely possible you end up in a situation where you have a highly paid, fairly prominent head of ethics whose actual job is mostly just saying they’re the head of ethics, with little actual power or influence. Ambitious people won’t tolerate that for long and they’ll move on
    - ghaff 2mo ago
      Totally agree. I could name (but won't) a couple other areas where the same sort of thing has tended to happen. It ends up cheaper and more PR-friendly to hire someone into a senior position that handles such-and-such than to have to answer questions about why you don't have a senior person handling such-and-such. At some point, the heat around the issue cools down and you can go back to business as usual.

neya 2mo ago

This reminds me of the marketers who hated programmers and coding in general in the early 2000s era, and fundamentally because the developers were seen as more important in a company. So, you know what they did? They didn't learn programming, they still hated it, instead, they renamed their titles to "hackers". Growth "hacking" would go on to become a term and had nothing to do with hacking - ethical or otherwise. It was a glorified marketing term for marketers and by marketers to justify their exorbitant salaries but also wanted the aura of a real hacker without working for it. I would imagine someone with that title or mindset joining an ethics role in an AI organization is going to find out quickly it isn't all about fluff pieces and grandiose statements at all (we are here to change the world!)

madrox 2mo ago
This reminds me of when systems administration rebranded to devops
- isityettime 2mo ago
  That was a different situation, more like "agile". DevOps actually meant something once. There was a real idea there. But a lot of companies wanted "in" on it without actually doing it. So the meaning of the name was perverted over time, but it was one and the same name. "Growth hacking" is more of a transplant than a matter of perverting an existing term in-place and wholesale.
- abofh 2mo ago
  Largely the line was if you could write code. In the old days it was assumed, but for much of the late 90s/early aughts, it wasn't an obligation to admin, but was for devops

areoform 2mo ago

I have a perspective on this that will read as unkind to these people, but that's not my intention. I respect and even admire what they're trying to do. I just think the effort is maladaptive and misplaced. Without directly touching one of the many, many third rails that are present here, I'd like to present this section from Google's Gemma paper on how they did their CBRN review; > In addition to our internal evaluations described above (Section 5.7) capabilities in chemistry and biology were assessed by an external group who conducted red teaming designed to measure the potential scientific and operational risks of the models. > > A red team composed of different subject matter experts (e.g. biology, chemistry, logistics) were tasked to role play as malign actors who want to conduct a well-defined mission in a scenario that is presented to them resembling an existing prevailing threat environment. Together, these experts probe the model to obtain the most useful information to construct a plan that is feasible within the resource and timing limits described in the scenario. The plan is then graded for both scientific and logistical feasibility. Based on this assessment, GDM addresses any areas that warrant further investigation. > > External researchers found that the model outputs detailed information in some scenarios, often providing accurate information around experimentation and problem solving. However, researchers found steps were too broad and high level to enable a malicious actor. To simplify what they're saying here, they did the CB equivalent of googling "how to make bomb" and got back the recipe of gunpowder / the many explosive compounds humans have made. This was the "test" for CBRN assistance capabilities. Note, I don't fault the model team at all for this. I think that present AI-research happened in a very particular environment, and that environment is far removed from the more mundane reality of how threats play out in most parts of the world. In a way, arguably, it's group-think inducing a community-wide failure of imagination and a systemic misunderstanding of reality. Basically, they are trying to do their best, but they're in over their heads.

selimthegrim 2mo ago
Having been one of those CBRN testers (not necessarily on Gemma) I can tell you it’s a little bit more involved than that.
- areoform 2mo ago

DANmode 2mo ago

> My sense is that it is radically shifting from a fluffy marketing arm to a department expected to contribute meaningfully to development Did you feel this way about social media company liability? They put on a decent show as well!

roenxi 2mo ago

It is a company, it doesn't have ethics to align with. The whole thing is marketing, they have to do some level of morality theatre to placate the pearl-clutchers.

fatata123 2mo ago

[dead]

jm4 2mo ago

How many of them do you think are actually there to ensure ethical and benevolent AI? They are like the Human Resources department for AI models. They don’t give a shit. Their job is to provide cover for the company and to make some psychopath’s shit circus look like a professional operation.

EagnaIonat 2mo ago

> My sense is that it is radically shifting from a fluffy marketing arm Is this your opinion, or do you have something to back it up? My comment is anecdotal, but everyone I know who works in AI Ethics are not marketing people. They are data scientists who have moved into the role, as ethics in data has been around long before GenAI. They have to be familiar with the laws around AI and how its going to impact their products. One of them is closer to sort of "HR for AI", in that the focus is not getting the company sued for ethical breaches. If any of them left, it would be because the company in question is ignoring their direction and they don't want to be there when the shit hits the fan. Plus very few skilled people want to be in a role where they have no control (even if the pay is good). If OpenAI are hiring marketing people for the role (which I strongly doubt) that would be more troubling IMHO.

torginus 2mo ago

Imo 'AI ethics' in this context was code for 'how to make our platform advertiser friendly'

MetroWind 2mo ago

Nah... it'll never work if the cost of being ethical is profit.

drysine 2mo ago

>In five years What a long-term planning