6 ms·
Organisationally it never works to have a group whose only job is to say no to some other group. The incentives are diametrically opposed and as a structure it
by seanhunter 2mo ago
Organisationally it never works to have a group whose only job is to say no to some other group. The incentives are diametrically opposed and as a structure it can’t last.
If you had an AI company and want it to be ethical you have to find a way to make ethics everyone’s responsibility, and have the consequences of poor ethics bite the people who make those bad decisions. If you just outsource it to the ethics group what happens is
1)everyone else thinks they don’t need to worry about ethics
2)the ethics group need to justify their existence so introduce a bunch of guidelines that everyone initially thinks are reasonable but over time people think are increasingly out of touch
3) The ethics group start to “make difficult calls” and say no to things. Initially everyone supports this and feels like the system is working as it should but over time everyone starts to just see them as an obstacle to work around
4)everyone else starts to try to work around what the ethics group says
5)The ethics group grows powerless and disconnected. The people who work around them “get things done” so get promoted etc whereas they only visibly put roadblocks in peoples’ way, so they get sidelined.
6)Eventually they get disbanded with some corporate announcement thanking them for their hard work, thought leadership etc. All that has been achieved is a lot of wasted time and bad blood.
- BloondAndDoom 2mo agoI get what you are saying but that’s exactly how security and compliance work. And I don’t think amount of people who want to be ethical in an organization ia that different than people who want to build secure software (but possibly no one gives a shit compliance other than it’s something that needs to be done)
- tgv 2mo agoThere are at least some penalties for failed security and compliance. Ethics is the first victim of greed.
- dannyw 2mo agoSecurity and compliance tends to be a lot less subjective than ethics. Not saying it’s objective, but there’s a huge difference. The downside is it can often feel like a box-checking exercise than actual security or compliance, but “you need 2FA” is less debatable than, say, AI and copyright.
- antupis 2mo agoBig corporations there is politics on play and too often you see CABs and other bureaucratic stuff instead of checkboxes for gated releases, 2FA etc.
- lnsru 2mo agoDuring my years at big corp we did tons of work just to go around the law department. While for core business it made absolute sense to have lawyers involved in daily business we were satellite office decoupled from core business. Local management failed to communicate that and we were stalked by lawyers from main office. Pretty sure others do the same with compliance, law, ethics departments. Just work around to get things done instead of stuck for weeks in stupid meetings.
- hnlmorg 2mo agoAnd organisations fail at security when security and compliance is only considered important by that one teamnn Security requires the whole business to buy in. And it requires processes that allow people to get shit done without people resorting to shadow IT; thus working around that one team. So the GPs point still works.
- ivan_gammel 2mo agoRule number one of CIO: become friends with CFO and chief lawyer. And nothing else matters.
- close04 2mo agoThe difference between "ethics" and "security and compliance" is that the latter is something that hits inside the company, while the former usually hits outside of the company. Poor security practices harm your teams, your data, and usually you make moderate savings at best. Poor ethics "only" harm your customers while making bank for the company. This is the real problem with ethics in a large corporation. You're not saying "no" to another team, you're saying no to large profits, you're saying no to the company's leadership. That is what never works.
- hnlmorg 2mo ago> The difference between "ethics" and "security and compliance" is that the latter is something that hits inside the company, while the former usually hits outside of the company. I don’t agree with this. Data breaches affect customers more than businesses. If your point were true, we’d see fewer breaches. Plus not all breaches are a result of software engineering teams. For example product managers sharing customer details. I’ve managed plenty of teams where I’ve had to instil the importance of secure best practices at all stages of development. So it’s definitely not something inherently important to all people who work in organisations. Just like with ethics. It’s very easy to dismiss either as an inconvenience if you don’t instil the right company culture at all levels of the organisation. This is why European financial organisations have such strict onboarding procedures to teach new hires about fraud, bribery and other financial misconduct even for issues that are ethical grey rather than outright illegal. Similarly many organisations will have onboarding procedures to teach new hires their security best practices too
- a_bonobo 2mo agoSecurity/compliance have the external hammer: if they fail, your org will have to pay fines and someone may end up being criminally liable (depends on country). The ethics department; if they fail, there may be some negative journalism, but who which AI company has positive journalism these days? There's no external hammer for ethics.
- nine_k 2mo agoThe structure is untenable when the work of one team is to say "no" to other teams, when these teams are not asking. A good infrastructure team would seek a competent security review that would say "no" to problematic things before an intruder says "aha" to them. If feedback from the ethics team is not sought, nobody is going to heed its opinion anyway.
- pyuser583 2mo agoI’ve never known a security team to be in the position to say “no.” The role of security teams vary a lot from one organization to another. But generally they can make you aware of the tradeoffs you are making, or serve as a kind of quality control that generally does not want to be asked questions.
- jiggawatts 2mo ago> exactly how security and compliance Not everywhere. I go out of my way to assist teams to achieve a secure outcome with less effort. Things like: “instead of admin access to the production servers the devs can have fully automated deployment pipelines combined with OpenTelemetry for observability so they don’t have to spend half the day scrolling through gigabytes of logs.” That’s more secure and and more better. Nobody had to be told “no”. Similarly, I replace key store access with secret-less managed identity, etc.
- seanhunter 2mo agoIn a high-functioning security and compliance team they tend to say “no” only in really dire circumstances. Good security and compliance teams spend a lot of time asking exactly what it is the people are really trying to do and then find a way to say “yes, and…” as in “yes you can and here’s how you do it without compromising security/breaking the law etc”. And as a sibling said, orgs fail at security when they make security only the infosec teams’ job. This is also why I said “whose only job”. In a good org, the security team doesn’t only say no to devsecops requests, they also do trainings to skill up other teams, keep the network secure, proactively seek out and understand external threats, work with external vendors etc etc …
- mnahkies 2mo agoI think the key part is about aligning incentives and outcomes - if your security and compliance departments are only judged on "were we breached, did we pass our audits" then there's a risk they won't weigh the tradeoffs associated with the decisions around controls implemented to achieve those outcomes. To use a ridiculous extreme you can't breach a web app that isn't exposed to the internet, but the users can't access it either. If you can connect/balance those goals to other metrics around cost and productivity, usability, and a realistic threat model, as guardrails then you incentatize cross-team collaboration to achieve the shared org outcomes.
- mchinen 2mo agoSRE is a 'say no to powerful people' job as well. I think for this to work the leadership needs to show support for it. The friction is still there, but in more tolerable areas. I bet for ethics this isn't the case at OAI, but everyone values security and stability. For an SRE there can be more directed hate received from the junior employees, that want to release new features they developed. Especially because there is less accountability across orgs. Security is an interesting one because it seems to have less of this friction, maybe because it's more clear cut what is an issue.
- nobodywillobsrv 2mo agoMarkets solve this by pricing risk. Ideally you would have some kind of notion of selling insurance internally and track things. But ultimately existential risk is hard to negotiate from the inside. And companies are supposed to go bust or succeed. They are not really the same as a population trying to survive forever. At least that is one take.
- scelerat 2mo agoThe compliance team seeks to maximize company profit/success within the confines of hard legal boundaries. There are no "hard" boundaries for ethics or "doing the right thing," and so those boundaries will always be pushed until they are useless.
- nakedrobot2 2mo agothis is a strong reminder of the character of Toby from The Office.... :-)
- hypfer 2mo ago> whose only job is to say no to some other group I suspect that people will miss just how precise you've been there. Your proposal - if I understand correctly - is not just to spread the "ethics team" onto everyone (which is correct), but also to have someone in a dedicated role. However, they need to have not _just_ that role, but also some skin in the game.
- bluehatbrit 2mo agoThis is also my view. These sorts of roles can have a very high impact and be very successful if everyone feels like they're rowing in the same direction. The people in those roles need make people feel like there is value in seeking their input. They also need to find ways to say yes that helps things happen in the right way, rather than stopping them entirely. This happens naturally in organisations where security is valued by everyone. InfoSec / CIO roles can operate very successfully and deliver a lot of value. As soon as people lose faith and see them as the "no" team, they start trying to hide from them. There are lots of ways to create this but it's as much about the people in the role, as it about the organisation itself. If either are skeptical about the other, it falls apart very quickly.
- nelox 2mo agoGeneral Counsel has entered the chat
- Rebuff5007 2mo agoShall we get rid of the FAA and the FDA then? Lets just have all pfizer employees make ethics their own responsibility. What your describing is the specific case of a company so paralyzed by short-term thinking that they see regulation as a burden. Some maturity in the organization would allow reframing this to be less antagonistic.
- erehweb 2mo agoCrucially, the FAA is not a part of United Airlines, and the FDA is not a part of Pfizer. If there weren't any FDA, then the safety group in Pfizer would be a lot weaker.
- benjiro29 2mo agoIts funny because "Shall we get rid of the FAA and the FDA then?" ... matches exactly with the current situation of the FDA and Points 3 > 5 ... The systematic nerfing of the FDA and other organizations their power.
- motbus3 2mo agoWell, not when the whole objective of the company is to find ways to say they are ethical while knowing they are not
- dspillett 2mo ago> Organisationally it never works to have a group whose only job is to say no to some other group. That is only true where the pressure to be [the thing that is inconvenient to the other group] is not from a source that can cause massive problems if non-compliance is spotted. When the blocking group is legally mandated or otherwise really has teeth or is defending the company against an external regulator with teeth, then it works better (though obviously not perfectly). Think legal and compliance in banking realms, where the company significantly fined and the people breaking the rules could be sacked & blackballed (though sometimes not the people ordering them to break the rules!) when something bad is noticed. That is quite different to an ethics officer in a company like OpenAI where the position is basically there for PR purposes (“look plebs, we care about doing the right thing, honest, we got a manager with a small team dedicated to it” and “look [government body], we are regulating ourselves, do you really need to spend time looking too?”) and therefore has no real teeth directly or indirectly especially as fault for non-compliance might not be easy to attribute.
- DrScientist 2mo agoAbsolutely - though the flip side of this is if something is everybody responsibility - it's nobodies. There are two things to successful organisations structure and people, you are focusing only on the structure. What you need for somebody responsible for health and safety, ethics, security, or compliance is a person who has courage and is willing to take calculated risks. However these roles do have a tendency to attract the risk adverse, or make them risk adverse if you punish risks that go wrong too harshly. And a key critical success factor is leadership from the top - companies have personalities and leaders in the organisation are very important in shaping that.
- bandrami 2mo agoHard disagree, as someone whose main job is telling developers "no". If it's everybody's responsibility it is in fact nobody's responsibility.
- mrweasel 2mo agoYou risk that it becomes like legal, where the lawyers are concerned with "what you can legally get away with", not what's right. If this person previously worked at Meta, then their experience in guiding company ethics isn't great and I suspect that they are more in the camp of explaining away ethical problems. Anecdotally I think the engineers and people doing the actual work have a better grasp of moral, ethics and the spirit of the law, than people dedicated to those areas, but not enough to walk away.
- plaidfuji 2mo agoIf this were true, food companies wouldn’t have a Regulatory team. Except every single one of them does, and they function similarly to how you describe, but without dissolving. They’re treated as a constraint that must be checked before major projects can advance. Sometimes they’re the longest-lead item on a new project and if you don’t get them involved early, your project can sink at a late stage after great expense. The only difference is AI isn’t regulated, so they just have a vague “ethics” department with no real teeth because it’s essentially PR and has no legal consequences to back up their stance.
- K0balt 2mo agoIn a situation where it doesn’t happen this way, the ethics team grows in power and staffing over time until it destroys the company, because ultimately all commercial activity can be construed as less than perfectly ethical at some level, or it morphs into an elaborate legalese department that masks unethical practices in a cloak of justification. Either way, you need third party regulation and then a department dedicated to ethical compliance, or you end up with corporate cancer. I say this as a person with a knee-jerk anti-regulation reflex.
- phoghed 2mo agoOperations was like this so we added Ops to the Dev. DevOps. Security was like this so we added Sec to the DevOps. DevSecOps. Ethics is now the problem, so we’re going to add Eth to the chain. DevEthSecOps. Legal and compliance is now the final hurdle to shipping fast. AI tools will allow us to implement DevEthSecLawOps at scale no doubt.
- HSO 2mo agoyup, like risk managers in investment banks with their own trading desks. just ngmi
- mwexler 2mo agoIsn't this the exact setup for every compliance, security, or other regulatory group in any company? To say no then "here's what's acceptable that's close" to guide the business or entity? Perhaps "ethics" is a more slippery slope, but the approach is well trodden. It's a choice of company culture from leadership on whether the company acks the suggestions or ignores them. And when they get ignored or drive little impact for the cost, then teams disband. Which is also true for any team at any company.
- deleted 2mo ago[deleted]
- sph 2mo agoIt is good PR to have what is largely ceremonial role that oversees an entire system, whose powers are in practice useless or completely defanged. I’m thinking of the role of the King or Queen of England, for example, or any country’s president. They have the power to disband parliament if it is deemed unfit for government, or reject a law that has been approved by the senate/Lords, yet in practice they are not expected to disagree with the ‘will of the people’ so to speak. Same applies to the head of an ethics department on a business which is obviously unethical (or they wouldn’t need one).
- MetroWind 2mo agoI don't get why in the comments this gets compared with regulations and security teams... Say no to regulations and security --> Fine, forced correction, fewer customers, shutdown. Say no to ethics --> money (at least for AI companies) Why is this so hard to understand?
- dostick 2mo agoIsn’t ethics incompatible with the nature of a commercial enterprise? Now, if OpenAI was a non-profit… as it was intended originally.