11 ms·

At the end of the day, we still don't know what exactly happened and probably, never will. However, it seems clear there was a rift between Rapid Commercializat

At the end of the day, we still don't know what exactly happened and probably, never will. However, it seems clear there was a rift between Rapid Commercialization (Team Sam) and Upholding the Original Principles (Team Helen/Ilya). I think the tensions were brewing for quite a while, as it's evident from an article written even before GPT-3 [1]. > Over time, it has allowed a fierce competitiveness and mounting pressure for ever more funding to erode its founding ideals of transparency, openness, and collaboration Team Helen acted in panic, but they believed they would win since they were upholding the principles the org was founded on. But they never had a chance. I think only a minority of the general public truly cares about AI Safety, the rest are happy seeing ChatGPT helping with their homework. I know it's easy to ridicule the sheer stupidity the board acted with (and justifiably so), but take a moment to think of the other side. If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? Honestly, I myself can't take the threat seriously. But, I do want to understand it more deeply than before. Maybe, it isn't without substance as I thought it to be. Hopefully, there won't be a day when Team Helen gets to say, "This is exactly what we wanted to prevent." [1]: https://www.technologyreview.com/2020/02/17/844721/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/ https://www.technologyreview.com/2020/02/17/844721/ai-openai...

loveparade 3y ago
> I think only a minority of the general public truly cares about AI Safety, the rest are happy seeing ChatGPT helping with their homework Not just the public, but also the employees. I doubt there are more than a handful of employees who care about AI Safety.
- deleted 3y ago
  [deleted]
- justrealist 3y ago
  the team is mostly e/acc so you could say they intentionally don't see safety as the end in itself, although I wouldn't quite say they don't care.
- deleted 3y ago
  [deleted]
- concordDance 3y ago
  Nah, a number do, including Sam himself and the entire leadership. They just have different ideas about one or more of: how likely another team is to successfully charge ahead while ignoring safety, how close we are to AGI, how hard alignment is.
est 3y ago
> Rapid Commercialization (Team Sam) and Upholding the Original Principles (Team Helen/Ilya) If you open up openai.com, the navigation menu shows Research, API, ChatGPT, Safety I believe they belong to @ilyasut, @gbd, @sama and Helen Toner respectively?
- ugh123 3y ago
  I have checked View Source and also inspected DOM. Cannot find that.
chrissnow2023 3y ago
[dead]
txnf 3y ago
well said, I would note that both sides recognize that "AGI" will require new uncertain R&D breakthroughs beyond merely scaling up another order of magnitude in compute. given this, i think it's crazy to blow the resources of azure on trying more scale. rapid commercialization at least buys more time for the needed R&D breakthrough to happen.
- consp 3y ago
  All commercialized R&D companies eventually become a hollowed out commercial shell. Why would this be any different?
- Galaxeblaffer 3y ago
  do we really know that scaling compute an order of magnitude won't at least get us close? what other "simple" techniques might actually work with that kind of compute ? at least i was a bit surprised by these first sparks, that seemingly was a matter of enough compute.
silenced_trope 3y ago
Honestly "Safety" is the word in the AI talk that nobody can quantify or qualify in any way when it comes to these conversations. I've stopped caring about anyone who uses the word "safety". It's vague and a hand-waive-y way to paint your opponents as dangerous without any sort of proof or agreed upon standard for who/what/why makes something "safety".
- antupis 3y ago
  I like alignment more it is pretty quantifiable and sometimes it goes against 'safety' because Claude and Openai are censoring models.
- fsloth 3y ago
  Exactly this. The ’safety’ people sound like delusional quacks. ”But they are so smart…” argument is bs. Nobody can be presumed to be super good outside their own specific niche. Linus Pauling and vitamin C. Until we have at least a hint of a mechanistic model if AI driven extinction event, nobody can be an expert on it, and all talk in that vein is self important delusional hogwash. Nobody is pro-apocalypse! We are drowning in things an AI could really help with. With the amount of energy needed for any sort of meaningfull AI results, you can always pull the plug if stuff gets too weird.
  - JumpCrisscross 3y ago
    Now do nuclear.
    - fsloth 3y ago
      War or power production?:) Those are different things. Nuclear war is exactly the kind of thing for which we do have excellent expertise. Unlike for AI safety which seems more like bogus cult atm. Nuclear power would be the best form of large scale power production for many situations. And smaller scale too in forms of emerging SMR:s.
      - 3y ago
swatcoder 3y ago
> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? FWIW, that's called zealotry and people do a lot of dramatic, disruptive things in the name of it. It may be rightly aimed and save the world (or whatever you care about), but it's more often a signal to really reflect on whether you, individually, have really found yourself at the make-or-break nexus of human existence. The answer seems to be "no" most of the time.
- jacobedawson 3y ago
  It's more often a signal to really reflect on whether you, individually as a Thanksgiving turkey, have really found yourself at the make-or-break nexus of turkey existence. The answer seems to be "no" most of the time.
- mlyle 3y ago
  Your comment perfectly justifies never worrying at all about the potential for existential or major risks; after all, one would be wrong most of the time and just engaging in zealotry.
  - RandomLensman 3y ago
    Probably not a bad heuristic: unless proven, don't assume existential risk.
    - altpaddle 3y ago
      Dude just think about that for a moment. By definition if existential risk has been proven. It's already too late
      - RandomLensman 3y ago
        Totally not true: take nuclear weapons, for example, or a large meteorite impact.
arketyp 3y ago
For all the talk about responsible progress, the irony of their inability to align even their own incentives in this enterprise deserves ridicule. It's a big blow to their credibility and questions whatever ethical concerns they hold.
- dmix 3y ago
  It's fear driven as much as moral, which in an emotional humans brain tends to triggers personal ambition to solve it ASAP. A more rational one would realize you need more than just a couple board members to win a major ideological battle. At a minimum something that doesn't immediately result in a backlash where 90% of the engineers most responsible for recent AI dev want you gone, when you're whole plan is to control what those people do.
- concordDance 3y ago
  Alignment is considered an extremely hard problem for a reason. It's already nigh impossible when you're dealing with humans. Btw: do you think ridicule eould be helpful here?
  - arketyp 3y ago
    I can see how ridicule of this specific instance could be the best medicine for an optimal outcome, even by a utilitarian argument, which I generally don't like to make by the way. It is indeed nigh impossible, which is kind of my point. They could have shown more humility. If anything, this whole debacle has been a moral victory for e/acc, seeing how the brightest of minds are at a loss dealing with alignment anyway.
    - FeepingCreature 3y ago
      I don't understand how the conclusion of this is "so we should proceed with AI" rather than "so we should immediately outlaw all foundation model training". Clearly corporate self-governance has failed completely.
      - deleted 3y ago
pknerd 3y ago
Not every sci-fi movie turn to a reality
cornholio 3y ago
What the general public thinks is irrelevant here. The deciding factor was the staff mutiny, without which the organization is an empty shell. And the staff sided with those who aim for rapid real world impact, with directly affects their career and stock options etc. It's also naive to think it was a struggle for principles. The rapid commercialization vs. principles is what the actors claim to rally their respective troops, in reality it was probably a naked power grab, taking advantage of the weak and confuse org structure. Quite an ill prepared move, the "correct" way to oust Altman was to hamstring him in the board and enforce a more and more ceremonial role until he would have quit by himself.
- upwardbound 3y ago
  I think this is an oversimplification and that although the decel faction definitely lost, there are still three independent factions left standing: https://news.ycombinator.com/edit?id=38375767 https://news.ycombinator.com/edit?id=38375767 It will be super interesting to see the subtle struggles for influence between these three.
  - ah765 3y ago
    Adam is likely still on the "decel" faction (although it's unclear whether this is an accurate representation of his beliefs) so I wouldn't really say they lost yet. I'm not sure what faction Bret and Larry will be on. Sam will still have power by virtue of being CEO and aligned with the employees.
- JumpCrisscross 3y ago
  > deciding factor was the staff mutiny The staff never mutinied. They threatened to mutiny. That's a big difference! Yesterday, I compared these rebels to Shockley's "traitorous eight" [1]. But the traitorous eight actually rebelled. These folk put their name on a piece of paper, options and profit participation units safely held in the other hand. [1] https://news.ycombinator.com/item?id=38348123 https://news.ycombinator.com/item?id=38348123
  - 3y ago
deleted 3y ago
[deleted]
nwiswell 3y ago
This is a coherent narrative, but it doesn't explain the bizarre and aggressively worded initial press release. Things perhaps could've been different if they'd pointed to the founding principles / charter and said the board had an intractable difference of opinion with Sam over their interpretation, but then proceeded to thank him profusely for all the work he'd done. Although a suitable replacement CEO out the gate and assurances that employees' PPUs would still see a liquidity event would doubtless have been even more important than a competent statement. Initially I thought for sure Sam had done something criminal, that's how bad the statement was.
- astrange 3y ago
  Apparently the FBI thought he'd done something wrong too, because they called up the board to start an investigation but they didn't have anything. https://x.com/nivi/status/1727152963695808865?s=46 https://x.com/nivi/status/1727152963695808865?s=46
  - gwern 3y ago
    The FBI doesn't investigate things like this on their own, and they definitely do not announce them in the press. The questions you should be asking are (1) who called in the FBI and has the clout to get them to open an investigation into something that obviously has 0% chance of being a federal felony-level crime worth the FBI's time, and (2) who then leaked that 'investigation' to the press?
    - astrange 3y ago
      Sorry, the SDNY. They do do things on their own. I expect the people they called leaked it.
  - dragonwriter 3y ago
    The FBI is not mentioned in that tweet. We don't need to telephone game anonymous leaks that are already almost certainly self-serving propaganda.
casebash 3y ago
Have you seen the Center for AI Safety letter? A lot of experts are worried AI safety could be an x-risk: https://www.safe.ai/statement-on-ai-risk https://www.safe.ai/statement-on-ai-risk
pug_mode 3y ago
I'm convinced there is a certain class of people who gravitate to positions of power, like "moderators", (partisan) journalists, etc. Now, the ultimate moderator role has now been created, more powerful than moderating 1000 subreddits - the AI safety job who will control what AI "thinks"/says for "safety" reasons. Pretty soon AI will be an expert at subtly steering you toward thinking/voting for whatever the "safety" experts want. It's probably convenient for them to have everyone focused on the fear of evil Skynet wiping out humanity, while everyone is distracted from the more likely scenario of people with an agenda controlling the advice given to you by your super intelligent assistant. Because of X, we need to invade this country. Because of Y, we need to pass all these terrible laws limiting freedom. Because of Z, we need to make sure AI is "safe". For this reason, I view "safe" AIs as more dangerous than "unsafe" ones.
- cyanydeez 3y ago
  All you're really describing is why this shouldn't be a non-proft and should just be a government effort. But I assume, from y our language, you'd also object to making this a government utility.
  - sethammons 3y ago
    > should just be a government effort And the controlling party de jour will totally not tweak it to side with their agenda, I'm sure. </s>
    - cyanydeez 3y ago
      uh. We're arguing about _who is controlling AI_. What do you image a neutral party does? If youu're talking about safety, don't you think there should be someone sitting on a boar dsomewhere, contemplating _what should the AI feed today?_ Seriously, why is a non profit, or a business or whatever any different than a government? I get it: there's all kinds of governments, but now theres all kind of businesses. The point of putting it in the governments hand is a defacto acknowledgement that it's a utility. Take other utilities, any time you give a prive org a right to control whether or not you get electricity or water, whats the outcome? Rarely good. If AI is suppose to help society, that's the purview of the government. That's all, you can imagine it's the chinese government, or the russian, or the american or the canadian. They're all _going to do it_, thats _going to happen_, and if a business gets there first, _what is the difference if it's such a powerful device_. I get it, people look dimly on governments, but guess what: they're just as powerful as some organization that gets billions of dollars to effect society. Why is it suddenly a boogeyman?
blackoil 3y ago
> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? No, if OpenAI is reaching singularity, so are Google, Meta, and Baidu etc. so proper course of action would be to loop in NSA/White House. You'll loop in Google, Meta, MSFT and will start mitigation steps. Slowing down OpenAI will hurt the company if assumption is wrong and won't help if it is true. I believe this is more a fight of ego and power than principles and direction.
- ragequittah 3y ago
  >Slowing down OpenAI will hurt the company if assumption is wrong and won't help if it is true. Personally as I watched the nukes be lobbed I'd rather not be the person who helped lob them. And hope to god others look at the same problem (a misaligned AI that is making insane decisions) with the exact same lens. It seems to have worked for nuclear weapons since WW2, one can that we learned a lesson there as a species. The Russian Stanislav Petrov who saved the world comes to mind."Well the Americans have done it anyways" was the motivation and he didn't launch. The cost of error was simply too great.
- concordDance 3y ago
  > so proper course of action would be to loop in NSA/White House Eh? That would be an awful idea. They have no expertise on this and government institutions like thus are misaligned with the rest of humanity by design. E.g. NSA recruits patriots and has many systems, procedures and cultural aspects in place to ensure it keeps up its mission of spying on everyone.
  - the_gipsy 3y ago
    And Google, Facebook, MSFT, Apple, are much more misaligned.
antupis 3y ago
I bet Team Helen will jump slowly to Anthropic, there is no drama, and probably no mainstream news will report this but down-to-line OpenAI will shell off the former self and competitors will catch up.
- tchbnl 3y ago
  With how much of a shitshow this was, I'm not sure Anthropic wants to touch that mess. Wish I was a fly on the wall when the board tried to ask the Anthropic CEO to come back/merge.
jkaplan 3y ago
I feel like the "safety" crowd lost the PR battle, in part, because of framing it as "safety" and over-emphasizing on existential risk. Like you say, not that many people truly take that seriously right now. But even if those types of problems don't surface anytime soon, this wave of AI is almost certainly going to be a powerful, society-altering technology; potentially more powerful than any in decades. We've all seen what can happen when powerful tech is put in the hands of companies and a culture whose only incentives are growth, revenue, and valuation -- the results can be not great. And I'm pretty sure a lot of the general public (and open AI staff) care about THAT. For me, the safety/existential stuff is just one facet of the general problem of trying to align tech companies + their technology with humanity-at-large better than we have been recently. And that's especially important for landscape-altering tech like AI, even if it's not literally existential (although it may be).
- concordDance 3y ago
  > Like you say, not that many people truly take that seriously right now. Eh? Polls on the matter show widespread public support for a pause due to safety concerns.
- cyanydeez 3y ago
  No one who wants to capitalize on AI appears to take it seriously. Especially how grey that safety is. I'm not concerned AI is going to nuke humanity, I'm more concerned it'll re-enforce racism, bias, and the rest of human's irrational activities because it's _blindly_ using existing history to predict future. We've seen it in the past decade in multiple cases. That's safety. The decision that the topic discusses means Business is winning, and they absolutely will re-enforce the idea that the only care is that these systems allow them to re-enforce the business cases. That's bad, and unsafe.
renewiltord 3y ago
This is what people need to understand. It's just like pro-life people. They don't hate you. They think they're saving lives. These people are just as admirably principled as them and they're just trying to make the world a better place.
YetAnotherNick 3y ago
> it seems clear there was a rift between Rapid Commercialization (Team Sam) and Upholding the Original Principles (Team Helen/Ilya) Is it? Why was the press release worded like that? And why did Ilya came up with two mysterious reasons of why board fired Sam if he had quite clearly better and more defendable reason if this goes to court. Also Adam is pro commercialization at least looking at public interviews, no? It's very easy to make the story in brain which involves one character being greedy, but it doesn't seem it is the exact case here.
eslaught 3y ago
Ok, serious question. If you think the threat is real, how are we not already screwed? OpenAI is one of half a dozen teams [0] actively working on this problem, all funded by large public companies with lots of money and lots of talent. They made unique contributions, sure. But they're not that far ahead. If they stumble, surely one of the others will take the lead. Or maybe they will anyway, because who's to say where the next major innovation will come from? So what I don't get about these reactions (allegedly from the board, and expressed here) is, if you interpret the threat as a real one, why are you acting like OpenAI has some infallible lead? This is not an excuse to govern OpenAI poorly, but let's be honest: if the company slows down the most likely outcome by far is that they'll cede the lead to someone else. [0]: To be clear, there are definitely more. Those are just the large and public teams with existing products within some reasonable margin of OpenAI's quality.
- davedx 3y ago
  I don't know. I think being realistic, only OpenAI and Google have the depth and breadth of expertise to develop general AI. Most of the new AI startups are one trick ponies obsessively focused on LLM's. LLM's are only one piece of the puzzle.
  - metanonsense 3y ago
    I would add Meta to this list, in particular because Yann LeCun is the most vocal critic of LLM one-ponyism.
  - MacsHeadroom 3y ago
    Anthropic is made up of former top OpenAI employees, has similar funding, and has produced similarly capable models on a similar timeline. The Claude series is neck and neck with GPT.
- concordDance 3y ago
  > If you think the threat is real, how are we not already screwed? That's the current Yudkowsky view. That it's essentially impossible at this point and we're doomed, but we might as well try anyway as its more "dignified" to die trying. I'm a bit more optimistic myself.
RHSman2 3y ago
Money, large amounts, will always win at scale (unfortunately).
AmericanOP 3y ago
It is a little amusing that we've crowned OpenAI as the destined mother of AGI long before the little sentient chickens have hatched.
theonemind 3y ago
I don't care about AI Safety, but: https://openai.com/charter https://openai.com/charter above that in the charter is "Broadly distributed benefits", with details like: """ Broadly distributed benefits We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power. Our primary fiduciary duty is to humanity. We anticipate needing to marshal substantial resources to fulfill our mission, but will always diligently act to minimize conflicts of interest among our employees and stakeholders that could compromise broad benefit. """ In that sense, I definitely hate to see rapid commercialization and Microsoft's hands in it. I feel like the only person on HN that actually wanted to see Team Sam lose, although it's pretty clear Team Helen/Ilya didn't have a chance, the org just looks hijacked by SV tech bros to me, but I feel like HN has a blindspot to seeing that at all and considering it anything other than a good thing if they do see it. Although GPT barely looks like the language module of AGI to me and I don't see any way there from here (part of the reason I don't see any safety concern). The big breakthrough here relative to earlier AI research is massive amounts more compute power and a giant pile of data, but it's not doing some kind of truly novel information synthesis at all. It can describe quantum mechanics from a giant pile of data, but I don't think it has a chance of discovering quantum mechanics, and I don't think that's just because it can't see, hear, etc., but a limitation of the kind of information manipulation it's doing. It looks impressive because it's reflecting our own intelligence back at us.
krisoft 3y ago
> Honestly, I myself can't take the threat seriously. But, I do want to understand it more deeply than before. I very much recommend reading the book “Superintelligence: Paths, Dangers, Strategies” from Nick Bostrom. It is a seminal work which provides a great introduction into these ideas and concepts. I found myself in the same boat as you do. I was seeing otherwise inteligent and rational people worry about this “fairy tale” of some AI uprising. Reading that book give me an appreciation of the idea as a serious intelectual excercise. I still don’t agree with everything contained in the book. And definietly don’t agree with everything the AI doomsayers write, but i believe if more people would read it that would elevate the discourse. Instead of rehashing the basics again and again we could build on them.
- Solvency 3y ago
  Who needs a book to understand the crazy overwhelming scale at which AI can dictate even online news/truth/discourse/misinformation/propaganda. And that's just barely the beginning.
  - krisoft 3y ago
    Not sure if you are sarcastic or not. :) Let’s assume you are not: The cool thing is that it doesn’t only talk about AIs. It talks about a more general concept it calls a superinteligence. It has a definition but I recommend you read the book for it. :) AIs are just one of the few enumerated possible implementations of a superinteligence. The other type is for example corporations. This is a usefull perspective because it lets us recognise that our attempts to control AIs is not a new thing. We have the same principal-agent control problem in many other parts of our life. How do you know the company you invest in has interests which align with yours? How do you know that politicians and parties you vote for represent your interests? How do you know your lawyer/accountant/doctor has your interest at their hearth? (Not all of these are superinteligences, but you get the gist.)
    - cyanydeez 3y ago
      I wonder how much this is connected to the "effective altruism" movement which seems to project this idea that the "ends justify the means" in a very complex matter, where it suggests such badly formulated ideas like "If we invest in oil companies, we can use that investment to fight climate change". I'd sayu the AI safety problem as a whole is similar to the safety problem of eugenics: Just because you know what the "goal" of some isolated system is, that does not mean you know what the outcome is of implementing that goal on a broad scale. So OpenAI has the same problem: They definitely know what the goal is, but they're not prepared _in any meaningful sense_ for what the broadscale outcome is. If you really care about AI safety, you'd be putting it under government control as utility, like everything else. That's all. That's why government exists.
two_in_one 3y ago
> there was a rift between Rapid Commercialization (Team Sam) and Upholding the Original Principles Seams very unlikely, board could communicate that. Instead they invented some BS reasons, which nobody took as a truth. It looks like more personal and power grab. The staff voted for monetization, people en mass don't care much about high principals. Also nobody wants to work under inadequate leadership. Looks like Ilya lost his bet, or Sam is going to keep him around?
nopinsight 3y ago
Both sides of the rift in fact care a great deal about AI Safety. Sam himself helped draft the OpenAI charter and structure its governance which focuses on AI Safety and benefits to humanity. The main reason of the disagreement is the approach they deem best: * Sam and Greg appear to believe OpenAI should move toward AGI as fast as possible because the longer they wait, the more likely it would lead to the proliferation of powerful AGI systems due to GPU overhang. Why? With more computational power at one's dispense, it's easier to find an algorithm, even a suboptimal one, to train an AGI. As a glimpse on how an AI can be harmful, this paper explores how LLMs can be used to aid in Large-Scale Biological Attacks https://www.rand.org/pubs/research_reports/RRA2977-1.html https://www.rand.org/pubs/research_reports/RRA2977-1.html? What if dozens other groups become armed with means to perform such an attack like this? https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack We know that there're quite a few malicious human groups who would use any means necessary to destroy another group, even at a serious cost to themselves. So the widespread availability of unmonitored AGI would be quite troublesome. * Helen and Ilya might believe it's better to slow down AGI development until we find technical means to deeply align an AGI with humanity first. This July, OpenAI started the Superalignment team with Ilya as a co-lead: https://openai.com/blog/introducing-superalignment https://openai.com/blog/introducing-superalignment But no one anywhere found a good technique to ensure alignment yet and it appears OpenAI's newest internal model has a significant capability leap, which could have led Ilya to make the decision he did. (Sam revealed during the APEC Summit that he observed the advance just a couple of weeks ago and it was only the fourth time he saw that kind of leap.)
- concordDance 3y ago
  So Sam wants to make AGI without working to be sure it doesn't have goals higher than the preservation of human value?! I can't believe that
  - nopinsight 3y ago
    No, I didn't say that. They formed the Superalignment team with Ilya as a co-lead (and Sam's approval) for that. https://openai.com/blog/introducing-superalignment https://openai.com/blog/introducing-superalignment I presume the current alignment approach is sufficient for the AI they make available to others and, in any event, GPT-n is within OpenAI's control.
_fizz_buzz_ 3y ago
I am still a bit puzzled that it is so easy to turn a non-profit into a for profit company. I am sure everything they did is legal, but it feels like it shouldn't be. Could Médecins Sans Frontières take in donations and then take that money to start a for profit hospital for plastics surgery? And the profits wouldn't even go back to MSF, but instead somehow private investors will get the profits. The whole construct just seems wrong.
- IanCal 3y ago
  Well, if it aligned with their goals, sure I think. Let's make the situation a little different. Could MSF pay a private surgery with investors to perform reconstruction for someone? Could they pay the surgery to perform some amount of work they deem aligns with their charter? Could they invest in the surgery under the condition that they have some control over the practices there? (Edit - e.g. perform Y surgeries, only perform from a set of reconstructive ones, patients need to be approved as in need by a board, etc) Raising private investment allows a non profit to shift cost and risk to other entities. The problem really only comes when the structure doesn't align with the intended goals - which is something distinct to the structure, just something non profits can do.
  - framapotari 3y ago
    The non-profit wasn't raising private investment.
    - IanCal 3y ago
      Nothing I've said suggests that or requires that.
      - framapotari 3y ago
        Apologies, I mistook this: "Raising private investment allows a non profit to shift cost and risk to other entities." for a suggestion of that.
- 3y ago
rurban 3y ago
Team Helen seems to be CIA and Military, if I glance over their safety paper. Controlling the narrative, not the damage.
shrikant 3y ago
> Upholding the Original Principles [of AI] There's a UtopAI / utopia joke in there somewhere, was that intentional on your part?
deleted 3y ago
[deleted]
ah765 3y ago
One funny thing about this mess is that "Team Helen" has never mentioned anything about safety, and Emmett said "The board did not remove Sam over any specific disagreement on safety". The reason everyone thinks it's about safety seems largely because a lot of e/acc people on Twitter keep bringing it up as a strawman. Of course, it might end up that it really was about safety in the end, but for now I still haven't seen any evidence. The story about Sam trying to get board control and the board retaliating seems more plausible given what's actually happened.
- rcMgD2BwE72F 3y ago
  >The story about Sam trying to get board control and the board retaliating seems more plausible given what's actually happened. What story? Any link?
mise_en_place 3y ago
A board still has a fiduciary duty to its shareholders. It’s materially irrelevant if those shareholders are of a public or private entity, or whether the company in question is a non-profit or for-profit. Laws mean something, and selective enforcement will only further the decay of the rule of law in the West.
deleted 3y ago
[deleted]
dlkf 3y ago
> I know it's easy to ridicule the sheer stupidity the board acted with (and justifiably so), but take a moment to think of the other side. If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? The real ”sheer stupidity” is this very belief.
sampo 3y ago
> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? In the 1990s and the 00s, it was no too uncommon for anti-GMO environmental activist / ecoterrorist groups to firebomb research facilities and to enter farms and fields to destroy planted GMO plants. Earth Liberation Front was only one of such activist groups [1]. We have yet to see even one bombing of an AI research lab. If people really are afraid of AIs, at least they do so more in the abstract and are not employing the tactics of more traditional activist movements. [1] https://en.wikipedia.org/wiki/Earth_Liberation_Front#Notable_attacks:_1998%E2%80%932009 https://en.wikipedia.org/wiki/Earth_Liberation_Front#Notable...
- concordDance 3y ago
  It's mostly that it's a can of worms no one wants to open. Very much a last resort as its very tricky to use uncoordinated violence effectively (just killing Sam, LeCunn and Greg doesnt do too much to move the needle and then everyond armors up) and very hard to coordinate violence without a leak.
pk-protect-ai 3y ago
> Honestly, I myself can't take the threat seriously. But, I do want to understand it more deeply than before. I believe this position reflects the thoughts of the majority of AI researchers, including myself. It is concerning that we do not fully understand something as promising and potentially dangerous as AI. I'm actually on Ilya's side; labeling his attempt to uphold the original OpenAI principles as an act of "coup" is what is happening now.
lewhoo 3y ago
I think only a minority of the general public truly cares about AI Safety That doesn't matter that much. If your analysis is correct then it means a (tiny) minority of OpenAI cares about AI safety. I hope this isn't the case.
soci 3y ago
The Technologyreview article mentioned in the parent’s first paragraph is the most insightful piece of content I’ve read about the tensions inside OpenAI.
wouldbecouldbe 3y ago
Would have been interesting if they appointed a co-ceo. That still might be the way to go.
cyanydeez 3y ago
I think you analysis is missing the key problem: Business interests. The public don't calculate into whats happening here. There's people using ChatGPT for real "business value" and _that_ is what was threatened. It's clear Business Interests could not be stopped.
nashashmi 3y ago
Helen could have one. She just had to publicly humiliate Sam. She didn't. Employees took over like a mob. Investors pressured board. Board is out. Sam is in. Employees look like they have say. But really, Sam has say. And MSFT is the kingmaker.
gandutraveler 3y ago
Honestly I feel that we will never be able to preemptively build safety without encountering the real risk or threat. Incrementally improving AI capabilities is the only way to do that.
deleted 3y ago
[deleted]
qudat 3y ago
> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? No, because it is an effort in futility. We are evolving into extinction and there is nothing we can do about it. https://bower.sh/in-love-with-a-ghost https://bower.sh/in-love-with-a-ghost

loveparade 3y ago

> I think only a minority of the general public truly cares about AI Safety, the rest are happy seeing ChatGPT helping with their homework Not just the public, but also the employees. I doubt there are more than a handful of employees who care about AI Safety.

deleted 3y ago
[deleted]
justrealist 3y ago
the team is mostly e/acc so you could say they intentionally don't see safety as the end in itself, although I wouldn't quite say they don't care.
deleted 3y ago
[deleted]
concordDance 3y ago
Nah, a number do, including Sam himself and the entire leadership. They just have different ideas about one or more of: how likely another team is to successfully charge ahead while ignoring safety, how close we are to AGI, how hard alignment is.

est 3y ago

> Rapid Commercialization (Team Sam) and Upholding the Original Principles (Team Helen/Ilya) If you open up openai.com, the navigation menu shows Research, API, ChatGPT, Safety I believe they belong to @ilyasut, @gbd, @sama and Helen Toner respectively?

ugh123 3y ago
I have checked View Source and also inspected DOM. Cannot find that.

chrissnow2023 3y ago

[dead]

txnf 3y ago

well said, I would note that both sides recognize that "AGI" will require new uncertain R&D breakthroughs beyond merely scaling up another order of magnitude in compute. given this, i think it's crazy to blow the resources of azure on trying more scale. rapid commercialization at least buys more time for the needed R&D breakthrough to happen.

consp 3y ago
All commercialized R&D companies eventually become a hollowed out commercial shell. Why would this be any different?
Galaxeblaffer 3y ago
do we really know that scaling compute an order of magnitude won't at least get us close? what other "simple" techniques might actually work with that kind of compute ? at least i was a bit surprised by these first sparks, that seemingly was a matter of enough compute.

silenced_trope 3y ago

Honestly "Safety" is the word in the AI talk that nobody can quantify or qualify in any way when it comes to these conversations. I've stopped caring about anyone who uses the word "safety". It's vague and a hand-waive-y way to paint your opponents as dangerous without any sort of proof or agreed upon standard for who/what/why makes something "safety".

antupis 3y ago
I like alignment more it is pretty quantifiable and sometimes it goes against 'safety' because Claude and Openai are censoring models.
fsloth 3y ago
Exactly this. The ’safety’ people sound like delusional quacks. ”But they are so smart…” argument is bs. Nobody can be presumed to be super good outside their own specific niche. Linus Pauling and vitamin C. Until we have at least a hint of a mechanistic model if AI driven extinction event, nobody can be an expert on it, and all talk in that vein is self important delusional hogwash. Nobody is pro-apocalypse! We are drowning in things an AI could really help with. With the amount of energy needed for any sort of meaningfull AI results, you can always pull the plug if stuff gets too weird.
- JumpCrisscross 3y ago
  Now do nuclear.
  - fsloth 3y ago
    War or power production?:) Those are different things. Nuclear war is exactly the kind of thing for which we do have excellent expertise. Unlike for AI safety which seems more like bogus cult atm. Nuclear power would be the best form of large scale power production for many situations. And smaller scale too in forms of emerging SMR:s.
    - 3y ago

swatcoder 3y ago

> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? FWIW, that's called zealotry and people do a lot of dramatic, disruptive things in the name of it. It may be rightly aimed and save the world (or whatever you care about), but it's more often a signal to really reflect on whether you, individually, have really found yourself at the make-or-break nexus of human existence. The answer seems to be "no" most of the time.

jacobedawson 3y ago
It's more often a signal to really reflect on whether you, individually as a Thanksgiving turkey, have really found yourself at the make-or-break nexus of turkey existence. The answer seems to be "no" most of the time.
mlyle 3y ago
Your comment perfectly justifies never worrying at all about the potential for existential or major risks; after all, one would be wrong most of the time and just engaging in zealotry.
- RandomLensman 3y ago
  Probably not a bad heuristic: unless proven, don't assume existential risk.
  - altpaddle 3y ago
    Dude just think about that for a moment. By definition if existential risk has been proven. It's already too late
    - RandomLensman 3y ago
      Totally not true: take nuclear weapons, for example, or a large meteorite impact.

arketyp 3y ago

For all the talk about responsible progress, the irony of their inability to align even their own incentives in this enterprise deserves ridicule. It's a big blow to their credibility and questions whatever ethical concerns they hold.

dmix 3y ago
It's fear driven as much as moral, which in an emotional humans brain tends to triggers personal ambition to solve it ASAP. A more rational one would realize you need more than just a couple board members to win a major ideological battle. At a minimum something that doesn't immediately result in a backlash where 90% of the engineers most responsible for recent AI dev want you gone, when you're whole plan is to control what those people do.
concordDance 3y ago
Alignment is considered an extremely hard problem for a reason. It's already nigh impossible when you're dealing with humans. Btw: do you think ridicule eould be helpful here?
- arketyp 3y ago
  I can see how ridicule of this specific instance could be the best medicine for an optimal outcome, even by a utilitarian argument, which I generally don't like to make by the way. It is indeed nigh impossible, which is kind of my point. They could have shown more humility. If anything, this whole debacle has been a moral victory for e/acc, seeing how the brightest of minds are at a loss dealing with alignment anyway.
  - FeepingCreature 3y ago
    I don't understand how the conclusion of this is "so we should proceed with AI" rather than "so we should immediately outlaw all foundation model training". Clearly corporate self-governance has failed completely.
    - deleted 3y ago

pknerd 3y ago

Not every sci-fi movie turn to a reality

cornholio 3y ago

What the general public thinks is irrelevant here. The deciding factor was the staff mutiny, without which the organization is an empty shell. And the staff sided with those who aim for rapid real world impact, with directly affects their career and stock options etc. It's also naive to think it was a struggle for principles. The rapid commercialization vs. principles is what the actors claim to rally their respective troops, in reality it was probably a naked power grab, taking advantage of the weak and confuse org structure. Quite an ill prepared move, the "correct" way to oust Altman was to hamstring him in the board and enforce a more and more ceremonial role until he would have quit by himself.

upwardbound 3y ago
I think this is an oversimplification and that although the decel faction definitely lost, there are still three independent factions left standing: https://news.ycombinator.com/edit?id=38375767 https://news.ycombinator.com/edit?id=38375767 It will be super interesting to see the subtle struggles for influence between these three.
- ah765 3y ago
  Adam is likely still on the "decel" faction (although it's unclear whether this is an accurate representation of his beliefs) so I wouldn't really say they lost yet. I'm not sure what faction Bret and Larry will be on. Sam will still have power by virtue of being CEO and aligned with the employees.
JumpCrisscross 3y ago
> deciding factor was the staff mutiny The staff never mutinied. They threatened to mutiny. That's a big difference! Yesterday, I compared these rebels to Shockley's "traitorous eight" [1]. But the traitorous eight actually rebelled. These folk put their name on a piece of paper, options and profit participation units safely held in the other hand. [1] https://news.ycombinator.com/item?id=38348123 https://news.ycombinator.com/item?id=38348123
- 3y ago

deleted 3y ago

[deleted]

nwiswell 3y ago

This is a coherent narrative, but it doesn't explain the bizarre and aggressively worded initial press release. Things perhaps could've been different if they'd pointed to the founding principles / charter and said the board had an intractable difference of opinion with Sam over their interpretation, but then proceeded to thank him profusely for all the work he'd done. Although a suitable replacement CEO out the gate and assurances that employees' PPUs would still see a liquidity event would doubtless have been even more important than a competent statement. Initially I thought for sure Sam had done something criminal, that's how bad the statement was.

astrange 3y ago
Apparently the FBI thought he'd done something wrong too, because they called up the board to start an investigation but they didn't have anything. https://x.com/nivi/status/1727152963695808865?s=46 https://x.com/nivi/status/1727152963695808865?s=46
- gwern 3y ago
  The FBI doesn't investigate things like this on their own, and they definitely do not announce them in the press. The questions you should be asking are (1) who called in the FBI and has the clout to get them to open an investigation into something that obviously has 0% chance of being a federal felony-level crime worth the FBI's time, and (2) who then leaked that 'investigation' to the press?
  - astrange 3y ago
    Sorry, the SDNY. They do do things on their own. I expect the people they called leaked it.
- dragonwriter 3y ago
  The FBI is not mentioned in that tweet. We don't need to telephone game anonymous leaks that are already almost certainly self-serving propaganda.

casebash 3y ago

Have you seen the Center for AI Safety letter? A lot of experts are worried AI safety could be an x-risk: https://www.safe.ai/statement-on-ai-risk https://www.safe.ai/statement-on-ai-risk

pug_mode 3y ago

I'm convinced there is a certain class of people who gravitate to positions of power, like "moderators", (partisan) journalists, etc. Now, the ultimate moderator role has now been created, more powerful than moderating 1000 subreddits - the AI safety job who will control what AI "thinks"/says for "safety" reasons. Pretty soon AI will be an expert at subtly steering you toward thinking/voting for whatever the "safety" experts want. It's probably convenient for them to have everyone focused on the fear of evil Skynet wiping out humanity, while everyone is distracted from the more likely scenario of people with an agenda controlling the advice given to you by your super intelligent assistant. Because of X, we need to invade this country. Because of Y, we need to pass all these terrible laws limiting freedom. Because of Z, we need to make sure AI is "safe". For this reason, I view "safe" AIs as more dangerous than "unsafe" ones.

cyanydeez 3y ago
All you're really describing is why this shouldn't be a non-proft and should just be a government effort. But I assume, from y our language, you'd also object to making this a government utility.
- sethammons 3y ago
  > should just be a government effort And the controlling party de jour will totally not tweak it to side with their agenda, I'm sure. </s>
  - cyanydeez 3y ago
    uh. We're arguing about _who is controlling AI_. What do you image a neutral party does? If youu're talking about safety, don't you think there should be someone sitting on a boar dsomewhere, contemplating _what should the AI feed today?_ Seriously, why is a non profit, or a business or whatever any different than a government? I get it: there's all kinds of governments, but now theres all kind of businesses. The point of putting it in the governments hand is a defacto acknowledgement that it's a utility. Take other utilities, any time you give a prive org a right to control whether or not you get electricity or water, whats the outcome? Rarely good. If AI is suppose to help society, that's the purview of the government. That's all, you can imagine it's the chinese government, or the russian, or the american or the canadian. They're all _going to do it_, thats _going to happen_, and if a business gets there first, _what is the difference if it's such a powerful device_. I get it, people look dimly on governments, but guess what: they're just as powerful as some organization that gets billions of dollars to effect society. Why is it suddenly a boogeyman?

blackoil 3y ago

> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? No, if OpenAI is reaching singularity, so are Google, Meta, and Baidu etc. so proper course of action would be to loop in NSA/White House. You'll loop in Google, Meta, MSFT and will start mitigation steps. Slowing down OpenAI will hurt the company if assumption is wrong and won't help if it is true. I believe this is more a fight of ego and power than principles and direction.

ragequittah 3y ago
>Slowing down OpenAI will hurt the company if assumption is wrong and won't help if it is true. Personally as I watched the nukes be lobbed I'd rather not be the person who helped lob them. And hope to god others look at the same problem (a misaligned AI that is making insane decisions) with the exact same lens. It seems to have worked for nuclear weapons since WW2, one can that we learned a lesson there as a species. The Russian Stanislav Petrov who saved the world comes to mind."Well the Americans have done it anyways" was the motivation and he didn't launch. The cost of error was simply too great.
concordDance 3y ago
> so proper course of action would be to loop in NSA/White House Eh? That would be an awful idea. They have no expertise on this and government institutions like thus are misaligned with the rest of humanity by design. E.g. NSA recruits patriots and has many systems, procedures and cultural aspects in place to ensure it keeps up its mission of spying on everyone.
- the_gipsy 3y ago
  And Google, Facebook, MSFT, Apple, are much more misaligned.

antupis 3y ago

I bet Team Helen will jump slowly to Anthropic, there is no drama, and probably no mainstream news will report this but down-to-line OpenAI will shell off the former self and competitors will catch up.

tchbnl 3y ago
With how much of a shitshow this was, I'm not sure Anthropic wants to touch that mess. Wish I was a fly on the wall when the board tried to ask the Anthropic CEO to come back/merge.

jkaplan 3y ago

I feel like the "safety" crowd lost the PR battle, in part, because of framing it as "safety" and over-emphasizing on existential risk. Like you say, not that many people truly take that seriously right now. But even if those types of problems don't surface anytime soon, this wave of AI is almost certainly going to be a powerful, society-altering technology; potentially more powerful than any in decades. We've all seen what can happen when powerful tech is put in the hands of companies and a culture whose only incentives are growth, revenue, and valuation -- the results can be not great. And I'm pretty sure a lot of the general public (and open AI staff) care about THAT. For me, the safety/existential stuff is just one facet of the general problem of trying to align tech companies + their technology with humanity-at-large better than we have been recently. And that's especially important for landscape-altering tech like AI, even if it's not literally existential (although it may be).

concordDance 3y ago
> Like you say, not that many people truly take that seriously right now. Eh? Polls on the matter show widespread public support for a pause due to safety concerns.
cyanydeez 3y ago
No one who wants to capitalize on AI appears to take it seriously. Especially how grey that safety is. I'm not concerned AI is going to nuke humanity, I'm more concerned it'll re-enforce racism, bias, and the rest of human's irrational activities because it's _blindly_ using existing history to predict future. We've seen it in the past decade in multiple cases. That's safety. The decision that the topic discusses means Business is winning, and they absolutely will re-enforce the idea that the only care is that these systems allow them to re-enforce the business cases. That's bad, and unsafe.

renewiltord 3y ago

This is what people need to understand. It's just like pro-life people. They don't hate you. They think they're saving lives. These people are just as admirably principled as them and they're just trying to make the world a better place.

YetAnotherNick 3y ago

> it seems clear there was a rift between Rapid Commercialization (Team Sam) and Upholding the Original Principles (Team Helen/Ilya) Is it? Why was the press release worded like that? And why did Ilya came up with two mysterious reasons of why board fired Sam if he had quite clearly better and more defendable reason if this goes to court. Also Adam is pro commercialization at least looking at public interviews, no? It's very easy to make the story in brain which involves one character being greedy, but it doesn't seem it is the exact case here.

eslaught 3y ago

Ok, serious question. If you think the threat is real, how are we not already screwed? OpenAI is one of half a dozen teams [0] actively working on this problem, all funded by large public companies with lots of money and lots of talent. They made unique contributions, sure. But they're not that far ahead. If they stumble, surely one of the others will take the lead. Or maybe they will anyway, because who's to say where the next major innovation will come from? So what I don't get about these reactions (allegedly from the board, and expressed here) is, if you interpret the threat as a real one, why are you acting like OpenAI has some infallible lead? This is not an excuse to govern OpenAI poorly, but let's be honest: if the company slows down the most likely outcome by far is that they'll cede the lead to someone else. [0]: To be clear, there are definitely more. Those are just the large and public teams with existing products within some reasonable margin of OpenAI's quality.

davedx 3y ago
I don't know. I think being realistic, only OpenAI and Google have the depth and breadth of expertise to develop general AI. Most of the new AI startups are one trick ponies obsessively focused on LLM's. LLM's are only one piece of the puzzle.
- metanonsense 3y ago
  I would add Meta to this list, in particular because Yann LeCun is the most vocal critic of LLM one-ponyism.
- MacsHeadroom 3y ago
  Anthropic is made up of former top OpenAI employees, has similar funding, and has produced similarly capable models on a similar timeline. The Claude series is neck and neck with GPT.
concordDance 3y ago
> If you think the threat is real, how are we not already screwed? That's the current Yudkowsky view. That it's essentially impossible at this point and we're doomed, but we might as well try anyway as its more "dignified" to die trying. I'm a bit more optimistic myself.

RHSman2 3y ago

Money, large amounts, will always win at scale (unfortunately).

AmericanOP 3y ago

It is a little amusing that we've crowned OpenAI as the destined mother of AGI long before the little sentient chickens have hatched.

theonemind 3y ago

I don't care about AI Safety, but: https://openai.com/charter https://openai.com/charter above that in the charter is "Broadly distributed benefits", with details like: """ Broadly distributed benefits We commit to use any influence we obtain over AGI’s deployment to ensure it is used for the benefit of all, and to avoid enabling uses of AI or AGI that harm humanity or unduly concentrate power. Our primary fiduciary duty is to humanity. We anticipate needing to marshal substantial resources to fulfill our mission, but will always diligently act to minimize conflicts of interest among our employees and stakeholders that could compromise broad benefit. """ In that sense, I definitely hate to see rapid commercialization and Microsoft's hands in it. I feel like the only person on HN that actually wanted to see Team Sam lose, although it's pretty clear Team Helen/Ilya didn't have a chance, the org just looks hijacked by SV tech bros to me, but I feel like HN has a blindspot to seeing that at all and considering it anything other than a good thing if they do see it. Although GPT barely looks like the language module of AGI to me and I don't see any way there from here (part of the reason I don't see any safety concern). The big breakthrough here relative to earlier AI research is massive amounts more compute power and a giant pile of data, but it's not doing some kind of truly novel information synthesis at all. It can describe quantum mechanics from a giant pile of data, but I don't think it has a chance of discovering quantum mechanics, and I don't think that's just because it can't see, hear, etc., but a limitation of the kind of information manipulation it's doing. It looks impressive because it's reflecting our own intelligence back at us.

krisoft 3y ago

> Honestly, I myself can't take the threat seriously. But, I do want to understand it more deeply than before. I very much recommend reading the book “Superintelligence: Paths, Dangers, Strategies” from Nick Bostrom. It is a seminal work which provides a great introduction into these ideas and concepts. I found myself in the same boat as you do. I was seeing otherwise inteligent and rational people worry about this “fairy tale” of some AI uprising. Reading that book give me an appreciation of the idea as a serious intelectual excercise. I still don’t agree with everything contained in the book. And definietly don’t agree with everything the AI doomsayers write, but i believe if more people would read it that would elevate the discourse. Instead of rehashing the basics again and again we could build on them.

Solvency 3y ago
Who needs a book to understand the crazy overwhelming scale at which AI can dictate even online news/truth/discourse/misinformation/propaganda. And that's just barely the beginning.
- krisoft 3y ago
  Not sure if you are sarcastic or not. :) Let’s assume you are not: The cool thing is that it doesn’t only talk about AIs. It talks about a more general concept it calls a superinteligence. It has a definition but I recommend you read the book for it. :) AIs are just one of the few enumerated possible implementations of a superinteligence. The other type is for example corporations. This is a usefull perspective because it lets us recognise that our attempts to control AIs is not a new thing. We have the same principal-agent control problem in many other parts of our life. How do you know the company you invest in has interests which align with yours? How do you know that politicians and parties you vote for represent your interests? How do you know your lawyer/accountant/doctor has your interest at their hearth? (Not all of these are superinteligences, but you get the gist.)
  - cyanydeez 3y ago
    I wonder how much this is connected to the "effective altruism" movement which seems to project this idea that the "ends justify the means" in a very complex matter, where it suggests such badly formulated ideas like "If we invest in oil companies, we can use that investment to fight climate change". I'd sayu the AI safety problem as a whole is similar to the safety problem of eugenics: Just because you know what the "goal" of some isolated system is, that does not mean you know what the outcome is of implementing that goal on a broad scale. So OpenAI has the same problem: They definitely know what the goal is, but they're not prepared _in any meaningful sense_ for what the broadscale outcome is. If you really care about AI safety, you'd be putting it under government control as utility, like everything else. That's all. That's why government exists.

two_in_one 3y ago

> there was a rift between Rapid Commercialization (Team Sam) and Upholding the Original Principles Seams very unlikely, board could communicate that. Instead they invented some BS reasons, which nobody took as a truth. It looks like more personal and power grab. The staff voted for monetization, people en mass don't care much about high principals. Also nobody wants to work under inadequate leadership. Looks like Ilya lost his bet, or Sam is going to keep him around?

nopinsight 3y ago

Both sides of the rift in fact care a great deal about AI Safety. Sam himself helped draft the OpenAI charter and structure its governance which focuses on AI Safety and benefits to humanity. The main reason of the disagreement is the approach they deem best: * Sam and Greg appear to believe OpenAI should move toward AGI as fast as possible because the longer they wait, the more likely it would lead to the proliferation of powerful AGI systems due to GPU overhang. Why? With more computational power at one's dispense, it's easier to find an algorithm, even a suboptimal one, to train an AGI. As a glimpse on how an AI can be harmful, this paper explores how LLMs can be used to aid in Large-Scale Biological Attacks https://www.rand.org/pubs/research_reports/RRA2977-1.html https://www.rand.org/pubs/research_reports/RRA2977-1.html? What if dozens other groups become armed with means to perform such an attack like this? https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack We know that there're quite a few malicious human groups who would use any means necessary to destroy another group, even at a serious cost to themselves. So the widespread availability of unmonitored AGI would be quite troublesome. * Helen and Ilya might believe it's better to slow down AGI development until we find technical means to deeply align an AGI with humanity first. This July, OpenAI started the Superalignment team with Ilya as a co-lead: https://openai.com/blog/introducing-superalignment https://openai.com/blog/introducing-superalignment But no one anywhere found a good technique to ensure alignment yet and it appears OpenAI's newest internal model has a significant capability leap, which could have led Ilya to make the decision he did. (Sam revealed during the APEC Summit that he observed the advance just a couple of weeks ago and it was only the fourth time he saw that kind of leap.)

concordDance 3y ago
So Sam wants to make AGI without working to be sure it doesn't have goals higher than the preservation of human value?! I can't believe that
- nopinsight 3y ago
  No, I didn't say that. They formed the Superalignment team with Ilya as a co-lead (and Sam's approval) for that. https://openai.com/blog/introducing-superalignment https://openai.com/blog/introducing-superalignment I presume the current alignment approach is sufficient for the AI they make available to others and, in any event, GPT-n is within OpenAI's control.

_fizz_buzz_ 3y ago

I am still a bit puzzled that it is so easy to turn a non-profit into a for profit company. I am sure everything they did is legal, but it feels like it shouldn't be. Could Médecins Sans Frontières take in donations and then take that money to start a for profit hospital for plastics surgery? And the profits wouldn't even go back to MSF, but instead somehow private investors will get the profits. The whole construct just seems wrong.

IanCal 3y ago
Well, if it aligned with their goals, sure I think. Let's make the situation a little different. Could MSF pay a private surgery with investors to perform reconstruction for someone? Could they pay the surgery to perform some amount of work they deem aligns with their charter? Could they invest in the surgery under the condition that they have some control over the practices there? (Edit - e.g. perform Y surgeries, only perform from a set of reconstructive ones, patients need to be approved as in need by a board, etc) Raising private investment allows a non profit to shift cost and risk to other entities. The problem really only comes when the structure doesn't align with the intended goals - which is something distinct to the structure, just something non profits can do.
- framapotari 3y ago
  The non-profit wasn't raising private investment.
  - IanCal 3y ago
    Nothing I've said suggests that or requires that.
    - framapotari 3y ago
      Apologies, I mistook this: "Raising private investment allows a non profit to shift cost and risk to other entities." for a suggestion of that.
3y ago

rurban 3y ago

Team Helen seems to be CIA and Military, if I glance over their safety paper. Controlling the narrative, not the damage.

shrikant 3y ago

> Upholding the Original Principles [of AI] There's a UtopAI / utopia joke in there somewhere, was that intentional on your part?

deleted 3y ago

[deleted]

ah765 3y ago

One funny thing about this mess is that "Team Helen" has never mentioned anything about safety, and Emmett said "The board did not remove Sam over any specific disagreement on safety". The reason everyone thinks it's about safety seems largely because a lot of e/acc people on Twitter keep bringing it up as a strawman. Of course, it might end up that it really was about safety in the end, but for now I still haven't seen any evidence. The story about Sam trying to get board control and the board retaliating seems more plausible given what's actually happened.

rcMgD2BwE72F 3y ago
>The story about Sam trying to get board control and the board retaliating seems more plausible given what's actually happened. What story? Any link?

mise_en_place 3y ago

A board still has a fiduciary duty to its shareholders. It’s materially irrelevant if those shareholders are of a public or private entity, or whether the company in question is a non-profit or for-profit. Laws mean something, and selective enforcement will only further the decay of the rule of law in the West.

deleted 3y ago

[deleted]

dlkf 3y ago

> I know it's easy to ridicule the sheer stupidity the board acted with (and justifiably so), but take a moment to think of the other side. If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? The real ”sheer stupidity” is this very belief.

sampo 3y ago

> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? In the 1990s and the 00s, it was no too uncommon for anti-GMO environmental activist / ecoterrorist groups to firebomb research facilities and to enter farms and fields to destroy planted GMO plants. Earth Liberation Front was only one of such activist groups [1]. We have yet to see even one bombing of an AI research lab. If people really are afraid of AIs, at least they do so more in the abstract and are not employing the tactics of more traditional activist movements. [1] https://en.wikipedia.org/wiki/Earth_Liberation_Front#Notable_attacks:_1998%E2%80%932009 https://en.wikipedia.org/wiki/Earth_Liberation_Front#Notable...

concordDance 3y ago
It's mostly that it's a can of worms no one wants to open. Very much a last resort as its very tricky to use uncoordinated violence effectively (just killing Sam, LeCunn and Greg doesnt do too much to move the needle and then everyond armors up) and very hard to coordinate violence without a leak.

pk-protect-ai 3y ago

> Honestly, I myself can't take the threat seriously. But, I do want to understand it more deeply than before. I believe this position reflects the thoughts of the majority of AI researchers, including myself. It is concerning that we do not fully understand something as promising and potentially dangerous as AI. I'm actually on Ilya's side; labeling his attempt to uphold the original OpenAI principles as an act of "coup" is what is happening now.

lewhoo 3y ago

I think only a minority of the general public truly cares about AI Safety That doesn't matter that much. If your analysis is correct then it means a (tiny) minority of OpenAI cares about AI safety. I hope this isn't the case.

soci 3y ago

The Technologyreview article mentioned in the parent’s first paragraph is the most insightful piece of content I’ve read about the tensions inside OpenAI.

wouldbecouldbe 3y ago

Would have been interesting if they appointed a co-ceo. That still might be the way to go.

cyanydeez 3y ago

I think you analysis is missing the key problem: Business interests. The public don't calculate into whats happening here. There's people using ChatGPT for real "business value" and _that_ is what was threatened. It's clear Business Interests could not be stopped.

nashashmi 3y ago

Helen could have one. She just had to publicly humiliate Sam. She didn't. Employees took over like a mob. Investors pressured board. Board is out. Sam is in. Employees look like they have say. But really, Sam has say. And MSFT is the kingmaker.

gandutraveler 3y ago

Honestly I feel that we will never be able to preemptively build safety without encountering the real risk or threat. Incrementally improving AI capabilities is the only way to do that.

deleted 3y ago

[deleted]

qudat 3y ago

> If you truly believed that Superhuman AI was near, and it could act with malice, won't you try to slow things down a bit? No, because it is an effort in futility. We are evolving into extinction and there is nothing we can do about it. https://bower.sh/in-love-with-a-ghost https://bower.sh/in-love-with-a-ghost