24 ms·

Inside The Chaos at OpenAI

281 points by maxutility 3y ago

m15i 3y ago
https://archive.md/PvbqA https://archive.md/PvbqA
- gigel82 3y ago
  That website looks like a Captcha farm scheme.
  - Sirizarry 3y ago
    It’s because of a dispute with cloudflare’s 1.1.1.1 DNS. Try another DNS
    - foobiekr 3y ago
      and don't use apple apple private relay
good_lord_dang 3y ago
[dead]
deleted 3y ago
[deleted]
gumballindie 3y ago
[flagged]
brucethemoose2 3y ago
Well that story is just sad, because it means the principle/research-oriented company structure they set up utterly failed in the face of profit motives. Clearly Altman was not dissuaded from doing things the power structure didn't want him to do.
- EMIRELADERO 3y ago
  What? We don't know what will happen, as the whole thing hasn't finished yet. Nobody can say for sure which side will "win" the "fight", or if there will be any winners at all.
  - brucethemoose2 3y ago
    If Altman is rehired its a total capitulation. If he's not, and literally all the other employees come back, its still a failure of the power structure that already happened. The threat of firing the CEO of the for-profit arm is supposed to be a unused threat, like a nuclear weapon.
    - EMIRELADERO 3y ago
      So what happens when the Altman and the board's views are concluded to be completely incompatible? How do you fix that clash without one side giving up?
      - brucethemoose2 3y ago
        > So what happens when the Altman and the board's views are concluded to be completely incompatible? He gets fired. Thats the enforcement mechanism. Hence if he comes back, that indicates the board essentially "agrees" with Altman's positions now, however much Altman and the board's positions have shifted.
- stevefan1999 3y ago
  In today's world, capitalism, or more like human hubris and greed for the desire of ever-growing process, always prevail in the end, no matter how venerate and honorable your initial goals would be
  - nickthegreek 3y ago
    It’s nice to have this modern clear example now to point to though.
jrm4 3y ago
Lol. So I didn't get past the few paragraphs before the paywall, and I didn't need to. I appreciate the idea of being a "not-greedy typical company," but there's a reason you e.g. separate university type research or non-profits and private companies. Trying to make up something in the middle is the exact sort of naivete you can ALWAYS expect from Silicon Valley.
- cmrdporcupine 3y ago
  Yeah: https://www.youtube.com/watch?v=j_JW9NwjLF8 https://www.youtube.com/watch?v=j_JW9NwjLF8 I'd say yes, Sutsksever is... naive? though very smart. Or just utopian. Seems he couldn't get the scale he needed/wanted out of a university (or Google) research lab. But the former at least would have bounded things better in the way he would have preferred, from an ethics POV. Jumping into bed with Musk and Altman and hoping for ethical non-profit "betterment of humanity" behaviour is laughable. Getting access to capital was obviously tempting, but ... As for Altman. No, he's not naive. Amoral, and likely proud of it. JFC ... Worldcoin... I can't even... I don't want either of these people in charge of the future, frankly. It does point to the general lack of funding for R&D of this type of thing. Or it's still too early to be doing this kind of thing at scale. I dunno. Bleak.
  - jrm4 3y ago
    Bleak? Why? Honestly, this seems like a pretty good outcome. Which is to say, I think the fearmongering sentient AI stuff is silly -- but I think we are all DEFINITELY better off with an ugly rough-and-tumble visible rocky start to the AI revolution. Weed out the BS; equalize out who actually has access to the best stuff BEFORE some jerk company can scale up fast and dominate market share; let a de-facto "open source" market have a go at the whole thing.
    - cmrdporcupine 3y ago
      The bleak reality is that OpenAI became that jerk company, and now in its effective demise, those reigns are handed over to Microsoft. Who have already demonstrated (with CoPilot) a similar lack of ... concern ... for IP rights / authorship, and various other ethical aspects. And bleak because there doesn't seem to be an alternative where the people making these decisions are responsible to an electorate or public in some democratic fashion. Just a bunch of people with $$ and influence who set themselves up to be arbiters ... and It's just might makes right. And bleak because in this case the "mighty" are often the very people who made fun of arts students who took the philosophy and ethics classes in school that could at least offer some insight in these issues.
maxutility 3y ago
A few interesting tidbits > The company pressed forward and launched ChatGPT on November 30. It was considered such a nonevent that no major company-wide announcement about the chatbot going live was made. Many employees who weren’t directly involved, including those in safety functions, didn’t even realize it had happened. Some of those who were aware, according to one employee, had started a betting pool, wagering how many people might use the tool during its first week. The highest guess was 100,000 users. OpenAI’s president tweeted that the tool hit 1 million within the first five days. The phrase low-key research preview became an instant meme within OpenAI; employees turned it into laptop stickers. > Anticipating the arrival of [AGI], Sutskever began to behave like a spiritual leader, three employees who worked with him told us. His constant, enthusiastic refrain was “feel the AGI,” a reference to the idea that the company was on the cusp of its ultimate goal. At OpenAI’s 2022 holiday party, held at the California Academy of Sciences, Sutskever led employees in a chant: “Feel the AGI! Feel the AGI!” The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack. > For a leadership offsite this year, according to two people familiar with the event, Sutskever commissioned a wooden effigy from a local artist that was intended to represent an “unaligned” AI—that is, one that does not meet a human’s objectives. He set it on fire to symbolize OpenAI’s commitment to its founding principles. In July, OpenAI announced the creation of a so-called superalignment team with Sutskever co-leading the research. OpenAI would expand the alignment team’s research to develop more upstream AI-safety techniques with a dedicated 20 percent of the company’s existing computer chips, in preparation for the possibility of AGI arriving in this decade, the company said.
- arrakeenrevived 3y ago
  > The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack. I know it's small and kind of a throw-away line, but statements like this make me take this author's interpretation of the rest of these events with a healthy dose of skepticism. At my company we have multiple company "memes" like this that have been turned into reacji, but the vast majority of them are not because it's "popular" but rather because we use it ironically or to make fun of the meme. Just the fact that an employee turned it into a reacji is a total non-event and I don't think you can read anything from it.
rmorey 3y ago
> For a leadership offsite this year, according to two people familiar with the event, Sutskever commissioned a wooden effigy from a local artist that was intended to represent an “unaligned” AI—that is, one that does not meet a human’s objectives. He set it on fire to symbolize OpenAI’s commitment to its founding principles. Honestly, pretty sick
- voidfunc 3y ago
  Very metal. I think that anecdote made me like this guy even if I disagree with him about the dangers of AI.
  - alienicecream 3y ago
    More like very mental. The geeks are drinking their own Kool aid and acting weird again.
- quickthrower2 3y ago
  Which particular human's objectives did it not align with?
  - frabcus 3y ago
    That is a secondary and huge problem, but the larger initial problem is making sure the AI aligns with values that nearly all humans have (e.g. don't kill all humans)
  - archon1410 3y ago
    The idea of "alignment" is pretty troubling to me. Do these people think that they, or those in power have achieved moral perfection, such that it would be good to have extremely powerful AI systems under their control, aligned with them? Imagine if the US or any other government of 1800s came gained so much power, 'locking-in' their repugnant values as the moral truth, backed by total control of the world.
    - 3y ago
truculent 3y ago
> with beliefs at times seemingly rooted in the realm of science fiction I don’t know how you can look at the development of generative AI tools in the past few years and write so dismissively about “science fiction” becoming reality
- tovej 3y ago
  Easy, you recognize the ability for the generative model to copy facets of data from its corpus as useful but boring in the AI sense, and wonder why people are even talking about sentience and AGI.
  - lyu07282 3y ago
    I suppose if this widely believed perception is true there can't possibly be any danger in scaling the model up a few thousand times. Except of course all those logical reasoning datasets the model is acing, but I guess it just memorized them all. Its only autocomplete on steroids why are all these people smarter than me worried about it, I don't understand.
    - tovej 3y ago
      There was just a post on this website where GPT4 failed to perform basic reasoning tasks better than minimum paid mechanical turk "microworkers".
      - andybak 3y ago
        And the comments section pointed out multiple flaws in that article.
        tovej 3y ago
        it didn't really. There were no fundamental flaws that I could see. Perhaps the only salient critique was the textual representation of the problem, but I think it was presented in a way where the model was given all the help it could get. You forget the result of the paper was actually improving the model's performance and still failing to get anywhere near decent results.
jph00 3y ago
FYI: one of the authors of this article, Karen Hao, just announced on Twitter that she's writing a book on OpenAI, and that this article is partly based on work done for that book.
- leobg 3y ago
  I hope her book will be as good as the article. Something like the earlier works of Walter Isaacson, rather than his latest Musk bio.
RationalDino 3y ago
Random thought. Let's suppose that AGI is about to be invented, and it will wind up having a personality similar to humans. The more that those are are doing the inventing are afraid of what they are inventing, the more that they will push it to be afraid of the humans in turn. This does not sound like a good conflict to start with. By contrast if the humans inventing it go full throttle to convincing it that humans are on its side, there is no such conflict at all. I don't know how realistic this model is. But it certainly suggests that the e/acc approach is more likely to create AI alignment than EA is.
- ChatGTP 3y ago
  I don’t think it’s a problem unless we workout how to teach them to feel emotions. I don’t think an LLM is ever going to be capable of feeling fear, boredom etc. If we did it would probably have many of the handicaps we do.
  - Davidzheng 3y ago
    Why can't it feel fear? The model itself doesn't have any built in mechanisms sure but it can simulate an agent capable of fear. In the same way the simulation can have other emotions needed to be a better model of a human
    - RationalDino 3y ago
      As I pointed out in a different comment, ChatGPT and friends are based on predicting the training data. As a result they learn to imitate what is in it. To the extent that we provide the training data for such models, we should expect it to internalize aspects of our behavior. And what is internalized won't just be what we expected and were planning on.
    - ChatGTP 3y ago
      Simulating fear and feeling actually fear, that can be fatal via nervous system shock are quite different things.
    - 3y ago
jimsimmons 3y ago
If Altman can use populism and VC pressure to overturn board decisions what is the point of OpenAI lol
minzi 3y ago
> The future, it seems, will be decided behind closed doors.
cmrdporcupine 3y ago
Can't find the article right now, but there was one circulating that heavily implied that various SV execs began their rounds of layoffs last fall at least partially or probably inspired by the demos they'd seen of OpenAI's tech. Microsoft in particular laid off 10,000 and then immediately turned around and invested billions more in OpenAI: https://www.sdxcentral.com/articles/news/microsoft-bets-billions-on-openai-following-layoffs/2023/01/ https://www.sdxcentral.com/articles/news/microsoft-bets-bill... -- last fall, just as the timeline laid out in the Atlnatic article was firing up. In that context this timeline is even more nauseating. Not only did OpenAI push ChatGPT at the expense of their own mission and their employee's well-being, they likely caused massive harm to our employment sector and the well-being of tens of thousands of software engineers in the industry at large. Maybe those layoffs would have happened anyways, but the way this all has rolled out and the way it's played out in the press and in the board rooms of the BigTech corporations... OpenAI is literally accomplishing the opposite of its supposed mission. And now it's about to get worse.
- jprete 3y ago
  I read a profile recently of an AI consultant for CEOs who mentioned in the interview that the CEOs all want to know when they can lay people off. I really don’t understand how tech people are so spectacularly naive.
  - cmrdporcupine 3y ago
    They were all making fun of Google for being "behind", when in reality it came down to there being more cautionary voices within Google around this stuff. Not cautionary enough, and they did and now are doing some terribly douchey things (the incident with Timnit Gebru was one of the early warnings there) but they did seem like they were trying to keep at least some of the Jinn corked up. I actually don't think these execs can replace us with LLMs, but the hype volcano certainly made them think they could. Which says more about the shit understanding they have of what their SWE staff does more than anything else. Idiots hired like crazy during COVID, and then were surprised that 9 women couldn't make a baby in 1 month... and now they think "AI" is going to fix this for them. I need a new career.
deleted 3y ago
[deleted]
tangjurine 3y ago
https://archive.is/Vqjpr https://archive.is/Vqjpr
- PieUser 3y ago
  This works better: https://12ft.io/https://www.theatlantic.com/technology/archive/2023/11/sam-altman-open-ai-chatgpt-chaos/676050/ https://12ft.io/https://www.theatlantic.com/technology/archi...
  - jack1243star 3y ago
    I like 12ft.io but it breaks Firefox reader mode for me.
dkarras 3y ago
The problem with the idealistic "we do research on alignment as we discover AGI, don't care about money" angle is that... you are not the only ones doing it. And OpenAI is trying to do it with its hands tied behind its back (non-profit status and vibes). There are and will be companies (like Anthropic) doing the same work themselves, they will do it for profit on the side, will rake in billions, possible become the most valuable country on Earth, build massive research and development labs etc. Then they will define what alignment is, not OpenAI. So for OpenAI to reach its goal, if they want to do it themselves that is, they need to compete on capitalistic grounds as well, there is no way around it.
- 0xDEAFBEAD 3y ago
  >There are and will be companies (like Anthropic) doing the same work themselves My understanding is that Anthropic is even more idealistic than OpenAI. I think it was founded by a bunch of OpenAI people who quit because they felt OpenAI wasn't cautious enough. In any case, ultimately it depends on the industry structure. If there are just a few big players, and most of them are idealistic, things could go OK. If there are a ton of little players, that's when you risk a race to the bottom, because there will always be someone willing to bend the rules to gain an advantage.
simonw 3y ago
This is really good journalism. There are a ton of interesting details in here that haven't been reported elsewhere, and it has all of the hallmarks of being well researched and sourced. The first clue is this: "In conversations between The Atlantic and 10 current and former employees at OpenAI..." When you're reporting something like this, especially when using anonymous sources (not anonymous to you, but sources that have good reasons not to want their names published), you can't just trust what someone tells you - they may have their own motives for presenting things in a certain way, or they may just be straight up lying. So... you confirm what they are saying with other sources. That's why "10 current and former employees" is mentioned explicitly in the article. Being published in the Atlantic helps too, because that's a publication with strong editorial integrity and a great track record.
- tkgally 3y ago
  > This is really good journalism. That was exactly my reaction. I’ve been following the news and rumors and speculation closely since Altman’s firing, and this is by far the most substantive account I have read. Kudos to the authors and to The Atlantic for getting it out so quickly.
- singularity2001 3y ago
  IDK it starts rather nonsensically: Sam Altman, the figurehead of the generative-AI revolution, —one must understand that OpenAI is not a technology company. EDIT: despite of the poor phrasing I agree that the article as a whole is of high quality Yet: "zealous doomers" is that how people cautious of the potential power of AI are now being labeled?
  - simonw 3y ago
    That makes a lot more sense in context: > To truly understand the events of the past 48 hours—the shocking, sudden ousting of OpenAI’s CEO, Sam Altman, arguably the figurehead of the generative-AI revolution, followed by reports that the company is now in talks to bring him back—one must understand that OpenAI is not a technology company. At least, not like other epochal companies of the internet age, such as Meta, Google, and Microsoft. The key piece here is "At least, not like other epochal companies of the internet age, such as Meta, Google, and Microsoft." - which then gets into the weird structure of OpenAI as a non-profit, which is indeed crucial to understanding what has happened over the weekend. This is good writing. The claim that "OpenAI is not a technology company" in the opening paragraph of the story instantly grabs your attention and makes you ask why they would say that... a question which they then answer in the next few sentences.
RadixDLT 3y ago
Frenzied speculation swirls after ousted OpenAI CEO Sam Altman teased breakthrough that pushed 'the frontier of discovery forward' - just ONE day before he was fired amid reports he was seeking investors for new AI chip venture
MichaelMoser123 3y ago
i suspect that 'AI safety' means a chatbot that is not hallucinating/making things up. Ilya Sutskever may want some sort of hybrid system, where the output of the LLM gets vetted by a second system, so as to minimize instances of hallucinations, whereas Sam Altman says 'screw it, lets make an even bigger LLM and just push it'. Is that right? Don't know if Altman or Sutskever is right, there seems to be a kind of arms race between the companies. OpenAI may be past the point where they can try out a radically different system, due to competition in the space. Maybe trying out new approaches could only work in a new company, who knows?
- pixl97 3y ago
  AI Safety means a lot of different things. Inward safety for people means their vulnerabilities are not exposed and/or open to attack. Outward safety means they are not attacking others and looking out for the general wellbeing of others. We have a lot of different social constructs with other people of which we attempt to keep this in balance. It doesn't work out for everyone, but in general it's somewhat stable. What does it mean to be safe if you're not at threat of being attacked and harm? What does attacking others mean if doing so is meaningless, just a calculation? This goes from everything from telling someone to kill themselves (it's just words) to issuing a set of commands to external devices with real world effects (print ebola virus or launch nuclear weapon). The concern here is the AI of the future will be extraordinarily powerful yet very risky when it comes to making decisions that could harm others.
  - MichaelMoser123 3y ago
    yes, still it boils down to the question of 'how well is the output of the chat bot aligned with reality' ? if you want to automate this, then you will likely need some system that is kind of censoring the output of the LLM, and that system should have a better model of what is real.
natch 3y ago
Does anyone else think it would make for a more healthy dynamic from the standpoint of AI safety if both sama and Ilya remained, despite their differences? Not that I know anything, but it seems a diversity of opinions at the top could have its benefits.
- leobg 3y ago
  In a culture where people talk, yes. In a culture where one pushes ahead and brushes the others aside, no. Marriage 101. (I don't know if that's what happened here. But many sources seem to point in that direction at the moment.)
- pdonis 3y ago
  Frankly, after seeing what has played out over the last few days, I don't see anybody anywhere in this that I would trust with AI safety.
deleted 3y ago
[deleted]
leobg 3y ago
Sutskever said something interesting in his Lex Fridman interview: "In an ideal world, humanity would be the board members, and AGI would be the CEO. Humanity can always press the reset button and say, 'Re-randomize parameters!'" This was 3 years ago. But that metaphor strikes me as too powerful for it not to have been at the back of Sutskever's mind when he pushed for Altman being ousted.
- breck 3y ago
  Interesting https://www.youtube.com/watch?v=13CZPWmke6A&t=5206s https://www.youtube.com/watch?v=13CZPWmke6A&t=5206s
- moberley 3y ago
  This does seem plausible, but I then wonder how the thought analogy plays out considering Sutskever's subsequent statement of regret for trying out the reset button in this case.
  - leobg 3y ago
    Well, of course he regrets it. That “reset params” button turned out to wipe the whole GitHub repo. :)
_lvbh 3y ago
The cult of tech bros and the cult of AGI
intimidated 3y ago
> Safety teams within the company pushed to slow things down. These teams worked to refine ChatGPT to refuse certain types of abusive requests and to respond to other queries with more appropriate answers. I wonder what this struggle means for the future of ChatGPT censorship/safety.
somewhereoutth 3y ago
> OpenAI’s president tweeted that the tool [ChatGPT] hit 1 million within the first five days. Perhaps the reason why ChatGPT has become so popular is because it provides entertainment. So it is not a great leap forward in AI or a path to AGI, but instead a incredibly convoluted way of keeping reasonable intelligent people occupied and amused. You enter a prompt, and it returns a result - what a fun game! Maybe that is it's primary contribution to society.
- okdood64 3y ago
  I’ve found it pretty useful to _assist_ in my engineering work.
- Loughla 3y ago
  It's not great at a lot of things that require genuine creativity for me, but it's AMAZING at providing d&d campaign ideas.
- maroonblazer 3y ago
  ChatGPT provides real utility for me. Some examples: 1. Drafting directional copy I can give to a real copywriter to create something we'd all be happy presenting to users. 2. Act as a sounding board for peer-personnel issues I'm dealing with at work. 3. "Dumb down" concepts in academic journals/articles such that I can make sense of them. 4. Just today it helped me build an app to drill a specific set of chord shapes/inversions on the guitar that I've been struggling with (programming has always been a very casual hobby and, consequently, I'm not very good at it).
- primax 3y ago
  Personally for me I've found that since Google search has been wrecked by SEO, or pivoted to targetting zoomers and gen alpha, it just doesn't work the same for me in my 40s. It used to be fantastic, but several times a day my search queries end in frustration like they did in the pre-Webcrawler search era. Increasingly now I use ChatGPT and sometimes Kagi. And they just work like I expect. I can think of one time that ChatGPT has failed me, which was when I was trying to remember the terms OLTP/OLAP in database architecture. But for a long time now it's been a very effective tool in my toolbox, while Google increasingly wears out.
tgsovlerkhgsel 3y ago
Looking at this article, the following theory would align with what I've seen so far: * Ilya Sutskever is concerned about the company moving too fast (without taking safety into account) under Sam Altman. * The others on the board that ended up supporting the firing are concerned about the same. * Ilya supports the firing because he wants the company to move slower. * The majority of the people working on AI don't want to slow down, either because they want to develop as fast as possible or because they're worried about missing out on profit. * Sam rallies the "move fast" faction and says "this board will slow us down horribly, let's move fast under Microsoft" * Ilya realizes that the practical outcome will be more speed/less safety, not more safety, as he hoped, leading to the regret tweet (https://nitter.net/ilyasut/status/1726590052392956028 https://nitter.net/ilyasut/status/1726590052392956028)
- paulddraper 3y ago
  That checks out. If so, a big issue was Ilya being so out of touch with what an overwhelming majority of his company wanted.
- jasfi 3y ago
  If they have AI doomers at the help of an AI company, then the company itself is doomed.
- rdtsc 3y ago
  Some hot takes there but this one I liked: > People building AGI unable to predict consequences of their actions 3 days in advance. It’s a reasonable point if these are the people building the future of humanity it’s a little concerning they can’t predict the immediate consequences of their own actions. On the other hand it shows some honesty being able to admit a mistake in public, also something you might want out of someone building the future of humanity.
Seanambers 3y ago
So the decels basically owned the board, and drove a 100B value destruction because they disagreed with letting people use GPT-4 in ChatGPT. Some colleagues that is. Is this decel movement just an extension of the wokeism that has been a problem in SV? Employees more focused on social issues than actually working.

281 points by maxutility 3y ago

m15i 3y ago

https://archive.md/PvbqA https://archive.md/PvbqA

gigel82 3y ago
That website looks like a Captcha farm scheme.
- Sirizarry 3y ago
  It’s because of a dispute with cloudflare’s 1.1.1.1 DNS. Try another DNS
  - foobiekr 3y ago
    and don't use apple apple private relay

good_lord_dang 3y ago

[dead]

deleted 3y ago

[deleted]

gumballindie 3y ago

[flagged]

brucethemoose2 3y ago

Well that story is just sad, because it means the principle/research-oriented company structure they set up utterly failed in the face of profit motives. Clearly Altman was not dissuaded from doing things the power structure didn't want him to do.

EMIRELADERO 3y ago
What? We don't know what will happen, as the whole thing hasn't finished yet. Nobody can say for sure which side will "win" the "fight", or if there will be any winners at all.
- brucethemoose2 3y ago
  If Altman is rehired its a total capitulation. If he's not, and literally all the other employees come back, its still a failure of the power structure that already happened. The threat of firing the CEO of the for-profit arm is supposed to be a unused threat, like a nuclear weapon.
  - EMIRELADERO 3y ago
    So what happens when the Altman and the board's views are concluded to be completely incompatible? How do you fix that clash without one side giving up?
    - brucethemoose2 3y ago
      > So what happens when the Altman and the board's views are concluded to be completely incompatible? He gets fired. Thats the enforcement mechanism. Hence if he comes back, that indicates the board essentially "agrees" with Altman's positions now, however much Altman and the board's positions have shifted.
stevefan1999 3y ago
In today's world, capitalism, or more like human hubris and greed for the desire of ever-growing process, always prevail in the end, no matter how venerate and honorable your initial goals would be
- nickthegreek 3y ago
  It’s nice to have this modern clear example now to point to though.

jrm4 3y ago

Lol. So I didn't get past the few paragraphs before the paywall, and I didn't need to. I appreciate the idea of being a "not-greedy typical company," but there's a reason you e.g. separate university type research or non-profits and private companies. Trying to make up something in the middle is the exact sort of naivete you can ALWAYS expect from Silicon Valley.

cmrdporcupine 3y ago
Yeah: https://www.youtube.com/watch?v=j_JW9NwjLF8 https://www.youtube.com/watch?v=j_JW9NwjLF8 I'd say yes, Sutsksever is... naive? though very smart. Or just utopian. Seems he couldn't get the scale he needed/wanted out of a university (or Google) research lab. But the former at least would have bounded things better in the way he would have preferred, from an ethics POV. Jumping into bed with Musk and Altman and hoping for ethical non-profit "betterment of humanity" behaviour is laughable. Getting access to capital was obviously tempting, but ... As for Altman. No, he's not naive. Amoral, and likely proud of it. JFC ... Worldcoin... I can't even... I don't want either of these people in charge of the future, frankly. It does point to the general lack of funding for R&D of this type of thing. Or it's still too early to be doing this kind of thing at scale. I dunno. Bleak.
- jrm4 3y ago
  Bleak? Why? Honestly, this seems like a pretty good outcome. Which is to say, I think the fearmongering sentient AI stuff is silly -- but I think we are all DEFINITELY better off with an ugly rough-and-tumble visible rocky start to the AI revolution. Weed out the BS; equalize out who actually has access to the best stuff BEFORE some jerk company can scale up fast and dominate market share; let a de-facto "open source" market have a go at the whole thing.
  - cmrdporcupine 3y ago
    The bleak reality is that OpenAI became that jerk company, and now in its effective demise, those reigns are handed over to Microsoft. Who have already demonstrated (with CoPilot) a similar lack of ... concern ... for IP rights / authorship, and various other ethical aspects. And bleak because there doesn't seem to be an alternative where the people making these decisions are responsible to an electorate or public in some democratic fashion. Just a bunch of people with $$ and influence who set themselves up to be arbiters ... and It's just might makes right. And bleak because in this case the "mighty" are often the very people who made fun of arts students who took the philosophy and ethics classes in school that could at least offer some insight in these issues.

maxutility 3y ago

A few interesting tidbits > The company pressed forward and launched ChatGPT on November 30. It was considered such a nonevent that no major company-wide announcement about the chatbot going live was made. Many employees who weren’t directly involved, including those in safety functions, didn’t even realize it had happened. Some of those who were aware, according to one employee, had started a betting pool, wagering how many people might use the tool during its first week. The highest guess was 100,000 users. OpenAI’s president tweeted that the tool hit 1 million within the first five days. The phrase low-key research preview became an instant meme within OpenAI; employees turned it into laptop stickers. > Anticipating the arrival of [AGI], Sutskever began to behave like a spiritual leader, three employees who worked with him told us. His constant, enthusiastic refrain was “feel the AGI,” a reference to the idea that the company was on the cusp of its ultimate goal. At OpenAI’s 2022 holiday party, held at the California Academy of Sciences, Sutskever led employees in a chant: “Feel the AGI! Feel the AGI!” The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack. > For a leadership offsite this year, according to two people familiar with the event, Sutskever commissioned a wooden effigy from a local artist that was intended to represent an “unaligned” AI—that is, one that does not meet a human’s objectives. He set it on fire to symbolize OpenAI’s commitment to its founding principles. In July, OpenAI announced the creation of a so-called superalignment team with Sutskever co-leading the research. OpenAI would expand the alignment team’s research to develop more upstream AI-safety techniques with a dedicated 20 percent of the company’s existing computer chips, in preparation for the possibility of AGI arriving in this decade, the company said.

arrakeenrevived 3y ago
> The phrase itself was popular enough that OpenAI employees created a special “Feel the AGI” reaction emoji in Slack. I know it's small and kind of a throw-away line, but statements like this make me take this author's interpretation of the rest of these events with a healthy dose of skepticism. At my company we have multiple company "memes" like this that have been turned into reacji, but the vast majority of them are not because it's "popular" but rather because we use it ironically or to make fun of the meme. Just the fact that an employee turned it into a reacji is a total non-event and I don't think you can read anything from it.

rmorey 3y ago

> For a leadership offsite this year, according to two people familiar with the event, Sutskever commissioned a wooden effigy from a local artist that was intended to represent an “unaligned” AI—that is, one that does not meet a human’s objectives. He set it on fire to symbolize OpenAI’s commitment to its founding principles. Honestly, pretty sick

voidfunc 3y ago
Very metal. I think that anecdote made me like this guy even if I disagree with him about the dangers of AI.
- alienicecream 3y ago
  More like very mental. The geeks are drinking their own Kool aid and acting weird again.
quickthrower2 3y ago
Which particular human's objectives did it not align with?
- frabcus 3y ago
  That is a secondary and huge problem, but the larger initial problem is making sure the AI aligns with values that nearly all humans have (e.g. don't kill all humans)
- archon1410 3y ago
  The idea of "alignment" is pretty troubling to me. Do these people think that they, or those in power have achieved moral perfection, such that it would be good to have extremely powerful AI systems under their control, aligned with them? Imagine if the US or any other government of 1800s came gained so much power, 'locking-in' their repugnant values as the moral truth, backed by total control of the world.
  - 3y ago

truculent 3y ago

> with beliefs at times seemingly rooted in the realm of science fiction I don’t know how you can look at the development of generative AI tools in the past few years and write so dismissively about “science fiction” becoming reality

tovej 3y ago
Easy, you recognize the ability for the generative model to copy facets of data from its corpus as useful but boring in the AI sense, and wonder why people are even talking about sentience and AGI.
- lyu07282 3y ago
  I suppose if this widely believed perception is true there can't possibly be any danger in scaling the model up a few thousand times. Except of course all those logical reasoning datasets the model is acing, but I guess it just memorized them all. Its only autocomplete on steroids why are all these people smarter than me worried about it, I don't understand.
  - tovej 3y ago
    There was just a post on this website where GPT4 failed to perform basic reasoning tasks better than minimum paid mechanical turk "microworkers".
    - andybak 3y ago
      And the comments section pointed out multiple flaws in that article.
      - tovej 3y ago
        it didn't really. There were no fundamental flaws that I could see. Perhaps the only salient critique was the textual representation of the problem, but I think it was presented in a way where the model was given all the help it could get. You forget the result of the paper was actually improving the model's performance and still failing to get anywhere near decent results.

jph00 3y ago

FYI: one of the authors of this article, Karen Hao, just announced on Twitter that she's writing a book on OpenAI, and that this article is partly based on work done for that book.

leobg 3y ago
I hope her book will be as good as the article. Something like the earlier works of Walter Isaacson, rather than his latest Musk bio.

RationalDino 3y ago

Random thought. Let's suppose that AGI is about to be invented, and it will wind up having a personality similar to humans. The more that those are are doing the inventing are afraid of what they are inventing, the more that they will push it to be afraid of the humans in turn. This does not sound like a good conflict to start with. By contrast if the humans inventing it go full throttle to convincing it that humans are on its side, there is no such conflict at all. I don't know how realistic this model is. But it certainly suggests that the e/acc approach is more likely to create AI alignment than EA is.

ChatGTP 3y ago
I don’t think it’s a problem unless we workout how to teach them to feel emotions. I don’t think an LLM is ever going to be capable of feeling fear, boredom etc. If we did it would probably have many of the handicaps we do.
- Davidzheng 3y ago
  Why can't it feel fear? The model itself doesn't have any built in mechanisms sure but it can simulate an agent capable of fear. In the same way the simulation can have other emotions needed to be a better model of a human
  - RationalDino 3y ago
    As I pointed out in a different comment, ChatGPT and friends are based on predicting the training data. As a result they learn to imitate what is in it. To the extent that we provide the training data for such models, we should expect it to internalize aspects of our behavior. And what is internalized won't just be what we expected and were planning on.
  - ChatGTP 3y ago
    Simulating fear and feeling actually fear, that can be fatal via nervous system shock are quite different things.
  - 3y ago

jimsimmons 3y ago

If Altman can use populism and VC pressure to overturn board decisions what is the point of OpenAI lol

minzi 3y ago

> The future, it seems, will be decided behind closed doors.

cmrdporcupine 3y ago

Can't find the article right now, but there was one circulating that heavily implied that various SV execs began their rounds of layoffs last fall at least partially or probably inspired by the demos they'd seen of OpenAI's tech. Microsoft in particular laid off 10,000 and then immediately turned around and invested billions more in OpenAI: https://www.sdxcentral.com/articles/news/microsoft-bets-billions-on-openai-following-layoffs/2023/01/ https://www.sdxcentral.com/articles/news/microsoft-bets-bill... -- last fall, just as the timeline laid out in the Atlnatic article was firing up. In that context this timeline is even more nauseating. Not only did OpenAI push ChatGPT at the expense of their own mission and their employee's well-being, they likely caused massive harm to our employment sector and the well-being of tens of thousands of software engineers in the industry at large. Maybe those layoffs would have happened anyways, but the way this all has rolled out and the way it's played out in the press and in the board rooms of the BigTech corporations... OpenAI is literally accomplishing the opposite of its supposed mission. And now it's about to get worse.

jprete 3y ago
I read a profile recently of an AI consultant for CEOs who mentioned in the interview that the CEOs all want to know when they can lay people off. I really don’t understand how tech people are so spectacularly naive.
- cmrdporcupine 3y ago
  They were all making fun of Google for being "behind", when in reality it came down to there being more cautionary voices within Google around this stuff. Not cautionary enough, and they did and now are doing some terribly douchey things (the incident with Timnit Gebru was one of the early warnings there) but they did seem like they were trying to keep at least some of the Jinn corked up. I actually don't think these execs can replace us with LLMs, but the hype volcano certainly made them think they could. Which says more about the shit understanding they have of what their SWE staff does more than anything else. Idiots hired like crazy during COVID, and then were surprised that 9 women couldn't make a baby in 1 month... and now they think "AI" is going to fix this for them. I need a new career.

deleted 3y ago

[deleted]

tangjurine 3y ago

https://archive.is/Vqjpr https://archive.is/Vqjpr

PieUser 3y ago
This works better: https://12ft.io/https://www.theatlantic.com/technology/archive/2023/11/sam-altman-open-ai-chatgpt-chaos/676050/ https://12ft.io/https://www.theatlantic.com/technology/archi...
- jack1243star 3y ago
  I like 12ft.io but it breaks Firefox reader mode for me.

dkarras 3y ago

The problem with the idealistic "we do research on alignment as we discover AGI, don't care about money" angle is that... you are not the only ones doing it. And OpenAI is trying to do it with its hands tied behind its back (non-profit status and vibes). There are and will be companies (like Anthropic) doing the same work themselves, they will do it for profit on the side, will rake in billions, possible become the most valuable country on Earth, build massive research and development labs etc. Then they will define what alignment is, not OpenAI. So for OpenAI to reach its goal, if they want to do it themselves that is, they need to compete on capitalistic grounds as well, there is no way around it.

0xDEAFBEAD 3y ago
>There are and will be companies (like Anthropic) doing the same work themselves My understanding is that Anthropic is even more idealistic than OpenAI. I think it was founded by a bunch of OpenAI people who quit because they felt OpenAI wasn't cautious enough. In any case, ultimately it depends on the industry structure. If there are just a few big players, and most of them are idealistic, things could go OK. If there are a ton of little players, that's when you risk a race to the bottom, because there will always be someone willing to bend the rules to gain an advantage.

simonw 3y ago

This is really good journalism. There are a ton of interesting details in here that haven't been reported elsewhere, and it has all of the hallmarks of being well researched and sourced. The first clue is this: "In conversations between The Atlantic and 10 current and former employees at OpenAI..." When you're reporting something like this, especially when using anonymous sources (not anonymous to you, but sources that have good reasons not to want their names published), you can't just trust what someone tells you - they may have their own motives for presenting things in a certain way, or they may just be straight up lying. So... you confirm what they are saying with other sources. That's why "10 current and former employees" is mentioned explicitly in the article. Being published in the Atlantic helps too, because that's a publication with strong editorial integrity and a great track record.

tkgally 3y ago
> This is really good journalism. That was exactly my reaction. I’ve been following the news and rumors and speculation closely since Altman’s firing, and this is by far the most substantive account I have read. Kudos to the authors and to The Atlantic for getting it out so quickly.
singularity2001 3y ago
IDK it starts rather nonsensically: Sam Altman, the figurehead of the generative-AI revolution, —one must understand that OpenAI is not a technology company. EDIT: despite of the poor phrasing I agree that the article as a whole is of high quality Yet: "zealous doomers" is that how people cautious of the potential power of AI are now being labeled?
- simonw 3y ago
  That makes a lot more sense in context: > To truly understand the events of the past 48 hours—the shocking, sudden ousting of OpenAI’s CEO, Sam Altman, arguably the figurehead of the generative-AI revolution, followed by reports that the company is now in talks to bring him back—one must understand that OpenAI is not a technology company. At least, not like other epochal companies of the internet age, such as Meta, Google, and Microsoft. The key piece here is "At least, not like other epochal companies of the internet age, such as Meta, Google, and Microsoft." - which then gets into the weird structure of OpenAI as a non-profit, which is indeed crucial to understanding what has happened over the weekend. This is good writing. The claim that "OpenAI is not a technology company" in the opening paragraph of the story instantly grabs your attention and makes you ask why they would say that... a question which they then answer in the next few sentences.

RadixDLT 3y ago

Frenzied speculation swirls after ousted OpenAI CEO Sam Altman teased breakthrough that pushed 'the frontier of discovery forward' - just ONE day before he was fired amid reports he was seeking investors for new AI chip venture

MichaelMoser123 3y ago

i suspect that 'AI safety' means a chatbot that is not hallucinating/making things up. Ilya Sutskever may want some sort of hybrid system, where the output of the LLM gets vetted by a second system, so as to minimize instances of hallucinations, whereas Sam Altman says 'screw it, lets make an even bigger LLM and just push it'. Is that right? Don't know if Altman or Sutskever is right, there seems to be a kind of arms race between the companies. OpenAI may be past the point where they can try out a radically different system, due to competition in the space. Maybe trying out new approaches could only work in a new company, who knows?

pixl97 3y ago
AI Safety means a lot of different things. Inward safety for people means their vulnerabilities are not exposed and/or open to attack. Outward safety means they are not attacking others and looking out for the general wellbeing of others. We have a lot of different social constructs with other people of which we attempt to keep this in balance. It doesn't work out for everyone, but in general it's somewhat stable. What does it mean to be safe if you're not at threat of being attacked and harm? What does attacking others mean if doing so is meaningless, just a calculation? This goes from everything from telling someone to kill themselves (it's just words) to issuing a set of commands to external devices with real world effects (print ebola virus or launch nuclear weapon). The concern here is the AI of the future will be extraordinarily powerful yet very risky when it comes to making decisions that could harm others.
- MichaelMoser123 3y ago
  yes, still it boils down to the question of 'how well is the output of the chat bot aligned with reality' ? if you want to automate this, then you will likely need some system that is kind of censoring the output of the LLM, and that system should have a better model of what is real.

natch 3y ago

Does anyone else think it would make for a more healthy dynamic from the standpoint of AI safety if both sama and Ilya remained, despite their differences? Not that I know anything, but it seems a diversity of opinions at the top could have its benefits.

leobg 3y ago
In a culture where people talk, yes. In a culture where one pushes ahead and brushes the others aside, no. Marriage 101. (I don't know if that's what happened here. But many sources seem to point in that direction at the moment.)
pdonis 3y ago
Frankly, after seeing what has played out over the last few days, I don't see anybody anywhere in this that I would trust with AI safety.

deleted 3y ago

[deleted]

leobg 3y ago

Sutskever said something interesting in his Lex Fridman interview: "In an ideal world, humanity would be the board members, and AGI would be the CEO. Humanity can always press the reset button and say, 'Re-randomize parameters!'" This was 3 years ago. But that metaphor strikes me as too powerful for it not to have been at the back of Sutskever's mind when he pushed for Altman being ousted.

breck 3y ago
Interesting https://www.youtube.com/watch?v=13CZPWmke6A&t=5206s https://www.youtube.com/watch?v=13CZPWmke6A&t=5206s
moberley 3y ago
This does seem plausible, but I then wonder how the thought analogy plays out considering Sutskever's subsequent statement of regret for trying out the reset button in this case.
- leobg 3y ago
  Well, of course he regrets it. That “reset params” button turned out to wipe the whole GitHub repo. :)

_lvbh 3y ago

The cult of tech bros and the cult of AGI

intimidated 3y ago

> Safety teams within the company pushed to slow things down. These teams worked to refine ChatGPT to refuse certain types of abusive requests and to respond to other queries with more appropriate answers. I wonder what this struggle means for the future of ChatGPT censorship/safety.

somewhereoutth 3y ago

> OpenAI’s president tweeted that the tool [ChatGPT] hit 1 million within the first five days. Perhaps the reason why ChatGPT has become so popular is because it provides entertainment. So it is not a great leap forward in AI or a path to AGI, but instead a incredibly convoluted way of keeping reasonable intelligent people occupied and amused. You enter a prompt, and it returns a result - what a fun game! Maybe that is it's primary contribution to society.

okdood64 3y ago
I’ve found it pretty useful to _assist_ in my engineering work.
Loughla 3y ago
It's not great at a lot of things that require genuine creativity for me, but it's AMAZING at providing d&d campaign ideas.
maroonblazer 3y ago
ChatGPT provides real utility for me. Some examples: 1. Drafting directional copy I can give to a real copywriter to create something we'd all be happy presenting to users. 2. Act as a sounding board for peer-personnel issues I'm dealing with at work. 3. "Dumb down" concepts in academic journals/articles such that I can make sense of them. 4. Just today it helped me build an app to drill a specific set of chord shapes/inversions on the guitar that I've been struggling with (programming has always been a very casual hobby and, consequently, I'm not very good at it).
primax 3y ago
Personally for me I've found that since Google search has been wrecked by SEO, or pivoted to targetting zoomers and gen alpha, it just doesn't work the same for me in my 40s. It used to be fantastic, but several times a day my search queries end in frustration like they did in the pre-Webcrawler search era. Increasingly now I use ChatGPT and sometimes Kagi. And they just work like I expect. I can think of one time that ChatGPT has failed me, which was when I was trying to remember the terms OLTP/OLAP in database architecture. But for a long time now it's been a very effective tool in my toolbox, while Google increasingly wears out.

tgsovlerkhgsel 3y ago

Looking at this article, the following theory would align with what I've seen so far: * Ilya Sutskever is concerned about the company moving too fast (without taking safety into account) under Sam Altman. * The others on the board that ended up supporting the firing are concerned about the same. * Ilya supports the firing because he wants the company to move slower. * The majority of the people working on AI don't want to slow down, either because they want to develop as fast as possible or because they're worried about missing out on profit. * Sam rallies the "move fast" faction and says "this board will slow us down horribly, let's move fast under Microsoft" * Ilya realizes that the practical outcome will be more speed/less safety, not more safety, as he hoped, leading to the regret tweet (https://nitter.net/ilyasut/status/1726590052392956028 https://nitter.net/ilyasut/status/1726590052392956028)

paulddraper 3y ago
That checks out. If so, a big issue was Ilya being so out of touch with what an overwhelming majority of his company wanted.
jasfi 3y ago
If they have AI doomers at the help of an AI company, then the company itself is doomed.
rdtsc 3y ago
Some hot takes there but this one I liked: > People building AGI unable to predict consequences of their actions 3 days in advance. It’s a reasonable point if these are the people building the future of humanity it’s a little concerning they can’t predict the immediate consequences of their own actions. On the other hand it shows some honesty being able to admit a mistake in public, also something you might want out of someone building the future of humanity.

Seanambers 3y ago

So the decels basically owned the board, and drove a 100B value destruction because they disagreed with letting people use GPT-4 in ChatGPT. Some colleagues that is. Is this decel movement just an extension of the wokeism that has been a problem in SV? Employees more focused on social issues than actually working.