6 ms·

In case people haven't noticed, this is the second resignation in as many days. https://news.ycombinator.com/item?id=40361128 https://news.ycombinator.com/item

In case people haven't noticed, this is the second resignation in as many days. https://news.ycombinator.com/item?id=40361128 https://news.ycombinator.com/item?id=40361128

the_mitsuhiko 2y ago
And entirely predictable from the first one: https://openai.com/index/introducing-superalignment/ https://openai.com/index/introducing-superalignment/
- uLogMicheal 2y ago
  Imagine trying to keep something so far above us in intelligence, caged. Scary stuff...
  - krainboltgreene 2y ago
    [flagged]
    - uLogMicheal 2y ago
      Surely there's a book on this? Maybe we shouldn't let the AI read it...
    - dvfjsdhgfv 2y ago
      Another funny thing is to actually try to go through all politically incorrect stuff in order to censor the model. To do that you need to actually make the list and make sure you haven't overlooked anything or else the vigilanti will punish you (on Twitter first, but then it will escalate quickly). So you need to make sure all references to differences between male end female brain, personality etc. get nullified. But there is always danger that you still missed some obscure and non-obvious thing that will make someone mad.
  - binarymax 2y ago
    I’m genuinely curious - do you actually believe that GPT is a super intelligence? Because I have the opposite experience. It consistently fails to be correct on following even the most basic instructions. For a little while I thought maybe I’m doing it wrong, and I need better prompts, but then I realized that its zero shot and few shot capabilities are really hit and miss. Furthermore, a superior intelligence shouldn’t need us to conform to its persnickety requirements, and it should be able to adapt far better than it actually does.
    - uLogMicheal 2y ago
      GPT does not need super-alignment. This refers to aligning artificial general and super intelligence.
- btown 2y ago
  Makes me wonder if that 20% compute commitment to superalignment research was walked back (or redesigned so as to be distant from the original mission). Or, perhaps the two deemed that even more commitment was necessary, and were dissatisfied with Altman's response. Either way, if it's enough to cause them both to think it's better to research outside of the opportunities and access to data that OpenAI provides, I don't see a scenario where this doesn't indicate a significant shift in OpenAI's commitment to superalignment research and safety. One hopes that, at the very least, Microsoft's interests in brand integrity incentivize at least some modicum of continued commitment to safety research.
  - dontupvoteme 2y ago
    Ironically Microsoft is the one that's notoriously terrible at checking their "AI" products before releasing them. Besides the infamous Tay there was that apparently un-aligned Wizard-2[or something like that] model from them which got released by mistake for about 12 hours
    - buildbot 2y ago
      As an MS employee working on LLMs, that entire saga is super weird. We need approval for everything! Releasing anything without approval is quite weird. We can’t just drop papers on arxiv. There is no way running your own twitter, github, etc as a separate group allowed. I checked fairly recently to see if the model was actually released again, it doesn’t seem to be; I find this telling.
    - fzzzy 2y ago
      I was able to download a copy of that before they took it down. Silly.
      - dontupvoteme 2y ago
        Yeah it was already mirrored pretty quickly. I expect enough people are now running cronjobs to archive whitelists of HF pages and auto-cloning anything that gets pushed out.
pfist 2y ago
I have noticed, and I am concerned that they were the leaders of the Superalignment team.
- zer00eyz 2y ago
  Sam Altman superalinged them right out the door...
- transcriptase 2y ago
  On the other hand, they clearly they weren’t concerned enough about the issue to continue working on it.
  - dontupvoteme 2y ago
    One could argue that at this point openai is being Extended and Embraced by Microsoft and is unlikely to have much autonomy or groundbreaking impact one way or another.
  - llamaimperative 2y ago
    Ah yes, a scientist refusing to work on the hydrogen bomb couldn't have been all that concerned about it.
  - scarmig 2y ago
    If your ostensible purpose is being sidelined by decision makers, trying to fight back is often a good option, but sometimes you fail. Admitting failure and focusing on other approaches is the right choice at that point.
  - HarHarVeryFunny 2y ago
    The Anthropic folk were concerned enough that they left, and are indeed continuing to work on it [AI safety]. Now, we've have the co-leads of the super-alignment/safety team leaving too. Certainly not a good look for OpenAI. There really doesn't seem to be much of a mission left at OpenAI - they have a CEO giving off used car salesman vibes who recently mentioned considering allowing their AI to generate porn, and now releasing a flirty AI-girlfriend as his gift to humanity.
hn_throwaway_99 2y ago
They resigned together on the same day - people are just announcing this like it's some type of "drip drip" of people leaving to build suspense. While Jan's (very pithy) tweet was later in the evening, I was reading other posts yesterday at the time of Ilya's announcement saying that Jan was also leaving.
dawoodee 2y ago
A few more people involved in the alignment efforts have left recently: https://x.com/ShakeelHashim/status/1790685752134656371 https://x.com/ShakeelHashim/status/1790685752134656371

In case people haven't noticed, this is the second resignation in as many days. https://news.ycombinator.com/item?id=40361128 https://news.ycombinator.com/item?id=40361128

the_mitsuhiko 2y ago

And entirely predictable from the first one: https://openai.com/index/introducing-superalignment/ https://openai.com/index/introducing-superalignment/

uLogMicheal 2y ago
Imagine trying to keep something so far above us in intelligence, caged. Scary stuff...
- krainboltgreene 2y ago
  [flagged]
  - uLogMicheal 2y ago
    Surely there's a book on this? Maybe we shouldn't let the AI read it...
  - dvfjsdhgfv 2y ago
    Another funny thing is to actually try to go through all politically incorrect stuff in order to censor the model. To do that you need to actually make the list and make sure you haven't overlooked anything or else the vigilanti will punish you (on Twitter first, but then it will escalate quickly). So you need to make sure all references to differences between male end female brain, personality etc. get nullified. But there is always danger that you still missed some obscure and non-obvious thing that will make someone mad.
- binarymax 2y ago
  I’m genuinely curious - do you actually believe that GPT is a super intelligence? Because I have the opposite experience. It consistently fails to be correct on following even the most basic instructions. For a little while I thought maybe I’m doing it wrong, and I need better prompts, but then I realized that its zero shot and few shot capabilities are really hit and miss. Furthermore, a superior intelligence shouldn’t need us to conform to its persnickety requirements, and it should be able to adapt far better than it actually does.
  - uLogMicheal 2y ago
    GPT does not need super-alignment. This refers to aligning artificial general and super intelligence.
btown 2y ago
Makes me wonder if that 20% compute commitment to superalignment research was walked back (or redesigned so as to be distant from the original mission). Or, perhaps the two deemed that even more commitment was necessary, and were dissatisfied with Altman's response. Either way, if it's enough to cause them both to think it's better to research outside of the opportunities and access to data that OpenAI provides, I don't see a scenario where this doesn't indicate a significant shift in OpenAI's commitment to superalignment research and safety. One hopes that, at the very least, Microsoft's interests in brand integrity incentivize at least some modicum of continued commitment to safety research.
- dontupvoteme 2y ago
  Ironically Microsoft is the one that's notoriously terrible at checking their "AI" products before releasing them. Besides the infamous Tay there was that apparently un-aligned Wizard-2[or something like that] model from them which got released by mistake for about 12 hours
  - buildbot 2y ago
    As an MS employee working on LLMs, that entire saga is super weird. We need approval for everything! Releasing anything without approval is quite weird. We can’t just drop papers on arxiv. There is no way running your own twitter, github, etc as a separate group allowed. I checked fairly recently to see if the model was actually released again, it doesn’t seem to be; I find this telling.
  - fzzzy 2y ago
    I was able to download a copy of that before they took it down. Silly.
    - dontupvoteme 2y ago
      Yeah it was already mirrored pretty quickly. I expect enough people are now running cronjobs to archive whitelists of HF pages and auto-cloning anything that gets pushed out.

pfist 2y ago

I have noticed, and I am concerned that they were the leaders of the Superalignment team.

zer00eyz 2y ago
Sam Altman superalinged them right out the door...
transcriptase 2y ago
On the other hand, they clearly they weren’t concerned enough about the issue to continue working on it.
- dontupvoteme 2y ago
  One could argue that at this point openai is being Extended and Embraced by Microsoft and is unlikely to have much autonomy or groundbreaking impact one way or another.
- llamaimperative 2y ago
  Ah yes, a scientist refusing to work on the hydrogen bomb couldn't have been all that concerned about it.
- scarmig 2y ago
  If your ostensible purpose is being sidelined by decision makers, trying to fight back is often a good option, but sometimes you fail. Admitting failure and focusing on other approaches is the right choice at that point.
- HarHarVeryFunny 2y ago
  The Anthropic folk were concerned enough that they left, and are indeed continuing to work on it [AI safety]. Now, we've have the co-leads of the super-alignment/safety team leaving too. Certainly not a good look for OpenAI. There really doesn't seem to be much of a mission left at OpenAI - they have a CEO giving off used car salesman vibes who recently mentioned considering allowing their AI to generate porn, and now releasing a flirty AI-girlfriend as his gift to humanity.

hn_throwaway_99 2y ago

They resigned together on the same day - people are just announcing this like it's some type of "drip drip" of people leaving to build suspense. While Jan's (very pithy) tweet was later in the evening, I was reading other posts yesterday at the time of Ilya's announcement saying that Jan was also leaving.

dawoodee 2y ago

A few more people involved in the alignment efforts have left recently: https://x.com/ShakeelHashim/status/1790685752134656371 https://x.com/ShakeelHashim/status/1790685752134656371