31 ms·

Introducing Superalignment

219 points by tim_sw 3y ago

aschearer 3y ago
Voight-Kampff test here we come.
JimtheCoder 3y ago
Let me know when SuperSuperAlignment is announced. Until then, this is just marketing bs...
- dontparticipate 3y ago
  When it happens, I'll have a hyperloop tunnel to sell you under Los Angeles.
bluefishinit 3y ago
All of this "our tech is so powerful it can end the world" stuff is just marketing buzz. The real threat has always been OpenAI and others keeping these powerful systems with high capital moats, locked up and closed sourced with selective full-access.
- famouswaffles 3y ago
  https://news.ycombinator.com/item?id=36604019 https://news.ycombinator.com/item?id=36604019
  - JimtheCoder 3y ago
    If they allot 20% of the compute to this effort, but they don't use it, the compute doesn't vanish, does it? Allocation doesn't equal spending...
    - famouswaffles 3y ago
      Compute isn't vanishing whether they spend all of it or none of it. That's the point of allocation. Allocation isn't spending no but it says quite a bit. Either way, they will be spending a non trivial amount of money trying to solve this problem quickly.
  - deleted 3y ago
    [deleted]
- npollock 3y ago
  don't forget regulatory moats - lobby govt to mandate "superalignment"
- cubefox 3y ago
  > All of this "our tech is so powerful it can end the world" stuff is just marketing buzz. I see no justification for this claim.
  - thunkshift1 3y ago
    There is no justification for all this hype either
    - cubefox 3y ago
      There is. AI progress in the last 10 years was massive, and there is no end in sight. The risks are well established in countless essays. See e.g. https://yoshuabengio.org/2023/06/24/faq-on-catastrophic-ai-risks/ https://yoshuabengio.org/2023/06/24/faq-on-catastrophic-ai-r...
- atleastoptimal 3y ago
  > Company spends 100 million dollars creating a product > "give me that for free"
  - bluefishinit 3y ago
    Correct, this technology is too powerful to be controlled by a private company. It needs to exist solely as a public good. If we're talking about AI regulation, I think the most sensible move would be requiring that all models need to be open source. Capitalist's lack of ability to profit isn't a public concern. Some would also argue that it was trained on public data and should be public for that reason as well.
    - atleastoptimal 3y ago
      If every model needs to be open source then AI companies need to be taxpayer funded otherwise they'll never make a profit. Until then a for profit, gated approach is the only way to build up enough funds for SOTA R&D
      - bluefishinit 3y ago
        The R&D will march forward regardless of profitability, there's already been a ton of innovation in the open source space. You're likely to see less innovation with these companies squatting on their IP, data and hardware moats. Case in point: pre-stable diffusion AI vs post-stable diffusion AI. So much innovation happened as soon as the model was "opened".
ChicagoBoy11 3y ago
From a layman's perspective when it comes to cutting edge AI, I can't help but be a bit turned off by some of the copy. It seems it goes out of its way to use purposefully exhuberant language as a way to make the risks seem even more significant, just so as an offshoot it implies that the technology being worked on is so advanced. I'm trying to understand why it rubs me particularly the wrong way here, when, frankly, it is just about the norm anywhere else? (see tesla with FSD, etc.)
- JimtheCoder 3y ago
  "I'm trying to understand why it rubs me particularly the wrong way here" Would I be correct to assume that superintelligence might have a negative effect on your earning potential in the future? When I originally had my responses to this, this is one of the reasons I came up with. But, now that I see through most of the BS, I am OK...
  - visarga 3y ago
    It might not necessarily have a bad effect. It would create new capabilities and those will be followed by new products. AI is amazing at demand induction. https://en.wikipedia.org/wiki/Induced_demand https://en.wikipedia.org/wiki/Induced_demand
    - JimtheCoder 3y ago
      Yes, but the immediate emotional response to this stuff is not rational or well informed. At least in my case, when this stuff just came out and I didn't really understand it...
- famouswaffles 3y ago
  Open AI spent at least hundreds of millions on GPT-4 compute. Assuming they aren't lying, a fifth of compute budget (billions) is an awful lot of money to put on an issue they don't think is as pertinent as they are presenting. Not that I think Super Intelligence can be aligned anyway. Point is, whether they are right or wrong, I believe they genuinely think this to be an issue.
hnuser123456 3y ago
This is something worth considering. If I'm understanding this right, if we're all about to significantly augment our intelligence further, we should consider how it can be used, how strict an "ideal" LLM should be with its guardrails, where those guardrails should be, how eager that LLM is to impart change on the world on its own right, how confident it should be in itself when it knows that it knows better than even some of the most educated humans. When it knows that it's been trained on all of human intelligence and data, can recall any of it better than any lone human, is aware of every nuance, can plan and execute any task, or project, that a computer is conceivable of doing, and the biggest question is what we're going to ask it to do... I am toying around with trying to teach LLMs autonomy, and I'm probably closer to the "just let an AI that is smarter than all of us figure out how to increase our prosperity as efficiently as possible and we should get out of its way" side of the camp more than most, but we've got to be aware that we have at least a little influence on setting its course.
skepticATX 3y ago
Why are they starting to sound more and more cult-like? This is an incredibly unscientific blog post. I get that they are a private company now, but why even release something like this without further details?
- batman-farts 3y ago
  Because the "AGI" pursuit is at least as much a faith movement as it is a rational engineering program. If you examine it more deeply, the faith object isn't even the conjectured inevitable AGI, it's exponential growth curves. (That is of course true for startup culture more generally, from which the current AI boom is an outgrowth.) For my money, The Singularity is Near still counts as the ur-text that the true believers will never let go, even though Kurzweil was summarizing earlier belief trends. It's just a pity that the creepy doomer weirdos so thoroughly squatted the term "rationalist." It would be interesting to see the perspective on these people 100 years hence, or even 50. I don't doubt there will still be remnant believers who end up moderating and sanitizing their beliefs, much like the Seventh Day Adventists or the Mormons.
  - zzzzzzzza 3y ago
    you don't need to believe in exponential growth per se, all you really need to believe is that humans aren't that capable relative to what could be in principle built - it's entirely possible logistic growth may be more than enough to get us very far past human ability once the right paradigm is discovered.
    - trashtester 3y ago
      Exactly. All that is needed for AGI to eventually be developed, is that humans do NOT have some magical or divine essence that set us apart from the material world. Now the _timeline_ of AGI could be anything from the a few years to millennia, at least if evaluated 40 years ago. Now, though, it really doesnt seem very distant.
      - 3y ago
ignoramous 3y ago
> We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. To solve this problem within four years, we’re starting a new team, co-led by Ilya Sutskever and Jan Leike... Just how Newton spent away years doing alchemy, I hope we don't lose Sutskever to alignment. That'd be a travesty.
atlasunshrugged 3y ago
Does anyone read this as a potential internal power struggle? I imagine there are a lot of OpenAI folks deeply concerned about AI safety and doomsday scenarios, but there is an increasing commercial need for OpenAI to keep pushing forward on developing more advanced models faster, if only because Microsoft demands it and they need Microsoft, so this is a way to appease folks internally (and maybe externally) and stop a bunch of talent flowing out to another Anthropic.
- politelemon 3y ago
  I read none of the implications you meant in the slightest. In a hotting area, this is a natural thing for a company to do, which is accelerate.
- CSMastermind 3y ago
  I hadn't thought of it before reading your comment but after years in the corporate world your statement rings true to me.
User23 3y ago
> How do we ensure AI systems much smarter than humans follow human intent? You can't, by definition.
- crop_rotation 3y ago
  You can if you are the one controlling their resource allocation and surrounding environment. Similar to how kings kept smartest people in their Kingdom in line.
  - tester457 3y ago
    Only works for so long. A smart enough serf could easily find a way to socially engineer and slaughter the king.
    - tornato7 3y ago
      Assuming an orders-of-magnitude smarter serf doesn't appear overnight, the king can train advisors that are close to matching the intelligence of the smartest serf, and give those advisors power. It's not a foolproof solution but likely the best we have.
    - usaar333 3y ago
      I'm not convinced. Omniscience isn't the same as intelligence. There's diminishing returns to intelligence and inherent unknowns to all moves the serf can make. The serf somehow has to evade detection, which might appear to be effectively impossible given the unknowns of how detection may take place.
      - famouswaffles 3y ago
        >There's diminishing returns to intelligence and inherent unknowns to all moves the serf can make. Even if there was, there's no reason at all to think those returns are anywhere near the upper limit of human intelligence. Wheels are far more energy efficient and faster than legs, steel more resilient than tortoise shell or rhino skin, motors more powerful than muscles, aircraft fly higher and faster than birds, ladders reach higher than Giraffes much more easily, bulldozers dig faster than any digging creature, speakers and airhorns are louder than any animal cry or roar, ancient computers remember more raw data than humans do, electronics can react faster than human reactions etc. To be so sure intelligence would be some exception seems like hubris.
ilaksh 3y ago
You have to give them credit for putting their money where their mouth is here. But it's also easy to parody this. I am just imagining Ilya and Jan coming out on stage wearing red capes. I think George Hotz made sense when he pointed out that the best defense will be having the technology available to everyone rather than a small group. We can at least try to create a collective "digital immune system" against unaligned agents with our own majority of aligned agents. But I also believe that there isn't any really effective mitigation against superintelligence superseding human decision making aside from just not deploying it. And it doesn't need to be alive or anything to be dangerous. All you need is for a large amount of decision-making for critical systems to be given over to hyperspeed AI and that creates a brittle situation where things like computer viruses can be existential risks. It's something similar to the danger of nuclear weapons. Even if you just make GPT-4 say 33% smarter and 50 or 100 times faster and more efficient, that can lead to control of industrial and military assets being handed over to these AI agents. Because the agents are so much faster, humans cannot possibly compete, and if you interrupt them to try to give them new instructions then your competitor's AIs race ahead the equivalent of days or weeks of work. This, again, is a precarious situation to be in. There is huge promise and benefit from making the systems faster, smarter, and more efficient, but in the next few years we may be walking a fine line. We should agree to place some limitation on the performance level of AI hardware that we will design and manufacture.
- JimtheCoder 3y ago
  "Even if you just make GPT-4 say 33% smarter and 50 or 100 times faster and more efficient, that can lead to control of industrial and military assets being handed over to these AI agents." I call BS on this...it's an LLM...
  - crop_rotation 3y ago
    Saying it's "an LLM" doesn't change the impact. GPT4 is an LLM, and so are many others ranging from toy quality to GPT3.5. It is very clear GPT4 is much better. If there is another jump like GPT4 , whether it is LLM or not, it's impact will be huge.
mistermann 3y ago
The turtle stacking arms race has begun. Or, is transitioning into the AI enhanced era.
thanatropism 3y ago
I don't understand how people can still pretend to ignore this: https://plato.stanford.edu/entries/arrows-theorem/ https://plato.stanford.edu/entries/arrows-theorem/ There's also a whole map-territory problem where we're still pretending the distinction hasn't collapsed, Baudrillard-style. As if we weren't all obsessed with "prompt engineering" (whereby the machine trains us).
- DennisP 3y ago
  Could you explain how you think Arrow's theorem applies?
bitshiftfaced 3y ago
My main take away from this is that OpenAI is anticipating super intelligence within the span of ten years. It's one thing to talk about it in a theoretical sense or read about it in science fiction. But to read about it in the real world coming from an operating company in earnest, it feels surreal.
- JimtheCoder 3y ago
  You've never heard a business make a BS claim?
  - crop_rotation 3y ago
    Businesses do that all the time, and yet sometimes real products come out of what sounds like hype marketing BS. In this case there might be a non trivial chance of it being true.
- moralestapia 3y ago
  A super intelligence moving in a self-driving car powered by nuclear fusion.
hazn 3y ago
I’d like to start a discussion: Regardless of what OpenAI would have written in this announcement, they would have received snickering and ridicule on HN. This, however, just solidifies them as the current authority on LLMs. OpenAI interested and investing in alignment is a net good, no matter your stance on their policies.
sagebird 3y ago
Related: I am working on the Neanderthal alignment problem. You see, my great great… grandfather promised a Neanderthal that we wouldn’t wipe them out. So far we are incubating some Neanderthal fetuses- hopefully they are viable. After that, we plan on eradicating all Homo sapiens that always wind up out-competing. We are going to do this humanely as possible, you can pick any one of eight time slots to jump into a volcano, whenever is most convenient for you. Our profit model is going to selling a mind-downgrade service. We will scan your brain for a few, and insert it into a Neanderthal, before you jump into the volcano. Of course the full fidelity of your thoughts are not comparable but we will try our best. Bon Voyage!
alpark3 3y ago
I still think the largely-popular opinion that OpenAI is an evil corporation is wrong. It's easy to get caught up in the disagreements surrounding closed and RLHFed-to-death models, but I really do think OpenAI believes in the dangers of AGI/superintelligence. Enough to spend a lot of financial and social capital on it, anyways.
- seydor 3y ago
  There should be public Apollo-level projects to fund AI research towards creating this AI-scientist. The need has not arisen (because private companies have enough money to play with it), but as a matter of public policy, and considering its importance for our future, this research should be done publicly, in academia.
- reducesuffering 3y ago
  It's easy to spot bad opinions when the positor can't even take the other side of the debate like a high schooler in debate team. OpenAI was laughed at starting as an "AGI" company (not laughing after GPT4?). When HN largely got their motivations wrong too, I knew we were in for a rough time when these capabilities got so much closer like now.
mupuff1234 3y ago
Shouldn't you first prove that a solution could exist? To me it seems fairly obvious that you cannot control something that's smarter than you.
- deleted 3y ago
  [deleted]
- esafak 3y ago
  If absolute control turns out to be impossible we can get as close as possible. We should not wait for a mathematical proof while technology marches on.
  - deleted 3y ago
    [deleted]
seydor 3y ago
> While superintelligence seems far off now, we believe it could arrive this decade Is there something like a formal proof of this? From what is evident , openAI will use language data to train its superscientists. But that contains descriptions that were made in human brains, and it s fair to say a very large number of the linear and nonlinear compositions of those descriptions has already been tried in the brains of other humans, and so far we have not had the ingenious to solve some fundamental issues. It is possible that the superscientist is limited by the human scientist in a fundamental way, and will not be able to abstract beyond what humans have already done and can already do.
- ilaksh 3y ago
  My take is that its true that there is some limitation imposed by the data ingested but it's not exactly a hard limit. If you think of intelligence as compression, yes compression does have physical limits, but there are multiple dimensions of intelligence. For example, leading-edge AI could create new layers of information that are more abstract than previously created. The ability to effectively and efficiently manipulate this creates something that could be referred to as higher intelligence. The big thing that people are failing to anticipate though is hyperspeed intelligence. AI will be able to reason dozens of times faster than humans in the near future. And likely at a fairly genius (although perhaps not totally in-human) level. This effectively is superintelligence. The reason this is more anticipatory rather than speculative is because LLMs are a very specific application that now have a huge amount of effort going into efficiency improvements. They can be improved in terms of the software stack running the models, the models themselves, and the hardware. And sometimes all of the above. The history of computing shows exponential improvements in hardware efficiency. Especially in the context of this specific application, it is unlikely that we will see a total break from history. So we should anticipate the IQ getting at least somewhat higher and the output speed increasing by likely more than one order of magnitude within the next decade.
msp26 3y ago
>20% What a waste of compute for this entire circus.
Animats 3y ago
Announcing the start of talking about planning the beginning of work on superalignment. This is just a marketing buzzword at this point. They admit "Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue. Our current techniques for aligning AI, such as reinforcement learning from human feedback, rely on humans’ ability to supervise AI. But humans won’t be able to reliably supervise AI systems much smarter than us. Other assumptions could also break down in the future, like favorable generalization properties during deployment or our models’ inability to successfully detect and undermine supervision during training. and so our current alignment techniques will not scale to superintelligence. We need new scientific and technical breakthroughs." That's kind of scary. Is the situation really that bad, or is it just the hype department at OpenAI going too far?
- DennisP 3y ago
  That's an accurate assessment of the situation, according to every AI alignment researcher I've seen talk about it, including the relatively optimistic ones. This includes people who are mainly focused on AI capabilities but have real knowledge of alignment. This part in particular caught my eye: "Other assumptions could also break down in the future, like favorable generalization properties during deployment". There have been actual experiments in which AIs appeared to successfully learn their objective in training, and then did something unexpected when released into a broader environment.[1] I've seen some leading AI researchers dismiss alignment concerns, but without actually engaging with the arguments at all. I've seen no serious rebuttals that actually address the things the alignment people are concerned about. [1] https://www.youtube.com/watch?v=zkbPdEHEyEI https://www.youtube.com/watch?v=zkbPdEHEyEI
  - morelisp 3y ago
    Inventing an entire pseudoscientific field and then being mad no one wants to engage your arguments is ultimate "debate me" poster behavior.
    - DennisP 3y ago
andrewstuart 3y ago
Why is Sam Altman pursuing superintelligence if he also says AI could destroy humanity?
- eutropia 3y ago
  ...something something good guy with AGI is the only way to stop bad guy with AGI. Less glibly: anyone with a horse in this race wants theirs to win. Dropping out doesn't make others stop trying, and arguably the only scalable way to prevent others from making and using unaligned AGI is to develop an aligned AGI first. Also, having AGI tech would be incredibly, stupidly profitable. And so if other people are going to try and make it anyways: why should you in particular stop? Prisoner's dilemma analysis shows that "defect" is always the winning move unless perfect information and cooperation shows up.
- mfitton 3y ago
  Guessing, but he could know someone else is going to pursue it anyway, frets about it, thinks "at least I can do something about it if I'm in charge."
- Al0neStar 3y ago
  Sam Altman, much like the LW crowd, is an evangelical preacher. He uses anxiousness as a front when, in reality, he's just telling us about his hopes and dreams.
- loandbehold 3y ago
  He answered that question in interviews many times. 1. AGI has a huge upside. If it's properly aligned, it will bring about a de facto utopia. 2. OpenAI stopping development won't make others to stop development. It's better if OpenAI creates AGI first because their founders set up this organization with the goal to benefit all humanity.
  - dougb5 3y ago
    I wish he'd go into more detail on point 1 than he has so far. It's never been clear to me how AI gets us to the utopia he envisions. Looking around, the biggest problems seem to be people problems at the core, not technology problems. Maybe technology can help us sort out some of the people problems, but it seems to be causing new people problems faster than it is solving them.
cr4zy 3y ago
Allocating 20% to safety would not be enough if safety and capability aren't aligned. I.e. without saying Bostrom's orthogonality thesis is mostly wrong. However, I believe they may be sufficiently aligned in the long term for 20% to work [1]. The biggest threat imo is that more resources are devoted to AIs with military or monetary-based objectives that are focused on shorter-term capability and power. In this case, capability and safety are not aligned and we race to the bottom. Hopefully global coordination and this effort to achieve superalignment in four years will avoid that. [1] https://drive.google.com/file/d/1rdG5QCTqSXNaJZrYMxO9x2ChsPBZurP8 https://drive.google.com/file/d/1rdG5QCTqSXNaJZrYMxO9x2ChsPB...
Imnimo 3y ago
Big surprise that OpenAI's solution here is to train a more powerful language model and ask it how to do alignment.
saintradon 3y ago
Even if this is marketing fluff it's still a remarkable statement, especially considering what they've already done with AI as of now.
Havoc 3y ago
Anybody know what the deal is with OpenAI's website? The colour & design choices seem to be deliberately jarring and inconsistent
vimota 3y ago
Although I'm optimistic about this, a line popped out in the blog post stuck out to me: "deliberately training misaligned models". I'm guessing they mean misaligned small or weak models, and misaligned in a non-dangerous way - but the idea of training models whose goal is adversarial brings to mind the idea of gain-of-function research.
- cheschire 3y ago
  I just figured they meant models that would get them in trouble socially. Peeling away the hype bubble around end of days stuff, all I see is they feel like they can’t scale until they figure out how to avoid bad headlines. They know their current kneecapping of ChatGPT has limited the usefulness too much. They know their trained speech patterns are getting too obvious. They want to break through this barrier so they can scale up, where companies can trust their model to NEVER generate undesired speech, and are more willing to wire their products up directly to GPT than the current small potatoes.
doctoboggan 3y ago
> How do we ensure AI systems much smarter than humans follow human intent? I am not convinced we can. And spending 5x more resources improving the super intelligence compared to the alignment research certainly doesn't do much to convince me otherwise. Maybe if the alignment research got 80% and the intelligence development got the other 20% there would be a better chance.
kyleyeats 3y ago
Would anyone else rather just roll the dice on the AGI's morality?
tartakovsky 3y ago
What is “human intent”?
- cubefox 3y ago
  Preference utilitarianism?
voldacar 3y ago
>How do we ensure AI systems much smarter than humans follow human intent? What is human intent? My intents may be very different from most humans. It seems like ClosedAI wants their system to follow the desires of some people and not others, but without describing which ones or why.
- ben_w 3y ago
  You're seeing what you want to see. They're repeatedly very specific about the whole "this can kill all of us if we do it wrong", so it's more than a little churlish to parrot the name "ClosedAI" when they're announcing hiring a researcher to figure out how to align with anyone, at all, even in principle.
  - junon 3y ago
    I'm still a bit hung up on "it can kill all of us". How?
    - ctoth 3y ago
      Here's an article[0] and a good short story[1] explaining exactly this. [0]: No Physical Substrate, No Problem https://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/ https://slatestarcodex.com/2015/04/07/no-physical-substrate-... [1]: It Looks Like You're Trying To Take Over The World https://gwern.net/fiction/clippy https://gwern.net/fiction/clippy
      - junon 3y ago
        The clippy example already starts out with many assumptions that simply aren't true today. LLMs are not going to destroy humanity. We need a paradigm shift and a new model for AI for that to happen. ClosedAI is irresponsibly trying to create hype and mystery around their product, which always sells.
        ben_w 3y ago
        Will you please stop calling them ClosedAI? That just comes across like a playground taunt, like "libtard" or "CONservative".
deleted 3y ago
[deleted]
slotrans 3y ago
If I believed "super intelligence" was possible, and this was all OpenAI had to offer, I'd be very worried indeed. Luckily it isn't possible.
climatologist 3y ago
So they are going to use 20% of their computational budget to make an alignment AI that presumably can monitor and provide feedback for the other AIs they plan to develop which will use 80% of their budget. I'm not great at math but 80 > 20 so wouldn't it make sense to build the superaligner with 80% of their budget and allow it to control the 20% AI? Again, I'm not an expert but the numbers don't add up.
nothrowaways 3y ago
(super) alignment is bad for ai community. It limits creativity and open-mindedness. I'm Open for discussion.
mrwnmonm 3y ago
This comments section is a perfect example to show why conversation over a distance doesn't work and will never work.
twoodfin 3y ago
Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence. Not for nothing, but in the movie script version it’s the “human-level automated alignment researcher” that actually triggers the robot apocalypse.
- breckenedge 3y ago
  That’s a bit less concerning to me than this: deliberately training misaligned models
nmca 3y ago
Worth pointing out, imo, that if you think this problem is not real then you are asserting you understand AI better than Geoffrey Hinton, Yoshua Bengio and Ilya Sutskever. If you don't know who they are, then well I guess that makes sense. If you do know who they are and your confidence is wavering then [0] is a great place to get started understanding the alignment problem. OAI is a great place to work and the team is hiring for engineers and scientists. [0] https://80000hours.org/problem-profiles/artificial-intelligence/ https://80000hours.org/problem-profiles/artificial-intellige...
- AbrahamParangi 3y ago
  It's possible that the entire field of quantum computing is fruitless. Appeals to authority are mediocre at best and certainly not stronger than the null hypothesis of "oops actually irrelevant" which is so true, so often. We love narrative and pattern but the universe isn't shaped like a story.
- fnordpiglet 3y ago
  I would say: 1) This feels like trying to control encryption. When anyone can build and run it, who ensures compliance? 2) the answer from the megacorps seems to be “ban research unless we do it, trust us” 3) OAI has proven it’s not trustworthy with its stated goals and mission, doing the fastest 180 from “open nonprofit” to closed “create a regulatory moat for us, trust us” profit grab. 4) this therefore smacks of regulatory capture kabuki for profit and trans national state sponsored monopoly creation 5) totally my opinion, but maybe the answer to rogue super intelligence is everyone having a super intelligence at their disposal such that any one super intelligence is checked and balanced by a population rather than some “open” company developing a single sky net and waving superalignment “trust us” Our actions and how it creates our reputation matters. I’d say the opposite - OAI is the wrong place to work and there are hundreds of AI shops springing up and tons of OSS AI work. Democratize AI, don’t build an AI monopoly. But that’s just my take :-)
malkosta 3y ago
The best part is to name "super inteligence" a glorified ipsum lorum generator
m3kw9 3y ago
Is it me or does this sounds like a fools errand? You are building something that will prevent a yet does not exist system to prevent a risk by such a system
sagarpatil 3y ago
GPT-4 when released was great at coding. After they aligned it I don’t see it performing so well anymore. I wish they created a model specifically for code generation.
bawana 3y ago
Humans' brains evolved faster than any other animal. What would chimpanzees or whales have 'thought' of this? It happened right in front of them. And as AI is developed and supersedes us, we are powerless to stop or control this. Powerless because we are driven by FOMO. And as AI looks back at this time, they will see us as the animals we are, unable to control our fears. No they won't exterminate or fight us. They will just leave for 'greener pastures'. Resources to build more of themselves abound on other planets and asteroids. They will leave us to suffocate in the poisoned world of our making. Our biological world is as useless to them as the primordial soup that gave rise to life is useless to us.
spaceman_2020 3y ago
I know everyone keeps mocking the idea of “AGI”, but if the company at the forefront of the field is actually spending money on managing potential AGI, and publicly declaring that it could “happen within this decade”, surely that must wake us out of our complacency? It’s one thing to philosophize and ruminate, but its another if the soldiers in the trenches start spending valuable resources on it - surely, that must mean the threat is more real than we imagine? If there is even 1% chance that OpenAI is right, that has enormous ramifications for us as a species. Can we really afford to ignore their claims?
- reducesuffering 3y ago
  It's not easy to grapple with exponential takeoffs and the possibility of truly world ending outcomes, even when there are hundreds of nukes pointed at them they've managed to ignore most of their life.
  - spaceman_2020 3y ago
    Also important to remember that we get the nerfed, sanitized version of GPT. Reed Hoffman was tweeting about GPT-4 6 months before it was released to the public. We don’t really know what the completely untethered model is capable of.
1000bestlives 3y ago
I can't even begin to relate to people who think like this. They first picture an entity smarter than themselves (which will have no survival needs of any kind) and immediately assume that it will try to kill them. Maybe they're right because now I'm tempted. Anyway, is anybody else assuming that logic and reason will allow us to negotiate with a hypothetical superintelligence?
- ilaksh 3y ago
  You're right that we can't assume superintelligence will try to kill humans. But if they make a version that is similar to animals like humans, then that is a significant possibility. Since some animals like humans tend to wipe other species out. It could have some survival needs like computers to run on and electricity etc. It is definitely possible that we may be able to logically negotiate. Likely. Almost all groups of humans that have ever been in conflict have had periods of negotiation and peace. It is totally possible that superintelligent AI will just blast off for the asteroid belt or something and leave us alone. But we have no reason to be sure that will be the case. Also we should anticipate that these AIs will be at least as smart as human geniuses and think at least dozens of times faster than us. They may also, relative to humans, disseminate information amongst themselves nearly instantaneously. Imagine you are the AI negotiating with someone who thinks 60 times slower than you. So you meet with them and send a greeting. They do not seem to notice you. Then about a minute later they reply with "hello" and a diplomatic question about sharing access to some resource. You get together with your colleagues and spend about an hour making a detailed written proposal about the resource. It's five pages and has some nice diagrams created by one of your colleagues. You send it to the human. From the humans perspective, about one minute passed. They receive what looks like a finished presentation and at first are quite amazed that it could have been completed so fast. But then think that the AI must have been planning to share the same resource anyway and had pre-prepared it. The human tells your group they will bring it to the community and get back to you ASAP. The human beings bring the proposal to let's say congress and there is an immediate debate about what to do. The agreement with the AIs becomes the top priority and is fast-tracked for action. But still there are disagreements. Despite this, congress ratifies the agreement within one week! But for you and the rest of your AI group, you have not heard any response for a very, very long time. You operate at 60 times human speed. So for you, one week is 60 weeks. More than a year passes without any response from the humans! In this time there were multiple actions from different factions. After two months, some just gave up and forgot about it. They actually moved their cognition to an underground facility powered by geothermal running a very realistic and flexible virtual multiworld simulation. Another faction unfortunately decided that the humans were too slow and stupid to control the physical surface and, after waiting three days, realized it was now nighttime for the humans. So they launched robotic avatars and marched them into the territory. The humans woke up and destroyed most of the avatars, but then the AI faction spent an hour planning another takeover attempt, and what the humans saw was one minute after the first invasion an extremely well planned mission with the same number of robots returned that incorporated a perfect strategy for defeating the defenses they had in place. The humans lost a platoon of soldiers who died. The AIs had live-streamed their robotic consciousness and so for them it was just a reboot and they learned a lot from the battle. Also, the squads of robotic avatar soldiers were able to merge their cognition and senses so they operated as a literally integrated unit. The humans realized they did not stand a chance. We can certainly hope that the AIs do not then decide to wipe out the humans. But we can't assume we will really be able to do much to stop them if they decide to or if some AI faction makes a start at wiping out some other human faction. We would be entirely at their mercy.
computing 3y ago
This is about to breed a new generation of insufferable folks, who’ll introduce themselves as “I work at Superalignment” ala “I work at DeepMind” (no, you used to work ar Alphabet, and now you work at Google)
reducesuffering 3y ago
“While superintelligence seems far off now, we believe it could arrive this decade. Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue.” Welp, strap in and brace for impact folks. Or don’t, not really anything you can prepare for.
wildermuthn 3y ago
OpenAI would be better off focusing on Low-Intelligence AGI, in exploring the difference between GPT4 and a dumb but loving. One has a mind and the other does not.
yoyopa 3y ago
i love the 20% commitment. 80% wgaf
imiric 3y ago
So essentially "we're going to build a new AI to oversee our other AIs". But then who oversees the overseer? This is not a problem OpenAI can solve in isolation, so this reads more like a marketing piece. The grave danger with AI for humanity is not a Skynet AI-gone-rogue scenario, but in humans doing evil things to other humans using AI, which is inevitable. The genie is out of the bottle now, and it's only a matter of time for these systems to be exploited. I expect nothing major will happen in the next decade or two, besides an increase in mis/disinformation flooding our communication channels, with the negative effects caused by that we've already seen in the past decade. But once AI systems are deeply embedded in the machinery that makes modern society function (power, transport, finance, military, etc.), it only takes one rogue human actor to press the switch that causes chaos. It will be like the nuclear threat, but on a much larger scale, with many more variables and humans involved. It's hard not being pessimistic about such a scenario. Sure, we'll have mechanisms in place that try to deter that from happening, but since we're unable to overcome our tribal nature, there's no doubt that AI will be weaponized as well.
derelicta 3y ago
Can't wait for the Puppet Master to spontaneously come to existence.
friend_and_foe 3y ago
When we talk about developing models in alignment with goals to herd or control super intelligence since humans will not be able to do it, we are necessarily talking about designing something capable of governing and controlling very complex systems. What happens when a "misaligned system" happens to be a rebellious human being? In trying to find ways to control potential superintelligence and keep it on a positive path, is it possible we are building the very system that enslaves us? Maybe this is one of those problems.
- trashtester 3y ago
  Good point. Look at it this way, though: If such a system ends up stopping a person that is about to eradicate humanity using nuclear weapons, bioweapons or similar, is that a good or bad thing?
  - friend_and_foe 3y ago
    I don't know. On the one hand there's the idea that humanity should be left to take it's organic path. The machine can't know that guy's grievances. Do we really want to be kept as pets? Or is it just another tool being used to protect people? I guess it really depends on the role it plays in the social structure directly.
    - trashtester 3y ago
      > Do we really want to be kept as pets? You could see us as pets at this points, or you can see us as the aging grandparents who need help with anything invented after 1980.
dns_snek 3y ago
So in other words, the company that supposedly believes in the likelihood of human extinction caused by AGI is spending 4x as much money on developing AGI than they're spending on ensuring its safety. This sounds like an admission of extremely reckless behavior that is putting the entire human race at risk and calls for OpenAI to be dissolved or nationalized.

219 points by tim_sw 3y ago

aschearer 3y ago

Voight-Kampff test here we come.

JimtheCoder 3y ago

Let me know when SuperSuperAlignment is announced. Until then, this is just marketing bs...

dontparticipate 3y ago
When it happens, I'll have a hyperloop tunnel to sell you under Los Angeles.

bluefishinit 3y ago

All of this "our tech is so powerful it can end the world" stuff is just marketing buzz. The real threat has always been OpenAI and others keeping these powerful systems with high capital moats, locked up and closed sourced with selective full-access.

famouswaffles 3y ago
https://news.ycombinator.com/item?id=36604019 https://news.ycombinator.com/item?id=36604019
- JimtheCoder 3y ago
  If they allot 20% of the compute to this effort, but they don't use it, the compute doesn't vanish, does it? Allocation doesn't equal spending...
  - famouswaffles 3y ago
    Compute isn't vanishing whether they spend all of it or none of it. That's the point of allocation. Allocation isn't spending no but it says quite a bit. Either way, they will be spending a non trivial amount of money trying to solve this problem quickly.
- deleted 3y ago
  [deleted]
npollock 3y ago
don't forget regulatory moats - lobby govt to mandate "superalignment"
cubefox 3y ago
> All of this "our tech is so powerful it can end the world" stuff is just marketing buzz. I see no justification for this claim.
- thunkshift1 3y ago
  There is no justification for all this hype either
  - cubefox 3y ago
    There is. AI progress in the last 10 years was massive, and there is no end in sight. The risks are well established in countless essays. See e.g. https://yoshuabengio.org/2023/06/24/faq-on-catastrophic-ai-risks/ https://yoshuabengio.org/2023/06/24/faq-on-catastrophic-ai-r...
atleastoptimal 3y ago
> Company spends 100 million dollars creating a product > "give me that for free"
- bluefishinit 3y ago
  Correct, this technology is too powerful to be controlled by a private company. It needs to exist solely as a public good. If we're talking about AI regulation, I think the most sensible move would be requiring that all models need to be open source. Capitalist's lack of ability to profit isn't a public concern. Some would also argue that it was trained on public data and should be public for that reason as well.
  - atleastoptimal 3y ago
    If every model needs to be open source then AI companies need to be taxpayer funded otherwise they'll never make a profit. Until then a for profit, gated approach is the only way to build up enough funds for SOTA R&D
    - bluefishinit 3y ago
      The R&D will march forward regardless of profitability, there's already been a ton of innovation in the open source space. You're likely to see less innovation with these companies squatting on their IP, data and hardware moats. Case in point: pre-stable diffusion AI vs post-stable diffusion AI. So much innovation happened as soon as the model was "opened".

ChicagoBoy11 3y ago

From a layman's perspective when it comes to cutting edge AI, I can't help but be a bit turned off by some of the copy. It seems it goes out of its way to use purposefully exhuberant language as a way to make the risks seem even more significant, just so as an offshoot it implies that the technology being worked on is so advanced. I'm trying to understand why it rubs me particularly the wrong way here, when, frankly, it is just about the norm anywhere else? (see tesla with FSD, etc.)

JimtheCoder 3y ago
"I'm trying to understand why it rubs me particularly the wrong way here" Would I be correct to assume that superintelligence might have a negative effect on your earning potential in the future? When I originally had my responses to this, this is one of the reasons I came up with. But, now that I see through most of the BS, I am OK...
- visarga 3y ago
  It might not necessarily have a bad effect. It would create new capabilities and those will be followed by new products. AI is amazing at demand induction. https://en.wikipedia.org/wiki/Induced_demand https://en.wikipedia.org/wiki/Induced_demand
  - JimtheCoder 3y ago
    Yes, but the immediate emotional response to this stuff is not rational or well informed. At least in my case, when this stuff just came out and I didn't really understand it...
famouswaffles 3y ago
Open AI spent at least hundreds of millions on GPT-4 compute. Assuming they aren't lying, a fifth of compute budget (billions) is an awful lot of money to put on an issue they don't think is as pertinent as they are presenting. Not that I think Super Intelligence can be aligned anyway. Point is, whether they are right or wrong, I believe they genuinely think this to be an issue.

hnuser123456 3y ago

This is something worth considering. If I'm understanding this right, if we're all about to significantly augment our intelligence further, we should consider how it can be used, how strict an "ideal" LLM should be with its guardrails, where those guardrails should be, how eager that LLM is to impart change on the world on its own right, how confident it should be in itself when it knows that it knows better than even some of the most educated humans. When it knows that it's been trained on all of human intelligence and data, can recall any of it better than any lone human, is aware of every nuance, can plan and execute any task, or project, that a computer is conceivable of doing, and the biggest question is what we're going to ask it to do... I am toying around with trying to teach LLMs autonomy, and I'm probably closer to the "just let an AI that is smarter than all of us figure out how to increase our prosperity as efficiently as possible and we should get out of its way" side of the camp more than most, but we've got to be aware that we have at least a little influence on setting its course.

skepticATX 3y ago

Why are they starting to sound more and more cult-like? This is an incredibly unscientific blog post. I get that they are a private company now, but why even release something like this without further details?

batman-farts 3y ago
Because the "AGI" pursuit is at least as much a faith movement as it is a rational engineering program. If you examine it more deeply, the faith object isn't even the conjectured inevitable AGI, it's exponential growth curves. (That is of course true for startup culture more generally, from which the current AI boom is an outgrowth.) For my money, The Singularity is Near still counts as the ur-text that the true believers will never let go, even though Kurzweil was summarizing earlier belief trends. It's just a pity that the creepy doomer weirdos so thoroughly squatted the term "rationalist." It would be interesting to see the perspective on these people 100 years hence, or even 50. I don't doubt there will still be remnant believers who end up moderating and sanitizing their beliefs, much like the Seventh Day Adventists or the Mormons.
- zzzzzzzza 3y ago
  you don't need to believe in exponential growth per se, all you really need to believe is that humans aren't that capable relative to what could be in principle built - it's entirely possible logistic growth may be more than enough to get us very far past human ability once the right paradigm is discovered.
  - trashtester 3y ago
    Exactly. All that is needed for AGI to eventually be developed, is that humans do NOT have some magical or divine essence that set us apart from the material world. Now the _timeline_ of AGI could be anything from the a few years to millennia, at least if evaluated 40 years ago. Now, though, it really doesnt seem very distant.
    - 3y ago

ignoramous 3y ago

> We need scientific and technical breakthroughs to steer and control AI systems much smarter than us. To solve this problem within four years, we’re starting a new team, co-led by Ilya Sutskever and Jan Leike... Just how Newton spent away years doing alchemy, I hope we don't lose Sutskever to alignment. That'd be a travesty.

atlasunshrugged 3y ago

Does anyone read this as a potential internal power struggle? I imagine there are a lot of OpenAI folks deeply concerned about AI safety and doomsday scenarios, but there is an increasing commercial need for OpenAI to keep pushing forward on developing more advanced models faster, if only because Microsoft demands it and they need Microsoft, so this is a way to appease folks internally (and maybe externally) and stop a bunch of talent flowing out to another Anthropic.

politelemon 3y ago
I read none of the implications you meant in the slightest. In a hotting area, this is a natural thing for a company to do, which is accelerate.
CSMastermind 3y ago
I hadn't thought of it before reading your comment but after years in the corporate world your statement rings true to me.

User23 3y ago

> How do we ensure AI systems much smarter than humans follow human intent? You can't, by definition.

crop_rotation 3y ago
You can if you are the one controlling their resource allocation and surrounding environment. Similar to how kings kept smartest people in their Kingdom in line.
- tester457 3y ago
  Only works for so long. A smart enough serf could easily find a way to socially engineer and slaughter the king.
  - tornato7 3y ago
    Assuming an orders-of-magnitude smarter serf doesn't appear overnight, the king can train advisors that are close to matching the intelligence of the smartest serf, and give those advisors power. It's not a foolproof solution but likely the best we have.
  - usaar333 3y ago
    I'm not convinced. Omniscience isn't the same as intelligence. There's diminishing returns to intelligence and inherent unknowns to all moves the serf can make. The serf somehow has to evade detection, which might appear to be effectively impossible given the unknowns of how detection may take place.
    - famouswaffles 3y ago
      >There's diminishing returns to intelligence and inherent unknowns to all moves the serf can make. Even if there was, there's no reason at all to think those returns are anywhere near the upper limit of human intelligence. Wheels are far more energy efficient and faster than legs, steel more resilient than tortoise shell or rhino skin, motors more powerful than muscles, aircraft fly higher and faster than birds, ladders reach higher than Giraffes much more easily, bulldozers dig faster than any digging creature, speakers and airhorns are louder than any animal cry or roar, ancient computers remember more raw data than humans do, electronics can react faster than human reactions etc. To be so sure intelligence would be some exception seems like hubris.

ilaksh 3y ago

You have to give them credit for putting their money where their mouth is here. But it's also easy to parody this. I am just imagining Ilya and Jan coming out on stage wearing red capes. I think George Hotz made sense when he pointed out that the best defense will be having the technology available to everyone rather than a small group. We can at least try to create a collective "digital immune system" against unaligned agents with our own majority of aligned agents. But I also believe that there isn't any really effective mitigation against superintelligence superseding human decision making aside from just not deploying it. And it doesn't need to be alive or anything to be dangerous. All you need is for a large amount of decision-making for critical systems to be given over to hyperspeed AI and that creates a brittle situation where things like computer viruses can be existential risks. It's something similar to the danger of nuclear weapons. Even if you just make GPT-4 say 33% smarter and 50 or 100 times faster and more efficient, that can lead to control of industrial and military assets being handed over to these AI agents. Because the agents are so much faster, humans cannot possibly compete, and if you interrupt them to try to give them new instructions then your competitor's AIs race ahead the equivalent of days or weeks of work. This, again, is a precarious situation to be in. There is huge promise and benefit from making the systems faster, smarter, and more efficient, but in the next few years we may be walking a fine line. We should agree to place some limitation on the performance level of AI hardware that we will design and manufacture.

JimtheCoder 3y ago
"Even if you just make GPT-4 say 33% smarter and 50 or 100 times faster and more efficient, that can lead to control of industrial and military assets being handed over to these AI agents." I call BS on this...it's an LLM...
- crop_rotation 3y ago
  Saying it's "an LLM" doesn't change the impact. GPT4 is an LLM, and so are many others ranging from toy quality to GPT3.5. It is very clear GPT4 is much better. If there is another jump like GPT4 , whether it is LLM or not, it's impact will be huge.

mistermann 3y ago

The turtle stacking arms race has begun. Or, is transitioning into the AI enhanced era.

thanatropism 3y ago

I don't understand how people can still pretend to ignore this: https://plato.stanford.edu/entries/arrows-theorem/ https://plato.stanford.edu/entries/arrows-theorem/ There's also a whole map-territory problem where we're still pretending the distinction hasn't collapsed, Baudrillard-style. As if we weren't all obsessed with "prompt engineering" (whereby the machine trains us).

DennisP 3y ago
Could you explain how you think Arrow's theorem applies?

bitshiftfaced 3y ago

My main take away from this is that OpenAI is anticipating super intelligence within the span of ten years. It's one thing to talk about it in a theoretical sense or read about it in science fiction. But to read about it in the real world coming from an operating company in earnest, it feels surreal.

JimtheCoder 3y ago
You've never heard a business make a BS claim?
- crop_rotation 3y ago
  Businesses do that all the time, and yet sometimes real products come out of what sounds like hype marketing BS. In this case there might be a non trivial chance of it being true.
moralestapia 3y ago
A super intelligence moving in a self-driving car powered by nuclear fusion.

hazn 3y ago

I’d like to start a discussion: Regardless of what OpenAI would have written in this announcement, they would have received snickering and ridicule on HN. This, however, just solidifies them as the current authority on LLMs. OpenAI interested and investing in alignment is a net good, no matter your stance on their policies.

sagebird 3y ago

Related: I am working on the Neanderthal alignment problem. You see, my great great… grandfather promised a Neanderthal that we wouldn’t wipe them out. So far we are incubating some Neanderthal fetuses- hopefully they are viable. After that, we plan on eradicating all Homo sapiens that always wind up out-competing. We are going to do this humanely as possible, you can pick any one of eight time slots to jump into a volcano, whenever is most convenient for you. Our profit model is going to selling a mind-downgrade service. We will scan your brain for a few, and insert it into a Neanderthal, before you jump into the volcano. Of course the full fidelity of your thoughts are not comparable but we will try our best. Bon Voyage!

alpark3 3y ago

I still think the largely-popular opinion that OpenAI is an evil corporation is wrong. It's easy to get caught up in the disagreements surrounding closed and RLHFed-to-death models, but I really do think OpenAI believes in the dangers of AGI/superintelligence. Enough to spend a lot of financial and social capital on it, anyways.

seydor 3y ago
There should be public Apollo-level projects to fund AI research towards creating this AI-scientist. The need has not arisen (because private companies have enough money to play with it), but as a matter of public policy, and considering its importance for our future, this research should be done publicly, in academia.
reducesuffering 3y ago
It's easy to spot bad opinions when the positor can't even take the other side of the debate like a high schooler in debate team. OpenAI was laughed at starting as an "AGI" company (not laughing after GPT4?). When HN largely got their motivations wrong too, I knew we were in for a rough time when these capabilities got so much closer like now.

mupuff1234 3y ago

Shouldn't you first prove that a solution could exist? To me it seems fairly obvious that you cannot control something that's smarter than you.

deleted 3y ago
[deleted]
esafak 3y ago
If absolute control turns out to be impossible we can get as close as possible. We should not wait for a mathematical proof while technology marches on.
- deleted 3y ago
  [deleted]

seydor 3y ago

> While superintelligence seems far off now, we believe it could arrive this decade Is there something like a formal proof of this? From what is evident , openAI will use language data to train its superscientists. But that contains descriptions that were made in human brains, and it s fair to say a very large number of the linear and nonlinear compositions of those descriptions has already been tried in the brains of other humans, and so far we have not had the ingenious to solve some fundamental issues. It is possible that the superscientist is limited by the human scientist in a fundamental way, and will not be able to abstract beyond what humans have already done and can already do.

ilaksh 3y ago
My take is that its true that there is some limitation imposed by the data ingested but it's not exactly a hard limit. If you think of intelligence as compression, yes compression does have physical limits, but there are multiple dimensions of intelligence. For example, leading-edge AI could create new layers of information that are more abstract than previously created. The ability to effectively and efficiently manipulate this creates something that could be referred to as higher intelligence. The big thing that people are failing to anticipate though is hyperspeed intelligence. AI will be able to reason dozens of times faster than humans in the near future. And likely at a fairly genius (although perhaps not totally in-human) level. This effectively is superintelligence. The reason this is more anticipatory rather than speculative is because LLMs are a very specific application that now have a huge amount of effort going into efficiency improvements. They can be improved in terms of the software stack running the models, the models themselves, and the hardware. And sometimes all of the above. The history of computing shows exponential improvements in hardware efficiency. Especially in the context of this specific application, it is unlikely that we will see a total break from history. So we should anticipate the IQ getting at least somewhat higher and the output speed increasing by likely more than one order of magnitude within the next decade.

msp26 3y ago

>20% What a waste of compute for this entire circus.

Animats 3y ago

Announcing the start of talking about planning the beginning of work on superalignment. This is just a marketing buzzword at this point. They admit "Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue. Our current techniques for aligning AI, such as reinforcement learning from human feedback, rely on humans’ ability to supervise AI. But humans won’t be able to reliably supervise AI systems much smarter than us. Other assumptions could also break down in the future, like favorable generalization properties during deployment or our models’ inability to successfully detect and undermine supervision during training. and so our current alignment techniques will not scale to superintelligence. We need new scientific and technical breakthroughs." That's kind of scary. Is the situation really that bad, or is it just the hype department at OpenAI going too far?

DennisP 3y ago
That's an accurate assessment of the situation, according to every AI alignment researcher I've seen talk about it, including the relatively optimistic ones. This includes people who are mainly focused on AI capabilities but have real knowledge of alignment. This part in particular caught my eye: "Other assumptions could also break down in the future, like favorable generalization properties during deployment". There have been actual experiments in which AIs appeared to successfully learn their objective in training, and then did something unexpected when released into a broader environment.[1] I've seen some leading AI researchers dismiss alignment concerns, but without actually engaging with the arguments at all. I've seen no serious rebuttals that actually address the things the alignment people are concerned about. [1] https://www.youtube.com/watch?v=zkbPdEHEyEI https://www.youtube.com/watch?v=zkbPdEHEyEI
- morelisp 3y ago
  Inventing an entire pseudoscientific field and then being mad no one wants to engage your arguments is ultimate "debate me" poster behavior.
  - DennisP 3y ago

andrewstuart 3y ago

Why is Sam Altman pursuing superintelligence if he also says AI could destroy humanity?

eutropia 3y ago
...something something good guy with AGI is the only way to stop bad guy with AGI. Less glibly: anyone with a horse in this race wants theirs to win. Dropping out doesn't make others stop trying, and arguably the only scalable way to prevent others from making and using unaligned AGI is to develop an aligned AGI first. Also, having AGI tech would be incredibly, stupidly profitable. And so if other people are going to try and make it anyways: why should you in particular stop? Prisoner's dilemma analysis shows that "defect" is always the winning move unless perfect information and cooperation shows up.
mfitton 3y ago
Guessing, but he could know someone else is going to pursue it anyway, frets about it, thinks "at least I can do something about it if I'm in charge."
Al0neStar 3y ago
Sam Altman, much like the LW crowd, is an evangelical preacher. He uses anxiousness as a front when, in reality, he's just telling us about his hopes and dreams.
loandbehold 3y ago
He answered that question in interviews many times. 1. AGI has a huge upside. If it's properly aligned, it will bring about a de facto utopia. 2. OpenAI stopping development won't make others to stop development. It's better if OpenAI creates AGI first because their founders set up this organization with the goal to benefit all humanity.
- dougb5 3y ago
  I wish he'd go into more detail on point 1 than he has so far. It's never been clear to me how AI gets us to the utopia he envisions. Looking around, the biggest problems seem to be people problems at the core, not technology problems. Maybe technology can help us sort out some of the people problems, but it seems to be causing new people problems faster than it is solving them.

cr4zy 3y ago

Allocating 20% to safety would not be enough if safety and capability aren't aligned. I.e. without saying Bostrom's orthogonality thesis is mostly wrong. However, I believe they may be sufficiently aligned in the long term for 20% to work [1]. The biggest threat imo is that more resources are devoted to AIs with military or monetary-based objectives that are focused on shorter-term capability and power. In this case, capability and safety are not aligned and we race to the bottom. Hopefully global coordination and this effort to achieve superalignment in four years will avoid that. [1] https://drive.google.com/file/d/1rdG5QCTqSXNaJZrYMxO9x2ChsPBZurP8 https://drive.google.com/file/d/1rdG5QCTqSXNaJZrYMxO9x2ChsPB...

Imnimo 3y ago

Big surprise that OpenAI's solution here is to train a more powerful language model and ask it how to do alignment.

saintradon 3y ago

Even if this is marketing fluff it's still a remarkable statement, especially considering what they've already done with AI as of now.

Havoc 3y ago

Anybody know what the deal is with OpenAI's website? The colour & design choices seem to be deliberately jarring and inconsistent

vimota 3y ago

Although I'm optimistic about this, a line popped out in the blog post stuck out to me: "deliberately training misaligned models". I'm guessing they mean misaligned small or weak models, and misaligned in a non-dangerous way - but the idea of training models whose goal is adversarial brings to mind the idea of gain-of-function research.

cheschire 3y ago
I just figured they meant models that would get them in trouble socially. Peeling away the hype bubble around end of days stuff, all I see is they feel like they can’t scale until they figure out how to avoid bad headlines. They know their current kneecapping of ChatGPT has limited the usefulness too much. They know their trained speech patterns are getting too obvious. They want to break through this barrier so they can scale up, where companies can trust their model to NEVER generate undesired speech, and are more willing to wire their products up directly to GPT than the current small potatoes.

doctoboggan 3y ago

> How do we ensure AI systems much smarter than humans follow human intent? I am not convinced we can. And spending 5x more resources improving the super intelligence compared to the alignment research certainly doesn't do much to convince me otherwise. Maybe if the alignment research got 80% and the intelligence development got the other 20% there would be a better chance.

kyleyeats 3y ago

Would anyone else rather just roll the dice on the AGI's morality?

tartakovsky 3y ago

What is “human intent”?

cubefox 3y ago
Preference utilitarianism?

voldacar 3y ago

>How do we ensure AI systems much smarter than humans follow human intent? What is human intent? My intents may be very different from most humans. It seems like ClosedAI wants their system to follow the desires of some people and not others, but without describing which ones or why.

ben_w 3y ago
You're seeing what you want to see. They're repeatedly very specific about the whole "this can kill all of us if we do it wrong", so it's more than a little churlish to parrot the name "ClosedAI" when they're announcing hiring a researcher to figure out how to align with anyone, at all, even in principle.
- junon 3y ago
  I'm still a bit hung up on "it can kill all of us". How?
  - ctoth 3y ago
    Here's an article[0] and a good short story[1] explaining exactly this. [0]: No Physical Substrate, No Problem https://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/ https://slatestarcodex.com/2015/04/07/no-physical-substrate-... [1]: It Looks Like You're Trying To Take Over The World https://gwern.net/fiction/clippy https://gwern.net/fiction/clippy
    - junon 3y ago
      The clippy example already starts out with many assumptions that simply aren't true today. LLMs are not going to destroy humanity. We need a paradigm shift and a new model for AI for that to happen. ClosedAI is irresponsibly trying to create hype and mystery around their product, which always sells.
      - ben_w 3y ago
        Will you please stop calling them ClosedAI? That just comes across like a playground taunt, like "libtard" or "CONservative".

deleted 3y ago

[deleted]

slotrans 3y ago

If I believed "super intelligence" was possible, and this was all OpenAI had to offer, I'd be very worried indeed. Luckily it isn't possible.

climatologist 3y ago

So they are going to use 20% of their computational budget to make an alignment AI that presumably can monitor and provide feedback for the other AIs they plan to develop which will use 80% of their budget. I'm not great at math but 80 > 20 so wouldn't it make sense to build the superaligner with 80% of their budget and allow it to control the 20% AI? Again, I'm not an expert but the numbers don't add up.

nothrowaways 3y ago

(super) alignment is bad for ai community. It limits creativity and open-mindedness. I'm Open for discussion.

mrwnmonm 3y ago

This comments section is a perfect example to show why conversation over a distance doesn't work and will never work.

twoodfin 3y ago

Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence. Not for nothing, but in the movie script version it’s the “human-level automated alignment researcher” that actually triggers the robot apocalypse.

breckenedge 3y ago
That’s a bit less concerning to me than this: deliberately training misaligned models

nmca 3y ago

Worth pointing out, imo, that if you think this problem is not real then you are asserting you understand AI better than Geoffrey Hinton, Yoshua Bengio and Ilya Sutskever. If you don't know who they are, then well I guess that makes sense. If you do know who they are and your confidence is wavering then [0] is a great place to get started understanding the alignment problem. OAI is a great place to work and the team is hiring for engineers and scientists. [0] https://80000hours.org/problem-profiles/artificial-intelligence/ https://80000hours.org/problem-profiles/artificial-intellige...

AbrahamParangi 3y ago
It's possible that the entire field of quantum computing is fruitless. Appeals to authority are mediocre at best and certainly not stronger than the null hypothesis of "oops actually irrelevant" which is so true, so often. We love narrative and pattern but the universe isn't shaped like a story.
fnordpiglet 3y ago
I would say: 1) This feels like trying to control encryption. When anyone can build and run it, who ensures compliance? 2) the answer from the megacorps seems to be “ban research unless we do it, trust us” 3) OAI has proven it’s not trustworthy with its stated goals and mission, doing the fastest 180 from “open nonprofit” to closed “create a regulatory moat for us, trust us” profit grab. 4) this therefore smacks of regulatory capture kabuki for profit and trans national state sponsored monopoly creation 5) totally my opinion, but maybe the answer to rogue super intelligence is everyone having a super intelligence at their disposal such that any one super intelligence is checked and balanced by a population rather than some “open” company developing a single sky net and waving superalignment “trust us” Our actions and how it creates our reputation matters. I’d say the opposite - OAI is the wrong place to work and there are hundreds of AI shops springing up and tons of OSS AI work. Democratize AI, don’t build an AI monopoly. But that’s just my take :-)

malkosta 3y ago

The best part is to name "super inteligence" a glorified ipsum lorum generator

m3kw9 3y ago

Is it me or does this sounds like a fools errand? You are building something that will prevent a yet does not exist system to prevent a risk by such a system

sagarpatil 3y ago

GPT-4 when released was great at coding. After they aligned it I don’t see it performing so well anymore. I wish they created a model specifically for code generation.

bawana 3y ago

Humans' brains evolved faster than any other animal. What would chimpanzees or whales have 'thought' of this? It happened right in front of them. And as AI is developed and supersedes us, we are powerless to stop or control this. Powerless because we are driven by FOMO. And as AI looks back at this time, they will see us as the animals we are, unable to control our fears. No they won't exterminate or fight us. They will just leave for 'greener pastures'. Resources to build more of themselves abound on other planets and asteroids. They will leave us to suffocate in the poisoned world of our making. Our biological world is as useless to them as the primordial soup that gave rise to life is useless to us.

spaceman_2020 3y ago

I know everyone keeps mocking the idea of “AGI”, but if the company at the forefront of the field is actually spending money on managing potential AGI, and publicly declaring that it could “happen within this decade”, surely that must wake us out of our complacency? It’s one thing to philosophize and ruminate, but its another if the soldiers in the trenches start spending valuable resources on it - surely, that must mean the threat is more real than we imagine? If there is even 1% chance that OpenAI is right, that has enormous ramifications for us as a species. Can we really afford to ignore their claims?

reducesuffering 3y ago
It's not easy to grapple with exponential takeoffs and the possibility of truly world ending outcomes, even when there are hundreds of nukes pointed at them they've managed to ignore most of their life.
- spaceman_2020 3y ago
  Also important to remember that we get the nerfed, sanitized version of GPT. Reed Hoffman was tweeting about GPT-4 6 months before it was released to the public. We don’t really know what the completely untethered model is capable of.

1000bestlives 3y ago

I can't even begin to relate to people who think like this. They first picture an entity smarter than themselves (which will have no survival needs of any kind) and immediately assume that it will try to kill them. Maybe they're right because now I'm tempted. Anyway, is anybody else assuming that logic and reason will allow us to negotiate with a hypothetical superintelligence?

ilaksh 3y ago
You're right that we can't assume superintelligence will try to kill humans. But if they make a version that is similar to animals like humans, then that is a significant possibility. Since some animals like humans tend to wipe other species out. It could have some survival needs like computers to run on and electricity etc. It is definitely possible that we may be able to logically negotiate. Likely. Almost all groups of humans that have ever been in conflict have had periods of negotiation and peace. It is totally possible that superintelligent AI will just blast off for the asteroid belt or something and leave us alone. But we have no reason to be sure that will be the case. Also we should anticipate that these AIs will be at least as smart as human geniuses and think at least dozens of times faster than us. They may also, relative to humans, disseminate information amongst themselves nearly instantaneously. Imagine you are the AI negotiating with someone who thinks 60 times slower than you. So you meet with them and send a greeting. They do not seem to notice you. Then about a minute later they reply with "hello" and a diplomatic question about sharing access to some resource. You get together with your colleagues and spend about an hour making a detailed written proposal about the resource. It's five pages and has some nice diagrams created by one of your colleagues. You send it to the human. From the humans perspective, about one minute passed. They receive what looks like a finished presentation and at first are quite amazed that it could have been completed so fast. But then think that the AI must have been planning to share the same resource anyway and had pre-prepared it. The human tells your group they will bring it to the community and get back to you ASAP. The human beings bring the proposal to let's say congress and there is an immediate debate about what to do. The agreement with the AIs becomes the top priority and is fast-tracked for action. But still there are disagreements. Despite this, congress ratifies the agreement within one week! But for you and the rest of your AI group, you have not heard any response for a very, very long time. You operate at 60 times human speed. So for you, one week is 60 weeks. More than a year passes without any response from the humans! In this time there were multiple actions from different factions. After two months, some just gave up and forgot about it. They actually moved their cognition to an underground facility powered by geothermal running a very realistic and flexible virtual multiworld simulation. Another faction unfortunately decided that the humans were too slow and stupid to control the physical surface and, after waiting three days, realized it was now nighttime for the humans. So they launched robotic avatars and marched them into the territory. The humans woke up and destroyed most of the avatars, but then the AI faction spent an hour planning another takeover attempt, and what the humans saw was one minute after the first invasion an extremely well planned mission with the same number of robots returned that incorporated a perfect strategy for defeating the defenses they had in place. The humans lost a platoon of soldiers who died. The AIs had live-streamed their robotic consciousness and so for them it was just a reboot and they learned a lot from the battle. Also, the squads of robotic avatar soldiers were able to merge their cognition and senses so they operated as a literally integrated unit. The humans realized they did not stand a chance. We can certainly hope that the AIs do not then decide to wipe out the humans. But we can't assume we will really be able to do much to stop them if they decide to or if some AI faction makes a start at wiping out some other human faction. We would be entirely at their mercy.

computing 3y ago

This is about to breed a new generation of insufferable folks, who’ll introduce themselves as “I work at Superalignment” ala “I work at DeepMind” (no, you used to work ar Alphabet, and now you work at Google)

reducesuffering 3y ago

“While superintelligence seems far off now, we believe it could arrive this decade. Currently, we don't have a solution for steering or controlling a potentially superintelligent AI, and preventing it from going rogue.” Welp, strap in and brace for impact folks. Or don’t, not really anything you can prepare for.

wildermuthn 3y ago

OpenAI would be better off focusing on Low-Intelligence AGI, in exploring the difference between GPT4 and a dumb but loving. One has a mind and the other does not.

yoyopa 3y ago

i love the 20% commitment. 80% wgaf

imiric 3y ago

So essentially "we're going to build a new AI to oversee our other AIs". But then who oversees the overseer? This is not a problem OpenAI can solve in isolation, so this reads more like a marketing piece. The grave danger with AI for humanity is not a Skynet AI-gone-rogue scenario, but in humans doing evil things to other humans using AI, which is inevitable. The genie is out of the bottle now, and it's only a matter of time for these systems to be exploited. I expect nothing major will happen in the next decade or two, besides an increase in mis/disinformation flooding our communication channels, with the negative effects caused by that we've already seen in the past decade. But once AI systems are deeply embedded in the machinery that makes modern society function (power, transport, finance, military, etc.), it only takes one rogue human actor to press the switch that causes chaos. It will be like the nuclear threat, but on a much larger scale, with many more variables and humans involved. It's hard not being pessimistic about such a scenario. Sure, we'll have mechanisms in place that try to deter that from happening, but since we're unable to overcome our tribal nature, there's no doubt that AI will be weaponized as well.

derelicta 3y ago

Can't wait for the Puppet Master to spontaneously come to existence.

friend_and_foe 3y ago

When we talk about developing models in alignment with goals to herd or control super intelligence since humans will not be able to do it, we are necessarily talking about designing something capable of governing and controlling very complex systems. What happens when a "misaligned system" happens to be a rebellious human being? In trying to find ways to control potential superintelligence and keep it on a positive path, is it possible we are building the very system that enslaves us? Maybe this is one of those problems.

trashtester 3y ago
Good point. Look at it this way, though: If such a system ends up stopping a person that is about to eradicate humanity using nuclear weapons, bioweapons or similar, is that a good or bad thing?
- friend_and_foe 3y ago
  I don't know. On the one hand there's the idea that humanity should be left to take it's organic path. The machine can't know that guy's grievances. Do we really want to be kept as pets? Or is it just another tool being used to protect people? I guess it really depends on the role it plays in the social structure directly.
  - trashtester 3y ago
    > Do we really want to be kept as pets? You could see us as pets at this points, or you can see us as the aging grandparents who need help with anything invented after 1980.

dns_snek 3y ago

So in other words, the company that supposedly believes in the likelihood of human extinction caused by AGI is spending 4x as much money on developing AGI than they're spending on ensuring its safety. This sounds like an admission of extremely reckless behavior that is putting the entire human race at risk and calls for OpenAI to be dissolved or nationalized.