4 ms·
OpenAI created a team to control 'superintelligent' AI – then let it wither
- slavetologic 2y agoThey dont have any super intelligence, which is why they don't care about these teams
- agucova 2y agoFor context, the point of the Superalignment team was to work on a problem known as scalable oversight: the problem of aligning models in a way that holds up as models become more capable [1]. The reason behind this is that current alignment techniques (like RLHF), have limitations which are expected to worsen as models are scaled up [2]. This is to say, the objective of the Superalignment team was precisely to work on techniques that would work for models which don't yet exist. They are of course aware that they don't yet have superintelligence. [1]: This paper by Anthropic is a good introduction to the problem; https://arxiv.org/abs/2211.03540 https://arxiv.org/abs/2211.03540 [2]: See, for example, Jan Leike's talk on this; https://www.youtube.com/watch?v=BtnvfVc8z8o https://www.youtube.com/watch?v=BtnvfVc8z8o
- consumer451 2y agoThank you very much for the informative non-hot-take context.
- andy99 2y agoLuckily that doesn't exist. It's about equivalent to spinning up a Ghostbusters unit and letting it wither. You can argue that we'll be in trouble if we encounter ghosts, but on likelihood weighted basis there are much better uses of time.
- lgas 2y agoProbability is only half of the expected value equation. When the outcome is excessive then you should care even if the probability is low as long as it is non-zero.
- XorNot 2y agoNobody knows what the outcome is though. Its literally bad science fiction being used to pay people to do... What exactly? About a technology which doesn't exist.
- manmal 2y agoPart of me thinks that’s exactly why this department has been sidelined - having such a department is necessary to create hype (“we are creating things so powerful we have to explore how to contain them”), but it doesn’t need to thrive either.
- ilrwbwrkhv 2y agoThe real answer is they have realized internally that agi is simply not possible. Even gpt 4o doesn't understand at all any logic.
- sigmoid10 2y agoThat's simply not true. We now have LLMs that perform similarly or better than average humans on general reasoning and logic benchmarks. So if you were to say that they don't understand logic at all, then by that definition most humans don't either (which can be debated, but it's a different topic).
- codexon 2y agoGPT4o still can't reason. It is super fancy autocomplete. https://i.imgur.com/z83umbk.jpeg https://i.imgur.com/z83umbk.jpeg Here I change a widely known riddle to the opposite answer, and I manage to make it state them both as the answer.
- jordanpg 2y agoAlso, even if this oft-repeated trump card is true, why are you so sure this is different from how our own brains functionally work?
- codexon 2y agoBecause if this is how our brains really worked, then chatgpt wouldn't be beating most humans at standardized tests and then failing this absurdly easy question that even an elementary school kid could pass. It is simply regurgitating this phrase without even considering that it is stating the exact opposite of the answer it just gave, simply because most answers to this riddle on the internet say this at the end. > This riddle plays on the assumption that a surgeon is typically male, but in this case, the surgeon is the boy's mother. So from this 1 failing, you can see that it is a copy and paste machine, and it doesn't even understand that it is contradicting itself.
- FormerBandmate 2y agoIt could theoretically be possible, but it's not gonna happen from training a machine learning model off Reddit. SolidGoldMagikarp is the canary in the coal mine if you doubt it's Reddit
- agucova 2y agoI find this kind of dismissive attitude annoying. There are good arguments in the literature for why you might want to care about these risks [1, 2], and I think there's lots of room for reasonable disagreement about whether these are arguments are any good, but pretending the entire field of AI Safety is just delusional is just bad faith at this point. Especially when companies like OpenAI, Anthropic or GDM were explicitly created to build AGI, and have been talking about these risks since they were first founded. [1]: An introductory paper I like is The Alignment Problem for a Deep Learning Perspective, from Ngo et al, https://openreview.net/forum?id=fh8EYKFKns https://openreview.net/forum?id=fh8EYKFKns. [2]: A broader, less technical introduction to AI Safety that I like is Hendricks et al's An Overview of Catastrophic AI Risks, https://arxiv.org/abs/2306.12001 https://arxiv.org/abs/2306.12001
- gfourfour 2y ago> This paper provides an overview of the main sources of catastrophic AI risks, which we organize into four categories: malicious use, in which individuals or groups intentionally use AIs to cause harm; AI race, in which competitive environments compel actors to deploy unsafe AIs or cede control to AIs; organizational risks, highlighting how human factors and complex systems can increase the chances of catastrophic accidents; and rogue AIs, describing the inherent difficulty in controlling agents far more intelligent than humans. The first three risks are completely reasonable and people should be thinking about them. No, ChatGPT should not be diagnosing patients and giving them medicine. Yes, we should be vigilant to a flood of disinformation and revenge porn made possible by AI generated content. But when people talk about “AI safety” in this context, it’s usually in reference to the fourth category, planning for a superintelligent malicious AI that evades detection, self-replicates, etc. That’s pure science fiction at that point, and it’s not a reason to slow down development of LLMs, which yes are basically glorified chatbots and will not lead to “AGI” in this threatening sense. If I recall correctly, when steam engines started being able to go 40-50 MPH, there were people who were concerned that human beings would not be able to survive travel at such speeds because we never had experienced them. This wasn’t completely irrational, I suppose, as there are speed-induced G forces that are fatal, and they had no way of knowing the threshold back then. But once it was clear that steam locomotives weren’t in any danger of putting us over that threshold, incessant worry about locomotive-induced speeds death was kooky. “Locomotive safety” involving derailment mitigation, track crossing markings, etc. - still legitimate. But if “locomotive safety” was associated with people making claims like “we’re headed for a mass casualty event when the first locomotive hits 60 mph,” then “locomotive safety” would be marginalized. It doesn’t help that the public faces of “AI safety” include autodidactic pseudointellectuals, clearly mentally unwell people, and philosophers too deep in their own “taken to its logical conclusion…” thought experiments.
- throwaway115 2y ago>What if someone builds a supermassive teapot in Earth's orbit and it breaks free and comes hurtling towards us? Ok, that seems ridiculous and infeasible and nothing that has happened so far indicates that this is a real threat. >Yes, but it would be so catastrophic to humanity, that we must take it seriously, regardless of how improbable it is! This is the AGI debate, in my view. If we're picking different extremely improbable events to get worked up about, why stop at AI?
- circuit10 2y agoIt’s arguable whether it’s actually improbable. Many people, including (and seemingly especially) experts think it’s likely or even inevitable. This video might be interesting to see some arguments for why people think it’s important: https://youtu.be/9i1WlcCudpU https://youtu.be/9i1WlcCudpU
- JKCalhoun 2y agoI feel like we'll know right away though. We can then act right away. I'm not sure what you would do now that is different because you are anticipating such a thing.
- circuit10 2y agoIt would be far too late by that point, as it can already take measures to stop us. This is a difficult problem and we really should have started putting a lot more effort into this a lot earlier. Of course we can’t test a superintelligence directly yet, but we should do as much theoretical work as we can Actually the video addresses this point, the comparison it makes is waiting until we’re already on Mars before we start thinking about spacesuits and airlocks
- Aloisius 2y agoInevitable? Sure assuming we don't extinct first. In the near future? No. This is just another in a long line of technology panics. Unfortunately, there always seems to be some "concerned" experts that are both overly optimistic about the speed of technological progress and overly pessimistic about where that progress will lead adding fuel to the fire. Eventually some other new technology will incite a new panic and this one will become another footnote in history like the fears over grey goo or genetically engineered superhumans.
- jijijijij 2y agoAt this point I am not sure, if it's all theatre to keep the hype up, or if these people actually drank the koolaid.
- agucova 2y agoIs your hypothesis that what, Jan Leike resigned as part of an elaborate conspiracy to boost OpenAI's prospect by... criticizing it? I find these theories to be extremely convoluted and implausible, and they often lack awareness of the history behind companies like OAI, Anthropic or GDM
- fumar 2y agoLLMs don't reason. Are LLMs on he path to AGI?
- addisonj 2y agoDo I think that we are likely to suddenly see a massive massive leap to AGI/ super AGI? Not really... But does it still seem like enough of a long tail risk in my/my kids lifetime that I hope someone is getting resources and access to think about this? I sure do. I know comparing AGI development to nuclear weapon development is a trope at this point, and in general, not that that useful of comparison... But I do think having a group with a diversity of how to think about the impact of desired outcomes is a good thing to have whenever you are working in the realm of world changing outcomes, and it seems like the changes at openAI are eroding that to some extent, even if it is really really unlikely to come out of this specific company at this time.
- j45 2y agoI like how you put this, things go slow before they go very fast. Also, competition search engine style.. this feels is getting ahead and staying ahead.
- soist 2y agoThere was one such group but they determined it was impossible because of Rice's theorem and other limitations of formal systems for computation. Logical incompleteness, Tarski's theorem, and Rice's theorem are the main meta-theoretical results that make alignment fundamentally unsolvable. If you're really concerned about robots taking over the world then understanding basic computational theory should be a prerequisite but most people are not willing to spend the time to learn the theory and instead focus on vague and ill-defined science fiction concepts which are very unlikely to be actually physically possible/implementable because of various physical and formal limitations of computers. I've decided anyone concerned about these issues knows almost nothing about computability theory so their theories are either nonsensical or just outright crazy. Very few understand the required formal concepts to have any useful ideas about how computers should be programmed to prevent "unsafe" results (which is often left just as ill-defined as most everything on AI safety and alignment research).
- nicklecompte 2y agoI don't think any of us will live to see AI smarter than a rat. So I am not concerned whether this team was going to superalign anything. The problem is that OpenAI's software, especially GPT-4o, is primed for dangerous misuse, and the demo videos of GPT-4o seemed "misaligned" with any reasonable standards of AI safety. Even if Leike/Sutskever have delusions of grandeur about AGI, at least they cared about the idea of AI safety. It seems like pushing the team out meant getting rid of a lot of internal critics (and implicitly threatening anyone else who might speak up).
- rahidz 2y agoIs this the same safety team that decided GPT-2 was too dangerous for us plebs to use? Good riddance.
- JumpCrisscross 2y agoThis is why finance needs regulators. A mistake happens, a team is empowered, five minutes go by and everyone forgets. Sometimes maliciously. More often because other priorities gain mindshare.
- EA-3167 2y agoI think everyone involved realized that LLM's are a dead end that has nothing to do with AGI, and we're no closer to AGI than we were before LLM's hit the scene. LLM's are nifty chatbots and have some other uses, but at a catastrophically high and unsustainable energy cost that means they will never be profitable. Who needs a safety team to control that?
- worik 2y ago> catastrophically high and unsustainable energy cost that means they will never be profitable. Reality differs from this opinion
- EA-3167 2y agoI vividly remember people saying that on this very forum when the topic was Crypto, the last "Make NVIDIA's stock and the market soar" technology was the darling. Proof of Work will give way to Proof of Stake, or some other thing will happen, trust us, the tech is just so good and useful! What makes this different? What do we actually have, stripping away all of the "it will be this someday" thinking? Pretty good chat bots, ok summarizers, things that can messily and somewhat unpredictably code at a low level, and worst of all, things that sometimes just make up what they're saying or otherwise fail in harmful ways. Then compare that to the cost to run these mediocre miracles.
- JumpCrisscross 2y ago> What do we actually have, stripping away all of the "it will be this someday" thinking? I can speak to a few restaurant owners who no longer need devs because an LLM can keep their website menu up to date. (Bonus: it’s now just a PDF.) That’s a tangible productivity change. Granted, it’s on par with crypto’s remittance-efficiency pitch. But unlike crypto, these advantages are growing. Speaking personally, Kagi’s AI search is far superior to a list of links in most cases.
- _giorgio_ 2y agoThere's no agi in sight, and by definition they have to bother you to justify themselves. Probably they even had something to say about how they legally download public data and videos.
- m3kw9 2y agoThey quickly find out there is no Skynet to control, it’s all just simple alignment so it doesn’t spit out how to make bombs and slurs. The ones that quit likely weren’t getting enough work to look important because AGI surely is thought to be coming, but everyone else inside knows they are very far off
- deleted 2y ago[deleted]
- neverokay 2y agoSomething is wrong with Sam Altman. You can’t find a single interview of him discussing technical details of anything. He may really be a tech version of George Santos. A grifter like that should be nowhere near in charge of tech like this.
- consumer451 2y agoSomewhat related discussion: "International Scientific Report on the Safety of Advanced AI" https://news.ycombinator.com/item?id=40400438 https://news.ycombinator.com/item?id=40400438 36 points | 53 comments
- bossyTeacher 2y agoHate to be that person but climate change is more of a threat than AI. Where is the team to reduce its impact? That should be the actual news. OpenAI realised that it is not getting closer to AGI anytime in the near future and it has more significant worries closer to the present. Including existential threats. Anyone worries about dangers to humanity should focus on climate change and jump off the sci fi hype train.
- darepublic 2y agoFrom the Jan leike tweet: > OpenAI is shouldering an enormous responsibility on behalf of all of humanity. To me this sounds delusional. It assumes all kinds of things, but primarily that openai will be the leader in this space up to and beyond smarter than human agi. This self important bs is also why I wouldn't trust open ai with this responsibility
- RayVR 2y agoAh, the wonderful lack of thought and skepticism we’ve come to expect from TC
- RcouF1uZ4gsC 2y ago> OpenAI’s Superalignment team, responsible for developing ways to govern and steer “superintelligent” AI systems, was promised 20% of the company’s compute resources, according to a person from that team. But requests for a fraction of that compute were often denied, blocking the team from doing their work. So… The team that was supposed to control and align super-human intelligence was unable to align merely human intelligence to give them compute? Maybe that was part of the test.
- jordanpg 2y agoThis big shakeup could just be a garden-variety "too many big egos in the room" situation. These folks have achieved a tech celebrity status that transcends anything else I can recall in recent memory.
- ChrisArchitect 2y ago[dupe] Lots more discussion: https://news.ycombinator.com/item?id=40390831 https://news.ycombinator.com/item?id=40390831 https://news.ycombinator.com/item?id=40391382 https://news.ycombinator.com/item?id=40391382 And Jan's post among others: https://news.ycombinator.com/item?id=40391412 https://news.ycombinator.com/item?id=40391412