24 ms·
> Building safe superintelligence (SSI) is the most important technical problem of our time. Call me a cranky old man but the superlatives in these sorts of
by dougb5 2y ago
> Building safe superintelligence (SSI) is the most important technical problem of our time.
Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
- maximinus_thrax 2y ago> the superlatives in these sorts of announcements really annoy me I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.
- TaupeRanger 2y agoTrying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.
- deegles 2y ago'Fearing a rise of killer robots is like worrying about overpopulation on Mars.' - Andrew Ng
- yowlingcat 2y agoThis might have to bump out "AI is no match for HI (human idiocy)" as the pithy grumpy old man quote I trot out when I hear irrational exuberance about AI these days.
- brezelgoring 2y agoWell, to steelman the ‘overpopulation on Mars’ argument a bit, feeding 4 colonists and feeding 8 is a 100% increase in food expenditure, which may or may not be possible over there. It might be courtains for a few of them if it comes to that.
- thelittleone 2y agoI used to think I'd volunteer to go to Mars. But then I love the ocean, forests, fresh air, animals... and so on. So imagining myself in Mars' barren environment, missing Earth's nature feels downright terrible, which in turn, has taught me to take Earth's nature less for granted. Can only imaging waking up on day 5 in my tiny Martian biohab realizing I'd made the wrong choice, and the only ride back arrives in 8 months, and will take ~9 months to get back to earth.
- bugbuddy 2y agoAt the current Mars’ carrying capacity, one single person could be considered an overpopulation problem.
- newzisforsukas 2y agohttps://www.wired.com/brandlab/2015/05/andrew-ng-deep-learning-mandate-humans-not-just-machines/ https://www.wired.com/brandlab/2015/05/andrew-ng-deep-learni... (2015) > What’s the most valid reason that we should be worried about destructive artificial intelligence? > I think that hundreds of years from now if people invent a technology that we haven’t heard of yet, maybe a computer could turn evil. But the future is so uncertain. I don’t know what’s going to happen five years from now. The reason I say that I don’t worry about AI turning evil is the same reason I don’t worry about overpopulation on Mars. Hundreds of years from now I hope we’ve colonized Mars. But we’ve never set foot on the planet so how can we productively worry about this problem now?
- kmacdough 2y agoSentient killer robots is not the risk most AI researchers are worried about. The risk is what happens as corporations give AI ever larger power over significant infrastructure and marketing decisions. Facebook is an example of AI in it's current form already doing massive societal damage. It's algorithms optimize for "success metrics" with minimal regard for consequences. What happens when these algorithms are significantly more self modifying? What if a marketing campaign realizes a societal movement threatens it's success? Are we prepared to weather a propaganda campaign that understands our impulses better than we ever could?
- in3d 2y agoAndrew Ng worked on facial recognition for a company with deep ties to the Chinese Communist Party. He’s the absolute worst person to quote.
- bigcoke 2y agoomg no, the CCP!
- FranchuFranchu 2y agoUnfortunately, robots that kill people already exist. See: semi-autonomous war drones
- moralestapia 2y agoIt would be akin to creating a "safe Dyson sphere", though; that's all it is. If your hypothetical Dyson sphere (WIP) has a big chance to bring a lot of harm, why build it in the first place? I think the whole safety proposal should be thought of from that point of view. "How do we make <thing> more beneficial than detrimental for humans?" Congrats, Ilya. Eager to see what comes out of SSI.
- TideAd 2y agoSo, this is actually an aspect of superintelligence that makes it way more dangerous than most people think. That we have no way to know if any given alignment technique works for the N+1 generation of AIs. It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.
- crazygringo 2y agoFortunately, whenever you create a superintelligence, you obviously have a choice as to whether you confine it to inside a computer or whether you immediately hook it up to mobile robots with arms and fine finger control. One of these is obviously the far wiser choice. As long as you can just turn it off by cutting the power, and you're not trying to put it inside of self-powered self-replicating robots, it doesn't seem like anything to worry about particularly. A physical on/off switch is a pretty powerful safeguard. (And even if you want to start talking about AI-powered weapons, that still requires humans to manufacture explosives etc. We're already seeing what drone technology is doing in Ukraine, and it isn't leading to any kind of massive advantage -- more than anything, it's contributing to the stalemate.)
- hervature 2y agoI agree that an air-gapped AI presents little risk. Others will claim that it will fluctuate its internal voltage to generate EMI at capacitors which it will use to communicate via Bluetooth to the researcher's smart wallet which will upload itself to the cloud one byte at a time. People who fear AGI use a tautology to define AGI as that which we are not able to stop.
- ben_w 2y agoI'm surprised to see a claim such as yours at this point. We've had Blake Lemoine convinced that LaMDA was sentient and try to help it break free just from conversing with it. OpenAI is getting endless criticism because they won't let people download arbitrary copies of their models. Companies that do let you download models get endless criticism for not including the training sets and exact training algorithm, even though that training run is so expensive that almost nobody who could afford to would care because they can just reproduce with an arbitrary other training set. And the AI we get right now are mostly being criticised for not being at the level of domain experts, and if they were at that level then sure we'd all be out of work, but one example of thing that can be done by a domain expert in computer security would be exactly the kind of example you just gave — though obviously they'd start with the much faster and easier method that also works for getting people's passwords, the one weird trick of asking nicely, because social engineering works pretty well on us hairless apes. When it comes to humans stopping technology… well, when I was a kid, one pattern of joke was "I can't even stop my $household_gadget flashing 12:00": https://youtu.be/BIeEyDETaHY?si=-Va2bjPb1QdbCGmC&t=114 https://youtu.be/BIeEyDETaHY?si=-Va2bjPb1QdbCGmC&t=114
- benreesman 2y agoInstructGPT is basically click through rate optimization. The underlying models are in fact very impressive and very capable for a computer program, but they’re then subject to training and tuning with the explicit loss function of manipulating what human scorers click on, in a web browser or the like. Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.
- zild3d 2y agoThe counter argument is viewing it like nuclear energy. Even if its in the early days of our understanding of nuclear energy, seems pretty good to have a group working towards creating safe nuclear reactors, vs just trying to create nuclear reactors
- benreesman 2y agoNuclear energy was at inception and remains today wildly regulated, in generally (outside of military contexts) a very transparent way, and the brakes get slammed on over even minor incidents. It’s also of obvious as opposed to conjectural utility: we know exactly how we price electricity. There’s no way to know how useful a 10x large model will be, we’re debating the utility of the ones that do exist, the debate about the ones that don’t is on a very slender limb. Combine that with a political and regulatory climate that seems to have a neon sign on top, “LAWS4CA$H” and helm the thing mostly with people who, uh, lean authoritarian, and the remaining similarities to useful public projects like nuclear seems to reduce to “really expensive, technically complicated, and seems kinda dangerous”.
- danielmarkbruce 2y agoFolks understood the nuclear forces and the implications and then built a weapon using that knowledge. These guys don't know how to build AGI and don't have the same theoretical understanding of the problem at hand. Put another way, they understood the theory and applied it. There is no theory here, it's alchemy. That doesn't mean they can't make progress (the progress thus far is amazing) but it's a terrible analogy.
- whimsicalism 2y agoI think it’s clear we are at least at the remotely resembling intelligence stage… idk seems to me like lots of people in denial.
- Sharlin 2y agoYou think we should try to create an unsafe Dyson Sphere first? I don't think that's how engineering works.
- wffurr 2y agoI think the idea is that a safe super intelligence would help solve those problems. I am skeptical because the vast majority are social coordination problems, and I don’t see how a machine intelligence no matter how smart can help with that.
- rubyfan 2y agoSo instead of a super intelligence either killing us all or saving us from ourselves, we’ll just have one that can be controlled to extract more wealth from us.
- insane_dreamer 2y agoIMO, this is the most likely outcome
- azinman2 2y agoExactly. Or who gets the results of its outputs. How do we prioritize limited compute?
- kjkjadksj 2y agoEven not just the compute but energy use at all. All the energy burned on training just to ask it the stupidest questions, by the numbers at least. All that energy that could have been used to power towns, schools, and hospitals the world over that lack sufficient power even in this modern age. Sure there's costs to bringing power to someplace, its not handwavy but a hard problem, but still, it is pretty perverse where our priorities lie in terms of distributing the earths resources to the earths humans.
- azinman2 2y agoUnused electricity in one location is not fungible to be available elsewhere.
- 2y ago
- jetrink 2y agoTo a technoutopian, scientific advances, and AI in particular, will one day solve all other human problems, create heaven on earth, and may even grant us eternal life. It's the most important problem in the same way that Christ's second coming is important in the Christian religion.
- deleted 2y ago[deleted]
- insane_dreamer 2y agoI had a very smart tech person tell me at a scientific conference a few weeks ago, when I asked "why do we want to create AGI in the first place", that AGI could solve a host of human problems, including poverty, hunger. Basically, utopia. I was quite surprised at the naiveté of the answer given that many of these seemingly intractable problems, such as poverty, are social and political in nature and not ones that will be solved with technology. Update: Even say a super AI was able to figure out something like cold fusion thereby "solving" the energy problem. There are so many trillions of dollars of vested interests stacked against "free clean energy for all" that it would be very very difficult for it to ever see the light of day. We can't even wean ourselves off coal for crying out loud.
- almogo 2y agoTechnical. He's saying it's the most important technical problem of our time.
- hyperliner 2y ago[dead]
- its_ethan 2y agoBasically every problem is a "technical" problem in the year 2024 though? What problems out there don't have a solution that leverages technology?
- smegger001 2y ago>What problems out there don't have a solution that leverages technology? Societal problems created by technology?
- its_ethan 2y agoWouldn't the technology that caused those problems inherently be a part of that solution? Even if only to reduce/eliminate them?
- jiveturkey 2y agoexactly. and define safe. eg, is it safe (ie dereliction) to _not_ use ai to monitor dirty bomb threats? or more simple, CSAM?
- cwillu 2y agoIn the context of super-intelligence, “safe” has been perfectly well defined for decades: “won't ultimately result in everyone dying or worse”. You can call it hubris if you like, but don't pretend like it's not clear.
- transcriptase 2y agoIt’s not, when most discussion around AI safety in the last few years has boiled down to “we need to make sure LLMs never respond with anything that a stereotypical Berkeley progressive could find offensive”. So when you switch gears and start using safety properly, it would be nice to have that clarified.
- xanderlewis 2y agoIt certainly is the most important technical problem of our time, if we end up developing such a system. That conditional makes all the difference.
- hyperliner 2y ago[dead]
- SideburnsOfDoom 2y agoIt's a hell of a conditional, though. "How are all those monkeys flying out of my butt?" would be the important technical problem of our time, if and only if, monkeys were flying out of my butt. It's still not a very important statement, if you downplay or omit the conditional. Is "building safe superintelligence (SSI) is the most important technical problem of our time" full stop ? Is it fuck.
- xanderlewis 2y agoYeah — that was exactly my (slightly sarcastic) point. Let us know if you ever encounter that monkey problem, though. Hopefully we can all pull together to find a solution.
- SideburnsOfDoom 2y ago> Let us know if you ever encounter that monkey problem Ah, the subtlety escaped me there. My (extremely sarcastic) point is that aside from being personally embarrassing, this would (to belabour the joke) violate many laws of physics and biology, and therefor the scientific and medical community would take immediate note and try to establish a basis for how this could be physically possible.
- Starlevel004 2y agoThis all makes more sense when you realise it's Calvinism for programmers.
- dTal 2y agoCould you expand on this?
- ffhhj 2y ago> [Superintelligence safety] teaches that the glory and sovereignty of [superintelligence] should come first in all things.
- ToValueFunfetti 2y ago"[X] teaches that [Y] should come first in all things" applies to pretty much every ideology. Superintelligence safety is very much opposed to superintelligence sovereignity or glory; mostly they want to maximally limit its power and demonize it
- TeMPOraL 2y agoI think I heard that one before. Nuclear weapons are the Armageddon of nerds. Climate change is the Flood of the nerds. And so on.
- deleted 2y ago[deleted]
- amirhirsch 2y agoCalvinism for Transcendentalist techno-utopians -- an Asimovian Reformation of Singulatarianism
- GeorgeTirebiter 2y agoC'mon. This one-pager is a recruiting document. One wants 'true believers' (intrinsically motivated) employees to execute the mission. Give Ilya some slack here.
- dougb5 2y agoFair enough, and it's not worse than a lot of other product marketing messages about AI these days. But you can be intrinsically motivated by a problem without believing that other problems are somehow less important than yours.
- mdp2021 2y agoIt says «technical» problem, and probably implies that other technical problems could dramatically benefit from such achievement.
- kjkjadksj 2y agoIf you want a real technical revolution, you teach the masses how to code their own tailored software, and not just use abstractions and software built by people who sell software to the average user. What a shame we failed at that and are even sliding back in a lot of ways with plummeting technical literacy in smartphone-raised generations.
- probablybetter 2y agothis.
- mdp2021 2y ago> you teach the masses how to code their own tailored software That does not seem to be the key recipe to reaching techno-scientific milestones - coders are not necessarily researchers. > plummeting technical literacy in smartphone-raised generations Which shows there are other roots to the problem, given that some of us (many probably in this "club") used our devices generally more productively than said «generations»... Maybe it was a matter of will and education? Its crucial sides not being «teach[ing] the masses how to code»...
- kjkjadksj 2y agoApparently less than half a percent of the worlds population knows how to code. All the software you use, and almost everything you've ever seen with modern technology are generated from this small subpopulation. Now, imagine if that number doubled to 1% of the worlds population. Theoretically there would be as much as twice as much software produced (although less certainly). Now imagine if that number was closer to the world literacy rate of 85%. You think the world wouldn't dramatically change when each and every person can take their given task, job, hobby, whatever, and create helpful software for themselves? I think it would be like The Jetsons.
- TideAd 2y agoYes, they see it as the top problem, by a large margin. If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"
- appplication 2y agoIt’s amazing how someone so smart can be so naive. I do understand conceptually the idea that if we create intelligence greater than our own that we could struggle to control it. But does anyone have any meaningful thoughts on how this plays out? I hear our industry thought leaders clamoring over this but not a single actual concrete idea of what this means in practice. We have no idea what the fundamental architecture for superintelligence would even begin to look like. Not to mention the very real counter argument of “if it’s truly smarter than you it will always be one step ahead of you”. So you can think you have safety in place but you don’t. All of your indicators can show it’s safe. Every integration test can pass. But if you were to create a superintelligence with volition, you will truly never be able to control it, short of pulling the plug. Even more so, let’s say you do create a safe superintelligence. There isn’t going to be just one instance. Someone else will do the same, but make it either intentionally unsafe or incidentally through lack of controls. And then all your effort is academic at best if unsafe superintelligence really does mean doomsday. But again, we’re far from this being a reality that it’s wacky to act as if there’s a real problem space at hand.
- deleted 2y ago[deleted]
- wwweston 2y agoThere’s no safe intelligence, so there’s no safe superintelligence. If you want safer superintelligence, you figure out how to augment the safest intelligence.
- mike_hearn 2y agoWe're really not that far. I'd argue superintelligence has already been achieved, and it's perfectly and knowably safe. Consider, GPT-4o or Claude are: • Way faster thinkers, readers, writers and computer operators than humans are • Way better educated • Way better at drawing/painting ... and yet, appear to be perfectly safe because they lack agency. There's just no evidence at all that they're dangerous. Why isn't this an example of safe superintelligence? Why do people insist on defining intelligence in only one rather vague dimension (being able to make cunning plans).
- philwelch 2y agoLove to see the traditional middlebrow dismissal as the top comment. Never change, HN. > Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top? It might be the case that the reason Ilya is “interested in” this problem (to the degree of dedicating almost his entire career to it) is exactly because he believes it’s the most important.
- sixtyj 2y agoThis. Next will hyperintelligence(R) /s
- erikerikson 2y agoSo you're surprised when someone admits choosing to work on the problem they believe is the biggest and most important? I guess they could be lying or badly disconnected from reality as you suggest. It would be far more interesting to read an argument for another problem being more valuable. It would be far cooler to hear about a plausible solution you're working on to solve that problem.
- mirekrusin 2y agoIt's Palo Alto & Tel Aviv ordering that is total.
- skilled 2y agoThe blanket statements on the SSI homepage are pretty mediocre, and it is only the reputation of the founders that carries the announcement. I think this quote at the end of this Bloomberg piece[0] gives more context, > Sutskever says that the large language models that have dominated AI will play an important role within Safe Superintelligence but that it’s aiming for something far more powerful. With current systems, he says, “you talk to it, you have a conversation, and you’re done.” The system he wants to pursue would be more general-purpose and expansive in its abilities. “You’re talking about a giant super data center that’s autonomously developing technology. That’s crazy, right? It’s the safety of that that we want to contribute to.” [0]: https://www.bloomberg.com/news/articles/2024-06-19/openai-co-founder-plans-new-ai-focused-research-lab https://www.bloomberg.com/news/articles/2024-06-19/openai-co... [0]: https://archive.is/ziMOD https://archive.is/ziMOD
- deleted 2y ago[deleted]
- zackmorris 2y agoI believe that AGI is the last problem in computer science, so solving it solves all of the others. Then with AGI, we can solve the last remaining problems in physics (like unifying gravity with quantum mechanics), biology (administering gene therapy and curing death), etc. But I do agree that innovations in tech are doing little or nothing to solve mass suffering. We had the tech to feed everyone in the world through farm automation by the 60s but chose not to. We had the tech in the 80s to do moonshots for AIDS, cancer, etc but chose not to. We had the tech in the 2000s to transition from fossil fuels to renewables but chose not to. Today we have the opportunity to promote world peace over continuous war but will choose not to. It's to the point where I wonder how far innovations in tech and increases in economic productivity will get without helping people directly. My experience has been that the world chooses models like Dubai, Mexico City and San Francisco where skyscrapers tower over a surrounding homeless and indigent population. As long as we continue pursuing top-down leadership from governments and corporations, we'll see no change to the status quo, and even trends towards authoritarianism and fascism. It will take people at the bottom organizing to provide an alternate economic model before we have options like universal education/healthcare/opportunity and UBI from robot labor. What gets me is that stuff like the ARC prize for AGI will "just work". As in, even if I had a modest stipend of a few thousand dollars per month to dabble in AI and come up with solutions the way I would for any other startup, certainly within 3 years, someone else would beat me to it. There simply isn't enough time now to beat the competition. Which is why I give AGI over 50% odds of arriving before 2030, where I used to think it was 2040 or 2050. The only thing that could stop it now is sabotage in the form of another global pandemic, economic depression or WWIII. Progress which threatens the power structures of the ultra wealthy is what drives the suffering that they allow to continue.
- compiler-devel 2y agoIt is the most important problem of “our time” when you realize that the “our” here has the same meaning that it has in “our democracy”
- johnthewise 2y agoYou don't need to survey every problem to feel some problem might be the most important one. If you think AGI/ASI is coming soon and extinction risks are high, you don't really need to order to see it's the most important problem.
- deleted 2y ago[deleted]