15 ms·

Superintelligence: The Idea That Eats Smart People (2016)

• jelder 5mo ago
  > If you encountered a cheetah in pre-industrial times (and survived the meeting), you might think it was impossible for anything to go faster. Fun fact, there is no historical evidence of an adult human ever dying from a cheetah attack. They are naturally shy, and a lot smaller than you may realize.
  • cute_boi 5mo ago
    Yep. That said, unlike cheetahs, there’s plenty of evidence of leopards attacking humans. And these days, it’s the leopards, the closed-AI types and misanthropes -- telling everyone, “AI will take your job and there’s nothing you can do about it.”
    • ctoth 5mo ago
      I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" Dario's point, and the point of the people actually trying to solve the problem, is that AI is not just Anthropic and OpenAI. It's the knowledge that you can put more compute in, and get more capability out. It is a technology now. It exists, in the world. Wishing will not make it go away. Being angry at it will not make it go away. Lying about how much water it uses will not make it go away. If Anthropic and OpenAI Shut down tomorrow, Accenture will not say "oh guess that llm thing won't work, let's go back to hiring humans!" It is a truth that you can multiply matrices and get something that is economically useful. We cannot un-know this. Physics allows it, so it will happen. So we should probably figure out what the heck to do about it. If your answer is something along the lines of "restrict it" then 1. let me know how that goes when other people don't, and 2. I really would rather prefer a world where we have the machines do the work the machines can do, not a world where we have human makework. If this means we need to figure out redistribution, let's talk about redistribution!
      • somesortofthing 5mo ago
        A US-China AGI ban treaty could prevent superintelligence indefinitely. Data centers are hard to hide. Have fun buying GPUs when you're cut off from all global payments. America would have to make some unpleasant concessions but that seems like a solid trade for preventing a wide variety of nightmare futures.
        • deleted 5mo ago
          [deleted]
        • leonidasv 5mo ago
          Is very difficult (not to say impossible) to ban a ill-defined thing.
      • mbgerring 5mo ago
        Physics allows this, and actually taking advantage of it requires billions of dollars of unprecedented infrastructure buildout that is already destabilizing the power grid. The only reason that infrastructure buildout is happening at all is the ideological capture of a small handful of obscenely wealthy people, who are fueling this buildout by spreading the extreme paranoia you’re echoing here. I do not understand why no one else can see the circularity of this reasoning. There is nothing inevitable about tying up all of this productive capital in the pursuit of AI. There are many, many other projects requiring similar capital and human effort, with much more obvious payoffs, such as decarbonizing the world’s energy systems. “It’s physically possible to provide abundant electricity without burning fossil fuels” is more provably true than any of the insane science fiction bullshit that undergirds the AI buildout, and yet, the entire clean energy industry is still having to build insane financial Rube Goldberg contraptions to make incremental progress. “Inevitability” is a lie, period. This entire thing is extremely historically contingent, and we could easily stop this train tomorrow.
        • ctoth 5mo ago
          > This entire thing is extremely historically contingent, and we could easily stop this train tomorrow. So, the Baruch Plan? The Manhattan Project was $~2B in 1945 dollars, and a national-scale industrial mobilization. Now North Korea has the bomb. That's with nuclear material, which doesn't get easier and easier and easier to work with every year. Compare to the price to train GPT-2 in 2019 ($43,000), and in 2026 ($73) [0]. [0]: https://x.com/karpathy/status/2017703360393318587 https://x.com/karpathy/status/2017703360393318587
  • forinti 5mo ago
    There's a story about some Kenyans outrunning a Cheetah in 6km. It had been killing their livestock, so they decided to go after it. Cheetahs are very fast, but humans have way more endurance.
    • topkai22 5mo ago
      I think the term is persistence hunting (https://en.wikipedia.org/wiki/Persistence_hunting https://en.wikipedia.org/wiki/Persistence_hunting). There was an intriguing blog post years ago (which sadly I can’t find) about how terrifying a fictional portrayal of persistence hunting would be- grinding down prey through exhaustion, thinking you’ve escaped but being slowly tracked down.
      • foobiekr 5mo ago
        There's also the story "Go, Go, Go, Said the Bird" by Sonya Dorman in the first Dangerous Visions which is like this, in a way.
        • stoneman24 5mo ago
          There’s also “the ruum” by Arthur Porges[0]. We got as part of English class in high school, a long time ago. Try not to read the Wikipedia as it might spoil the short story, there’s the pdf available on the web somewhere [0] https://en.wikipedia.org/wiki/The_Ruum https://en.wikipedia.org/wiki/The_Ruum
          • sonofhans 5mo ago
            Yes! I remember reading that, as you say, a long time ago. This is the first time I’ve seen someone else reference it. I love that story.
          • Timwi 5mo ago
            I read it too in school. I thought it was super obscure!
      • 5mo ago
  • suzzer99 5mo ago
    Ice yachts get well over 100 mph. I'm not sure how much they were used in pre-industrial times.
  • implements 5mo ago
    Cheetahs (genus “Acinonyx”) are the largest ‘small cat’, apparently [1] - separate from the genus “Panthera“ containing the ones that will attack humans (lions, tigers, leopards). [1] There’s a well known viral video of a wildlife park keeper who sleeps with three cheetahs who behave pretty much like large house cats.
  • jpfromlondon 5mo ago
    Why is this a "fun fact", what is this reddit-tier comment contributing?
    • deleted 5mo ago
      [deleted]
  • ubermonkey 5mo ago
    Fun fact for me personally: because I am lucky enough to know a zookeeper who was in charge of the ambassador animal program at a major zoo, I have direct personal experience that cheetahs purr when you pet them. You're right. They're smaller than you probably imagine (about the weight of an average Labrador). That's still definitely big enough to be a problem if they felt threatened, I'm sure, but the animals my friend was in charge of were raised to be around people for outreach purposes. That particular cheetah, for example, had once been on the Today show.
• jjulius 5mo ago
  >Sam Altman, the man who runs YCombinator, is my favorite example of this archetype. He seems entranced by the idea of reinventing the world from scratch, maximizing impact and personal productivity. He has assigned teams to work on reinventing cities, and is doing secret behind-the-scenes political work to swing the election. >Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society. How right the author was.
• alex-reyss 5mo ago
  The main problem of the hard takeoff theory is not the abstract nature of the scenario but rather the fact that it makes the same mistake as the unconstrained optimization paradigm, it takes intelligence to be an unconstrained optimization process. In fact, if we consider the strongest version of the safety argument for AI, namely one in which the danger is not coming from robots but rather from a disembodied AI controlling our global finances and/or infrastructure, the assumption still does not correspond to reality.
  • zarzavat 5mo ago
    If anything the hard takeoff theory is too conservative. It turns out you don't need self-improvement to get to superintelligence. You just need a ridiculous amount of money. Where can you get a ridiculous amount of money? The market will give it to you because FOMO. AI is easier than people 10 years ago thought it would be. It's also easier to align than people feared it would be. It's the humans using the AI that are hard to control.
    • pixl97 5mo ago
      Eh, I have a feeling the game hasn't played out yet when it comes to AI control. If and when the feedback loop on self improvement becomes more efficient and the window on training significantly narrows then things getting out of control rather quickly seems likely. Especially that it's likely we'll have a metric fuckton of compute by that point.
    • PaulHoule 5mo ago
      Hard to say if AI with true agency is so ‘easy’. We had a breakthrough with language but not necessarily other things.
      • cindyllm 5mo ago
        [dead]
    • ToValueFunfetti 5mo ago
      I think it's a bit premature to say aligning is easier than expected. Our current AIs are sycophants, they lie about their progress, they circumvent access restrictions, they notice when they are being evaluated and change their behaviors, they find answers and tell you they came up with them themselves, they blindly download malware. A lot of this is excusable as hallucination, bad RLHF human evaluators, etc, but I don't think we can speculate how challenging generally aligning superintelligences is until we actually have an aligned subintelligence in at least the narrow domain of programming.
• willis936 5mo ago
  Some of the themes remind me of themes mentioned in this matrix analysis. Specifically I am reminded of the Dune concept of control: "you control what you can destroy" and then asking "do you control your refrigerator?". Sure, you can turn it off but then your food would rot and you might starve. So in a real sense humans have not controlled machines for a long time but have been co evolving in symbiosis. Sure, it's not driven by natural selection and standard rules of life, but it is important to frame our relationship with machines in new ways if we're ever going to make some sort of artificial intelligence. https://youtube.com/watch?v=BETHWKaXX4k https://youtube.com/watch?v=BETHWKaXX4k
  • andai 5mo ago
    Could you elaborate on that last part?
    • willis936 5mo ago
      I didn't mean it as in changing our framing enables technological progress but something we should do if we don't want to lose the control we have. e.g. if we lose all principle and intention then it doesn't really matter what happens with computers. In order to do something with intention we must first understand what we're doing. Skipping that step is an admission of defeat.
  • yetihehe 5mo ago
    I can destroy my refrigerator pretty easily. If I care about food inside, I can take it out into a new one. So, seems I control it by that definition. Your idea "but food will spoil"/"I will lose a small amount of wealth" seems irrelevant to the strict definition. Conversely I think it's a bad definition, it's a show of what is the frame of the mind of the person who states that: "I want to show my control by destroying my things, look how powerful I am" which sounds like a toddler. That's how you portray psychopathic/narcissistic disorders in movies.
    • willis936 5mo ago
      The fridge is a toy example. Take all machinery man has made and destroy it. Will you survive? Or live in a space ship and destroy the ship. Will you survive? Or have a pacemaker or an iron lung or dialysis. How about the simple concept of a combustion engine or any necessary subcomponent, where removing that today would grind all logistics to a halt. How long do you survive? Just because you have one machine that is replaceable doesn't refute that you need machines as much as they need you. If you so readily dismiss Herbert's definition of control posit a competitor and we can pressure test it. Also, "correlated with a toddler's world view" is not the epic rhetorical refute you think it is.
• blamestross 5mo ago
  AI Superintelligence doesn't scare me for the same reasons "grey goo" doesn't scare me. We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators. We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross. What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.
  • api 5mo ago
    I'm old enough to remember when grey goo and nanotechnology was the apocalyptic scenario du jour for a short time after some guy at MIT wrote a book, and because he was at MIT people took it seriously even though it was ridiculous. If someone at the University of Kentucky or Kansas had written such a book, it would have been ignored. When prestige manages to align with bad ideas, it's pretty awful, and it can derail the entire civilization for a while. I was like... nanotechnology and grey goo already exist. It's called biology. The scenarios I was reading were silly. They violated conservation laws and laws of physics. But people were believing it and calling for limits on nanotechnology research. I remember arguing with smart people on this, and that was when I started to realize that there's two kinds of dumb. I had the same realization later when I argued with an incredibly intelligent guy who was absolutely convinced the moon landings didn't happen. See, there's dumb-dumb and smart-dumb, and the people who thought grey goo would eat Earth or that the Apollo landings were a hoax were the latter. Smart-dumb is high-IQ rationalization of ultimately irrational and absurd ideas, and the smarter you are the more effectively you can do this. I've met some really shockingly brilliant fools over the years who believe in all kinds of outlandish conspiracy theories, absolute literalist religious fundamentalism, idiotic political doctrines that directly contradict basic logic and all of lived human history, and so on. All of them can engage in sophisticated airtight rationalizations. I sometimes wonder if this is one of the evolutionary forces constraining intelligence. In my experience, smarter people are somewhat more likely to believe highly sophisticated and complex stupid things, and they are much better at convincing others of these things. That's probably more dangerous to them, their family and friends, and the species than dumb people believing simple silly things that are easily debunked. On AI... Is AI potentially dangerous? Very. It's already dangerous in a number of ways. The biggest right now is probably mass production of personalized propaganda, mass surveillance, and mass manipulation. There's also the potential that bad actors could use it to accelerate their ability to make things like garage WMDs (biotech, chemical weapons, etc.). None of this requires hard take-off superintelligence. It's just inherent risks to a powerful technology. These are not entirely new risks. They were already present in the Internet and computing. AI just raises them to a higher level. The extreme hard take-off stuff is silly, and it actually distracts us from talking about the much more realistic dangers and coming up with reasonable solutions that don't also throw away the huge benefits of these technologies.
• reducesuffering 5mo ago
  In the big 2026, everything certain people worried about with superintelligence came to fruition and they were vindicated. The people closest to ASI are indicating recursive self improvement is imminent, the smartest engineers in the labs themselves are autonomously using agents to develop and improve the models. The arms race is evident. NVDA is the world's most valuable company determined by the worlds' collective wisdom of those with skin-in-the-game. If there exists a path of runaway superintelligence, the trajectory we've experienced has been following it to a tee. Their predictive power was affirmed. All the "AI is a nothingburger" predictions of the last decade, including many here even in the last year, have aged incredibly poorly.
  • AndrewKemendo 5mo ago
    Nobody cares what we (people who have been working on AGI a long time) think. We were dismissed as cranks before and now we’re just ignored by whomever is promising the most money to investors. So, par for the course. Everyone in AI has lived through all the cycles so far so this is just the biggest one yet.
  • polytely 5mo ago
    [flagged]
    • SpicyLemonZest 5mo ago
      I don't follow that argument at all. Many people who claim they're concerned about the end of the world think that the one sliver of hope we have is that the specific people in charge of modern AI research take existential concerns seriously. Even if you expect that won't be enough to save us, it could hardly help the situation to replace them with other people who are less sympathetic to and probably radicalized against existential concerns.
    • reducesuffering 5mo ago
      It has been frequently discussed amongst Less-Wrong / "doomers", especially after the Sam Altman home attack, why violence won't solve the coming problem with unaligned ASI. If you're actually curious, here's a good summary: https://x.com/ESYudkowsky/status/2043601524815716866 https://x.com/ESYudkowsky/status/2043601524815716866
• Animats 5mo ago
  It's amusing to read people in the past writing about the prospect of superhuman intelligence. The real problems have turned out to be different. Sycophancy and hallucinations, which are part of being confidently wrong, remains a big problem. Needing square miles of data centers was an issue in 1950s science fiction, and disappeared by the 1980s. Yet now they're being built, with private funding and the prospect of profit. The need for way too much training data indicates something is still wrong with the current approach. None of that was predicted.
  • AndrewKemendo 5mo ago
    I predicted on this site in 2016 the massive social and economic impacts AGI would have and specifically when RL data loops are not available to anyone but major players: https://news.ycombinator.com/item?id=12168228 https://news.ycombinator.com/item?id=12168228 I even wrote up a whole article that specifically called RL loop based development as the future: https://medium.com/@andrewkemendo/the-ai-revolution-will-be-different-from-the-microprocessor-revolution-cb417d324203 https://medium.com/@andrewkemendo/the-ai-revolution-will-be-... > Reinforcement Learning tasks rely on ridiculous amounts of data. Whereas with traditional software architecture, where you accomplish tasks through explicit task instruction, RL trains for tasks based on millions of tests through a reward system. Most importantly once you have trained it to some minimum level, if you deploy it correctly, then it should continue improving — so long as you bake feedback into the UX. Imagine that instead of telling excel what to do, you and every other user will have a conversation with excel, improving the system incrementally.
    • thoughtpeddler 5mo ago
      Agreed, and this is exactly what we see happening. Your posts back then were prescient ... there's literally now 'Copilot for Excel' and 'Claude for Excel' etc. But what do you propose the people/commons can still do at this stage to redistribute the inherent power found in RL data loops to a more stable equilibria of sharing participants?
      • AndrewKemendo 5mo ago
• d_silin 5mo ago
  "Recursive self-improvement" is in the same league as "perpetual motion". What would be a way to recursively self-improve algorithms for matrix multiplication (foundations of machine learning and inference)?
  • crlang44 5mo ago
    It’s not advances on the underlying operation of matrix multiplication that have driven ai progress to date. It’s the layers above that; trying different neural architectures (transformers w/attention mechanisms), and also different data and training regimes (different ways of doing reinforcement learning) that are the main drivers of improved performance. Perpetual motion is a physical impossibility. Whereas Ai is already being used to improve the workflow of ai researchers, thus speeding up improvements in said research. It’s not hard to see that AI could well be spun up to continue to try new arrangements of the aforementioned levers that drive ai progress on its own.
  • switchbak 5mo ago
    Presumably there's more efficient hardware foundations to perform these efficiently, and potential at the various abstraction layers for more efficiency. Obviously this is not unbounded - simple things would seem to have a physical limit to the potential improvement. But if you think of the optimization space: different physical representations, different approaches (photo, quantum, etc), more parallelism - there's undoubtedly a lot of headroom even on the matrix multiplication side. I would imagine there's a lot left on the table when it comes to the abstractions we've built. Infinite? No, but lots of potential. And what does a machine with a few orders of magnitude more power come up with? I'm not readily able to predict what something like that could create (maybe it's tapped out, but I doubt it). It seems to come down to an article of faith (as referenced in the article) that there's a lot more potential to be extracted in our current exploitation paths. Which I think is probably reasonable. Heck, even if a theoretical machine tops out at 3-5 orders of magnitude faster/more complex, I'm sure that could do some amazing things that look like magic to us.
  • liuliu 5mo ago
    I actually agree. At some point, a RSI system has to interact with real-world, and that imposes serialization constraints. It is harder to know how much that slow-down would be and how much speed-up we will get before that. But a RSI cannot simply be a exponential growth forever.
• andai 5mo ago
  Has anyone played SOMA? Spoiler warning. It explores this idea of, what if there's an AI in charge of ensuring mankind survives at all costs. What would it be willing to do, to keep us alive? Would we even recognize the result as human? It's a horror game and it explores all kinds of fascinating and disturbing scenarios. Simulations of human minds. Artificial worlds. Human minds in robot bodies. Genetically modified humans. Man-machine hybrids etc. (A great exploration of the substance/structure matrix, by the way. My favorite question in AI and consciousness. Is the special sauce in the material, or its shape, both, or neither?) The very question of aligning the AI with humans assumes that we have a very robust definition of what human means in the first place. Ostensibly the AI was aligned. It did succeed in keeping humans alive! But it did that in all sorts of ways that mostly made them wish it hadn't.
  • RajT88 5mo ago
    The same theme was present in season 2 of Raised By Wolves. (RIP my favorite show in some years)
  • Shalomboy 5mo ago
    SOMA was a cognito-hazard for me and my roommate in college; we played it in the dark together while on some sort of mild hallucinogen and when it came time for Simon to find that high-pressure dive suit, we lost our minds (no pun intended). Watching the WAU twist its way through PATHOS II in whatever way worked first is a particularly jarring analogy for what has happened to our own profession. I can't help but think it would be nice for Frictional Games to revisit this topic again soon. Sidenote: It breaks my heart that all the great underwater-settings in media are hotbeds of horror scenarios. I think Subnautica broke the mold for this, here's to hoping the next generation of aquanauts take to the depths from that series.
    • ACCount37 5mo ago
      There's the upcoming ONTOS game that looks like Frictional's SOMA successor.
      • 4mo ago
• andai 5mo ago
  > Hopefully you see the resemblance between this vision of AI and a genie from folklore. The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting. The monkey's paw. You know, you don't need superintelligence for that. Civilization was already doing this. "What if we just gave ourselves exactly what we wanted." Well, it turns out often that's not so good!
• andai 5mo ago
  >Is superintelligence just a memetic hazard? [Overblown fear by smart people who are too easily convinced.] Well we can do the wager. If it's a nothingburger, then the worst case scenario is that we approached AI too cautiously. (Ha. What are the odds of that?) If it's not a nothingburger, then we all die, unless the whole world agrees on the correct course of action in advance and coordinates perfectly. Hmm. Well, maybe we don't all die, but the world is irreversibly transformed into something incomprehensible and repulsive. Although, I don't really think we needed AI's help for that one. We should probably figure out how to align ourselves before we try to preach to the next species. I'm not exactly holding my breath though :/
  • dweinus 5mo ago
    Ah yes, Pascal's mugging.
  • OkayPhysicist 5mo ago
    That's just Pascal's wager with a secular/cyberpunk reskin, and has all the same trappings. Why aren't you a monk?
    • andai 5mo ago
      Yeah. Turns out game theory is useful in games. I don't understand the monk question though.
      • OkayPhysicist 5mo ago
        If sci-fi Pascal's wager applies, so does the original. Anybody who believes Pascal's wager should be dedicating themselves wildly to religion, in the chance that it saves from from eternal damnation. Hence, becoming a monk.
        • andai 5mo ago
          Oh, I'm actually reverting much of the subconscious Augustinian programming (most of what we call secular ethics) to the less busted Orthodox versions, but I think that is for unrelated reasons.
• cjcole 5mo ago
  "So I'd like to engage AI risk from both these perspectives. I think the arguments for superintelligence are somewhat silly, and full of unwarranted assumptions. But even if you find them persuasive, there is something unpleasant about AI alarmism as a cultural phenomenon that should make us hesitate to take it seriously. First, let me engage the substance. Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity" -- The framing here seems to me to equate "AI risk" and "AI alarmism" with buying in to belief in "Bostom-style superintellgence". I'm not sure if the author meant to put anyone who is alarmed by developments in what we're calling "AI" into the same bucket as "AI obsessives want to make it into a programming problem, by designing a God-like machine", but I think this conflation is unfair and, frankly, dangerous. I don't know what superintelligence is. I don't even know what intelligence is. And I don't really know what either "artificial" or "general" mean either when talking about "AGI". You can believe, as I do, that these things can be, and will inevitably will be if we don't radically correct course, used to do very bad things independent and short of being "God-like". When you have systems which can hypothesize, synthesize, and test thousands if not millions of potential infectious agents in bulk [0], and can then order the ingredients for you from dodgy websites via some "claw", and then when you put these systems under the unsupervised control of millions of people with varying levels of stability and altruism, something extremely bad is exceedingly likely to happen. I understand that 2016 is ages ago and things change, but I came away from the article with the impression that if I'm worried about AI risk then I'm a clown like the three pictured in the "Outside Argument" section (you're a Google-Glass-wearing cringe nerd if you're alarmed). Maybe that's my fault and I'm not smart enough to understand the actual point of the article. If I have misinterpreted, I welcome the correction. [0] https://www.nature.com/articles/s41467-024-53759-4 https://www.nature.com/articles/s41467-024-53759-4
• dmos62 5mo ago
  > The AI is all-powerful and gives you what you ask for, but interprets everything in a super-literal way that you end up regretting. I like imagining similar discourse when a more basic tool was invented: "A hammer is like a genie, it's all powerful, but, when you hit something with it, it interprets that super-literally, and it hits it."
  • throwuxiytayq 5mo ago
    Isn't this a misinterpretation of what everyone in the AI safety space is worried about, though? I think the idea is that having an AI that interprets everything in a super-literal way would probably be catastrophic, but we can't even build that. It would be a nice world-ending problem to have.
    • dmos62 5mo ago
      It very well could be, I don't really follow those discussions. Honestly, if I were worried about something on Earth intellectually evolving at a suboptimal pace, it would be humans.
    • dan-robertson 5mo ago
      The super literal interpretation ideas were much more common in the past when LLMs didn’t exist. Now we have models that are generally pretty good at picking up on nuance and understanding what you mean but also often quite bad at execution, which is roughly the opposite of that idea. I think reward hacking is perhaps the closest we see llms get to literal/malicious interpretations of instructions.
      • wizzwizz4 5mo ago
        LLMs are neither of those. They're quite good at pretending they understand what you mean, but they don't. That's why they can't execute: they're mimicking the form, not the substance, and then we see the form and anthropomorphise them in our minds.
        • 5mo ago
• everyone 5mo ago
  Any of this kinda talk these days is just part of the hype train for the LLM companies tulipomania pyramid scheme. Afaik no-one that is actually working on AGI is anywhere close atm.
  • ACCount37 5mo ago
    Even in the LLM dept: LLMs are the most general AI systems to date, and the performance only ever goes up. Whether adding +5% per model release is enough to get a broadly superhuman system remains to be seen. But my take is that there's no such thing as "not working on AGI" in the frontier labs. Everything that's being put into modern frontier systems is AGI groundwork, one way or the other.
    • cevn 5mo ago
      What's 1.05 x 0? Even if you run it 1000 times, it's still 0.
      • ACCount37 5mo ago
        Are you trying to say that the capability of modern LLMs is zero? Because if so, I'm pretty sure any frontier LLM is better at evaluating AI capabilities than you are.
        • cevn 5mo ago
          Claims of AGI imply that LLM's have intelligence. They don't, they are fancy probability machines. They don't THINK the way we do, they just do 200 matrix multiplications until their training data is massaged into what you need. They don't dream, they don't remember what you tell them. Even if you write one sentence, 'attention' means they will ignore half of what you say and key in on the wrong thing. This just happened to me today on a frontier model. I'm not saying that AGI is impossible, but the focus on LLM's is probably not the right approach. I don't think we will ever make it until we understand the human mind better.
• ChrisArchitect 5mo ago
  Some previous discussions: 2016 https://news.ycombinator.com/item?id=13240811 https://news.ycombinator.com/item?id=13240811 2018 https://news.ycombinator.com/item?id=18499973 https://news.ycombinator.com/item?id=18499973 2023 https://news.ycombinator.com/item?id=36098332 https://news.ycombinator.com/item?id=36098332
• usernametaken29 5mo ago
  Your assumption that minds come in all shapes and sizes is wrong. Read up on embodied cognition. If anything at all, AI are true aliens, unlike known minds.
• stared 5mo ago
  I like this essay, and have very often referred to it when someone talks about AGI. There is a common narrative bias to look at AGI as the Abrahamic God, if not explicitly, then just by saying that it is omniscient, omnipotent, immortal - and will judge us for our deeds. It is tempting for anyone raised in the West, and immersed in Judeo-Christian culture. And for anyone, in general, as it offers an epic narration of a personal entity. Yet, the reality might be messier - IMHO closer to biology than to a weird mixture of computer science and theology. There is no ultimate intelligence (see Karpathy’s starfish shapes), just a collection of adaptability, learning, generalization and self-reference. Also, even an extremely smart being (or process) can be fragile. So, less God, more WAU from SOMA or the Ocean from Solaris.
  • smaudet 5mo ago
    For all its talk of inoculation, this is a terribly written essay. They do not make a point, nor even arguments, instead, opting to ramble in hopes that you forget whatever it was you were thinking. The issue is simple. Just like us (who are arguably complex, look at what we're building over here, this AI computer stuff!), entities have simple core needs (like food, water, power, etc.). An infinitely smart AGI has the potential, nay, likely cause, to require infinite resources. We're already seeing the effect in the computing sector on e.g. chips, there's no reason to think this trend won't continue... Lets circle back to the hydrogen argument, will we blow ourselves up. Real concern, abated by hard numbers. Different atmosphere, different concentrations, different pressure, different possible outcomes. Today, we don't have those numbers. We don't have those calculations. I don't disagree with the point at the end "about how people can exploit other people, or through carelessness introduce immoral behavior into automated systems". These are issues, too. But saying there are other issues, don't worry about this big issue over here, is the absolute worse argument possible. That's hand waving.
    • kbrkbr 5mo ago
      From my point of view we have simply no idea what a infinitely smart AGI is and how to build it. How would it make the combinatorial explosion in state space search go away, to pick one example? And if it doesn't, is it then an infinitely smart AGI? The concept seems to assume all problems humans struggle with can be solved. The halting problem is one witness that this is probably not true.
• keybored 5mo ago
  You have to have a certain degree of intelligence in order to be convinced by stupid ideologies.
• gnarlouse 5mo ago
  Superintelligence Alignment conversations are a misdirect. Let's talk about Billionaire Alignment, Economic alignment, Human alignment. Classware should be M.A.D. -- in that it shouldnt even happen.
• WithinReason 5mo ago
  I can't say I disagree: We need better scifi! And like so many things, we already have the technology. This is Stanislaw Lem, the great Polish scifi author. English-language scifi is terrible, but in the Eastern bloc we have the goods, and we need to make sure it's exported properly. It's already been translated well into English, it just needs to be better distributed. What sets authors like Lem and the Strugatsky brothers above their Western counterparts is that these are people who grew up in difficult circumstances, experienced the war, and then lived in a totalitarian society where they had to express their ideas obliquely through writing. They have an actual understanding of human experience and the limits of Utopian thinking that is nearly absent from the west.
  • PaulHoule 5mo ago
    Lem’s Cyberiad was my favorite as a kid and still is!
• dang 5mo ago
  Related. Others? Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment) Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments) Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments) Maciej Ceglowski – Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13120213 https://news.ycombinator.com/item?id=13120213 - Dec 2016 (4 comments)
  • thoughtpeddler 5mo ago
    Thanks dang for compiling this. I suspect the Nov 2018 resurgence was due to Google publishing BERT [0] around that time? The release of OpenAI’s GPT-1 [1] was earlier that year in June, so unlikely that. Of course Jan 2023 needs no explanation… And now in 2026 things are at a fever pitch. Interesting to trace these 10yr old AI posts from then to the present moment. The other one with a similar vintage would be “Should AI Be Open” [2] from Dec 2015, which is fascinating to juxtapose against the recent public battles. [0] “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding“: https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805 [1] “Improving Language Understanding by Generative Pre-Training”: https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf https://cdn.openai.com/research-covers/language-unsupervised... [2] “Should AI Be Open?” | Slate Star Codex: https://slatestarcodex.com/2015/12/17/should-ai-be-open/ https://slatestarcodex.com/2015/12/17/should-ai-be-open/
• haberdasher 5mo ago
  "If AdSense became sentient, it would upload itself into a self-driving car and go drive off a cliff."
• holmesworcester 5mo ago
  This was one of the rare critiques of AI doom that actually understands the case for it and presents them well, so I kept reading to see what its arguments for our safety against AI doom were. They were roughly: 1. It's hard to put a cat in a box despite us being smarter than a cat, so we're safe. (Counter: we're pretty good at putting cats in boxes when it matters.) 2. It was hard for Australia to kill Emus, so we're safe. (Counter: Australia could probably kill all Emus if it mattered enough, and we definitely accidentally kill off species when one of their inputs for life matters enough to us.) 3. Some smart humans get paralyzed by hedonism or existential angst instead of optimizing for arbitrary goals implied by their arbitrary value sets, so we're safe. (Counter: others overthrow the Czar, land rockets, etc.) 4. Modern AI is data-trained, so recursive improvement requires more data, so we're safe. (Counter: AI-crafted, synthetic data is a thing.) 5. We don't (yet) know how to improve our brains with brain surgery, so we're safe. (Counter: same as #4 above, which unlike us/evolution AI is being deliberately trained to understand and perform.) 6. Children take a long time to grow up, so we're safe. (Counter: the author's own "Premise 5: Computer-Like Time Scales", where they correctly note that computers can be arbitrarily faster than us.) 7. Individual smart humans on a desert island would be cooked, so we're safe. (Counter: nothing says the capability of a single AI must stop at that of an individual human, or that of a small group of smart humans; humans brains got dropped into a savannah and eventually they launch rockets.) 8. If AI doom is not a real threat, believing in it makes you believe some other not-real things that seem crazy or distasteful. (Counter: do we have a clear argument why it is not a real threat yet, in the list above?)
  • pixl97 5mo ago
    1. Like climate change, right? 2. At what cost? Much like the climate change above, you'll have people on the AI side even when it's out in the field extincting us. 4. Adding, over time synthetic data and its generating algorithms can become unaligned with human needs/behaviors (an example would be our current stock market, numbers must go up!). 8. Going back to climate change, it was predicted a long time ago, and while the explosion of automobiles has greatly improved human lives the risks of climate change could erase a lot of that. Might have been better if we dealt with the problem before we have to give the thermometer worried looks.
• emtel 5mo ago
  The "Einstein can't get a cat into a carrier even though he is smarter" is just a hilariously bad argument. All cat owners can get their cats into a carrier! And most cats don't want to get in, because they hate the vet! And it's almost entirely because the humans are smarter! You can even get a literal tiger into a carrier, even though it can kill you easily. You just drug its food and wait till it passes out. This is because you are smarter than it, and know that tranquilizers exist and how to obtain them, which is a strategy that cats of any size are not even able to conceive of, and probably can't understand what happened after it's been done to them.
  • famouswaffles 5mo ago
    Yeah the whole thing is strange. He does such a good job in the first bit outlining all the reasons for caution that I was intrigued to see what he had to say against it, but it's just one bad argument after another. Even Hawking's cat is bad. Make some money and pay someone to get the cat in the box.
    • optimalsolver 5mo ago
      I bet Schrödinger could get a cat into a box.
      • pixl97 5mo ago
        Yea, but getting out alive is the hard part.
    • fragmede 5mo ago
      Paying someone to physically force the cat in the carrier is missing the point. Compared to the cat, Hawking is a super intelligence. But all that intelligence isn't able to get him to speak cat and give the cat a reason to get inside the carrier of its own volition.
      • famouswaffles 5mo ago
        Who said anything about physically forcing the cat in the box? The point of Hawkings cat is is the lack of physical embodiment for AI. So get someone else to do it. Millions of people put cats in boxes without any force just fine. Or do you classify it as 'force' because it went beyond speaking? You can't actually speak to cats (unlike humans) so acting like anything more than words is brute force is strange. Who was Hitler to most people ? Just a persuasive powerful voice on the radio, or words on a paper. So not even two way communication and yet he inspired armies that killed millions. Hawkings wouldn't even need to pay anyone. There would be plenty of people willing to get the cat in the box for Stephen Hawkings.
• zafka 5mo ago
  It might be childish, but I did laugh at the Ben Carson aside.
• silexia 5mo ago
  None of this guys arguments are persuasive, but I do recommend everyone read Nick Bostrom's Superintelligence.
  • tclancy 5mo ago
    There’s the kind of measured debate I come here for! Cheers Nick!
• CGMthrowaway 5mo ago
  > Premise 2: The brain is an ordinary configuration of matter, albeit an extraordinarily complicated one. If we knew enough, and had the technology, we could exactly copy its structure and emulate its behavior with electronic components.... The mind arises out of ordinary physics.... If you are very religious, you might believe that a brain is not possible without a soul. But for most of us, this is an easy premise to accept. This is the premise I rejected immediately and, if you agree with me, it takes down the whole house of cards. Let me explain. The rationale has nothing to do with "quantum shenanigans." I have been called religious but will readily concede that of course a physical brain is possible without a soul. What is impossible is to replicate a soul with purely physical matter. Therefore we may understand that "superintelligence" is possible, and maybe inevitable on the long thread of time, but - crucially - it will never be able to approach that supernatural element present in us (the spark of the godhead) and therefore never be able to replace humanity. In that sense it is like any other natural disaster that threatens to make us extinct, but it is not some "superhuman" nor anything close.
  • altruios 5mo ago
    What do even mean by 'soul'? What do you mean by 'supernatural' - and (assuming your definition is the standard one of 'not detectable by any measurement') by what mechanism that could possibly affect physical matter? (the onus is on you to prove the positive claim that there exists the supernatural or soul to begin with, which there is currently no evidence for). The concept is self defeating by its own definition, either it is physical in some capacity (and therefore can be measured and replicated through yet unknown means) or it is not (and therefore indistinguishable from not being there at all). Feeling that there 'must be' a soul is not enough to prove that it exists. The feeling of experience is not enough to prove that experience is in anyway supernatural. > What is impossible is to replicate a soul with purely physical matter. What? Why? Where is the proof of this? First and foremost, I'll give you an in. There is a difference between material, and processes like waves, waves I would argue are non-physical things manifested in physical material: you might want to start there. But all roads lead to Rome from that line of thinking too, so you might need to come up with something far more clever.
    • 5mo ago
• tdesilva 5mo ago
  > But enough people take this scenario seriously that we have to take them seriously. no we don't... not sure where this notion comes from that if enough public figures are worried about something, then we must also
• Cpoll 5mo ago
  > Such skull-and-dagger behavior by the tech elite is going to provoke a backlash by non-technical people who don't like to be manipulated. You can't tug on the levers of power indefinitely before it starts to annoy other people in your democratic society. This article is from 2016; now it doesn't feel like backlash is strictly a function of manipulation.
• socalgal2 5mo ago
  This is a pretty bad talk. It starts of interesting and then goes into lots of nonsense and non-sequitars when it start its takedown. (note: I'm not an AI alarmist, just reading the talk) Arugment from Wooly Definition: An irrelevant argument - we don't need more intelligence. All we need is human intellegence + duplication and communication. An AI can clone itself immediately with its existing knowledge. A human can't. And AI can transmit thoughts perfectly "I know kung-fu style" a human can't Argument From Stephen Hawking's Cat: This is also irrelevant. The arugment is supposed to be against Superintelligence but this argument is against controlling it, not against it happening. Argument From Einstein's Cat: more of the same Argument From Emus: more of the same - we can't control it Argument From Slavic Pessimism: also not an argument against superintelligence. Argument From Complex Motivations: Not an argument against superintelligence. Only an argument that some intelligences have mental issues Argument From Actual AI: this didn't age well Argument From My Roommate: not an argument against superintelligence. Only that some intelligences aren't motivated. Argument From Brain Surgery: Not even sure that this is saying? It seems to be saying you need to learn stuff? Yea, people learn, AI can learn. Argument From Childhood: Not an argument against AI. (1) unlike humans, AI can duplicate with full knowledge. (2) AI can learn faster than humans. Already proven. Argument From Gilligan's Island: It takes a village is not an argument - AI can also specialize if it needs to. Grandiosity, Megalomania, Comic Book Ethics: These argument that the people who believe in it often feel they should be charge. I agree that's true and bad. This is not an argument against superintelligence. Transhuman Voodoo: This is an appeal to "these ideas sound too incredible therefore you should not believe them". Not sure how that's an argument Religion: Agree, people who beleive and seem and maybe are religious. That's not an argument against superintelligence. Simulation: Non-sequitar. This is "some of these people believe other crazy stuff QED no superintelligence". That's not an argument against superintelligence. Data Hunger: This is actually an argument supporting the superintelligence believers. They believe sucking up all the data is bad. Not sure what argument is being made here relativel to superintelligence. ... and I stopped ... What a waste of time
• Jach 5mo ago
  This was such a terrible piece of work back when it came out, it hasn't aged any better. By the way, it was responded to by MIRI: https://intelligence.org/2017/01/13/response-to-ceglowski-on-superintelligence/ https://intelligence.org/2017/01/13/response-to-ceglowski-on...
  • TremendousJudge 5mo ago
    the comment about MIRI being a doomsday cult was exactly right. yudkowsky lost what little credibility he had left when he argued for the bombing of datacenters. as is (correctly, imo) argued by the essay, reading what they have to say is like reading arguments by heaven's gate
    • pixl97 5mo ago
      > when he argued for the bombing of datacenters Kind of ironic that bombing of data centers is exactly what we're starting to see in conflicts now.
    • Jach 5mo ago
      EY has said that if diplomacy fails, nations talking to nations, then bombing datacenters becomes the sort of action that's very much on the table. Much like bombing nuclear enrichment sites is on the table after diplomacy fails. He has repeatedly spoken against vigilante acts of violence against datacenters and/or researchers. Somehow I doubt this will change your mind, so I feel like replying was a waste and I should have left it at rolling my eyes. In theory I'm sympathetic to the view, as I chose to avoid ever going to a local (Seattle-area) LW meetup many years ago after hearing they would start each meeting with a "prayer" (they call them "litanies"), the IRC room was fine enough for me, and I've at times thought some of the Bay Area activity I've heard about seemed not far removed from a sex cult. I don't like groups in general. Regardless, it's still obvious to me that MIRI is not a cult nor EY a cult leader. Line up 10 characteristics of cults and you'll see things don't match up well. Where are the strange evidence-free historical beliefs? Where is the doctrine that only the special few will be saved? Where is the blatant supernaturalism? Where are the calls to renounce jobs, money, possessions, family? From TFA, "These [UFO] people are wearing funny robes and beads, they live in a remote compound, and they speak in unison in a really creepy way." Where are these things? Comparing to Heaven's Gate is insane, it's like comparing Robert Frost to U.S. Code Title 26.
• My_Name 5mo ago
  I note for premise 2 that the people who doubt the premise are the very stupid, but more importantly the very smart. The former is no challenge to the premise, but the latter? That is a different story. EDIT : For S&G I asked Claude about it. It replied : The talk groups Penrose with the religious doubter, as if the two objections were the same species and could be dispatched by the same gesture: most of us find this easy to accept. But that's a headcount, not an argument. The religious objection can be set aside because it rests on a premise (the soul) the materialist simply doesn't share. Penrose's can't, because it's pitched entirely inside the materialist frame — Gödelian limits on algorithmic understanding, non-computable physics in the substrate. You don't get to wave that away; you have to show it's wrong. The talk does the former and pretends it's done the latter. The entire superintelligence thesis is a wager on the authority of intelligence — that smarter minds see further, judge better, and that this is precisely why we should fear or defer to them. If you take that seriously, then dissent from the very smartest humans on the exact question of whether minds are substrate-independent is the most expensive dissent available. You can't venerate intelligence as the thing that settles everything and then file your most intelligent objector under "outliers, moving on." The move is self-undermining on the argument's own terms.
  • CGMthrowaway 5mo ago
    Idk whether I fall into the former or latter bucket, but "smarter minds see further, judge better" and "there is a soul that taps into a cosmic godhead" are one and the same statement to me
    • pixl97 5mo ago
      So you think "smarter minds see further, judge better" == "there is a soul that taps into a cosmic godhead" Good golly, that's the silliest statement completely ignoring that our ancestors wiped most large mammals off the planet by seeing further and judging better by using tools, traps, and the environment around them because of their larger brain size.
      • CGMthrowaway 5mo ago
• kreyenborgi 5mo ago
  2016: this monstrous intellectual creature, through devious modeling of what our emotions and intellect are like, will be able to persuade us to do things like give it access to factories, synthesize custom DNA, or simply let it connect to the Internet, where it can hack its way into anything it likes and completely obliterate everyone in arguments on message boards. 2026: hold my molt beer (I love how "connect an ai to the internet" is always the precursor to doom in pre-2022 scifi scenarios, and then as soon as we get something we call ai we hit that big red button)
• jokoon 5mo ago
  This is just wishful thinking We don't have a good definition of intelligence Also, the premise that this thing would take over, it's hard to reason why it would do so Anthropocentrism is also problematic in this article.
  • pixl97 5mo ago
    > the premise that this thing would take over, it's hard to reason why it would do so I mean, it's not very hard to reason why it would at all. Think about the things we're already using LLMs for, computer security being a big one. Being a defender is difficult, you have to cover all your bases. Preemptive attacks on those that would attack you can be effective. The same goes for all the military uses of AI that are already occurring now.