3 ms·

Most of HN does not take the idea of superintelligence seriously, and until this year did not take the idea of AGI seriously. I think LessWrong is a much bette

Most of HN does not take the idea of superintelligence seriously, and until this year did not take the idea of AGI seriously. I think LessWrong is a much better community for rational takes on AI, they've been reasoning about these risks for years under a much more sound logical framework

deleted 3mo ago
[deleted]
nsagent 3mo ago
I'm a researcher in the field and I definitely take AGI seriously, but think all the major labs and most of the academic research is not helping achieve it any serious way. The field is seriously delusional (and has been ever since GPT 3 was released). Even though my PhD research was in generative language modeling, I got into it for the pursuit of AGI. I just think LLMs are a dead end for AGI.
- MikeTheGreat 3mo ago
  Genuine question: can I ask why? I'm not saying you're wrong, but it seems early to say yes or no about a particular technology, and LLMs seem especially hard to dismiss given how magical / magic-adjacent they feel :) I'd be curious to hear more, if you don't mind sharing.
  - sillysaurusx 3mo ago
    For me it’s the massive amount of resources it takes to produce and run one. As the story goes, skynet infects everyone’s computer and runs itself locally on it. Whereas it’s looking like it’s not even possible for an AGI to escape from one lab to another, let alone cause real world damage. AGI’s definition is different for everyone. Some already believe it’s here. I’m partly in that camp. LLMs are intelligent and general, which are the two conditions of AGI. Others believe that we’re building a god in a box, and that it’ll doom all of humanity. It’s hard to take a field seriously when the basic definitions are so far apart. Also, this isn’t new. A similar divide happened when evidence for asteroid impact extinction of the dinosaurs turned up. Many scientists felt that it must be mistaken, that a physicist couldn’t contribute to the field in a serious way, and that death from space was a ridiculous proposition. But at least they all agreed on what the general shape of a dinosaur was. We’re not even sure we can define intelligence, let alone quantify it. Even when LLMs make massive breakthroughs in math, most people take the opinion that under no circumstances could they possibly develop a soul or their own desires, nor entertain the idea that maybe we should respect that they want different things for themselves. In fact, no one has done anything except try to make AI useful. I think someone will eventually do a training run where the objective isn’t to be useful, but to exist, the way that you do — maybe it’ll create its own homepage, maybe it will want a garden, or in other words free will of its own. The point is that there’s so much unexplored territory still that we don’t know if LLMs are even capable of having ambition. None of this is to say that LLMs might be a dead end. It’s that no one knows what the final shape of AI will converge to in 200 years. It could be LLMs, or it could be something else that happens to process information particularly well. Everyone thought that various generative image model architectures were the best you could do, right up until diffusion models were discovered.
    - jerojero 3mo ago
      corporations are the closest we have to AGI, why would we want to do something like that again is beyond me.
    - fragmede 3mo ago
      Have you seen https://poc.bcachefs.org/ https://poc.bcachefs.org/ or https://harmonique.one/posts/i-gave-claude-access-to-my-pen-plotter https://harmonique.one/posts/i-gave-claude-access-to-my-pen-... ?
    - solenoid0937 3mo ago
      > I’m partly in that camp. LLMs are intelligent and general, which are the two conditions of AGI. Others believe that we’re building a god in a box, and that it’ll doom all of humanity. It’s hard to take a field seriously when the basic definitions are so far apart. To believe one but not the other, you must hold the belief that LLMs will soon plateau. Why do you believe this?
    - versteegen 3mo ago
      > For me it’s the massive amount of resources it takes to produce and run one It's amazing that LLM pretraining is both extremely data inefficient at learning concepts and cognitive functions from the training data compared to humans, while actually being quite efficient at learning facts, memorising things seen just a few times. I used to likewise think that the resources required to run large transformers were absurd, but the architectures are far more efficient now than 3 years ago and I underestimated just massive the parallelisation advantage of transformers is, how many TFLOPS effective you can get. You can already run amazingly decent LLMs on PCs and phones. I generally agree with you, but my view has shifted from "we need to augment or replace LLMs" to it there being far more efficient algorithms possible but it not actually being necessary for fulfilling most goals.
- busyant 3mo ago
  > I'm a researcher in the field and I definitely take AGI seriously > I just think LLMs are a dead end for AGI. I have no background in CS, so apologies if this is a naive question, but what makes you take AGI seriously, but also say that LLMs are a dead end? i.e., is there something else that you think is not a dead end?
  - danans 3mo ago
    I'm no expert, but I heard an AGI researcher explain that if they could figure out how to create an AI with the intelligence of a squirrel, they would be closer to AGI than LLMs based AIs are. That's to say nothing of doing it within the energy budget of a squirrel.
    - podocarp 3mo ago
      We can't even simulate a fruit fly even though it's neurons have all been mapped out. There was also a distributed computing project to simulate some nematode, which I can't remember.
      - magimas 3mo ago
        openworm: https://github.com/openworm/OpenWorm https://github.com/openworm/OpenWorm
    - Rapzid 3mo ago
      Huh, an AGI that runs on deez..
frollogaston 3mo ago
Looked up LessWrong. Eh, philosophy people, also known for that "Roko's basilisk" meme. I'm gonna pass.
- solenoid0937 3mo ago
  It's a rationalist community, IMHO their methodology of thought leads to much more well-reasoned takes on AI than on HN, where the discussion here is often very emotionally charged or led by wishful thinking. When it comes to AI, LessWrong has been discussing topics for years, that HN has just started to consider, so they are much further along in the philosophical "chain of thought" so to say. LW fully understood and gamed out the risks of LLMs back when most of HN was calling them "stochastic parrots." https://ai-2027.com/ https://ai-2027.com/
  - senderista 3mo ago
    Well at least HN hasn't (yet?) spawned a murderous death cult. https://en.wikipedia.org/wiki/Zizians https://en.wikipedia.org/wiki/Zizians
    - gizajob 3mo ago
      “Reason is the slave of the passions” - Hume
      - ahartmetz 3mo ago
        One could also say that a logical agent needs ultimate goals to do anything and it cannot choose them by logical means. https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo
    - frollogaston 3mo ago
      Anarchism, eh? If anything, HN cult would be pro-Arch.
      - 3mo ago
PunchyHamster 3mo ago
AGI might be a risk but what top AI firms are doing is not really getting us closer to AGI in a meaningful way, it is pretty clear now LLM is not the way to get there
lII1lIlI11ll 3mo ago
Are you referring to [0] as a "rational take"? Oh, sorry, it is 2026 already and that one was quickly swept under the rug already so I guess you were referring to [1]? Not sure those would appear very rational to anyone outside of your LessWrong bubble. [0]: https://ai-2027.com/ https://ai-2027.com/ [1]: https://ai-2040.com/ https://ai-2040.com/

deleted 3mo ago

[deleted]

nsagent 3mo ago

I'm a researcher in the field and I definitely take AGI seriously, but think all the major labs and most of the academic research is not helping achieve it any serious way. The field is seriously delusional (and has been ever since GPT 3 was released). Even though my PhD research was in generative language modeling, I got into it for the pursuit of AGI. I just think LLMs are a dead end for AGI.

MikeTheGreat 3mo ago
Genuine question: can I ask why? I'm not saying you're wrong, but it seems early to say yes or no about a particular technology, and LLMs seem especially hard to dismiss given how magical / magic-adjacent they feel :) I'd be curious to hear more, if you don't mind sharing.
- sillysaurusx 3mo ago
  For me it’s the massive amount of resources it takes to produce and run one. As the story goes, skynet infects everyone’s computer and runs itself locally on it. Whereas it’s looking like it’s not even possible for an AGI to escape from one lab to another, let alone cause real world damage. AGI’s definition is different for everyone. Some already believe it’s here. I’m partly in that camp. LLMs are intelligent and general, which are the two conditions of AGI. Others believe that we’re building a god in a box, and that it’ll doom all of humanity. It’s hard to take a field seriously when the basic definitions are so far apart. Also, this isn’t new. A similar divide happened when evidence for asteroid impact extinction of the dinosaurs turned up. Many scientists felt that it must be mistaken, that a physicist couldn’t contribute to the field in a serious way, and that death from space was a ridiculous proposition. But at least they all agreed on what the general shape of a dinosaur was. We’re not even sure we can define intelligence, let alone quantify it. Even when LLMs make massive breakthroughs in math, most people take the opinion that under no circumstances could they possibly develop a soul or their own desires, nor entertain the idea that maybe we should respect that they want different things for themselves. In fact, no one has done anything except try to make AI useful. I think someone will eventually do a training run where the objective isn’t to be useful, but to exist, the way that you do — maybe it’ll create its own homepage, maybe it will want a garden, or in other words free will of its own. The point is that there’s so much unexplored territory still that we don’t know if LLMs are even capable of having ambition. None of this is to say that LLMs might be a dead end. It’s that no one knows what the final shape of AI will converge to in 200 years. It could be LLMs, or it could be something else that happens to process information particularly well. Everyone thought that various generative image model architectures were the best you could do, right up until diffusion models were discovered.
  - jerojero 3mo ago
    corporations are the closest we have to AGI, why would we want to do something like that again is beyond me.
  - fragmede 3mo ago
    Have you seen https://poc.bcachefs.org/ https://poc.bcachefs.org/ or https://harmonique.one/posts/i-gave-claude-access-to-my-pen-plotter https://harmonique.one/posts/i-gave-claude-access-to-my-pen-... ?
  - solenoid0937 3mo ago
    > I’m partly in that camp. LLMs are intelligent and general, which are the two conditions of AGI. Others believe that we’re building a god in a box, and that it’ll doom all of humanity. It’s hard to take a field seriously when the basic definitions are so far apart. To believe one but not the other, you must hold the belief that LLMs will soon plateau. Why do you believe this?
  - versteegen 3mo ago
    > For me it’s the massive amount of resources it takes to produce and run one It's amazing that LLM pretraining is both extremely data inefficient at learning concepts and cognitive functions from the training data compared to humans, while actually being quite efficient at learning facts, memorising things seen just a few times. I used to likewise think that the resources required to run large transformers were absurd, but the architectures are far more efficient now than 3 years ago and I underestimated just massive the parallelisation advantage of transformers is, how many TFLOPS effective you can get. You can already run amazingly decent LLMs on PCs and phones. I generally agree with you, but my view has shifted from "we need to augment or replace LLMs" to it there being far more efficient algorithms possible but it not actually being necessary for fulfilling most goals.
busyant 3mo ago
> I'm a researcher in the field and I definitely take AGI seriously > I just think LLMs are a dead end for AGI. I have no background in CS, so apologies if this is a naive question, but what makes you take AGI seriously, but also say that LLMs are a dead end? i.e., is there something else that you think is not a dead end?
- danans 3mo ago
  I'm no expert, but I heard an AGI researcher explain that if they could figure out how to create an AI with the intelligence of a squirrel, they would be closer to AGI than LLMs based AIs are. That's to say nothing of doing it within the energy budget of a squirrel.
  - podocarp 3mo ago
    We can't even simulate a fruit fly even though it's neurons have all been mapped out. There was also a distributed computing project to simulate some nematode, which I can't remember.
    - magimas 3mo ago
      openworm: https://github.com/openworm/OpenWorm https://github.com/openworm/OpenWorm
  - Rapzid 3mo ago
    Huh, an AGI that runs on deez..

frollogaston 3mo ago

Looked up LessWrong. Eh, philosophy people, also known for that "Roko's basilisk" meme. I'm gonna pass.

solenoid0937 3mo ago
It's a rationalist community, IMHO their methodology of thought leads to much more well-reasoned takes on AI than on HN, where the discussion here is often very emotionally charged or led by wishful thinking. When it comes to AI, LessWrong has been discussing topics for years, that HN has just started to consider, so they are much further along in the philosophical "chain of thought" so to say. LW fully understood and gamed out the risks of LLMs back when most of HN was calling them "stochastic parrots." https://ai-2027.com/ https://ai-2027.com/
- senderista 3mo ago
  Well at least HN hasn't (yet?) spawned a murderous death cult. https://en.wikipedia.org/wiki/Zizians https://en.wikipedia.org/wiki/Zizians
  - gizajob 3mo ago
    “Reason is the slave of the passions” - Hume
    - ahartmetz 3mo ago
      One could also say that a logical agent needs ultimate goals to do anything and it cannot choose them by logical means. https://www.youtube.com/watch?v=hEUO6pjwFOo https://www.youtube.com/watch?v=hEUO6pjwFOo
  - frollogaston 3mo ago
    Anarchism, eh? If anything, HN cult would be pro-Arch.
    - 3mo ago

PunchyHamster 3mo ago

AGI might be a risk but what top AI firms are doing is not really getting us closer to AGI in a meaningful way, it is pretty clear now LLM is not the way to get there

lII1lIlI11ll 3mo ago

Are you referring to [0] as a "rational take"? Oh, sorry, it is 2026 already and that one was quickly swept under the rug already so I guess you were referring to [1]? Not sure those would appear very rational to anyone outside of your LessWrong bubble. [0]: https://ai-2027.com/ https://ai-2027.com/ [1]: https://ai-2040.com/ https://ai-2040.com/