5 ms·

Despite what Sam Altman (a high-school graduate) might want to be true, human cognition is not just a massive pile of intuition; there are critical deliberative

by bbor 2y ago

Despite what Sam Altman (a high-school graduate) might want to be true, human cognition is not just a massive pile of intuition; there are critical deliberative and intentional aspects to cognition, which is something we've seen come to the fore with the hubbub around "reasoning" in LLMs. Any AGI design will necessarily take these facts into account--hardcoded or no--and will absolutely be capable of forming plans and executing them over time, as Simon & Newell described the best back in '71: The problem solver’s search for a solution is an odyssey through the problem space, from one knowledge state to another, until… [they] know the answer. With this in mind, I really don't see any basis to attack the intelligence explosion hypothesis. I linked a Yudkowsky paper above examining how empirically feasible it might be, which is absolutely an unsolved question at some level. But the utility of the effort itself is just downright obvious, even if we didn't have reams of internet discussions like this one to nudge any nascent agent in that direction. [1] Simon & Newell, 1971: Human Problem Solving https://psycnet.apa.org/record/1971-24266-001 https://psycnet.apa.org/record/1971-24266-001

marcus0x62 2y ago
> Despite what Sam Altman (a high-school graduate) might want to be true > I linked a Yudkowsky paper above examining how empirically feasible it might be ...
- bbor 2y ago
  Lol I was wondering if anyone would comment on that! To be fair Yudkowsky is a self-taught scholar, AFAIK Altman has never even half-heartedly attempted to engage with any academy, much less 5 at once. I'm not a huge fan of Yudkowsky's overall impact, but I think it's hard to say he's not serious about science.
  - nradov 2y ago
    Yudkowsky is not serious about science. His claims about AI risks are unscientific and rely on huge leaps of faith; they are more akin to philosophy or religion than any real science. You could replace "AI" with "space aliens" in his writings and they would make about as much sense.
    - concordDance 2y ago
      His argument is of the form "if we get a Thing(s) with these properties you most likely get these outcomes for these reasons". He avoids over and over again making specific timeline claims or stating how likely an extrapolation of current systems could become a Thing with those proporties. Each individual bit of the puzzle (such as the orthogonaly thesis or human value complexity and category decoherence at high power) seems sound, problem is the entire argument-counterargument tree is hundreds of thousands of words, scattered about in many places.
      - greentxt 2y ago
        "problem is the entire argument-counterargument tree is hundreds of thousands of words, scattered about in many places" An llm could solve that.
    - gjm11 2y ago
      If we encountered space aliens, I think it would in fact be reasonable to worry that they might behave in ways catastrophic for the interests of humanity. (And also to hope that they might bring huge benefits.) So "Yudkowsky's arguments for being worried about AI would also be arguments for being worried about space aliens" doesn't seem to me like much of a counter to those arguments. If the point isn't that he's wrong about what the consequences of AI might be, but that he's wrong about whether there's ever going to be such a thing as AI, well, that's an empirical question and it seems like the developments of the last few years are pretty good evidence that (1) something at least very AI-like is possible and (2) substantially superhuman[1] AI is at least plausible. [1] Yes, intelligence is a complicated thing and not one-dimensional; a machine might be smarter than a human in one way and stupider in another (and of course that's already the case). By substantially superhuman, here, I mean something like "better than 90th-percentile humans at all things that could in principle be done by a human in a locked room with only a textual connection to the rest of the world". Though I would be very very surprised if in the next 1-20 years we do get AI systems that are superhuman in this sense and don't put some of them into robots, and very surprised if doing that doesn't produce systems that are also better than humans at most of the things that are done by humans with bodies.
dartos 2y ago
> Sam Altman (a high-school graduate) “People who didn’t pass a test aren’t worth listening to” I have no love for Altman, but this is kind of elitism is insulting.
- vixen99 2y ago
  More tellingly it betokens a lack of critical thought. It's just silly.
- bbor 2y ago
  Hmm, don't want to be elitist. More like "people who don't put any time into studying science shouldn't be listened to about science".
  - dartos 2y ago
    > people who don't put any time into studying Degrees don’t mean that either. I’ve been studying textbooks and papers on real time rendering techniques for the past 4 or so years. I think one could learn something from listening to me explain rasterization or raytracing. I have no degree in math or graphic computing.

by bbor 2y ago

marcus0x62 2y ago

> Despite what Sam Altman (a high-school graduate) might want to be true > I linked a Yudkowsky paper above examining how empirically feasible it might be ...

bbor 2y ago
Lol I was wondering if anyone would comment on that! To be fair Yudkowsky is a self-taught scholar, AFAIK Altman has never even half-heartedly attempted to engage with any academy, much less 5 at once. I'm not a huge fan of Yudkowsky's overall impact, but I think it's hard to say he's not serious about science.
- nradov 2y ago
  Yudkowsky is not serious about science. His claims about AI risks are unscientific and rely on huge leaps of faith; they are more akin to philosophy or religion than any real science. You could replace "AI" with "space aliens" in his writings and they would make about as much sense.
  - concordDance 2y ago
    His argument is of the form "if we get a Thing(s) with these properties you most likely get these outcomes for these reasons". He avoids over and over again making specific timeline claims or stating how likely an extrapolation of current systems could become a Thing with those proporties. Each individual bit of the puzzle (such as the orthogonaly thesis or human value complexity and category decoherence at high power) seems sound, problem is the entire argument-counterargument tree is hundreds of thousands of words, scattered about in many places.
    - greentxt 2y ago
      "problem is the entire argument-counterargument tree is hundreds of thousands of words, scattered about in many places" An llm could solve that.
  - gjm11 2y ago
    If we encountered space aliens, I think it would in fact be reasonable to worry that they might behave in ways catastrophic for the interests of humanity. (And also to hope that they might bring huge benefits.) So "Yudkowsky's arguments for being worried about AI would also be arguments for being worried about space aliens" doesn't seem to me like much of a counter to those arguments. If the point isn't that he's wrong about what the consequences of AI might be, but that he's wrong about whether there's ever going to be such a thing as AI, well, that's an empirical question and it seems like the developments of the last few years are pretty good evidence that (1) something at least very AI-like is possible and (2) substantially superhuman[1] AI is at least plausible. [1] Yes, intelligence is a complicated thing and not one-dimensional; a machine might be smarter than a human in one way and stupider in another (and of course that's already the case). By substantially superhuman, here, I mean something like "better than 90th-percentile humans at all things that could in principle be done by a human in a locked room with only a textual connection to the rest of the world". Though I would be very very surprised if in the next 1-20 years we do get AI systems that are superhuman in this sense and don't put some of them into robots, and very surprised if doing that doesn't produce systems that are also better than humans at most of the things that are done by humans with bodies.

dartos 2y ago

> Sam Altman (a high-school graduate) “People who didn’t pass a test aren’t worth listening to” I have no love for Altman, but this is kind of elitism is insulting.

vixen99 2y ago
More tellingly it betokens a lack of critical thought. It's just silly.
bbor 2y ago
Hmm, don't want to be elitist. More like "people who don't put any time into studying science shouldn't be listened to about science".
- dartos 2y ago
  > people who don't put any time into studying Degrees don’t mean that either. I’ve been studying textbooks and papers on real time rendering techniques for the past 4 or so years. I think one could learn something from listening to me explain rasterization or raytracing. I have no degree in math or graphic computing.