3 ms·

I think it is bad writing on the part of the author. Or maybe good writing for getting us engaged with the blog post, bad for making an argument though. They i

I think it is bad writing on the part of the author. Or maybe good writing for getting us engaged with the blog post, bad for making an argument though. They include a line that they don’t believe in the possibility of AGI: > I don’t really believe in the threat of AGI (Artificial General Intelligence—human-level intelligence) partly because I don’t believe in the possibility of AGI and I’m highly skeptical that the current technology underpinning LLMs will provide a route to it. This is a basically absurd position to hold, I mean humans physically exist so our brains must be possible to build within the existing laws of physics. It is obviously far beyond our capabilities to replicate a human brain (except via the traditional approach), but unless brains hold irreproducible magic spirits (well we must at least admit the possibility) they should be possible to build artificially. Fortunately they immediately throw that all away anyway. Next, they get to the: > and I’m highly skeptical that the current technology underpinning LLMs will provide a route to it. Which is, of course, at least a plausible thing to believe. I mean there are a bunch of philosophical questions about what “intelligence” even means so there’s plenty of room to quibble here. Then we have, > But I also think there’s something we should actually be afraid of long before AGI, if it ever comes. […] > Now, if you equip humans with a hammer, or sword, or rifle, or AI then you’ve just made the scariest monster in the woods (that’s you) even more terrifying. […] > We don’t need to worry about AI itself, we need to be concerned about what “humans + AI” will do. Which is like, yeah, this is a massively worrying problem that doesn’t involve any sci-fi bullshit, and I think it is what most(?) anybody who’s thought about this seriously at all (or even stupid people who haven’t, like myself). Artificial Sub-intelligences, things that are just smart enough to make trouble and too dumb or too “aligned” to their owner (instead of society in general) to push back are a big currently happening problem.

andy99 1y ago
> humans physically exist so our brains must be possible to build within the existing laws of physics This is an unscientific position to take. We have no idea how our brains work, or how life, consciousness, and intelligence work. It could very well be that’s because our model of the world doesn’t account for these things and they are not in fact possible based on what we know. In fact I think this is likely. So it really could be that AI is not possible, for example on a Turing machine or our approximation of them. This is at least as likely as it being possible. At some point we’ll hopefully refine our theories to have a better understanding, for now we have no idea and I think it’s useful to acknowledge this.
- zeroonetwothree 1y ago
  Turing machines have been universal as far as we have found. So while I acknowledge it’s possible, I would definitely not say it’s more likely that brains cannot be simulated by TMs. I would personally weight this as under 10%. Of course it doesn’t speak to how challenging it will be to actually do that. And I don’t believe that LLMs are sufficient to reach AGI.
- bee_rider 1y ago
  I think my main mistake, which I agree is a legitimate mistake, was to write “the existing laws of physics.” It is definitely possible that our current understanding of the laws of physics is insufficient to build a brain. Of course the actual underlying laws of the universe that we’re trying (unsuccessfully so far, it is a never ending process) to describe admit the existence of brains. But that is not what I said. Sorry for the error.
- gpderetta 1y ago
  >> humans physically exist so our brains must be possible to build within the existing laws of physics > This is an unscientific position to take The universe being constrained observable and understandable, natural laws is pretty much a fundamental axiom of the scientific method.
kayodelycaon 1y ago
I don’t think we’ll be able to replicate consciousness until we’re able to make things alive at a biological level. We can certainly make systems smart enough and people complicit enough to destroy society well before we reach that point.
- forgotoldacc 1y ago
  I guess we also need to define what biological life means. Even biologists have debated whether viruses should be considered life. And if we determine it must be something with cells that can sustain themselves, we run into a challenge should we encounter extraterrestrials that don't share our evolutionary path. When we get self-building machines that can repair themselves, move, analyze situations, and respond accordingly, I don't think it's unfair to consider them life. But simply being life doesn't mean it's inherently good. Humans see syphilis bacteria and ticks as living things, but we don't respect them. We acknowledge that polar bears have a consciousness, but they're at odds with our existence if we're put in the same room. If we have autonomous machines that can destroy humans, I think those could be considered life. But it's life that opposes our own.
- myrmidon 1y ago
  > I don’t think we’ll be able to replicate consciousness until we’re able to make things alive at a biological level. But why? What gives you any confidence in that? This is a very popular notion that I frequently encounter but I'm convinced that it is just barely disguised human exceptionalism. It is humbling to accept that the operation of your mind could be replicated by a machine, similar to how it was difficult for us to accept that the earth is not the center of the universe or that we evolved from animals.

andy99 1y ago

> humans physically exist so our brains must be possible to build within the existing laws of physics This is an unscientific position to take. We have no idea how our brains work, or how life, consciousness, and intelligence work. It could very well be that’s because our model of the world doesn’t account for these things and they are not in fact possible based on what we know. In fact I think this is likely. So it really could be that AI is not possible, for example on a Turing machine or our approximation of them. This is at least as likely as it being possible. At some point we’ll hopefully refine our theories to have a better understanding, for now we have no idea and I think it’s useful to acknowledge this.

zeroonetwothree 1y ago
Turing machines have been universal as far as we have found. So while I acknowledge it’s possible, I would definitely not say it’s more likely that brains cannot be simulated by TMs. I would personally weight this as under 10%. Of course it doesn’t speak to how challenging it will be to actually do that. And I don’t believe that LLMs are sufficient to reach AGI.
bee_rider 1y ago
I think my main mistake, which I agree is a legitimate mistake, was to write “the existing laws of physics.” It is definitely possible that our current understanding of the laws of physics is insufficient to build a brain. Of course the actual underlying laws of the universe that we’re trying (unsuccessfully so far, it is a never ending process) to describe admit the existence of brains. But that is not what I said. Sorry for the error.
gpderetta 1y ago
>> humans physically exist so our brains must be possible to build within the existing laws of physics > This is an unscientific position to take The universe being constrained observable and understandable, natural laws is pretty much a fundamental axiom of the scientific method.

kayodelycaon 1y ago

I don’t think we’ll be able to replicate consciousness until we’re able to make things alive at a biological level. We can certainly make systems smart enough and people complicit enough to destroy society well before we reach that point.

forgotoldacc 1y ago
I guess we also need to define what biological life means. Even biologists have debated whether viruses should be considered life. And if we determine it must be something with cells that can sustain themselves, we run into a challenge should we encounter extraterrestrials that don't share our evolutionary path. When we get self-building machines that can repair themselves, move, analyze situations, and respond accordingly, I don't think it's unfair to consider them life. But simply being life doesn't mean it's inherently good. Humans see syphilis bacteria and ticks as living things, but we don't respect them. We acknowledge that polar bears have a consciousness, but they're at odds with our existence if we're put in the same room. If we have autonomous machines that can destroy humans, I think those could be considered life. But it's life that opposes our own.
myrmidon 1y ago
> I don’t think we’ll be able to replicate consciousness until we’re able to make things alive at a biological level. But why? What gives you any confidence in that? This is a very popular notion that I frequently encounter but I'm convinced that it is just barely disguised human exceptionalism. It is humbling to accept that the operation of your mind could be replicated by a machine, similar to how it was difficult for us to accept that the earth is not the center of the universe or that we evolved from animals.