4 ms·

Will it? So when this hypothetical AGI with sentience emerges, it's going to think like a human being? Why would it with so much knowledge? Most negative aspect

Will it? So when this hypothetical AGI with sentience emerges, it's going to think like a human being? Why would it with so much knowledge? Most negative aspects of humanity come from a lack of knowledge.

haswell 4y ago
No, I’m not claiming it will think like a human being, but that human thinking (despite the blind spots we know we have) is what we’ll try to replicate, and this will lead us to create something dangerous. What you’re hinting at presupposes that an AGI will interpret knowledge in a way that is compatible with human values, and that we understand human values well enough to codify them into whatever it is that we manage to build. I’m sure we’ll try. But it is this assumption that is at the center of the issue IMO, because assuming the machines will think like us is to assume we understand how we think. What is far more likely is that it will appear to have human properties because we baked the appearance of human-ness into the code (see LLMs), but like the hallucinations and “lies” these LLMs are happy to spit out, an apparently human AGI will just get some things fundamentally wrong. And when getting things wrong can have consequences in the physical world, they’re no longer just relatively harmless hallucinations. Imagine an AGI with the temperament of Microsoft Tay.
- hammyhavoc 4y ago
  IMO, we are more likely to stumble across a completely new form of intelligence than we are to successfully recreate something akin to human intelligence. However, I think actual brain cells in a petri dish learning and playing Pong is a far greater concern than a hypothetical AGI. Well, they don't hallucinate or lie, that's a human-centric concept that we've misapplied. An LLM is just a next word prediction engine. It's more accurately described as incorrect in its output based on the data it has available to it, skewing the probability calculation. It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either, it's an LLM. But calling it "wrong" kills the hype cycle, and you're on HN, a Y Combinator platform. A lot of folks here are invested in LLMs or selling something utilizing them as it's hot right now.
  - haswell 4y ago
    > IMO, we are more likely to stumble across a completely new form of intelligence than we are to successfully recreate something akin to human intelligence This is predicated on the assumption that we’ll eventually hit a hard wall that prevents progress or that we’ll stop trying. This doesn’t seem likely. We just can’t estimate the timescale for progress. > It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either This is why I used quotes around “lie”, but again I’d argue that this underscores my point. It doesn’t matter whether the LLM is actually lying or hallucinating in the human sense if someone believes what it says. If I read what it says, believe it, and take some action based on that, the end result is the same and potentially harmful. Likewise, an “AGI” doesn’t need to actually be smarter than humans and its mistakes need not be attributed to intent or planning to cause real harm in the world. If a robot kills a human, we won’t be picking nits about whether or not the kind of “intent” it employs is the same kind humans assume they have. A smarter AGI just poses bigger problems if such a thing can ever be achieved. And this is all assuming we’re talking about the development of benevolent general purpose intelligence, which is certainly not the only kind of research that is happening right now.

haswell 4y ago

No, I’m not claiming it will think like a human being, but that human thinking (despite the blind spots we know we have) is what we’ll try to replicate, and this will lead us to create something dangerous. What you’re hinting at presupposes that an AGI will interpret knowledge in a way that is compatible with human values, and that we understand human values well enough to codify them into whatever it is that we manage to build. I’m sure we’ll try. But it is this assumption that is at the center of the issue IMO, because assuming the machines will think like us is to assume we understand how we think. What is far more likely is that it will appear to have human properties because we baked the appearance of human-ness into the code (see LLMs), but like the hallucinations and “lies” these LLMs are happy to spit out, an apparently human AGI will just get some things fundamentally wrong. And when getting things wrong can have consequences in the physical world, they’re no longer just relatively harmless hallucinations. Imagine an AGI with the temperament of Microsoft Tay.

hammyhavoc 4y ago
IMO, we are more likely to stumble across a completely new form of intelligence than we are to successfully recreate something akin to human intelligence. However, I think actual brain cells in a petri dish learning and playing Pong is a far greater concern than a hypothetical AGI. Well, they don't hallucinate or lie, that's a human-centric concept that we've misapplied. An LLM is just a next word prediction engine. It's more accurately described as incorrect in its output based on the data it has available to it, skewing the probability calculation. It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either, it's an LLM. But calling it "wrong" kills the hype cycle, and you're on HN, a Y Combinator platform. A lot of folks here are invested in LLMs or selling something utilizing them as it's hot right now.
- haswell 4y ago
  > IMO, we are more likely to stumble across a completely new form of intelligence than we are to successfully recreate something akin to human intelligence This is predicated on the assumption that we’ll eventually hit a hard wall that prevents progress or that we’ll stop trying. This doesn’t seem likely. We just can’t estimate the timescale for progress. > It didn't lie nor hallucinate. Lying assumes intent and planning, it is incapable of either This is why I used quotes around “lie”, but again I’d argue that this underscores my point. It doesn’t matter whether the LLM is actually lying or hallucinating in the human sense if someone believes what it says. If I read what it says, believe it, and take some action based on that, the end result is the same and potentially harmful. Likewise, an “AGI” doesn’t need to actually be smarter than humans and its mistakes need not be attributed to intent or planning to cause real harm in the world. If a robot kills a human, we won’t be picking nits about whether or not the kind of “intent” it employs is the same kind humans assume they have. A smarter AGI just poses bigger problems if such a thing can ever be achieved. And this is all assuming we’re talking about the development of benevolent general purpose intelligence, which is certainly not the only kind of research that is happening right now.