5 ms·

He’s an excellent writer and has spent a long time trying to come up with high level easily understandable ways to try to measure alignment abstractly through p

He’s an excellent writer and has spent a long time trying to come up with high level easily understandable ways to try to measure alignment abstractly through pure logic/outside of any given system. Essentially him and his crowd have been trying to create a rationally grounded purely logical non contradictory moral framework for action from axiomatic first principles (he’s not the first or the last to do this), and they believe such a system is essential if AGI comes to be. I personally think AGI is impossible, his purely logical moral system is impossible, and think what he’s working on is not applicable to current AI (I think he knows this, but his conclusion seems to be that the path to AGI is out of the bag/was made incredibly foolishly and should have been constructed differently with more ability to add guardrails) But I do think the high level abstract way he thinks is fruitful/very valuable, and a good moderating counterbalance to the frenzied “harder better faster stronger” ML stampede that’s not really thinking about the ramifications of what they’re doing with any sort of discipline.

adastra22 4y ago
> I personally think AGI is impossible That’s a very weird statement to make. Do you think you have a soul or something? Otherwise how is your own Intelligence something a computer can’t replicate?
- sebastianz 4y ago
  A computer could indeed replicate our intelligence, but our intelligence might not be sufficient to write the AGI sourcecode :)
- drdeca 4y ago
  I do think I have a soul, and I don’t think that AGI is impossible. So, I really don’t get why people think AGI is impossible. I mean, I have hope that it won’t ever happen, but not because I suspect it is impossible.
  - didericis 4y ago
    I think it’s impossible because I don’t think you can leapfrog evolution, and there’s almost definitely hidden context essential to our intelligence and perception that you can only get through a very very long history. However I do think it’s quite possible and likely that we’ll create mimics that convince most people they’re intelligent. But I think that’s a weird type of reflection/anything we can make is limited to mirroring our observable past and existing collective thought and can never be truly introspective, because true introspection requires evolutionary context and hidden motivation you can’t transfer.
    - swagmoney1606 4y ago
      Evolution is a process that works without an intelligent steward. In a way it's a brute force technique. Plus, nothing is optimizing for intelligence in evolution, it is merely a happy accident that humans ended up with the brains that we did. A different environment could yield drastically different evolutionary history It doesn't seem very logical to think that because evolution took so long to get us to where we are now, we consequently won't be able to design an intelligent AGI system.
      - didericis 4y ago
        It’d take a while to justify this argument to the extent I think it’s justified, but I think we’re in an inescapable frame set by evolution, and our adaptation to our environment probably goes a lot deeper than what we can see. I don’t think the visible context is the full context, and think true intelligence probably requires an implicit understanding of context which is invisible to us.
- didericis 4y ago
  I don’t think true intelligence is propositional/logic based. I also think the observable universe is inherently limited by our perceptual framework, and there will always be something outside it. Imagine an amoeba that can only detect light and dark. Its observable universe exists on a simple gradient. But the cells that are used in its perceptual system cannot be described on a gradient. I think that probably scales up indefinitely. We confuse what we see for all that exists because of our recent history and scientific advances. The triumph of science comes from focusing solely on what we can see and reason about and improving our vision because thats what we have control over, but it doesn’t mean we see everything. I think true intelligence originates from something we can’t see. You can call it a soul, or call it a scaled up amoeba cell, but I personally think the origin of true intelligence and qualia is from some weird very very old evolutionarily created thing we can’t see (and I think other animals have similar invisible perceptual machinery somewhere, I don’t think it’s just a human thing).
  - deleted 4y ago
    [deleted]
  - breuleux 4y ago
    Current AI as exemplified by LLMs like ChatGPT is not propositional/logic based, though, so I'm unsure what your objection is. Artificial intelligence can be just as messy and alogical as natural intelligence. It's not because it runs on a computer that we fully see or understand what it does. I suppose there is a discussion to be had about what "true intelligence" is and whether some or most human beings have it in the first place.
    - didericis 4y ago
      It’s still propositional and logic based. Every instruction a computer can interpret is propositional and logic based. AI models are chaotic, but deterministic. All computers are deterministic. The distinction I’m making between propositional systems and intelligence is like the distinction between being able to generate a transcendental number like pi (which is propositionally generated/you can calculate it without “understanding” the sequence/being able to reduce it into a smaller pattern) and being able to know whether your procedure to generate pi is correct or incorrect. That latter ability is what I mean by true intelligence, and this transformers approach doesn’t solve that. I believe this is called the “grounding” problem. Gödel proved it’s impossible for any deterministic system to determine its own correctness close to a hundred years ago. Whatever we’re doing when we say an output A is “right” and output B is “wrong” is fundamentally different than what a propositional deterministic system does, regardless of how large or chaotic or well fitted to expected output it is. What qualifies as “true intelligence” is the core issue here, yes, and transformers don’t qualify. That doesn’t mean they aren’t valuable or can’t give correct results to things much faster and better than humans, but I think anything we could deliberately design inevitably ends up being deterministic and subject to the introspective limitations of any such system, no matter how big or all encompassing. I think we’re going to create better and more comprehensive “checkpoint”/“savepoints” for the results of collective intelligence, but that’s different than the intelligence itself.
- astrange 4y ago
  One answer to this would be that humans aren't general intelligences either, but existing in the real world/human society keeps you honest enough you have to try and act like one.
  - deleted 4y ago
    [deleted]
  - adastra22 4y ago
    One of the many ways AGI is often defined is “human-level intelligence,” so that seems like a tautological impossibility.
- YeGoblynQueenne 4y ago
  >> Do you think you have a soul or something? Otherwise how is your own Intelligence something a computer can’t replicate? Strictly speaking, it is for you to say why you think that a digital computer, which is nothing like a human, can be intelligent like a human. Btw, I see this quip about souls often on HN, in response to comments that AGI is impossible and it is invariably introduced in the conversation by the person arguing that AGI is possible. I have never, ever seen anyone argue that "digital computers can't be intelligent because they don't have a soul". I don't even know where that "soul" bit comes from, probably from something someone said centuries ago? In any case it's irrelevant to most modern conversations where most people don't believe in the supernatural, and have other reasons to think that AGI may be impossible. And why wouldn't they? For the time being, we don't have anything anyone is prepared to call "AGI". We also don't have a crystal ball to see into the future and know whether it is possible, or not. For all we know, it might be a theoretical possibility, but a practical impossibility, like stable wormholes, or time travel, or the Alcubierre Drive. We will know when we know, not before. Until then, invoking "souls", or any other canned reply, only seems to serve to shut down curious conversation and should be avoided. To say it in very plain English: you don't know if AGI is possible, you only assume it to be, so let's hear the other person say what they assume, too. Their opinion is just as interesting as yours, and just as an opinion.

adastra22 4y ago

> I personally think AGI is impossible That’s a very weird statement to make. Do you think you have a soul or something? Otherwise how is your own Intelligence something a computer can’t replicate?

sebastianz 4y ago
A computer could indeed replicate our intelligence, but our intelligence might not be sufficient to write the AGI sourcecode :)
drdeca 4y ago
I do think I have a soul, and I don’t think that AGI is impossible. So, I really don’t get why people think AGI is impossible. I mean, I have hope that it won’t ever happen, but not because I suspect it is impossible.
- didericis 4y ago
  I think it’s impossible because I don’t think you can leapfrog evolution, and there’s almost definitely hidden context essential to our intelligence and perception that you can only get through a very very long history. However I do think it’s quite possible and likely that we’ll create mimics that convince most people they’re intelligent. But I think that’s a weird type of reflection/anything we can make is limited to mirroring our observable past and existing collective thought and can never be truly introspective, because true introspection requires evolutionary context and hidden motivation you can’t transfer.
  - swagmoney1606 4y ago
    Evolution is a process that works without an intelligent steward. In a way it's a brute force technique. Plus, nothing is optimizing for intelligence in evolution, it is merely a happy accident that humans ended up with the brains that we did. A different environment could yield drastically different evolutionary history It doesn't seem very logical to think that because evolution took so long to get us to where we are now, we consequently won't be able to design an intelligent AGI system.
    - didericis 4y ago
      It’d take a while to justify this argument to the extent I think it’s justified, but I think we’re in an inescapable frame set by evolution, and our adaptation to our environment probably goes a lot deeper than what we can see. I don’t think the visible context is the full context, and think true intelligence probably requires an implicit understanding of context which is invisible to us.
didericis 4y ago
I don’t think true intelligence is propositional/logic based. I also think the observable universe is inherently limited by our perceptual framework, and there will always be something outside it. Imagine an amoeba that can only detect light and dark. Its observable universe exists on a simple gradient. But the cells that are used in its perceptual system cannot be described on a gradient. I think that probably scales up indefinitely. We confuse what we see for all that exists because of our recent history and scientific advances. The triumph of science comes from focusing solely on what we can see and reason about and improving our vision because thats what we have control over, but it doesn’t mean we see everything. I think true intelligence originates from something we can’t see. You can call it a soul, or call it a scaled up amoeba cell, but I personally think the origin of true intelligence and qualia is from some weird very very old evolutionarily created thing we can’t see (and I think other animals have similar invisible perceptual machinery somewhere, I don’t think it’s just a human thing).
- deleted 4y ago
  [deleted]
- breuleux 4y ago
  Current AI as exemplified by LLMs like ChatGPT is not propositional/logic based, though, so I'm unsure what your objection is. Artificial intelligence can be just as messy and alogical as natural intelligence. It's not because it runs on a computer that we fully see or understand what it does. I suppose there is a discussion to be had about what "true intelligence" is and whether some or most human beings have it in the first place.
  - didericis 4y ago
    It’s still propositional and logic based. Every instruction a computer can interpret is propositional and logic based. AI models are chaotic, but deterministic. All computers are deterministic. The distinction I’m making between propositional systems and intelligence is like the distinction between being able to generate a transcendental number like pi (which is propositionally generated/you can calculate it without “understanding” the sequence/being able to reduce it into a smaller pattern) and being able to know whether your procedure to generate pi is correct or incorrect. That latter ability is what I mean by true intelligence, and this transformers approach doesn’t solve that. I believe this is called the “grounding” problem. Gödel proved it’s impossible for any deterministic system to determine its own correctness close to a hundred years ago. Whatever we’re doing when we say an output A is “right” and output B is “wrong” is fundamentally different than what a propositional deterministic system does, regardless of how large or chaotic or well fitted to expected output it is. What qualifies as “true intelligence” is the core issue here, yes, and transformers don’t qualify. That doesn’t mean they aren’t valuable or can’t give correct results to things much faster and better than humans, but I think anything we could deliberately design inevitably ends up being deterministic and subject to the introspective limitations of any such system, no matter how big or all encompassing. I think we’re going to create better and more comprehensive “checkpoint”/“savepoints” for the results of collective intelligence, but that’s different than the intelligence itself.
astrange 4y ago
One answer to this would be that humans aren't general intelligences either, but existing in the real world/human society keeps you honest enough you have to try and act like one.
- deleted 4y ago
  [deleted]
- adastra22 4y ago
  One of the many ways AGI is often defined is “human-level intelligence,” so that seems like a tautological impossibility.
YeGoblynQueenne 4y ago
>> Do you think you have a soul or something? Otherwise how is your own Intelligence something a computer can’t replicate? Strictly speaking, it is for you to say why you think that a digital computer, which is nothing like a human, can be intelligent like a human. Btw, I see this quip about souls often on HN, in response to comments that AGI is impossible and it is invariably introduced in the conversation by the person arguing that AGI is possible. I have never, ever seen anyone argue that "digital computers can't be intelligent because they don't have a soul". I don't even know where that "soul" bit comes from, probably from something someone said centuries ago? In any case it's irrelevant to most modern conversations where most people don't believe in the supernatural, and have other reasons to think that AGI may be impossible. And why wouldn't they? For the time being, we don't have anything anyone is prepared to call "AGI". We also don't have a crystal ball to see into the future and know whether it is possible, or not. For all we know, it might be a theoretical possibility, but a practical impossibility, like stable wormholes, or time travel, or the Alcubierre Drive. We will know when we know, not before. Until then, invoking "souls", or any other canned reply, only seems to serve to shut down curious conversation and should be avoided. To say it in very plain English: you don't know if AGI is possible, you only assume it to be, so let's hear the other person say what they assume, too. Their opinion is just as interesting as yours, and just as an opinion.