4 ms·

> if and only if it wanted be friendly and respect those constraints in the first place No, not if it "wanted" to - if it was written that way. > otherwise yo

> if and only if it wanted be friendly and respect those constraints in the first place No, not if it "wanted" to - if it was written that way. > otherwise you should expect for it to figure out how to work around them to achieve whatever goal function was implemented Yes, if the goal function did not include preserving certain (behavioral) invariants in its own programming, then yes. But, it would. > Making a spec "AI, do X" that isn't catastrophically bad is a very hard unsolved problem. No it's not. That's just something you throw out there. I just gave an example of exactly how to do it. Write a simple test to check whether a certain outcome is endorsed or rejected by the A.I. If you are the designer, you will not stop there either. You will want to check that the reason it is rejected is because the machine is working in exactly the way you predicted. Do this for a couple hundred scenarios, and make sure that the clock ticks right, and I see no reason to introduce any talk of "hard unsolved problems".

PeterisP 12y ago
This is not something that I just throw out there - these are real research problems with no known good solutions (but a lot of published attempts and discussions) despite having a lot of phd-years thrown at it already. For example, the obvious solutions for making self-modifying code that would preserve behavioral invariants cannot work because it rather obviously ends up at the halting problem. For very restrictive subsets of self-modifying code that can preserve invariants it's not clear if it can be powerful enough to allow the required level of self-introspection and self-improvement. You also cannot make a test if a certain outcome will be endorsed or rejected by an AI that the AI will make that the AI will make until the 'seed' AI has actually made the next versions of it - and by that time it's too late. If the AI can run some arbitrary code before you make that test (and it pretty much has to), then a single buffer overflow in your OS means that it can do arbitrary things before that test is made. Etc. This is not the space to discuss this - the exact suggestions that you give above have been proposed many times, analyzed, and they don't really work as well as you'd expect. Any answer that's not very simplifying cannot fit in comments and requires a bunch of academic reading. https://intelligence.org/research/ https://intelligence.org/research/ may be a place to start, if you're intereseted.
- jsnathan 12y ago
  Having spent years researching AGI myself (and as I intend to start a venture based on my research in the near future) I can say that I really do not care for taking thought-experiments about specific toy architectures and trying to extrapolate from these to actual designed systems. If you set your model up to be naive (intentionally or not) and then conclude that the problem is oh-so-difficult instead of concluding - like an engineer would - that we need another conditional check right around here, I'm not surprised that you are daunted by the problem. You're not telling me anything new here. To address your points, if I understood them correctly: First of all the code doesn't have to be self-modifying - it can create a new separate version, which is then subject to testing administered by the old version (both the goal function, which designs the new code to spec, and a hard-coded control mechanism as a backup). Secondly, I see no correlation between producing a next iteration of itself and running the test suite. What do you mean here? Thirdly, the AI does not run "arbitrary code", it runs inside of a sandbox/VM, with tightly controlled inputs and outputs. Same as we think inside of our brains, but act using our bodies, arbitrary code execution would be bodily action. But the AI does not have a "body" at this point, and it cannot interact with the OS. The solution I pointed out cannot not work. Only magic could make it break. And there is no magic. If someone reasoned out that actual _testing_ of the code does not work, and his reasoning is valid, then he must have made some unnecessary assumptions in his model of the computation. If you have any specific paper in mind, please point me to it and I will try to tell you how to fix the toy-model to remove the problem.
  - PeterisP 12y ago
    An automatically testable scenario requires a perfect formal definition of the thing that you want to test. For example, testing "don't do things that cause lots of humans to die" is automatically testable only if it relies on exact formal (nonverbal) definition of the concept of what is and isn't "human" and what is and and isn't "death" (i.e. is a transformation that fundamentally changes the personality 'death' of the previous one? Is a replacement of decaying biological brain with a "prosthesis" that keeps the same personality 'death'?). We're not really able to write code that defines that - we'd need an intelligence to learn them from the available inputs; but different intelligences (or buggy intelligences) may understand them completely differently. This would result in a machine that passes a test "in scenario X, don't kill jsnathan" because by it's previous version's understanding it's not really killing you but doing you a favor. Furthermore, if you're writing a "dumb-almost-AI" to self-improve into a proper GAI, then that first version is not smart enough to formulate these concepts yet, so it can't check the next version. Furthermore - can you give an example (a fragment?) of what you'd put in the goal function? Perhaps that can help me understand what direction do you have in mind?

PeterisP 12y ago

This is not something that I just throw out there - these are real research problems with no known good solutions (but a lot of published attempts and discussions) despite having a lot of phd-years thrown at it already. For example, the obvious solutions for making self-modifying code that would preserve behavioral invariants cannot work because it rather obviously ends up at the halting problem. For very restrictive subsets of self-modifying code that can preserve invariants it's not clear if it can be powerful enough to allow the required level of self-introspection and self-improvement. You also cannot make a test if a certain outcome will be endorsed or rejected by an AI that the AI will make that the AI will make until the 'seed' AI has actually made the next versions of it - and by that time it's too late. If the AI can run some arbitrary code before you make that test (and it pretty much has to), then a single buffer overflow in your OS means that it can do arbitrary things before that test is made. Etc. This is not the space to discuss this - the exact suggestions that you give above have been proposed many times, analyzed, and they don't really work as well as you'd expect. Any answer that's not very simplifying cannot fit in comments and requires a bunch of academic reading. https://intelligence.org/research/ https://intelligence.org/research/ may be a place to start, if you're intereseted.

jsnathan 12y ago
Having spent years researching AGI myself (and as I intend to start a venture based on my research in the near future) I can say that I really do not care for taking thought-experiments about specific toy architectures and trying to extrapolate from these to actual designed systems. If you set your model up to be naive (intentionally or not) and then conclude that the problem is oh-so-difficult instead of concluding - like an engineer would - that we need another conditional check right around here, I'm not surprised that you are daunted by the problem. You're not telling me anything new here. To address your points, if I understood them correctly: First of all the code doesn't have to be self-modifying - it can create a new separate version, which is then subject to testing administered by the old version (both the goal function, which designs the new code to spec, and a hard-coded control mechanism as a backup). Secondly, I see no correlation between producing a next iteration of itself and running the test suite. What do you mean here? Thirdly, the AI does not run "arbitrary code", it runs inside of a sandbox/VM, with tightly controlled inputs and outputs. Same as we think inside of our brains, but act using our bodies, arbitrary code execution would be bodily action. But the AI does not have a "body" at this point, and it cannot interact with the OS. The solution I pointed out cannot not work. Only magic could make it break. And there is no magic. If someone reasoned out that actual _testing_ of the code does not work, and his reasoning is valid, then he must have made some unnecessary assumptions in his model of the computation. If you have any specific paper in mind, please point me to it and I will try to tell you how to fix the toy-model to remove the problem.
- PeterisP 12y ago
  An automatically testable scenario requires a perfect formal definition of the thing that you want to test. For example, testing "don't do things that cause lots of humans to die" is automatically testable only if it relies on exact formal (nonverbal) definition of the concept of what is and isn't "human" and what is and and isn't "death" (i.e. is a transformation that fundamentally changes the personality 'death' of the previous one? Is a replacement of decaying biological brain with a "prosthesis" that keeps the same personality 'death'?). We're not really able to write code that defines that - we'd need an intelligence to learn them from the available inputs; but different intelligences (or buggy intelligences) may understand them completely differently. This would result in a machine that passes a test "in scenario X, don't kill jsnathan" because by it's previous version's understanding it's not really killing you but doing you a favor. Furthermore, if you're writing a "dumb-almost-AI" to self-improve into a proper GAI, then that first version is not smart enough to formulate these concepts yet, so it can't check the next version. Furthermore - can you give an example (a fragment?) of what you'd put in the goal function? Perhaps that can help me understand what direction do you have in mind?