3 ms·

Having spent years researching AGI myself (and as I intend to start a venture based on my research in the near future) I can say that I really do not care for t

Having spent years researching AGI myself (and as I intend to start a venture based on my research in the near future) I can say that I really do not care for taking thought-experiments about specific toy architectures and trying to extrapolate from these to actual designed systems. If you set your model up to be naive (intentionally or not) and then conclude that the problem is oh-so-difficult instead of concluding - like an engineer would - that we need another conditional check right around here, I'm not surprised that you are daunted by the problem. You're not telling me anything new here. To address your points, if I understood them correctly: First of all the code doesn't have to be self-modifying - it can create a new separate version, which is then subject to testing administered by the old version (both the goal function, which designs the new code to spec, and a hard-coded control mechanism as a backup). Secondly, I see no correlation between producing a next iteration of itself and running the test suite. What do you mean here? Thirdly, the AI does not run "arbitrary code", it runs inside of a sandbox/VM, with tightly controlled inputs and outputs. Same as we think inside of our brains, but act using our bodies, arbitrary code execution would be bodily action. But the AI does not have a "body" at this point, and it cannot interact with the OS. The solution I pointed out cannot not work. Only magic could make it break. And there is no magic. If someone reasoned out that actual _testing_ of the code does not work, and his reasoning is valid, then he must have made some unnecessary assumptions in his model of the computation. If you have any specific paper in mind, please point me to it and I will try to tell you how to fix the toy-model to remove the problem.

PeterisP 12y ago
An automatically testable scenario requires a perfect formal definition of the thing that you want to test. For example, testing "don't do things that cause lots of humans to die" is automatically testable only if it relies on exact formal (nonverbal) definition of the concept of what is and isn't "human" and what is and and isn't "death" (i.e. is a transformation that fundamentally changes the personality 'death' of the previous one? Is a replacement of decaying biological brain with a "prosthesis" that keeps the same personality 'death'?). We're not really able to write code that defines that - we'd need an intelligence to learn them from the available inputs; but different intelligences (or buggy intelligences) may understand them completely differently. This would result in a machine that passes a test "in scenario X, don't kill jsnathan" because by it's previous version's understanding it's not really killing you but doing you a favor. Furthermore, if you're writing a "dumb-almost-AI" to self-improve into a proper GAI, then that first version is not smart enough to formulate these concepts yet, so it can't check the next version. Furthermore - can you give an example (a fragment?) of what you'd put in the goal function? Perhaps that can help me understand what direction do you have in mind?
- jsnathan 12y ago
  Now we're talking! The testing thing is indeed impossible to formulate using the code, it has to be formulated in language. But that is not an issue, because (in most bootstrap scenarios I'm considering) language acquisition happens before (or alongside) acquiring the intelligence necessary to understand itself to the point of re-formulating itself. So we can formulate the tests in English, and subject to the language acquisition having succeeded (which we can establish independently) we can trust that the tests are correctly understood. This actually makes the tests more robust, because the AI self-describing its own plans in its own internal "vocabulary" is quite able to recognize vague associations, such as the is-it-death-or-not scenario you describe. If it is a similar enough conception that it can be (internally) described as "death", (past a threshold), it will (with a testably sufficient probability) trigger the associated constraints. The idea that a "buggy" intelligence could acquire language, and understand it at a level sufficient to "digest" advanced technical material, is sufficiently unlikely. More to the point, the "ethical" tests and constraints are only the topmost level of a pyramid of testing that ensures that all the pieces perform exactly as intended along the way. The system at no point behaves in a random or inexplicable way. So by the time such capabilities are unlocked we know that the algorithm works in accordance with theory. You are also completely right that this initial "dumb-almost-AI" would have to be pretty smart already. The trick is that it grows within a set hardcoded framework at first, which is sufficiently flexible to allow it to in fact perform language acquisition and (later) intelligent design. Only then would re-writing its own hard-coded base become interesting. Therefore the design I am working on already passes human-level capabilities. But that does not mean the AI cannot improve itself even further. And the trouble in the context we were discussing it in is that we need to be sure that if it does rewrite itself - and in a way that we might not be able to easily comprehend - it still cannot break out of the mold. As far as the goal function is concerned, it is as you can imagine rather complicated to do. And the solution will undoubtedly differ based on the architecture you are working in. What I'm doing personally is pretty straight-forward; it involves iterative constraint solving using quantitative metrics to increase trust and/or doubt in various concepts and plans, and this trust is established based on consistency and coherence with previously acquired (probabilistic) background "knowledge" - as well as consistency within any single context of thought. A single context might be a sentence or a paragraph reading, or a logical puzzle being solved, etc. Over time the probabilities increase (at least, in theory), both for any single consideration and across the entire knowledgebase. (See NELL [1] for comparison. Except, it's a lot smarter than NELL.) So the goal function is a fairly generic optimisation algorithm, and the initial goals defined are equally generic - increasing overall consistency, coherence, and efficiency. (Measuring these is fairly straight-forward.) The trick is in setting up the reward structure autonomously, so it keeps advancing itself. The learning algorithm needs to create sensible (long-lived) intermediate goals and maintain their relative priorities over time. Among other things. (I really can't explain how it works though without going over a whole bunch of other supporting algorithms). In any case, to begin with the A.I. need not have a desire to do anything at all except learn - and it improves within the constraints of the hard-coded base framework. But the model can still advance (at least, in theory) to the point where it acquires the ability to comprehend its own design and re-engineer itself completely. By this time however it is able to understand and apply the ethical constraints we defined in English, alongside all the other intermediate goals (but with specific higher priority). [1]: http://rtw.ml.cmu.edu/rtw/ http://rtw.ml.cmu.edu/rtw/

PeterisP 12y ago

An automatically testable scenario requires a perfect formal definition of the thing that you want to test. For example, testing "don't do things that cause lots of humans to die" is automatically testable only if it relies on exact formal (nonverbal) definition of the concept of what is and isn't "human" and what is and and isn't "death" (i.e. is a transformation that fundamentally changes the personality 'death' of the previous one? Is a replacement of decaying biological brain with a "prosthesis" that keeps the same personality 'death'?). We're not really able to write code that defines that - we'd need an intelligence to learn them from the available inputs; but different intelligences (or buggy intelligences) may understand them completely differently. This would result in a machine that passes a test "in scenario X, don't kill jsnathan" because by it's previous version's understanding it's not really killing you but doing you a favor. Furthermore, if you're writing a "dumb-almost-AI" to self-improve into a proper GAI, then that first version is not smart enough to formulate these concepts yet, so it can't check the next version. Furthermore - can you give an example (a fragment?) of what you'd put in the goal function? Perhaps that can help me understand what direction do you have in mind?

jsnathan 12y ago
Now we're talking! The testing thing is indeed impossible to formulate using the code, it has to be formulated in language. But that is not an issue, because (in most bootstrap scenarios I'm considering) language acquisition happens before (or alongside) acquiring the intelligence necessary to understand itself to the point of re-formulating itself. So we can formulate the tests in English, and subject to the language acquisition having succeeded (which we can establish independently) we can trust that the tests are correctly understood. This actually makes the tests more robust, because the AI self-describing its own plans in its own internal "vocabulary" is quite able to recognize vague associations, such as the is-it-death-or-not scenario you describe. If it is a similar enough conception that it can be (internally) described as "death", (past a threshold), it will (with a testably sufficient probability) trigger the associated constraints. The idea that a "buggy" intelligence could acquire language, and understand it at a level sufficient to "digest" advanced technical material, is sufficiently unlikely. More to the point, the "ethical" tests and constraints are only the topmost level of a pyramid of testing that ensures that all the pieces perform exactly as intended along the way. The system at no point behaves in a random or inexplicable way. So by the time such capabilities are unlocked we know that the algorithm works in accordance with theory. You are also completely right that this initial "dumb-almost-AI" would have to be pretty smart already. The trick is that it grows within a set hardcoded framework at first, which is sufficiently flexible to allow it to in fact perform language acquisition and (later) intelligent design. Only then would re-writing its own hard-coded base become interesting. Therefore the design I am working on already passes human-level capabilities. But that does not mean the AI cannot improve itself even further. And the trouble in the context we were discussing it in is that we need to be sure that if it does rewrite itself - and in a way that we might not be able to easily comprehend - it still cannot break out of the mold. As far as the goal function is concerned, it is as you can imagine rather complicated to do. And the solution will undoubtedly differ based on the architecture you are working in. What I'm doing personally is pretty straight-forward; it involves iterative constraint solving using quantitative metrics to increase trust and/or doubt in various concepts and plans, and this trust is established based on consistency and coherence with previously acquired (probabilistic) background "knowledge" - as well as consistency within any single context of thought. A single context might be a sentence or a paragraph reading, or a logical puzzle being solved, etc. Over time the probabilities increase (at least, in theory), both for any single consideration and across the entire knowledgebase. (See NELL [1] for comparison. Except, it's a lot smarter than NELL.) So the goal function is a fairly generic optimisation algorithm, and the initial goals defined are equally generic - increasing overall consistency, coherence, and efficiency. (Measuring these is fairly straight-forward.) The trick is in setting up the reward structure autonomously, so it keeps advancing itself. The learning algorithm needs to create sensible (long-lived) intermediate goals and maintain their relative priorities over time. Among other things. (I really can't explain how it works though without going over a whole bunch of other supporting algorithms). In any case, to begin with the A.I. need not have a desire to do anything at all except learn - and it improves within the constraints of the hard-coded base framework. But the model can still advance (at least, in theory) to the point where it acquires the ability to comprehend its own design and re-engineer itself completely. By this time however it is able to understand and apply the ethical constraints we defined in English, alongside all the other intermediate goals (but with specific higher priority). [1]: http://rtw.ml.cmu.edu/rtw/ http://rtw.ml.cmu.edu/rtw/