4 ms·

What's the behavior your trying to get the AI to do in this example? Learn how to compute the power of 2? This is a task that can be accomplished much more si

by enkid 5y ago

What's the behavior your trying to get the AI to do in this example? Learn how to compute the power of 2? This is a task that can be accomplished much more simply with a different reward system. For example, have A always equal 1 and B equal 2 if it is a power if 2 and 0 otherwise. I understand you can use non real numbers, that's not what I was asking. I'm asking what's a behaviour you can't replicate using a reward system based on real numbers.

xamuel 5y ago
>I'm asking what's a behaviour you can't replicate using a reward system based on real numbers So glad you asked! I can give an answer which people will love who take the necessary time to understand it. It's complicated, you might have to re-read it a few times and really ponder it. It's about automatic code generation (though it might not look like it at first). Definition 1: Define the "Intuitive Ordinal Notations" (IONs) to be the smallest set P of computer programs such that for every computer program p, if all the things p outputs are IONs, then p is an ION. See https://github.com/semitrivial/IONs https://github.com/semitrivial/IONs for some ION examples in python. Definition 2: Inductively associate an ordinal |p| with every ION p as follows: |p| is defined to be smallest ordinal which is bigger than every ordinal |q| such that q is an output of p. Say that p "notates" |p|. Finally, to answer your question, I want the AGI to write programs which are IONs notating large ordinals, accompanied by arguments convincing me they really are IONs. An easy way to incentivize this with RL would be as follows. If the AGI writes an ION p and an argument that convinces me it's an ION, I will grant the AGI reward |p|. If the AGI does anything else (including if its argument does not convince me), then I'll give it reward 0. You can't correctly incentivize this behavior using reals. The computable ordinals are too non-Archimedean to do so.

by enkid 5y ago

xamuel 5y ago

>I'm asking what's a behaviour you can't replicate using a reward system based on real numbers So glad you asked! I can give an answer which people will love who take the necessary time to understand it. It's complicated, you might have to re-read it a few times and really ponder it. It's about automatic code generation (though it might not look like it at first). Definition 1: Define the "Intuitive Ordinal Notations" (IONs) to be the smallest set P of computer programs such that for every computer program p, if all the things p outputs are IONs, then p is an ION. See https://github.com/semitrivial/IONs https://github.com/semitrivial/IONs for some ION examples in python. Definition 2: Inductively associate an ordinal |p| with every ION p as follows: |p| is defined to be smallest ordinal which is bigger than every ordinal |q| such that q is an output of p. Say that p "notates" |p|. Finally, to answer your question, I want the AGI to write programs which are IONs notating large ordinals, accompanied by arguments convincing me they really are IONs. An easy way to incentivize this with RL would be as follows. If the AGI writes an ION p and an argument that convinces me it's an ION, I will grant the AGI reward |p|. If the AGI does anything else (including if its argument does not convince me), then I'll give it reward 0. You can't correctly incentivize this behavior using reals. The computable ordinals are too non-Archimedean to do so.