4 ms·

Thanks for the links. It looks like I'm behind a bit - this space has been moving just crazy fast and months make the difference. Just to clarify.. > Finally

Thanks for the links. It looks like I'm behind a bit - this space has been moving just crazy fast and months make the difference. Just to clarify.. > Finally, as already mentioned by others, agents finding unexpected ways to maximize their reward is a well known problem of reinforcement learning: This much I agree with, but when it comes to the chain of reasoning stuff - my understanding was that the current state of the art wasn't capable of proper abstract logic given a sufficiently complex domain. Specifically, the air force test rang false because it seemed like a total stretch that an NN would be able to reason in that way. Am I just plain wrong or has there been movement in that space in particular recently?

TeMPOraL 3y ago
> This much I agree with, but when it comes to the chain of reasoning stuff - my understanding was that the current state of the art wasn't capable of proper abstract logic given a sufficiently complex domain. How complex is "sufficiently complex"? Or put another way - GPT-4 already seems quite good at abstract reasoning, as long as you translate things to avoid too obscure concepts, and keep things under its context window. For this specific case, a small experiment I did once convinced me that GPT-4 is rather good at planning ahead, when playing an interactive text game. So imagine yourself narrating an UAV camera feed over the radio, like it was a baseball match or a nature documentary. That's close enough, and embedded enough in real context, that GPT-4 would be able to provide good responses to "What to do next? List updated steps of you plan.". As for the "rest of the owl" involved in piloting an UAV, that's long ago been solved. Classical algorithms can handle flying, aiming and shooting just fine. Want something extra fancy? Videogame developers have you covered - game AI as a field is mostly several decades worth of experience in using a mix of cheesy hacks and bleeding-edge algorithms to make virtual agents good at planning ahead to best navigate a dynamic world and kill other agents in it. Recent Deep Learning AI work would be mostly helpful in keeping "sensory inputs" accurate. That said, despite being the bleeding edge of AI research, I don't think LLMs would be able to achieve the effect this air force story is describing - they understand too much. You'd have to go out of your way to get something like GPT-4 to confuse its own operator with an enemy missile launcher. This scenario smells like the work of an algorithm that doesn't work with high-level concepts, and instead has numeric outputs plugged in directly to UAV's low-level controls, and is trained on simulated scenarios. I.e. generic NNs, especially pre-deep learning, genetic algorithms, etc. - the simple stuff, plugged in as feedback controllers - essentially smarter PLCs. Those are prone to finding cheesy local minima of the cost function. In short: think tool-assisted speed runs. Or fuzzers. Or whatever that web demo was that evolved virtual 2-dimensional cars, by generating a few "vehicles" with randomly-sized wheels and bodies, making them ride a randomly-generated squiggly line, waiting until all of them get stuck, and then using the few that traveled the farthest to "breed" the next generation. This is the stuff that you put on a drone, if you want something that can start shooting at you just because "it seemed like a good idea at that moment".
T-A 3y ago
It's not abstract logic, it's rollout [1]: repeatedly simulating different action sequences N steps into the future and comparing their score after the last step. A brute force approach would just simulate all possible sequences and go with the highest-scoring one; RL algorithms pseudo-randomly sample the (typically intractably large) search space and use the results to update their policy (these days, typically implemented as a neural network). In the Air Force examples, as long as shooting the human controller or the communication tower is not explicitly prohibited, there is nothing surprising about an RL agent trying that course of action (along with other random things like shooting rocks, prairie dogs and even itself). Doing so requires no abstract reasoning or understanding of the causal mechanism between shooting the human controller and getting a better score, just random sampling and score-keeping. If the rollout score consistently goes up after the action "shoot the human controller", any RL algorithm worth its salt will update its policy accordingly and start shooting the human controller. [1] https://robotics.stackexchange.com/questions/16596/what-is-the-definition-of-rollout-in-neural-network-or-openai-gym https://robotics.stackexchange.com/questions/16596/what-is-t...

TeMPOraL 3y ago

> This much I agree with, but when it comes to the chain of reasoning stuff - my understanding was that the current state of the art wasn't capable of proper abstract logic given a sufficiently complex domain. How complex is "sufficiently complex"? Or put another way - GPT-4 already seems quite good at abstract reasoning, as long as you translate things to avoid too obscure concepts, and keep things under its context window. For this specific case, a small experiment I did once convinced me that GPT-4 is rather good at planning ahead, when playing an interactive text game. So imagine yourself narrating an UAV camera feed over the radio, like it was a baseball match or a nature documentary. That's close enough, and embedded enough in real context, that GPT-4 would be able to provide good responses to "What to do next? List updated steps of you plan.". As for the "rest of the owl" involved in piloting an UAV, that's long ago been solved. Classical algorithms can handle flying, aiming and shooting just fine. Want something extra fancy? Videogame developers have you covered - game AI as a field is mostly several decades worth of experience in using a mix of cheesy hacks and bleeding-edge algorithms to make virtual agents good at planning ahead to best navigate a dynamic world and kill other agents in it. Recent Deep Learning AI work would be mostly helpful in keeping "sensory inputs" accurate. That said, despite being the bleeding edge of AI research, I don't think LLMs would be able to achieve the effect this air force story is describing - they understand too much. You'd have to go out of your way to get something like GPT-4 to confuse its own operator with an enemy missile launcher. This scenario smells like the work of an algorithm that doesn't work with high-level concepts, and instead has numeric outputs plugged in directly to UAV's low-level controls, and is trained on simulated scenarios. I.e. generic NNs, especially pre-deep learning, genetic algorithms, etc. - the simple stuff, plugged in as feedback controllers - essentially smarter PLCs. Those are prone to finding cheesy local minima of the cost function. In short: think tool-assisted speed runs. Or fuzzers. Or whatever that web demo was that evolved virtual 2-dimensional cars, by generating a few "vehicles" with randomly-sized wheels and bodies, making them ride a randomly-generated squiggly line, waiting until all of them get stuck, and then using the few that traveled the farthest to "breed" the next generation. This is the stuff that you put on a drone, if you want something that can start shooting at you just because "it seemed like a good idea at that moment".

T-A 3y ago

It's not abstract logic, it's rollout [1]: repeatedly simulating different action sequences N steps into the future and comparing their score after the last step. A brute force approach would just simulate all possible sequences and go with the highest-scoring one; RL algorithms pseudo-randomly sample the (typically intractably large) search space and use the results to update their policy (these days, typically implemented as a neural network). In the Air Force examples, as long as shooting the human controller or the communication tower is not explicitly prohibited, there is nothing surprising about an RL agent trying that course of action (along with other random things like shooting rocks, prairie dogs and even itself). Doing so requires no abstract reasoning or understanding of the causal mechanism between shooting the human controller and getting a better score, just random sampling and score-keeping. If the rollout score consistently goes up after the action "shoot the human controller", any RL algorithm worth its salt will update its policy accordingly and start shooting the human controller. [1] https://robotics.stackexchange.com/questions/16596/what-is-the-definition-of-rollout-in-neural-network-or-openai-gym https://robotics.stackexchange.com/questions/16596/what-is-t...