This 'layering' of new functionality reminds me of the Subsumption robots architecture[0]. My main takeaway was that you could build complex behavior by layeri

This 'layering' of new functionality reminds me of the Subsumption robots architecture[0]. My main takeaway was that you could build complex behavior by layering together reflexes. You wouldn't actually disable an existing layer, just add an overlay to disable it in certain circumstances. I heard in a lecture once that our hands are wired similarly; we have a nerve that triggers a grasping motion involving all fingers, then we have overlay nerves that can disable the signal on a per-finger basis. It may be that this is a fine way to build complex systems that's not optimal for _human_ reasoning patterns, but may for evolution (or other hypothetical non-human reasoning processes). [0] - https://en.wikipedia.org/wiki/Subsumption_architecture https://en.wikipedia.org/wiki/Subsumption_architecture