Try to implement a rational agent (the AI or mathematically formalized variety) in software, and you'll see what I mean. You have to define some utility functio

Try to implement a rational agent (the AI or mathematically formalized variety) in software, and you'll see what I mean. You have to define some utility function precisely, and some algorithm to maximize it. The problem with this is it won't really produce a "general intelligence" that can think outside the box, because it will always be maximizing some utility function defined in some rigid formal system. In other words, it will be completely unable to "understand" things outside of the formal system in which you define it.