The way I see it, it's first a question of how much agency is given to any computer program over the real world, regardless of how advanced it is cognitively sp

The way I see it, it's first a question of how much agency is given to any computer program over the real world, regardless of how advanced it is cognitively speaking. Technically a bash script can be nightmare-ish if plugged into a nuclear launch station. Then comes the issue of predictability and reasonability. Given a set of inputs to this program, will it produce an output deemed reasonable and expected to an acceptable amount, compared to what the society or the government would have decided in this situation (or whoever is in charge of that particular decision). This is still not exclusively within the realm of AI (a bash script's ways might be impenetrable). But humoring TFA, current gen AI is limited in this regard in most of its incarnations, barring some approaches (decision trees, random forests) which also happen to be the least popular, or cutting edge research. The more we bring black box software (AI or not) in our decision loops, the closer we'll get the bash god to write its bible, post-singularity or not.