Long article. I found this passage insightful: … the problem with the way we build AI systems now is we we give them a fixed objective. The algorithms require

Long article. I found this passage insightful: … the problem with the way we build AI systems now is we we give them a fixed objective. The algorithms require us to specify everything in the objective. And if you say: can we fix the acidification of the oceans? Yeah, you could have a catalytic reaction that does that extremely efficiently, but consumes a quarter of the oxygen in the atmosphere … the main point is if we build systems that know that they don't know what the objective is, then they start to exhibit these behaviours, like asking permission before getting rid of all the oxygen in the atmosphere