I agree, it's doing spectacularly well. My point isn't "AI research is doing badly", it's "we should probably expect AI systems not to be perfectly effective op

I agree, it's doing spectacularly well. My point isn't "AI research is doing badly", it's "we should probably expect AI systems not to be perfectly effective optimizers of the things we try to build them to optimize; we may try to design them to do X but that doesn't necessarily mean they will actually do X, any more than we consistently do the things evolution has tried to design us to do".