I would say it's more a commentary on the fact the the gradient is effectively based on L2 distance in parameter space, and so can be a bad/inefficent direction

I would say it's more a commentary on the fact the the gradient is effectively based on L2 distance in parameter space, and so can be a bad/inefficent direction to move in even if you have access to the full gradient. Hence the motivation for momentum and second-order optimization.