This paper makes the points that it's the saddles and not local minima that are the problem: https://arxiv.org/abs/1406.2572 https://arxiv.org/abs/1406.2572 It

This paper makes the points that it's the saddles and not local minima that are the problem: https://arxiv.org/abs/1406.2572 https://arxiv.org/abs/1406.2572 It was the basis for adding 'momentum' to optimizers - so that you could skate across the saddles.