Modern DL architectures are complicated. If you want to gain insight if a loss function is convex (bowl shaped), you can calculate the second derivatives matri

Modern DL architectures are complicated. If you want to gain insight if a loss function is convex (bowl shaped), you can calculate the second derivatives matrix (Hessian) at arbitrary points and test that all its eigenvalues are non negative. If you have a simple linear regression then that’s true, but for modern DL architectures things are not always convex and of course Hessians in very high dimensions are unrealistic. A simpler numerical test is to take lines connecting two arbitrary points of the loss function and showing that the value of the function along the line of the function variable connecting the two points is always lower than or equal to the value of the line connecting the two points. It is also hard to make this test rigorous in a practical high dimensional setting but one could be lucky and eg connect local minima to show that the function is not convex. The practical reason people didn’t use annealing on the DL problems at scale has been the satisfactory empirical results of the simpler/faster methods.