They aren't technically using just steepest descent, there is momentum, adaptive rate, stochastic pieces in the most used Adam optimizer, so it's not like it's

They aren't technically using just steepest descent, there is momentum, adaptive rate, stochastic pieces in the most used Adam optimizer, so it's not like it's the most basic version of gradient descent. There are also some structural rules on top, for example if one wants to model discrete variables, they create some relaxations, resembling simplex, tightened by simulated annealing, hoping by the time temperature drops to 0 the optimum is similar to the discrete optimum they wanted to find. It would be super surprising if the gradient descent by itself did all the difficult part, which we know for sure isn't because otherwise shallow fully-connected networks would be sufficient to figure out everything. Instead the whole craft nowadays consists of figuring out how to assemble certain primitive pieces together so that steepest descent actually does a good job, like how convolutions reduce searchable space considerably or embedings allow to group related terms together.