I think you may underestimate the scaling of global optimization with dimension. You split each simplex into n+1 smaller ones in n dimensions, so even in 99 dim

I think you may underestimate the scaling of global optimization with dimension. You split each simplex into n+1 smaller ones in n dimensions, so even in 99 dimensions, the number of areas to consider grows as 100^k! After only a few subdivisions, the regions may be still pretty large, but suddenly you're dealing with thousands or millions of reasonable candidates. If your optimization problem is particularly easy, you might manage it, but in tricky cases it just cannot scale. This isn't a flaw in your algorithm - global optimization of non 'nice' functions in high dimensions is essentially impossible for the same reasons. Luckily, turns out that in a lot of cases you don't need a global maximum, only something good enough, so the NN folks aren't bothered.