Other points to note: this is a lot like evolution strategies or CMA-ES if you're familiar with those, so inherits their strengths and weaknesses; it's sample/c

Other points to note: this is a lot like evolution strategies or CMA-ES if you're familiar with those, so inherits their strengths and weaknesses; it's sample/compute-inefficient because it doesn't use real gradients, but that also means you can use non-differentiable tools, so this might be useful in a RL context or a tool/API context, which are increasingly big deals for LLM usecases. And it's a lot less sample/compute-inefficient than you might expect from such low-quality optimization applied to such highly-parameterized models; so this is another example of the blessings of scale, and how very large neural net models are, in some sense, simpler, more linear/interpretible, more generalizable, and more efficient than they 'ought' to be.