Along the same lines, I read something (on here but I don't think I can find it) suggesting that llms + prompting are kind of overfitting, where if you know wha

Along the same lines, I read something (on here but I don't think I can find it) suggesting that llms + prompting are kind of overfitting, where if you know what you achieve, you can try few things and get it - which would be equivalent to optimizing on a train set. Whereas if you try to get it to do something you don't already know how to do, all bets are off.