4 ms·

In my experience, "the bitter lesson" is just wrong. Yes, a hypothetical AI model that has a suitable architecture so that it can memorize your handcrafted feat

In my experience, "the bitter lesson" is just wrong. Yes, a hypothetical AI model that has a suitable architecture so that it can memorize your handcrafted features can improve upon them, if it is perfectly trained. But that's a huge if. There's a reason that large AI models only work with some architectures and you need to randomly initialize tens to hundreds of times and you need exactly the right optimizer with the right hyperparameters and then the right training data in the right order... and the reason is that generic function approximators get stuck in a local minima incredibly easy. People used to write handcrafted features to limit the optimization base area. Now people use specific pretraining data and initialization functions to do the same. It's just a different way to express the same constraints. In short, the "bitter lesson" assumes a change that did not happen yet.

imtringued 2y ago
>and the reason is that generic function approximators get stuck in a local minima incredibly easy. The key insight in massively over parameterized models (aka LLMs) has been that all the local minima are very similar. Picking one over the other doesn't actually benefit you that much.
- fxtentacle 2y ago
  Absolutely not! LLM training quite easily gets stuck. People call it "divergence" or "loss spike". See for example: https://dl.acm.org/doi/10.5555/3524938.3525354 https://dl.acm.org/doi/10.5555/3524938.3525354 "Gradient optimization with attention layers can be notoriously difficult requiring tricks such as learning rate warmup to prevent divergence.“ "we then propose a new weight initialization scheme with theoretical justification, that enables training without warmup" Or "Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes" https://arxiv.org/html/2410.05052v1 https://arxiv.org/html/2410.05052v1 If all local minima were similar, then these papers would represent a lot of wasted effort. Or said in a different way: If labs like DeepMind are willing to pay for large teams that are trying to optimize initialization, then surely this represents a real existing problem.
  - noosphr 2y ago
    There are basically no people who have done pre-training of LLMs. What you're describing is absolutely the case, but everyone not in a top 100 AI lab would have never had the displeasure of having to deal with that. That said I strongly suggest everyone follow along with a GPT2 tutorial like: https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT to get a feel for what the big boys have to deal with. Any $100,000 training run can end in failure for no reason what so ever because you just got unlucky with your random seed. If you want something that is both SOTA and cheap: https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert you can get from pre-training to fine tuning for a couple hundred dollars with a model that is better than anything available to the public. Bert models are so under researched that you legitimately push the state of the art for them on a work station. Also feel free to drop a line at my profile's address.
  - kristjansson 2y ago
    nit: a divergent training run is somewhat separate from the question of local minima. The latter observes the loss curve stabilize and asks 'is the _best_ (loss-minimizing) set of parameters for this loss/data/model/hypers/seed/etc?'. The former observes the loss curve explode, and asks for another 10k GPU hours to try again with a different choice of loss/data/model/... . They're definitely related in some instances e.g. 'flat' vs 'steep' local minima with comparable losses.
cma 2y ago
> There's a reason that large AI models only work with some architectures and you need to randomly initialize tens to hundreds of times and you need exactly the right optimizer with the right hyperparameters and then the right training data in the right order.. GPT4 used the tensor programs (Greg Yang,now at X. Ai) research to effectively transfer hyper parameters from smaller models where experiments can go fast to the larger one in a predictable way and got a very smooth loss curve throughout the training.
- noosphr 2y ago
  But that's OPs point. You need to care about those hyper parameters. Good on them for finding them in smaller models, but they still had to find them, because without those hyper parameters pre-training is at best unstable and at worst divergent. Also small here means hundreds of millions to billions of parameters - that's still larger than any model pre 2020.
  - cma 2y ago
    Prior to that work on hyperparameter transfer though things would go wrong much more often training the big expensive model and require restarts back to checkpoints and retuning parameters much more often though. Yeah it's still a lot of fiddling around but it is orders of magnitude less expensive. But that work also went far beyond transfer of fiddling around and mathematically solved a lot of stuff with avoiding overfitting, great video overview of it here: https://www.youtube.com/watch?v=1aXOXHA7Jcw https://www.youtube.com/watch?v=1aXOXHA7Jcw

imtringued 2y ago

>and the reason is that generic function approximators get stuck in a local minima incredibly easy. The key insight in massively over parameterized models (aka LLMs) has been that all the local minima are very similar. Picking one over the other doesn't actually benefit you that much.

fxtentacle 2y ago
Absolutely not! LLM training quite easily gets stuck. People call it "divergence" or "loss spike". See for example: https://dl.acm.org/doi/10.5555/3524938.3525354 https://dl.acm.org/doi/10.5555/3524938.3525354 "Gradient optimization with attention layers can be notoriously difficult requiring tricks such as learning rate warmup to prevent divergence.“ "we then propose a new weight initialization scheme with theoretical justification, that enables training without warmup" Or "Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes" https://arxiv.org/html/2410.05052v1 https://arxiv.org/html/2410.05052v1 If all local minima were similar, then these papers would represent a lot of wasted effort. Or said in a different way: If labs like DeepMind are willing to pay for large teams that are trying to optimize initialization, then surely this represents a real existing problem.
- noosphr 2y ago
  There are basically no people who have done pre-training of LLMs. What you're describing is absolutely the case, but everyone not in a top 100 AI lab would have never had the displeasure of having to deal with that. That said I strongly suggest everyone follow along with a GPT2 tutorial like: https://github.com/karpathy/minGPT https://github.com/karpathy/minGPT to get a feel for what the big boys have to deal with. Any $100,000 training run can end in failure for no reason what so ever because you just got unlucky with your random seed. If you want something that is both SOTA and cheap: https://huggingface.co/blog/modernbert https://huggingface.co/blog/modernbert you can get from pre-training to fine tuning for a couple hundred dollars with a model that is better than anything available to the public. Bert models are so under researched that you legitimately push the state of the art for them on a work station. Also feel free to drop a line at my profile's address.
- kristjansson 2y ago
  nit: a divergent training run is somewhat separate from the question of local minima. The latter observes the loss curve stabilize and asks 'is the _best_ (loss-minimizing) set of parameters for this loss/data/model/hypers/seed/etc?'. The former observes the loss curve explode, and asks for another 10k GPU hours to try again with a different choice of loss/data/model/... . They're definitely related in some instances e.g. 'flat' vs 'steep' local minima with comparable losses.

cma 2y ago

> There's a reason that large AI models only work with some architectures and you need to randomly initialize tens to hundreds of times and you need exactly the right optimizer with the right hyperparameters and then the right training data in the right order.. GPT4 used the tensor programs (Greg Yang,now at X. Ai) research to effectively transfer hyper parameters from smaller models where experiments can go fast to the larger one in a predictable way and got a very smooth loss curve throughout the training.

noosphr 2y ago
But that's OPs point. You need to care about those hyper parameters. Good on them for finding them in smaller models, but they still had to find them, because without those hyper parameters pre-training is at best unstable and at worst divergent. Also small here means hundreds of millions to billions of parameters - that's still larger than any model pre 2020.
- cma 2y ago
  Prior to that work on hyperparameter transfer though things would go wrong much more often training the big expensive model and require restarts back to checkpoints and retuning parameters much more often though. Yeah it's still a lot of fiddling around but it is orders of magnitude less expensive. But that work also went far beyond transfer of fiddling around and mathematically solved a lot of stuff with avoiding overfitting, great video overview of it here: https://www.youtube.com/watch?v=1aXOXHA7Jcw https://www.youtube.com/watch?v=1aXOXHA7Jcw