4 ms·

I’m surprised we weren’t doing this already. I’d like to see what happens if you train a small language model on preschool level reading material, and ramp up b

by valine 3y ago

I’m surprised we weren’t doing this already. I’d like to see what happens if you train a small language model on preschool level reading material, and ramp up both the model size and training data complexity as you go. My hope would be you’d need less data to train a model in this fashion than you would with our current approach of throwing the entire internet at the model.

two_in_one 3y ago
> I’m surprised we weren’t doing this already. Because they don't work very well in real life. Article doesn't have any experimental proof it works, saves training time, or size, or anything.. And it doesn't mention another promising approach: mix of experts. There are many ways of implementing it. I mean the general idea of specialized fragments of the NN which are selectively called. They don't have to be trained all at once.
- tempusalaria 3y ago
  It’s weird a DeepMind author is on this as they experimentally studies these methods in Gopher and found it was useless. Even the layer duplication trick they were using doesn’t work. I tried a refinement of that where you fine tuned the activations of the added layers to match the original activations which helped a bit relative to what DeepMind found but it still wasn’t material enough to be worth bothering with
  - two_in_one 3y ago
    I tried adding layer. The conclusion is, it's useful if you cannot train the whole model at once. Then you can split it in 2 and train the first half, with embedding/deembedding layers. After it stops improving add the second half of the layers, and train only it, freezing everything else. This worked in my case, when model params number < 10% of the number of training tokens. Splitting model in 3 is significantly less effective. Don't know why, but the third part adds almost nothing in accuracy and a lot in training time. One thing where additive training works is fine tuning, like LoRA. But that's different. It's like poor man's way. For those who cannot train the whole model. Most likely uptrainig it whole is more efficient, just my guess.
jncraton 3y ago
You might be interested in TinyStories: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759 > In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities.
SomewhatLikely 3y ago
The general term for this idea is curriculum learning.

by valine 3y ago

two_in_one 3y ago

> I’m surprised we weren’t doing this already. Because they don't work very well in real life. Article doesn't have any experimental proof it works, saves training time, or size, or anything.. And it doesn't mention another promising approach: mix of experts. There are many ways of implementing it. I mean the general idea of specialized fragments of the NN which are selectively called. They don't have to be trained all at once.

tempusalaria 3y ago
It’s weird a DeepMind author is on this as they experimentally studies these methods in Gopher and found it was useless. Even the layer duplication trick they were using doesn’t work. I tried a refinement of that where you fine tuned the activations of the added layers to match the original activations which helped a bit relative to what DeepMind found but it still wasn’t material enough to be worth bothering with
- two_in_one 3y ago
  I tried adding layer. The conclusion is, it's useful if you cannot train the whole model at once. Then you can split it in 2 and train the first half, with embedding/deembedding layers. After it stops improving add the second half of the layers, and train only it, freezing everything else. This worked in my case, when model params number < 10% of the number of training tokens. Splitting model in 3 is significantly less effective. Don't know why, but the third part adds almost nothing in accuracy and a lot in training time. One thing where additive training works is fine tuning, like LoRA. But that's different. It's like poor man's way. For those who cannot train the whole model. Most likely uptrainig it whole is more efficient, just my guess.

jncraton 3y ago

You might be interested in TinyStories: https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759 > In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities.

SomewhatLikely 3y ago

The general term for this idea is curriculum learning.