4 ms·
So can you take this: (...) experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and w
by mirekrusin 3y ago
So can you take this:
(...) experiments suggest that having abundant long texts in the pretrain dataset is not the key to achieving strong performance, and we empirically verify that long context continual pretraining is more efficient and similarly effective compared to pretraining from scratch with long sequences (...)
...to the extreme and arrange training into passes with increasing context length and decreasing number of samples?
- tysam_and 3y agoYes, empirically I've noticed that if you do not exchange that sequence length gain for huge batch sizes up front, however, your performance will go down overall....
- mirekrusin 3y agoI'd imagine performance will suffer but how does it relate to gains in overall training costs? Ie. with same compute budget, which approach would produce better overall performance? Expanding on this idea, do you think it makes sense to explore what could be called "bootstrapping phase" or "progressive training" for foundational model training: - starting with small number of weights that are being increased with further training - arranging training data with basics first - short sentences, logic, grammar, arithmetic, naive knowledge ("bear is animal" etc) that increases in complexity as training progresses. - increasing context length - ideally implicit, based on increasing sample sizes