3 ms·
>My next goal is to reduce 7B llama2 to 10-100M without making it much dumber. That is going to be hard as the 7B model was trained on 2T tokens. Maybe if you
by GaggiX 3y ago
>My next goal is to reduce 7B llama2 to 10-100M without making it much dumber.
That is going to be hard as the 7B model was trained on 2T tokens. Maybe if you heavily restrict the range in which the model should operate.
- ljlolel 3y ago1. It’s faster and cheaper to train a smaller model 2. Better than tokens is to train on probability distributions (distillation) and trees of probability distributions
- nickpsecurity 3y agoI've never seen anything about training on probability distributions or trees of them. Do you have articles with examples you could share with us? I did try a quick search for it. Found some interesting papers. The links to them are below in case anyone finds them interesting. https://arxiv.org/abs/2212.11481 https://arxiv.org/abs/2212.11481 https://towardsdatascience.com/a-new-way-to-predict-probability-distributions-e7258349f464 https://towardsdatascience.com/a-new-way-to-predict-probabil... https://arxiv.org/pdf/1912.07913.pdf https://arxiv.org/pdf/1912.07913.pdf https://dukespace.lib.duke.edu/dspace/bitstream/handle/10161/25822/Awaya_duke_0066D_16951.pdf?sequence=1 https://dukespace.lib.duke.edu/dspace/bitstream/handle/10161...