4 ms·

Nice catch! We sampled the github dataset to match the total # tokens seen by LLaMA during training: ~64B tokens (they only pass through 0.64 of their total Git

by csris 3y ago

Nice catch! We sampled the github dataset to match the total # tokens seen by LLaMA during training: ~64B tokens (they only pass through 0.64 of their total Github dataset according to the paper). We have a lot of Github data and will make them available soon. Note, we also have not built this for compute optimal training. We are following LLaMA's lead and are training on more data for longer to optimize for quality, not compute.

doctoboggan 3y ago
Very good to hear that you are optimizing for inference rather than training. I’ve tried llama and its various instruction tuned siblings and have yet to get equivalent performance to gpt-3.5 on coding tasks. Seeing how the base model performed relative to gpt-3 on the various benchmarks gives me hope that the difference is just in RLHF or other fine tuning steps. I really hope the community is able to get there, Especially if the resulting model is able to be quantized with minimal loss.
rwl4 3y ago
I wonder if it would make sense to create tokens for each emoji so they don't have to be multi-token. Especially considering people have experimented with using them for makeshift compression.
kir-gadjello 3y ago
Thank you for developing the pipeline and amassing considerable compute for gathering and preprocessing this dataset! I'm not sure if this is the right place to ask about this, but could you consider training an LLM using a more advanced, sparse transformer architecture (specifically, "Terraformer" from this paper https://arxiv.org/abs/2111.12763 https://arxiv.org/abs/2111.12763 and this codebase https://github.com/google/trax/blob/master/trax/models/research/terraformer.py https://github.com/google/trax/blob/master/trax/models/resea... by Google Brain and OpenAI)? I understand the pressure to focus on training a straightforward LLaMA replication, but of course you see that it's a legacy dense architecture which limits its inference performance. This new architecture is not just an academic curiosity but is already validated at scale by Google, providing 10x+ inference performance boost on the same hardware. Frankly, the community's compute budget - for training and for inference - isn't infinite, and neither is the public's interest in models that do not have advantage (at least in convenience) over closed-source ones; and so we should utilize both those resources as efficiently as possible. It could be a big step forward if you trained at least LLaMA-Terraformer-7B and 13B foundation models on the whole dataset.

by csris 3y ago

doctoboggan 3y ago

Very good to hear that you are optimizing for inference rather than training. I’ve tried llama and its various instruction tuned siblings and have yet to get equivalent performance to gpt-3.5 on coding tasks. Seeing how the base model performed relative to gpt-3 on the various benchmarks gives me hope that the difference is just in RLHF or other fine tuning steps. I really hope the community is able to get there, Especially if the resulting model is able to be quantized with minimal loss.

rwl4 3y ago

I wonder if it would make sense to create tokens for each emoji so they don't have to be multi-token. Especially considering people have experimented with using them for makeshift compression.

kir-gadjello 3y ago

Thank you for developing the pipeline and amassing considerable compute for gathering and preprocessing this dataset! I'm not sure if this is the right place to ask about this, but could you consider training an LLM using a more advanced, sparse transformer architecture (specifically, "Terraformer" from this paper https://arxiv.org/abs/2111.12763 https://arxiv.org/abs/2111.12763 and this codebase https://github.com/google/trax/blob/master/trax/models/research/terraformer.py https://github.com/google/trax/blob/master/trax/models/resea... by Google Brain and OpenAI)? I understand the pressure to focus on training a straightforward LLaMA replication, but of course you see that it's a legacy dense architecture which limits its inference performance. This new architecture is not just an academic curiosity but is already validated at scale by Google, providing 10x+ inference performance boost on the same hardware. Frankly, the community's compute budget - for training and for inference - isn't infinite, and neither is the public's interest in models that do not have advantage (at least in convenience) over closed-source ones; and so we should utilize both those resources as efficiently as possible. It could be a big step forward if you trained at least LLaMA-Terraformer-7B and 13B foundation models on the whole dataset.