3 ms·

From a modeling perspective, I am impressed with the effects of training on 2T tokens rather than 1T. Seems like this was able to get LLAMA v2 7b param models e

From a modeling perspective, I am impressed with the effects of training on 2T tokens rather than 1T. Seems like this was able to get LLAMA v2 7b param models equivalent to LLAMA v1's 13b performance, and the 13b similar to 30b. I wonder how far this can be scaled up - if it can, we can get powerful models on consumer GPUs that are easy to fine tune with QLORA. A RTX 4090 can serve an 8-bit quantized 13b parameter model or a 4-bit quantized 30b parameter model. Disclaimer - I work on Databricks' ML Platform and open LLMs are good for our business since we help customers fine-tune and serve.

brucethemoose2 3y ago
At some point, higher quality tokens will be far more important than more tokens. No telling how much junk is in that 2T. But I wonder if data augmentations could help? For instance, ask LLaMA 70B to reword everything in a dataset, and you can train over the same data multiple times without repeats.
- visarga 3y ago
  A great idea. If we are at it, why don't we search all topics and then summarise with a LLM? It would be like an AI made wikipedia 1000x times larger indexing all things, concepts and events, or a super knowledge graph. It would create a lot of training data, and maybe add a bit of introspection to the model - it explicitly knows what it knows. Could help reduce hallucinations, learn attribution, ability to recognise copyrighted content, and fact checking.
  - gaogao 3y ago
    I have this pet proposal that LLMs would be pretty nice to help fill out WikiData https://friend.computer/jekyll/update/2023/04/30/wikidata-llms.html https://friend.computer/jekyll/update/2023/04/30/wikidata-ll..., as the technique of getting LLMs to write queries, instead of directly giving data, has worked really well so far for me.
- joshhart 3y ago
  You are totally right - both more and better matters. There are many good papers on the importance of data quality, Textbooks Are All You Need is one that comes to mind - https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644

brucethemoose2 3y ago

At some point, higher quality tokens will be far more important than more tokens. No telling how much junk is in that 2T. But I wonder if data augmentations could help? For instance, ask LLaMA 70B to reword everything in a dataset, and you can train over the same data multiple times without repeats.

visarga 3y ago
A great idea. If we are at it, why don't we search all topics and then summarise with a LLM? It would be like an AI made wikipedia 1000x times larger indexing all things, concepts and events, or a super knowledge graph. It would create a lot of training data, and maybe add a bit of introspection to the model - it explicitly knows what it knows. Could help reduce hallucinations, learn attribution, ability to recognise copyrighted content, and fact checking.
- gaogao 3y ago
  I have this pet proposal that LLMs would be pretty nice to help fill out WikiData https://friend.computer/jekyll/update/2023/04/30/wikidata-llms.html https://friend.computer/jekyll/update/2023/04/30/wikidata-ll..., as the technique of getting LLMs to write queries, instead of directly giving data, has worked really well so far for me.
joshhart 3y ago
You are totally right - both more and better matters. There are many good papers on the importance of data quality, Textbooks Are All You Need is one that comes to mind - https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644