3 ms·
Is it a common practice in LLMs to give different weights to different training data sources? For instance I might want to say that all training data that come
by galaxyLogic 2y ago
Is it a common practice in LLMs to give different weights to different training data sources?
For instance I might want to say that all training data that comes from my inhouse emails take precedence over anything that comes from the internet?
- sebzim4500 2y agoYes it is. IIRC back when OpenAI was open and they published the breakdown they were significantly overweighting wikipedia.
- Tenoke 2y agoYes, though it's not about taking precedence but about sampling frequency. So for example, if you have 1 GB emails, 10 GB external data, you can instead sample your emails twice as much and effectively change the ratio of what the model was trained on from 1:10 to 2:10.