3 ms·
> Knowledge cutoff: August 2024. Could this mean training time is generally around 6 month, with 2 month of Q/A?
by clueless 2y ago
> Knowledge cutoff: August 2024.
Could this mean training time is generally around 6 month, with 2 month of Q/A?
- bertil 2y agoCouldn’t you gradually include more recent documents as you train?
- soulofmischief 2y agoThat makes it harder to analyze the results of training and draw conclusions for the next round.
- changoplatanero 2y agoYou can do that but the amount of incremental data will be negligible compared to the rest of the data. Think of the knowledge cutoff more like a soft value.
- nickysielicki 2y agoIt scales depending on the dataset you want exposure on and the compute you have available, so any specific time box is kind of meaningless if you don’t know the rest of the inputs that went into it. The llama 3 paper went into a lot of this and how these decisions were made (see section 3 and onward): https://ai.meta.com/research/publications/the-llama-3-herd-of-models/ https://ai.meta.com/research/publications/the-llama-3-herd-o... tl;dr: llama 3 was 54 days, but it’s more complicated than that.
- jhugg 2y agoI wish my knowledge cutoff was August 2024.
- steenandersson 2y agoThis made me LOL louder than I have for a long time! Agree.