5 ms·
GPT-4 was rumored to be trained on 13 trillion tokens. https://www.kdnuggets.com/2023/07/gpt4-details-leaked.html https://www.kdnuggets.com/2023/07/gpt4-details
by binarymax 1y ago
GPT-4 was rumored to be trained on 13 trillion tokens. https://www.kdnuggets.com/2023/07/gpt4-details-leaked.html https://www.kdnuggets.com/2023/07/gpt4-details-leaked.html
Not sure where you’re getting the 45Gb number.
Also, Google doesn’t use GPT-4 for summaries. They use a custom version of their Gemini model family.
- acoustics 1y agoThe number of tokens trained on is separate from the model size. Gemma 3 270M was trained on 6 trillion tokens but can be loaded into a few hundred million bytes of memory. But yeah GPT-4 is certainly way bigger than 45GB.
- jerf 1y agoIt wouldn't matter if they trained on a quadrillion tokens, or another ten orders of magnitude. There's only so much information you call stuff into a given set of numbers. But once again I am reminded, never make arguments based on information theory. Nobody understands it.