5 ms·

15 million GPU hours is interesting, llama 2 had about 1.7 million hours for the 70B (3.3M for all). Their GPU hours are on LUMI which is AMD MI250 GPU, which M

by huac 3y ago

15 million GPU hours is interesting, llama 2 had about 1.7 million hours for the 70B (3.3M for all). Their GPU hours are on LUMI which is AMD MI250 GPU, which Mosaic reports is about 75% of the A100 (https://www.mosaicml.com/blog/amd-mi250 https://www.mosaicml.com/blog/amd-mi250) but AFAIK hasn't been tested at this kind of scale. so, validating if the MI250 in a cluster can effectively train a large scale LLM will be useful. not that it really matters to any other group, you won't get these in the cloud anytime soon. either way - say that they have about 10M A100 hours. if they use half their budget for the biggest model as FB did with Llama (rest on testing / smaller models as they scale), then they'll budget 5M A100 hours for the biggest model, or 3x llama-2's 1.7M A100 hours (2k A100 for 21 days). However, why wouldn't Llama-3 also be 3x bigger than Llama-2? FB's H100's are coming online; they apparently have 4k H100 in a cluster (https://www.nextplatform.com/2023/09/26/meta-platforms-is-determined-to-make-ethernet-work-for-ai/ https://www.nextplatform.com/2023/09/26/meta-platforms-is-de...), and each H100 is about 3x the speed of an A100 (https://www.mosaicml.com/blog/coreweave-nvidia-h100-part-1 https://www.mosaicml.com/blog/coreweave-nvidia-h100-part-1). So if they provided the same 'calendar time' of 21 days over a single cluster of 4096 H100, that would be about 6.2M A100 hours for a 70B model. there is an entirely separate question of, LLM's are functions of compute _and_ data, will they also find enough high quality _and_ kosher tokens in their low resource languages?

vidarh 3y ago
If this gets enough of an official stamp of approval from the EU, then access to enough tokens in smaller languages may not be a big issue. For that matter, even for OpenAI it's likely largely down to how much they care to negotiate access. E.g. for Norwegian the Norwegian national library and state archives sits on at least a couple of magnitudes more of digitized material in Norwegian than OpenAI appears to have found Norwegian material for GPT3. How much exactly would depend on willingness to give access to still in-Copyright data, but it'd still be far more even if you only give access to what is currently on their website. And ChatGPT is well versed enough in Norwegian even without that to be able to convincingly use several regional dialects that are rarely used in writing (I've tested several) so it won't take all that much. But frankly being able to ask it about the contents of these archives would be amazing because they include a couple of centuries of newspapers and almost every book ever published in Norwegian. The level of digitization will differ, and Norway was early with that, but most countries have depositary requirements for printed materials, and countries with smaller languages like are often more anal about them...
- crakenzak 3y ago
  Does training an LLM on a variety of different languages improve its reasoning and understanding in other languages? For example, if GPT-5 (or whatever next gen model) was trained on a staggering amount of high quality Norwegian tokens that talk about medicine, city life, romance, maths, etc... would it be able to apply those concepts & learnings to other languages? I'm trilingual and have ~100% of transfer of knowledge/reading I learn/read in one language into the others, curious to see if LLMs are like that.
  - vidarh 3y ago
    I think how well it handles Norwegian despite the small number of Norwegian tokens present for GPT3 (I'm not aware of up to date overviews of token distributions in the training data) is a good indicator that a lot of it does transfer.
    - huac 3y ago
      the 'low resource languages jailbreak gpt-4' paper (https://arxiv.org/abs/2310.02446v1 https://arxiv.org/abs/2310.02446v1) shows that the models, when prompted in a low resource language, return highly coherent and sophisticated responses. it's possible that RLHF does not do a great job here but the base models are clearly cross-applying intelligence from high resource languages to low resource languages.

by huac 3y ago

vidarh 3y ago

If this gets enough of an official stamp of approval from the EU, then access to enough tokens in smaller languages may not be a big issue. For that matter, even for OpenAI it's likely largely down to how much they care to negotiate access. E.g. for Norwegian the Norwegian national library and state archives sits on at least a couple of magnitudes more of digitized material in Norwegian than OpenAI appears to have found Norwegian material for GPT3. How much exactly would depend on willingness to give access to still in-Copyright data, but it'd still be far more even if you only give access to what is currently on their website. And ChatGPT is well versed enough in Norwegian even without that to be able to convincingly use several regional dialects that are rarely used in writing (I've tested several) so it won't take all that much. But frankly being able to ask it about the contents of these archives would be amazing because they include a couple of centuries of newspapers and almost every book ever published in Norwegian. The level of digitization will differ, and Norway was early with that, but most countries have depositary requirements for printed materials, and countries with smaller languages like are often more anal about them...

crakenzak 3y ago
Does training an LLM on a variety of different languages improve its reasoning and understanding in other languages? For example, if GPT-5 (or whatever next gen model) was trained on a staggering amount of high quality Norwegian tokens that talk about medicine, city life, romance, maths, etc... would it be able to apply those concepts & learnings to other languages? I'm trilingual and have ~100% of transfer of knowledge/reading I learn/read in one language into the others, curious to see if LLMs are like that.
- vidarh 3y ago
  I think how well it handles Norwegian despite the small number of Norwegian tokens present for GPT3 (I'm not aware of up to date overviews of token distributions in the training data) is a good indicator that a lot of it does transfer.
  - huac 3y ago
    the 'low resource languages jailbreak gpt-4' paper (https://arxiv.org/abs/2310.02446v1 https://arxiv.org/abs/2310.02446v1) shows that the models, when prompted in a low resource language, return highly coherent and sophisticated responses. it's possible that RLHF does not do a great job here but the base models are clearly cross-applying intelligence from high resource languages to low resource languages.