4 ms·
Rule of thumb is that you need ~20 tokens per parameter. The average token size is ~4 characters, probably more for larger models where you want larger dictiona
by EvgeniyZh 3y ago
Rule of thumb is that you need ~20 tokens per parameter. The average token size is ~4 characters, probably more for larger models where you want larger dictionary, but for simplicity I'll say it's 5 bytes to make numbers round. So you need 100 bytes of text data per parameter, or 10 PB for 100T model. Now, recent research says that you can reuse the same data like 4 times before it becomes hindering performance but it doesn't help much in our case.
But in this case what is really ridiculous is the compute requirement. The required compute for optimal model growth roughly quadratically (both your model and your data grow linearly). So for 100T model you need 1e30 FLOPs. This machine gives you 1e18 FLOPs per second. It will take 30k years to train this model on one of these (or 30k of these to train it in a year, but then utilization will start kicking in).
- samstave 3y agoI am not going to be embarrassed for the following Q ; Please ELI5 where I can have a glossary of AI/ML terms - where do I get fluency in speaking about Tokens, Models, Training, Parameters, etc... Please dont be Snarky - This is info that everyone younger than I am needs as well. Is there a Canon? Where is it?
- EvgeniyZh 3y agoI have no idea tbh. I learned these a while ago (~7 years ago), and the materials I used then are heavily outdated and also I won't be able to remember what they were. I guess any intro course to deep learning should talk about these. Stanford ones used to be good. Maybe someone else can be more useful about it.
- rocmcd 3y agoI'd recommend starting here: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/ https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-... It's pretty lengthy but doesn't require a PhD to understand. If you can get to the end of it you'll have a much better understanding of what's going on.
- eamsen 3y agoAt the risk of sounding snarky, https://chat.openai.com https://chat.openai.com would be a good introduction, followed by books, which GPT could recommend.
- Crye 3y agoTo be honest, I'd start with some introduction to Transformer YouTube videos. They'll cover a lot of these terms and you'll then have a better understanding to find additional resources.
- causalmodels 3y ago> Rule of thumb is that you need ~20 tokens per parameter. That rule of thumb is wrong. The chinchilla paper has it anywhere between 1 and 100 tokens per parameter.
- moffkalast 3y ago"The best time to start training a 100T param model was 30k years ago, the second best time is now."
- EvgeniyZh 3y agoProbably the best time to start train 100T model is never
- samstave 3y agoWhat if you could train an AI with a desired outcome to their answers? I.E. ; "answer this question where the outcome is the most beneficial to quality of life"
- moffkalast 3y ago> life A strange game. The only winning move is not to play. unplugs self
- chromoblob 3y agoA properly designed AI agent would do exactly that.
- EvgeniyZh 3y agoI'll take your question further: what if we have unlimited data (say some crazy rich RL environment or way to produce high quality and diverse synthetic data)? You still have to get these 1e30 FLOPs. Lets say you can connect 100 of these bad boys together with 40% utilization, with total 4e19 FLOPs/s. Assume also Moore's law keeps working indefinitely. When should we start training 100T model on it to get is as early as possible? We wait x years and the start training on machine with 4e19*2^(x/2) FLOPs/s. Turns out the answers is ~16 years, after which we'll have 1e22 FLOPs/s and 1e30 FLOPs will take another 3 years.
- rowanG077 3y agoThat's obviously false under the assumption computing power will increase as it has in the past.
- deleted 3y ago[deleted]