7 ms·
Llama 3 8B is almost as good as Wizard 2 8x22B
- DalasNoin 3y agoI think it is interesting that llama 3 still does not use Mixture of Experts. Command R+ is also not using MoE. On paper you get 4x less training and inference compute cost. But maybe there is a good reason not to use it, this kind of supports this. Might be wasteful to create these huge MoE models if they can't outperform smaller dense models.
- samus 3y agoMany MoE models like Mixtral were not created from scratch. Somebody could still upcycle that 8B into an 8x8B. It should ideally be done by Meta since they have access to the original training data.
- CuriouslyC 3y agoMoE models are harder to fine tune, and they don't solve the biggest problem which is GPU memory use.
- fnands 3y agoLlama 3 8B basically answers the question of: "What happens if you train a small(-ish) model for a very long time". I think we've seen this trend start with some of the Mistral models going beyond the Chinchilla optimal point, and now with Llama 3 even further. 15T tokens for a 8B param model is a lot more than we've seen so far (for context, Llama 2 was 2T tokens), and it seems to be paying off. If anything, this release makes me excited for the quality of smaller models going forward.
- Tenoke 3y ago>If anything, this release makes me excited for the quality of smaller models going forward. While there's going to be more that can be milked out of them, we are already clearly fairly deep in the diminishing returns portion on small models.
- MyFirstSass 3y agoSeems like it when for 7b models: (but also wow things are moving fast!); Llama - 1.4 Trillion tokens - feb. 2023, Llama 2 - 2 Trillion tokens - july. 2023, Mistral - 8 Trillion tokens - sep. 2023 (this was the first big impressive leap where local models really became useful for chat) Llama 3 - 15 Trillion tokens So we went 1.4x, to 4x, to 2x. I wonder if there's even unused data out there? Haven't tried the new model enough to see how much better it is than Mistral, that will be the real SOTA test for now.
- Escapado 3y agoI wondered about the same thing but at the same time only about 5% of the training data is non-English and I would be surprised if the total amount of published text from all non English languages combined was also just 5% of all published text. So my intuition tells me there is still heaps of data but what might be tricky is to properly access and asses it’s quality. Also the redpyjama v2 dataset has 30T tokens and is based on common crawl. Now I don’t know too much about common crawl but I doubt it has in it all published scientific books in all the different languages as these are often not freely crawlable. I remember when I studied physics there were at least 20 different 200-800 page long books on particle physics in German alone in our campus library. That must amount to 5million token by itself, from just one niche of physics. The Hamburg public library hosts about 5 million books and 90 million scientific articles mostly in English and German. If the average length of a scientific article is 6000 tokens and the average book about 100000 then that alone is already 1 trillion token. I bet there are significantly larger libraries and this is before even crawling the internet and looking at other languages or even generating training data.
- sheepscreek 3y agoStill quite impressive to think these models are trained on 15x the content in a large public library. That is insane.
- vidarh 3y agoDeutsche Nationalbibliothek appears to have 43.2 million "items", of which apparently 17.3 million are books. If we assume ~60,000 tokens for an average book, which seems very conservative given average word length in German and a novel typically being considered anything above ~40k words), that's another trillion just for their books, so I'm guessing the total German language content available in major libraries will be many times that. E.g. the Norwegian National Library has somewhere between 3x and 10x as many tokens in Norwegian newspapers as in books (at one point I think GPT3 breakdown of training data by language surfaced, and the Norwegian data was a tiny fraction of what was available in the national library, even before trying to estimate online/digital content). While I'm sure there's overlap [1] between the languages, a lot of it will help translation, and I think even for smaller languages the ratio of local content seems to dwarf translations. E.g. the "bestsellers" from English, French, and German all get translated to Norwegian, but most of the "long tail" content is local. [1] I was tickled to a find one of my uncles represented in Deutsche Nationalbibliothek; he was a professor in statistics, so it was a translation of some of his research
- kleiba 3y agoIsn't overfitting the usual result of training a model for a long time?
- bjornsing 3y agoAs I understand it the Chinchilla “optimal point” is severely under-fitted. It’s optimal in the sense that if you only care about training cost it would have been better to make the model bigger and even more under-fitted. But clearly we care about inference cost too (or even primarily), so it makes sense to train for longer. Also, these models are trained on trillions of tokens, so I’m not sure an 8B model even can overfit.
- candiodari 3y ago1) obviously, when the people doing the inferencing are different from the ones doing training, training cost does not matter to inference. 2) Almost, Chinchilla concerns itself with minimizing cost(training) + cost(inference). The expectation most people have is that there's an insane amount of inference compute, and training is maybe 1% of that. The point of the Chinchilla paper is that that's not true, training uses such insane amounts of compute that despite all model inference by half the internet for a year or two is a huge amount of compute, training is still a very decent percentage of that. I believe in one of the examples they pointed out that even the whole internet inferring with a model for years was still only 20% of the cost of training that model. People expect it works like compilers, that making a compiler produce 1% faster code is worth 50 highly-paid SWEs because while an individual program run isn't exactly expensive, the time and resources spent running programs is astronomically larger than the time and resources spent developing compilers. The thing is most of the optimizations we know don't work during training. You can't quantize, you can't MoE (well, you can, obviously, but it doesn't save any training computation. In fact it increases training cost) At 50-50, having a 10% cheaper-to-train model justifies 10% more expensive inference. 3) Combining both arguments ... at this point people should probably realize that Facebook's LLama is really an attack on Google (which is at least partially working, elon musk is tweeting about it) If Facebook really doesn't care about AI (or ... about as much as, say, netflix does. Not zero, but as long as they beat reddit's efforts they feel very comfortable), but Zuck does care about destroying Google, the calculus changes. Zuckerberg may not want the best possible AI, he may want as many scammers as possible trying to Game the Google search quality team, to present them with challenges faster than they can adapt. Then the training cost becomes a moot point. Hmmm, I should send my CV to meta ...
- machiaweliczny 3y agoThere's still lot's of low hanging fruit in data preparation for these models it seems. I wonder if we will reach insane quality when we will be able to allow models to generate not exact same text but something similar (thus allowing for more compression). I think math leans a lot to this type of training as one can super easily verify if generated result is fine or not. We probably should do similar thing for language when other model judges if generation is ok or not (but doesn't require exact generation). This probably can be done as fine tuning? Anyone know if this has been tested already?
- threatripper 3y agoCould this be implemented by making the loss zero if the output for the expected token is above some threshold? E.g. if the threshold is 0.1 it can output 10 likely tokens with 0.1 probability each and if any of them is correct it will not modify the weights.
- wongarsu 3y agoOne issue is that this doesn't punish the model of the top prediction produces complete nonsense. I can imagine getting good results when fine tuning an existing model though
- wongarsu 3y agoSo something like having a smaller model generate embeddings, and use the embedding distance between the predicted text and the expected text for the loss computation? That sounds like it would be incredibly expensive. But maybe it allows you to fine tune on less data, offsetting the cost a bit (and allowing you to fine tune on just your very best data)
- zitterbewegung 3y agoYou could derive similar data by looking at the embedding and then synthesize similar data and then add it to the training dataset. Right now fine tuning a model before synthesizing the similar data could be done.
- 3y ago
- deleted 3y ago[deleted]
- freehorse 3y agoI wonder if quantisation in these models actually reduces the quality more than the less overtrained ones we were used to? Anybody knows if that could be the case?
- WithinReason 3y agoThat would make sense
- imjonse 3y agoToo bad they did not train or release a Llama 3 2B to see how it fares against Phi and Gemma.
- cchance 3y agoim surprised heirs no 14b or 16b
- Aissen 3y ago> very long time On one of Meta's 24k H100 clusters running at 95% efficiency, it's 2.3 days.
- WithinReason 3y agoIt was closer to 40% efficiency AFAIK
- Aissen 3y agoI'm curious, do you have any pointers? This is what this article mentions: > Those improvements resulted in an overall effective training time of more than 95% https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/
- logicchains 3y agoThe known SOTA for GPU flops utilisation for training on that many GPUs is somewhere between 50-60%, e.g. https://github.com/NVIDIA/Megatron-LM https://github.com/NVIDIA/Megatron-LM . If they really managed to get 95%, it's a huge advance in the state of the art. I guess by training time they meant with respect to downtime of the hardware, not utilisation of the hardware potential.
- Aissen 3y ago> I guess by training time they meant with respect to downtime of the hardware, not utilisation of the hardware potential. That was my understanding as well, those are two different levels of efficiency.
- WithinReason 3y agoThat means that 5% of the time GPUs were not doing any training, e.g. the server was busy with checkpointing so the training paused. 40% is according to Karpathy: https://twitter.com/karpathy/status/1781028605709234613 https://twitter.com/karpathy/status/1781028605709234613
- jasonjmcghee 3y agoZuck mentioned they also cut entire classes of data from training Llama 2 that they included in Llama 3 as Llama 2 was intended to be used for social / meta related services. So things like code were almost entirely skipped. He only mentioned code, but wouldn't be surprised if math etc was treated similarly. My assumption here is adding these missing subject matter areas had a larger impact than raw token counts.
- UrineSqueegee 3y agoI am using 8b-instruct and I am honestly blown away, this almost feels like early GPT4 levels of logic I am absolutely pumped for whats in store for small models!
- kromem 3y agoI dunno, I'm less than dazzled. One of my gotos is a variation of a classic logic puzzle, as models naturally tend to try to provide the original form of the solution because of token similarities and by seeing how well they can identify where steps are wrong or seeing if they can solve the variation when breaking token similarities with representative symbols or other techniques like mapping out object relationships in the initial answer before providing a solution it tends to reveal the underlying capacity for logic outside of simply similarities to the training data. Mistral, GPT-4, Claude, Gemini can all solve it with various degrees of competency and hand holding. Llama 3 just cannot handle it. I think what we're seeing is that the massive training data set is leading to a much more capable stochastic parrot but not necessarily as capable world modeling. Or else there's an advantage to MoE for this kind of task that Llama 3 lacks. But it's tracking with my subjective sense of being somewhere between GPT-3 and 4 in terms of "critical thinking" ability, but not in the same class as more modern models, even though it seems to have a better breadth of training to draw on than most other medium sized models.
- MyFirstSass 3y agoBoth GPT-4 and llama 3 8b fail actual reasoning still: jack has 15 sisters, each sister has 3 brothers. How many brothers does jack have? (real answer is 2) ------------------- Llama 3 8b Let's break it down step by step: Jack has 15 sisters. Each of his sisters has 3 brothers. To find the total number of brothers, we need to multiply the number of sisters (15) by the number of brothers each sister has (3). 15 sisters x 3 brothers/sister = 45 brothers So, Jack has 45 brothers! ------------------- GPT-4 Jack has 3 brothers. Here's why: If each of Jack's sisters has 3 brothers, these brothers are the same for all sisters, including Jack himself. Therefore, Jack and his 3 brothers make up the total count of brothers in the family. Since the problem states that each sister has 3 brothers, we know that Jack himself must be counted as one of these brothers. So, Jack has 3 brothers. ------------------- Lol. You can tweak the questions numbers and names to avoid training data. Can Opus solve this?
- toxypol 3y agoit is a huge jump, testing for last hours
- tosh 3y agoI wonder to what degree these results come from more training and emergent capability from that vs these kinds of riddles now being known and well represented in training material? Adjacent: one thing that makes llama3 special is that they spent extra effort on deduplicating training data (quality) + increased training data (quantity) and used 4x more code in training material (=> good for reasoning). So perhaps we can still see huge capability gains in these small models by spending more effort on training data curation and generation?
- wrasee 3y agoExactly this. I only casually follow this space and I recognised many of those questions as essentially ‘classic’ questions by now. Surely any recently trained model taking recent data is going to have a ton of references to these now, especially those that are not officially part of some test set that might have been controlled for to some extent?
- nl 3y agoOne thing I hadn't really thought through is that the reason these models are trained to point X is that they just decide to release it then. But most of these companies _keep training_ the same model. A lot of the time there is no "ok we are done, we will do a new thing now" - it just keeps going. Obviously thinking this though it explains a lot of the vast spend on GPUs. They need new compute because the existing compute is occupied _and always will be_. Model training will never finish!
- petesergeant 3y ago> But most of these companies _keep training_ the same model Do they? Genuine question. Why do you think so?
- sva_ 3y agoIf you look at the number of different gpt4 models, it seems like a reasonable thing to say: https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-4 https://platform.openai.com/docs/models/gpt-4-turbo-and-gpt-... While it appears like they all share the same gpt4 base, it isn't really known how these are trained (fine-tuned?)
- JKCalhoun 3y agoHow many models do they have? Is model availability becoming the bottleneck?
- idle_zealot 3y agoSay I wanted to run this on my phone today. What's the best path one could take to actually use it as mkre than a toy? I suspect step 1 is to get llama.cpp running in Termux and a ggml or other model file loaded. That gets you a local web interface for inference. Then... how do you hook that up to your system such that it can do anything useful?
- moffkalast 3y agoMLC might be a decent option, I think they have a way to deploy for mobile and might actually compile for the NPU.
- maksimur 3y agoAnnoying that I have to wait to get accepted before seeing details about the files. I'm not planning to download them but was just interested in knowing their size.
- syntaxing 3y agoAs in Llama3 8B? You can see the size of the files from Ollama using the FP16 one https://ollama.com/library/llama3:8b-text-fp16 https://ollama.com/library/llama3:8b-text-fp16
- miven 3y agoAny guesses as to why they bumped the parameter count up from 7B to 8B?
- matrix2596 3y agoI also wondered the same and check the model configs. they are using bigger vocab size and the intermediate size of fully connected layer seems to be bigger.
- alecco 3y agoI wonder how did they get the 15T tokens for training.
- rvnx 3y agoIllegally acquired content on the internet, like search engines do
- deleted 3y ago[deleted]
- pytness 3y agoSeems like a yapper. And it makes some wrong(?) assumptions? >>> hi Hi! It's nice to meet you. Is there something I can help you with, or would you like to chat? >>> if tilt my head upwards so my face is looking at the sky and I spit, what will happen? That's an interesting question! If you tilt your head upwards so your face is looking at the sky and then spit, a few things might happen: 1. **Spit will go flying upwards**: Since your face is now pointing towards the sky, the trajectory of your spit will be upward too. It might even reach the clouds (if there are any nearby) or disperse in mid-air. 2. **Some spittle might fall back down**: Depending on air currents, wind direction, and the distance between you and the ground, some of the spittle might fall back down towards the earth. 3. **You might get a bit of a face-full**: If you're not careful, some of that spit might splash back onto your own face, which could be... interesting. Remember to be mindful of where you are and who's around when doing this experiment. It's always good to clean up after yourself and respect those nearby! Now, I'm curious – what inspired this question? EDIT: >>> How would the spit reach the clouds? I was getting a bit carried away with the imagination there. In reality, it's highly unlikely that your spit would actually reach the clouds. Clouds are typically several thousand feet (or meters) above ground level, and even if you were to tilt your head at an extreme angle, the spittle would likely be deflected or absorbed by air currents before reaching such heights. In practice, the spittle would probably fall back down towards the earth due to gravity, and might land on the ground, a nearby surface, or even get blown away by the wind.
- pennomi 3y agoI’ve also noticed it’s “overactive imagination“ on some questions.
- andrewstuart 3y agoWeird all these measures of what makes a good AI. The only really great one is OpenAI ChatGPT 3.5
- saghul 3y agoKinda surprised the 8B gets this wrong: "what's heavier a kilo of steel or two kilos of feathers?" GPT-3.5 gets it wrong too. The 70B model does get it right, so does GPT-4.
- wongarsu 3y agoThat seems to be the general experience. Maybe 8B are just too few parameters to achieve higher level reasoning.
- brrrrrm 3y agoMaybe depth rather than parameter count.
- pennomi 3y agoMy pet question is “Which weighs more, 1000cm^3 of styrofoam or 1cm^3 of tungsten?” Most LLMs go through the calculation and find the styrofoam is heavier, then confidently announces that the tungsten weighs more. Strange considering it’ll say something very nearly like “The styrofoam weighs 50 g and the tungsten weighs 19.3 g, therefore the tungsten is heavier.”
- fransje26 3y agoThat's not how it responded to my query. > What's heavier? 1 kg of lead or 2 kg of feathers? That's a classic trick question! The answer is: 2 kg of feathers. Why? Because 2 kg is heavier than 1 kg, regardless of the material. The density of the material doesn't matter in this case, only the weight. So, 2 kg of feathers would weigh more than 1 kg of lead.
- fransje26 3y agoI stand corrected. I was inadvertently on the 70b model.
- syntaxing 3y agoThe biggest issue when you have local LLMs is that people “use” them in different way which can cause different impression. As a somewhat PSA, try loading the Q6 KM if possible. Q5 KM at a minimum. Anything lower like the default of ollama Q4_0 will have noticeable quality degradation.
- earth2mars 3y agoExcept GPT-4 no other model answered following correctly. Why Name five countries that starts and end with same letter
- kennyadam 3y agoGemini Advanced was able to answer that no problem: https://g.co/gemini/share/fee3410e018a https://g.co/gemini/share/fee3410e018a
- furyofantares 3y agoThey operate on tokens and so are particularly bad at knowing what letters are in words beyond the first letter or so.
- ijustlovemath 3y agoOpus did pretty well with this slight modification: write a self contained Python program (no I/O, no dependencies) that prints a list of all countries that start and end in the same letter. Heres the output (which looks correct!) ['Albania', 'Algeria', 'Andorra', 'Angola', 'Antigua and Barbuda', 'Argentina', 'Armenia', 'Australia', 'Austria', 'Central African Republic', 'Czech Republic', 'Saint Kitts and Nevis', 'Saint Vincent and the Grenadines', 'Seychelles', 'Solomon Islands']
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- dimask 3y ago> I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river? > You now have 34 apples! Actually, it forgot to add one coin when the person lost 4 apples, and the apples should be 34+2=36 in the end, so the model got it wrong. I tried on phind.com with their 70B model and it actually got it correct. Still, quite impressive for a 8B model, but it does show again the problems that come with trusting such problems to LLMs, that actually finding mistakes is so hard because it all sounds correct and confidently well written.
- exe34 3y agoI tried chatgpt: > Let's break down the events step by step: > > Starting Apples: 10 apples > Find 3 Gold Coins: > +3 gold coins > Total apples: 10 > ... I keep thinking about how to reliably make it come up with code to do the job correctly. edit: oh I think chatgpt did it. https://chat.openai.com/share/f1742187-ab06-46c7-abaf-9a7981138f06 https://chat.openai.com/share/f1742187-ab06-46c7-abaf-9a7981...
- isusmelj 3y agoDon't forget that this is 24k H100. They are getting 10x the compute: https://www.cnbc.com/2024/01/18/mark-zuckerberg-indicates-meta-is-spending-billions-on-nvidia-ai-chips.html https://www.cnbc.com/2024/01/18/mark-zuckerberg-indicates-me... So gpt-4 level 8B models running on phones and notebooks seems feasible within the next 5 years. I imaging having (voice) assistans running locally. Crazy how fast we progress.
- 0DayFlaw 3y agoStill lags behind article summarization vs the leaders. Not the 70b or the 8b were able to produce consistent results for the below prompt Prepare a threat intelligence advisory based on below article, provide following sections" summary (maximum 300 words), technologies affected, TTPs (if any) and TA groups (if any), vulnerabilities exploited (if any) and recommendations for mitigation and detection. https://www.bleepingcomputer.com/news/security/mitre-says-state-hackers-breached-its-network-via-ivanti-zero-days/ https://www.bleepingcomputer.com/news/security/mitre-says-st...