8 ms·
JetMoE: Reaching LLaMA2 performance with 0.1M dollars
- born-jre 3y agoTrain a decent model, but market with wrong emphasis. Discussion derailed by people doing napkin maths of how it doesn't make sense than actual model discussion. Classic marketing mistake, classic HN response.
- helloericsf 3y agoX thread: https://x.com/qinzytech/status/1775916338822709755?s=20 https://x.com/qinzytech/status/1775916338822709755?s=20
- plufz 3y agoYou’ve been in tech for too long when 1 million USD is your smallest unit.
- deleted 3y ago[deleted]
- noodlesUK 3y agoI wonder why they decided to call it 0.1M USD rather than 100k USD. For many of us, a million dollars is a large amount of money, even for a business.
- mattlondon 3y agoSame reason things are x.99 - the 0.1 decimal "feels" smaller than seeing 100,000 - "holy fuck!" Etc
- plufz 3y agoI’m sure they had their reasons, but all I can see is the Simpsons meme with Mr Burns at the ATM saying “What’s the smallest amount of money I can think of? A thousand dollars.” ;)
- IshKebab 3y agoIt's to imply that it costs other people in the millions, but they did it for only 0.1 million, which is a small number of millions. Just a rhetorical trick.
- oceanplexian 3y ago100k isn’t worth anywhere near what it used to due to inflation. It might get you a nice pickup truck or a kitchen remodel. If your business is doing research and can’t spend then it’s more of a hobby than a business.
- uptownfunk 3y agoWell, it's interesting to think about how much has been invested into BigModel companies (Anthropic, Perplexity, OpenAI) when it's very rapidly becoming commoditized.
- ipsum2 3y agoI'm skeptical, expect data contamination was the reason for high benchmark scores.
- hiddencost 3y agoYeah. IBM especially has a history of fudging the numbers on reports like this. Research puts together reports which are aggressively p-hacked and ensembled and overfit, and then sales uses those reports to boondoggle clients into using IBM.
- throwitaway222 3y agoThis stuff is just going to keep getting pushed down.
- lolinder 3y ago> JetMoE-8B is trained with less than $ 0.1 million1 cost but outperforms LLaMA2-7B from Meta AI, who has multi-billion-dollar training resources. LLM training can be much cheaper than people generally thought. They want you to read this as "we spent $100k compared to Meta's spending billions", but that's not actually what this says. It says that they spent $100k and Meta has the resources to spend billions if they wanted to. We don't know what Facebook spent on training LLaMA 2, but they say that it took them 184320 A100-80GB GPU-hours to train the 7B model [0]. AWS charges $14.46/hour for an instance that has 8 of those [1], which amounts to $1.81/GPU/hr. At that rate and assuming they paid something resembling AWS's list price, LLaMA 2 7B cost ~$333k. That's more than $100k, but not by orders of magnitude, and it's likely that Facebook wasn't paying the full price AWS is charging today. [0] https://github.com/meta-llama/llama/blob/main/MODEL_CARD.md#hardware-and-software https://github.com/meta-llama/llama/blob/main/MODEL_CARD.md#... [1] https://aws.amazon.com/ec2/instance-types/p4/ https://aws.amazon.com/ec2/instance-types/p4/
- rvba 3y agoTraining data costs zero? What is the training data anwyay? Books? (Is it legal to use non public domain, wait whom am I kidding) Reddit posts scrapped from the site? (Without clean up there are lots of bad subreddits) Wikipedia? Of course I understand why labour costs + employee cost related costs are ignored.
- samus 3y agoIt costs more the more you care about squeaky clean training data. Of course you get a better model in return. ChatGPT used a crawl of the internet and patches things up with alignment and DPO. Big boys like Microsoft might have deals with publishers to get textbooks in bulk. Contents from sites with moderation can be filtered using the platform's mechanism, e.g., only include text with a certain length and count of upvotes. LLMs can be used to generate and filter data as well. Humans have been used to do this, they might have to do this less in the future. Mostly to review what the LLMs are suggesting.
- nuz 3y agoMeta has their own data centers so they definitely didn't pay the equivalent to what AWS costs
- tosh 3y agoAnyone got a ballpark figure for what Meta spent on Llama 2 training for the 7B model?
- deleted 3y ago[deleted]
- davidcollantes 3y agoCan't wait for the GGUF to play with it. I tried the demo (https://www.lepton.ai/playground/chat?model=jetmoe-8b-chat https://www.lepton.ai/playground/chat?model=jetmoe-8b-chat), and the results were very good!
- antimatter15 3y agoIt looks like Llama 2 7B took 184,320 A100-80GB GPU-hours to train[1]. This one says it used a 96×H100 GPU cluster for 2 weeks, for 32,256 hours. That's 17.5% of the number of hours, but H100s are faster than A100s [2] and FP16/bfloat16 performance is ~3x better. If they had tried to replicate Llama 2 identically with their hardware setup, it'd cost a little bit less than twice their MoE model. [1] https://github.com/meta-llama/llama/blob/main/MODEL_CARD.md#hardware-and-software https://github.com/meta-llama/llama/blob/main/MODEL_CARD.md#... [2] https://blog.ori.co/choosing-between-nvidia-h100-vs-a100-performance-and-costs-considerations https://blog.ori.co/choosing-between-nvidia-h100-vs-a100-per...
- anizan 3y agoThey mention the cost was ~80,000k USD so for 32,256 hours it comes to ~2.48$ an hour. Amazing how cost effective the compute actually is.
- kleiba 3y agoI've been out of academia for a bit, but in my day 100k USD would not have been considered academia-friendly in my neck of the woods...
- dheera 3y agoThat's about the cost of 1 grad student year including all overhead, I believe. It's definitely far less than what many physics and biology labs spend on equipment in a year. I mean, you're an idiot of a PI if you have $500K/year of grants and spend it on 5 students and no compute.
- kleiba 3y ago$500k/year of grants for a single research institute would have been outrageous, and I used to be in NLP.
- deleted 3y ago[deleted]
- barkingcat 3y agoThis kind of assumption is super deceptive. The Facebook budget includes money to pay off people they've ripped off (in private settlements) and money for lawyers to shield the developers so they can feel free to rip off copyrighted content without having to pay personal penalty or be imprisoned for infringement. It also includes the price of buying lobbyists to alter laws to let this practice continue. Also, unless the authors work inside Facebook, they have no idea how much Facebook spent on training that model specifically.
- ein0p 3y agoAt $DAY_JOB nowadays we run 128x H100 runs without thinking twice nowadays. Only takes a few days to train a small-ish LLM with that to test out some ideas.
- uptownfunk 3y agoWhere are they hosted?
- ein0p 3y agoAWS and GCP both.
- turnsout 3y agoOut of curiosity, what leads you to train models from the ground up rather than fine tuning existing models?
- ein0p 3y agoWe do both. You can’t just fine tune if you’re trying a different model architecture, or even change some of the hyperparameters on an existing one. Every now and again you might be able to reuse some of the weights, but that’s about it. That’s part of the reason research is so incredibly expensive and time consuming in this field. I bet that $80k is only a fraction of the overall cost for the model described in the article, too.
- moffkalast 3y agoecho $DAY_JOB
- YetAnotherNick 3y ago> It only uses public datasets for training, and the code is open-sourced Looking at the repo, there is no training or data processing code.
- _ea1k 3y agoIt was trained with "1 trillion tokens from large-scale open-source pretraining datasets, including RefinedWeb, Pile, Github data, etc." I guess it is good that they mentioned some of it, but yeah, that isn't exceptionally helpful when making claims of it being 100% open source. I'm not sure why they feel the need to be so secretive if all of the sources are open.
- YetAnotherNick 3y ago"etc." is the most important part here. There is NL SFT and code SFT data which guessing by the names are instruction data very likely from GPT-4. It is known in finetuning community that training with GPT-4 data is the easiest way of improving the model. If that's the case base JetMoE should be compared to finetuned llama, not base llama.
- kristianp 3y agoPile includes books3, is there a test prompt to see whether the books are present in the training data?
- vertis 3y agoIt might be equivalent to LLaMA2 but it's still not capable of even simple reasoning: > If two apples cost 2 dollars and 1 apple costs 1.20, what is the discount you're getting > To calculate the discount, you need to compare the original price of the apples to the price after the discount. > The original price for two apples is: 2 apples * $1.20/apple = $2.40 > The price for one apple after the discount is: 1 apple * $2.00/apple = $2.00 > Now, let's calculate the discount for one apple: > Original price for one apple - Discounted price for one apple = Discount amount > $2.40 - $2.00 = $0.40 > The discount for one apple is $0.40. Since you're buying two apples, you'll get the discount on both: > Discount amount for two apples = 2 * $0.40 = $0.80 > So, the discount you're getting for two apples is $0.80.
- _ea1k 3y agoI wonder if the reasoning capability of a moe model is limited by the max size of the individual experts?
- moffkalast 3y agoThe simple fact that they did not list the current SOTA for the size class in their comparison table tells you all you need to know about their confidence. And listing Gemma-2B is like shooting fish in a barrel, might as well also put RedPajama on there. It's good to see MoE being attempted at the smaller sizes, and it may scale well downwards as well given their results. But regardless, 1.25T is very little training data compared to the 6T that Mistral 7B received and even that makes it barely usable and likely not yet saturated. Before it, the sub-13B size class was considered basically an academic exercise.
- ravetcofx 3y agoI'm kind of impressed it was able to do basic math even if the reasoning isn't correct. That seems like an impressive emergent behavior for a small cheap model like this.
- vertis 3y agoLlama2:7b makes the same mistakes. It's not until you use something like Mixtral or Llama2:13b that it actually gets the correct results (in my one example). Interestingly Llama2:13b objects that there is no discount until I clarify: "the discount you're getting [with 2 apples]" It's not just math though it's any kind of complex reasoning and ambiguity. Comparing to humans is always complex, but humans for the most part wouldn't balk at me asking what discount you're getting without specifying that it's the 2 apples that have the discount in this example. A more advanced model often states the assumptions. There are lots of nuances in this question as well. I'm still paying 80c more than buying one apple, so I should only buy two apples if I would use two apples.
- operator-name 3y agoSadly no matter what kind of prompting that I try it fails the sisters question: > Sally (a girl) has three brothers. Each of her brothers has two sisters. How many sisters does Sally have? Sally (a girl) has three brothers. Each of her brothers has two sisters. How many sisters does Sally have? Sally has two sisters. Here's the reasoning: Sally has three brothers. Each of her brothers has two sisters. Since her brothers are the ones who have two sisters each, and we know there are three brothers, we can conclude that each brother has two sisters. Therefore, there are two sisters in total (one for each brother). So, Sally has two sisters. https://www.lepton.ai/playground/chat?model=jetmoe-8b-chat https://www.lepton.ai/playground/chat?model=jetmoe-8b-chat
- moffkalast 3y agoNot exactly something you'd expect a model of this size to solve, Mixtral fails it too (if you switch the names/genders that is, since it's contaminated with a few versions). It does at least indicate that their training data might indeed be as clean as they say.
- vertis 3y agoMore complex models fail this question, so it would be exceptional if it passed it: ollama run llama2:13b >>> Sally (a girl) has three brothers. Each of her brothers has two sisters. How many sisters does Sally have? This is a classic lateral thinking puzzle that requires you to think outside the box and challenge your assumptions. The answer is not what you might expect at first glance. To solve this puzzle, we need to break it down step by step: 1. Sally has three brothers. 2. Each of her brothers has two sisters. So, let's count the number of sisters that each of Sally's brothers has: Brother 1: 2 sisters Brother 2: 2 sisters Brother 3: 2 sisters Now, let's add up all the sisters that Sally has: Sister 1 + Sister 2 + Sister 3 + Sister 4 + Sister 5 = 2 + 2 + 2 + 2 + 2 = 8 Therefore, Sally has 8 sisters.
- patrick-fitz 3y agoOut of curiosity, looking at the cheapest price for a H100 that I could find online. Lambda Reserved Cloud [1] starts at $1.89 per H100 per hour. It could be possible to get the cost down to a lower amount: $1.89 * 96GPUs * 24hours * 14days = ~$61k 1 - https://lambdalabs.com/deep-learning/servers/hyperplane https://lambdalabs.com/deep-learning/servers/hyperplane
- logicchains 3y agoIt also depends on the interconnect speed. If you don't have fast enough interconnect between the machines, you won't get linear speedup with N gpus.
- avrionov 3y agoThis is the price of training if nothing fails.
- gavin_guo 3y agomodel https://huggingface.co/jetmoe/jetmoe-8b https://huggingface.co/jetmoe/jetmoe-8b
- eurekin 3y agodemo https://www.lepton.ai/playground/chat?model=jetmoe-8b-chat https://www.lepton.ai/playground/chat?model=jetmoe-8b-chat Just had a quick chat about kubernetes, very simple discussion. It held well