5 ms·
> JetMoE-8B is trained with less than $ 0.1 million1 cost but outperforms LLaMA2-7B from Meta AI, who has multi-billion-dollar training resources. LLM training
by lolinder 3y ago
> JetMoE-8B is trained with less than $ 0.1 million1 cost but outperforms LLaMA2-7B from Meta AI, who has multi-billion-dollar training resources. LLM training can be much cheaper than people generally thought.
They want you to read this as "we spent $100k compared to Meta's spending billions", but that's not actually what this says. It says that they spent $100k and Meta has the resources to spend billions if they wanted to.
We don't know what Facebook spent on training LLaMA 2, but they say that it took them 184320 A100-80GB GPU-hours to train the 7B model [0]. AWS charges $14.46/hour for an instance that has 8 of those [1], which amounts to $1.81/GPU/hr.
At that rate and assuming they paid something resembling AWS's list price, LLaMA 2 7B cost ~$333k. That's more than $100k, but not by orders of magnitude, and it's likely that Facebook wasn't paying the full price AWS is charging today.
[0] https://github.com/meta-llama/llama/blob/main/MODEL_CARD.md#hardware-and-software https://github.com/meta-llama/llama/blob/main/MODEL_CARD.md#...
[1] https://aws.amazon.com/ec2/instance-types/p4/ https://aws.amazon.com/ec2/instance-types/p4/
- rvba 3y agoTraining data costs zero? What is the training data anwyay? Books? (Is it legal to use non public domain, wait whom am I kidding) Reddit posts scrapped from the site? (Without clean up there are lots of bad subreddits) Wikipedia? Of course I understand why labour costs + employee cost related costs are ignored.
- samus 3y agoIt costs more the more you care about squeaky clean training data. Of course you get a better model in return. ChatGPT used a crawl of the internet and patches things up with alignment and DPO. Big boys like Microsoft might have deals with publishers to get textbooks in bulk. Contents from sites with moderation can be filtered using the platform's mechanism, e.g., only include text with a certain length and count of upvotes. LLMs can be used to generate and filter data as well. Humans have been used to do this, they might have to do this less in the future. Mostly to review what the LLMs are suggesting.
- nuz 3y agoMeta has their own data centers so they definitely didn't pay the equivalent to what AWS costs
- freedomben 3y agoGood point, although it's possible that with the extreme price of GPUs that it cost more to train by buying hardware than it would to rent. For example it might take two to three years before the GPUs are paid for by customers.
- samus 3y agoSince the GPUs can be rented out afterwards, they amortize very quickly with prices in the order of $1/h.
- fleischhauf 3y agoI think AWS prices scale with hardware price
- greenavocado 3y agoLinux reserved cost of p3.16xlarge is $146,362.0800 annually. On-demand cost is $214,444.8000 annually I am pretty damn sure I could build a 8 GPU Intel Xeon E5-2686 v4 (Broadwell) (that's what Amazon uses - it's $30 to $75 on eBay) server for less than that and come out ahead on electricity even at full throttle. RTX 4090 are just under $2000 each on eBay. 8 GPU × $2000 (RTX 4090) + $1000 (for the rest of the computer) = $17,000 If pulling 2kW continuously at 15 cents per kW*hr for 1 year that's 2000 watts × 365 days × (0.15/(kW×hr)) or $2,628 In total the computer will cost $19,628 if you throw it in the dumpster at the end of each calendar year of using it. If you stack internet cost of $200 a month on top, that's $2400 a year, which raises your annual cost to: $22,028 This is still $124,334 cheaper per year than one AWS 8-GPU server if you fully depreciate your own hardware at the end of year 1 to $0. I could hire an engineer in America to babysit it with the money left over.
- ghshephard 3y ago
- DalasNoin 3y agoThis entire difference can be explained due to their double mixture of experts architecture. So only 1/4 MLP and attention blocks are used at any time. Maybe this should be the headline, Moe reduces compute by a factor of 4 without losing accuracy. But this is already known. Still interesting to see a smaller Moe model. This could be the ideal size for many local applications.
- brianarbuckle 3y agoAgreed. In inference, I could even imagine a 4-bit conversion for certain edge devices.
- Centigonal 3y agoMoE reduces compute cost for inference at scale, but not for training. You still have to train the whole model (plus the router)
- rileyphone 3y agoWith an MoE you only need to train a smaller model which you can then combine into an x8 and finetune/train the router. Mistral used their 7B base to make Mixtral, Qwen's new MoE uses their 1.8B model upscaled to 2.7B, pretty sure Grok also trained a smaller model first.
- samus 3y agoYou're only correct about Qwen's MoE. I presume that Chinese model builders feel more pressure to be efficient about using their GPU time because of sanctions.
- phree_radical 3y agoVery incorrect! The "8x7b" in the name regularly confuses people into some similar conclusion, but there are not eight 7b "experts" in Mixtral 8x. It's more apt to think of all 256 FFN's as the "experts," as each expert FFN on a given layer has no relation to the expert FFN's on other layers. You need to train them all within the MoE architecture, as combining existing models ("clown car MoE") works, but isn't gaining anything from the architecture/sparsity
- dannyw 3y agoAny company as big as Meta have teams working on optimisation (eg optimised kernels), usually with direct engagement with NVIDIA engineers. These kind of things are usually only selectively shared.
- benreesman 3y agoI’ll agree with your general point even though there are some subtleties to FBNY. More important, we let an awful lot of self-promotion from the big guys slide around here. I can live with the guys and gals doing this on a shoestring getting a little of that sweet hype love. This seems pretty legit.