10 ms·
How Meta trains large language models at scale
- jauntywundrkind 2y agoRandom q, I wonder if gloo is used in these systems? https://github.com/facebookincubator/gloo https://github.com/facebookincubator/gloo RDMA and GPUDirect capable. Coordinates over MPI or (hi)redia.
- runeblaze 2y ago̶I̶I̶R̶C̶ ̶g̶l̶o̶o̶ ̶i̶s̶ ̶C̶P̶U̶ ̶t̶e̶n̶s̶o̶r̶s̶ ̶o̶n̶l̶y̶ ̶s̶o̶ ̶l̶i̶k̶e̶l̶y̶ ̶n̶o̶t̶ Edit: I had a brain freeze or something... gloo is not CPU only but for whatever reason I don't see it outside of CPU-comms
- whalesalad 2y agointeresting that their domain is still engineering.fb.com
- mike_d 2y agoPosts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle flunkies into leadership positions.
- coralreef 2y agoHow do TPUs perform compared to GPUs on LLMs and image generation?
- smarterclayton 2y agoPretty well. Anthropic runs some of Claude inference on GKE and TPU v5e - this talk at the last GCP conference has some details: https://youtu.be/b87I1plPeMg?si=T4XSFUzXG8BwpphR https://youtu.be/b87I1plPeMg?si=T4XSFUzXG8BwpphR Ecosystem support for GPU is very strong and so smaller teams might find TPUs to have a steeper learning curve. Sophisticated users can definitely get wins today on TPU if they can get capacity. And it’s not as if nvidia is standing still, so how the strengths of each change over future generations isn’t set either. Ie TPU are “simple” matrix multipliers and also optimized for operation at scale, GPU are more general purpose and have strong ecosystem power. Disclaimer - work on GKE on enabling AI workloads.
- zdyn5 2y agoH100s are far from consumer video cards
- stygiansonic 2y agoYeah, ops comment makes it seem like they are building racks of RTX 4090s, when this isn’t remotely true. Tensor Core performance is far different on the data center class devices vs consumer ones.
- mike_d 2y agoThey are building racks of 4090s. Nobody can get H100s in any reasonable volume. Hell, Microsoft is renting GPUs from Oracle Cloud to get enough capacity to run Bing.
- kkielhofner 2y agoWho is "they"? RTX 4090s are terrible for this task. Off the top of my head: - VRAM (obviously). Isn't that where the racks come in? Not really. Nvidia famously removed something as basic as NVLink between two cards from the 3090 to the 4090. When it comes to bandwidth between cards (crucial) even 16 lanes of PCIe 4 isn't fast enough. When you start talking about "racks" unless you're running on server grade CPUs (contributing to cost vs power vs density vs perf) you're not going to have nearly enough PCIe lanes to get very far. Even P2P over PCIe requires a hack geohot developed[0] and needless to say that's umm, less than confidence inspiring for what you would lay out ($$$) in terms of hardware, space, cooling, and power. The lack of ECC is a real issue as well. - Form factor. Remember PCIe lanes, etc? The RTX 4090 is a ~three slot beast when using air cooling and needless to say rigging up something like the dual slot water cooled 4090s I have at scale is another challenge altogether... How are people going to wire this up? What do the enclosures/racks/etc look like? This isn't like crypto mining where cheap 1x PCIe risers can be used without dramatically limiting performance to the point of useless. - Performance. As grandparent comment noted 4090s are not designed for this workload. In typical usage for training I see them as 10-20% faster than an RTX 3090 at much higher cost. Compared to my H100 with SXM it's ridiculously slow. - Market segmentation. Nvidia really knows what they're doing here... There are all kinds of limitations you run into with how the hardware is designed (like Tensor Core performance for inference especially). - Issues at scale. Look at the Meta post - their biggest issues are things that are dramatically worse with consumer cards like the RTX 4090, especially when you're running with some kind of goofy PCIe cabling issue (like risers). - Power. No matter what power limiting you employ an RTX 4090 is pretty bad for power/performance ratio. The card isn't fundamentally designed for these tasks - it's designed to run screaming for a few hours a day so gamers can push as many FPS at high res as possible. Training, inference, etc is a different beast and the performance vs power ratio for these tasks is terrible compared to A/H100. Now lets talk about the physical cabling, PSU, etc issues. Yes miners had hacks for this as well but it's yet another issue. - Fan design. There isn't a single "blower" style RTX 4090 on the market. There was a dual-slot RTX 3090 at one point (I have a bunch of them) but Nvidia made Gigabyte pull them from the market because people were using them for this. Figuring out some kind of air-cooling setup with the fan and cooling design of the available RTX 4090 cards sounds like a complete nightmare... - Licensing issues. Again, laying out the $$$ for this with a deployment that almost certainly violates the Nvidia EULA is a risky investment. Three RTX 4090s (at 9 slots) to get "only" 72GB of VRAM, talking over PCIe, using 48 PCIe lanes, multi-node over sloooow ethernet (hitting CPU - slower and yet more power), using what likely ends up at ~900 watts (power limited) for significantly reduced throughput and less VRAM is ridiculous. Scaling the kind of ethernet you need for this (100 gig) comes at a very high per-port cost and due to all of these issues the performance would still be terrible. I'm all for creativity but deploying "racks" of 4090s for AI tasks is (frankly) flat-out stupid. [0] - https://github.com/tinygrad/open-gpu-kernel-modules https://github.com/tinygrad/open-gpu-kernel-modules
- tflol 2y ago[flagged]
- tflol 2y ago[flagged]
- brainfog 2y agoBecause it's completely irrelevant.
- tflol 2y ago[flagged]
- Jabrov 2y agoBecause you're making a baseless claim that contradicts the linked article with nothing to back it up
- jauer 2y agoand deceptive if not inaccurate. Meta's Model Cards specifically call out that they were trained on publicly available datasets and NOT any Meta user data. For example: https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md
- yellow_postit 2y agoThis model card seems at odds with what they claim you can and cannot opt out of [1] > We use information that is publicly available online and licensed information. We also use information shared on Meta’s Products and services. This information could be things like posts or photos and their captions. [1] https://www.facebook.com/privacy/genai/?_rdr https://www.facebook.com/privacy/genai/?_rdr
- WheatMillington 2y ago
- ai4ever 2y ago[flagged]
- kdot 2y agoHow will Meta leverage LLMs at scale to drive revenue? It's not clear.
- dweekly 2y agoAsk the AI to come up with a business model.
- threeseed 2y agoI still believe that a VR future is coming once the technology commoditises i.e. costs come down 10x and we have 3090 level GPUs in the headset. At that point we will have photo-realistic experiences like concerts etc that anyone can afford. And at that point having a lot of LLM based avatars that can help "fill in the space" will be valuable.
- sangnoir 2y agoIf only Meta had a way to monetize engagement with generative AI content in a way that scales with quantity of generated content.
- tucnak 2y agoRevolutionary!
- OsrsNeedsf2P 2y agoLLMs aren't a monetizable product themselves. For the foreseeable future, that will always be ads. LLMs (and VR) are just big bets on getting ahead of future technology.
- HDThoreaun 2y agoI think the nearish term plan is chatbot customer assistants on whatsapp. Doesnt seem like theyre that close to releasing them but who knows
- altdataseller 2y agoAt the end of the day, everything eventually comes down to chatbots
- deleted 2y ago[deleted]
- samspenc 2y agoOK this was a bit funny: Top HW failure modes: * GPU falling off the bus I honestly thought "do they mean GPUs falling off a bus entering the data center" and then realized its actually the connectivity, as they mention in the next line GPUs falling off: In this case, GPUs are not detected by the host on PCIe.
- cachvico 2y agoBrings a whole new meaning to bus factor
- throwup238 2y agoI’ve never met a GPU that could survive getting hit by a bus.
- ChuckMcM 2y agoThe bits on the bus go round and round! There is a lot of interesting yet unpublished work on 'data center' scale compute complexes. It was a rabbit hole I fell into several times while at Google.
- falcor84 2y agoThey do publish some of that, or at least they used to. In particular "The Datacenter as a Computer" [0] was a very interesting read. [0] https://research.google/pubs/the-datacenter-as-a-computer-an-introduction-to-the-design-of-warehouse-scale-machines-second-edition/ https://research.google/pubs/the-datacenter-as-a-computer-an...
- martin-adams 2y agoA GPU falling off the bus would be one mega flop
- TeMPOraL 2y agoThe audience in the back goes clap clap clap, chapeau bas.
- xvector 2y ago> So we decided to build both: two 24k clusters, one with RoCE and another with InfiniBand. Our intent was to build and learn from the operational experience. I love how they built two completely insane clusters just to learn. That's badass.
- riku_iki 2y agoMore like Mark gave them 100k GPUs, and they are not sure what exactly to do with them..
- logicchains 2y agoIt's not just to learn; an RoCE ethernet cluster with Aristas is way cheaper to build and maintain than a fancy InfiniBand cluster with Mellanox/NVidia networking, so proving that the former is good enough at scale will eventually save Meta a huge amount of money. InfiniBand cards are much more expensive than ethernet because there's few vendors, that have a quasi-monopoloy, and because overall far fewer of them are produced so there's less economy of scale.
- idkdotcom 2y agoThese seem classic challenges with running distributed systems loads that are not specific to training LLMs. Anyone of the super computers listed here https://en.wikipedia.org/wiki/TOP500 https://en.wikipedia.org/wiki/TOP500 suffers from the same issues. Think about it. While the national labs use these systems to model serious stuff -such as climate or nuclear weapons- Meta uses them to train LLMs. What a joke, honestly!
- whiplash451 2y agoA lot of serious things look like a toy or a joke at first.
- idkdotcom 2y ago[dead]
- mhandley 2y agoOn the other hand, Meta just rapidly built two different training networks in existing datacenter buildings, with existing cooling constraints, using mostly commodity components (albeit expensive commodity components) each of which would place at #3 on that top500 list in terms of GPU power. Compare that with how long it took to get any of the other supercomputers from design to being fully commissioned.
- idkdotcom 2y ago[dead]
- _zoltan_ 2y agoFor profit is not less serious than what research labs do. I'd even say it's more important: they drive the economy.
- idkdotcom 2y ago[dead]
- yosito 2y agoI wish that instead of just training another stupid LLM, Meta would use it to improve their search and help me find the content I'm actually interested in.
- TeMPOraL 2y agoTheir revenue depends on it being hard (but not impossible) for you to find the content you're actually interested in. Would be nice if it didn't, but in this reality, money on the Internet is made by wasting users' lives. That is what attention economy is about.
- rmbyrro 2y agoIt's actually a mix. They need to disappoint the user for the right amount of time, to then please it at the right moment and dose. This maximizes the dopamine release and increases addictiveness. When you find good content depends on when the algo judges you're already primed to a colorful dopamine intake.
- rldjbpin 2y agothis is like asking disney to reduce wait times for their rides. in other words, it is against content aggregation platforms' interest to let you get what you want directly.
- dudus 2y ago> Since we did not have time to change the cooling infrastructure, we had to remain in an air-cooled environment. The mechanical and thermal designs had to change to accommodate this, and that triggered a validation cycle to support a large-scale deployment. > All of these hardware-related changes were challenging because we had to find a solution that fit within the existing resource constraints, with a very small degree of freedom to change and meet a tight schedule. Seems like the time constraints put into the team impacted the overall quality of the model.
- vessenes 2y agoThis sounds to me like standard CYA / perhaps good-natured complaining from the tech team. The last tech team to have no budget and time constraints to pursue their vision? I don’t know, the Xanadu team? Romero’s original Daikatana team?
- deleted 2y ago[deleted]
- nomel 2y agoWhat's the cost of letting that hardware sit idle?
- Muhtasham 2y agoit’s serious crime
- koolala 2y agowhat a title...
- Oras 2y agoWould be nice to read how do they collect/prepare data for training. Which data sources? How much of Meta users data (fb, instagram… etc). How do they sanitize PII?
- OsrsNeedsf2P 2y ago> How do they sanitize PII? I can't comment on how things like faces get used, but in my experience, PII at Meta is inaccessible by default. Unless you're impersonating a user on the platform (to access what PII they can see), you have to request special access for logs or database columns that contain so much as user IDs, otherwise the data simply won't show up when you query for it. This is baked into the infrastructure layer, so I doubt the GenAI teams are using something else.
- actionfromafar 2y agoFor a convenient definition of PII. Isn’t everything a user does in aggregate PII?
- robertlagrant 2y agoI don't think it's PII. If you had someone's movements, you could go and spy on them, find out who they were (i.e. their PII) and then link that back and say "I now know this identified person's movements". I don't think the movements themselves are PII. Things that aren't PII aren't "convenient" definitions. Doesn't mean everything that isn't PII is fine to share. It's like saying a kidnapping isn't a murder. That's not a convenient definition of murder; it's just a different thing. We shouldn't start talking like witch hunters as soon as we encounter a situation that we haven't memorised a reasonable response to. We should be able to respond reasonably to new situations.
- KaiserPro 2y agoPII is pretty intuitive to define. Obvious examples: data that easily identifies a person (Photo, name, number, UUID, etc) Thats trivial to block. Where it gets harder is stuff that on it's own isn't PII, but combined with another source, would be For example, aggregating public comments on a celeb's post. (ie stripping out usernames and likes and assigning a new UUID to each person.) For a single post, thats good enough. You're very unlikley to be able to identify a single person. But over multiple posts, thats where it gets tricky. As with large companies, the process for getting permission to use that kind of data is righty difficult, so it often doesn't get used like that.
- lokimedes 2y agoYikes, the little Infiniband+A100 cluster I installed for my previous company seemed useful at the time (12 GPUs) and that was at a cost of around $300k. With LLMs it feels like game over for non-cloud applications if you are not a mega-corp.
- lannisterstark 2y agoWell, yes, but Not all models need to be "super large." Smaller models, specialized in specific tasks, working together - and then reporting to a slightly larger model is the way to go. Think of everything being connected to a "Home Computer" in those "Future House of 2020" videos that were out there in 70s or what not. Another example (very rough) would be something like "Weather data gets to a small model via an API, model looks at it, updates the home dashboard, also sees if there's any alerts, if so, adds x or y to home dashboard appropriately as to what it thinks best." We can probably achieve the latter example today. (without any significant 'coding' on anyone's part except the API owner)
- afro88 2y ago> Well, yes, but Not all models need to be "super large." Smaller models, specialized in specific tasks, working together - and then reporting to a slightly larger model is the way to go. I want to believe, but I'm still yet to see this kind of set up being anywhere near GPT-4 level. The weather example seems quite contrived. Why not just display the alerts for your area? Why is a complex system of smaller models reporting up to a slightly larger model necessary?
- dudus 2y agoBecause cost and speed. Smaller models can run on your phone for free, or on the cloud for pennies. An API call for a large LLM with a lot of context can cost orders of magnitude more and incur network latency
- lannisterstark 2y ago>Why not just display the alerts for your area? Because "Flood warning on roads" is very different than "hey there is a possible tornado/aftershocks/*" and I'd like the 2nd one to take up the entire home-assistant dashboard. I can either code it myself, or let the model figure it out after passing it the YAML file for the dashboard. >Why is a complex system of smaller models reporting up to a slightly larger model necessary? because as the other poster said, cost and speed. thousands of queries every day (potentially) at 30 cents per million tokens is very different than 15 dollars per million tokens.
- radarsat1 2y agoFrustratingly little information. For example, I'm exceedingly curious how they deal with scheduling jobs on such a huge array of machines. The article: > Efficient scheduling helps ensure that our resources are used optimally. This involves sophisticated algorithms that can allocate resources based on the needs of different jobs and dynamic scheduling to adapt to changing workloads. Wow thanks for that, captain obvious. So how do you do it?
- p4ul 2y agoI usually assume these companies are using some of the popular schedulers (e.g., Slurm, MOAB, SGE) that have existed in the HPC community for many years. I have anecdotally also heard that some are using k8s, but I've not seen that myself. Slurm [1] is basically built for this stuff; that's definitely what I would use! [1] https://slurm.schedmd.com/documentation.html https://slurm.schedmd.com/documentation.html
- claytonjy 2y agoSlurm is definitely still dominant, but OpenAI has been using k8s for training for many years now¹, and there are various ways to run slurm on top of Kubernetes, including the recent SUNK from coreweave² at my company we use slurm "directly" for static compute we rent or own (i.e. not in a public cloud), but are considering using Kubernetes because that's how we run the rest of the company, and we'd rather invest more effort into being better at k8s than becoming good slurm admins. ¹: https://openai.com/index/scaling-kubernetes-to-2500-nodes/ https://openai.com/index/scaling-kubernetes-to-2500-nodes/ ²: https://www.coreweave.com/blog/sunk-slurm-on-kubernetes-implementations https://www.coreweave.com/blog/sunk-slurm-on-kubernetes-impl...
- p4ul 2y agoVery cool! Thank for this, claytonjy!!