3 ms·
> Is it exclusively used for LLM training? While LLMs is what cool kids all talk about nowadays, there's way, way, way more to the machine learning than them.
by justapassenger 3y ago
> Is it exclusively used for LLM training?
While LLMs is what cool kids all talk about nowadays, there's way, way, way more to the machine learning than them. I'd say that minority of GPUs are used for LLMs in the ML space. There's likely way more used them where they directly bring billions of dollars to the companies (like all the rankers across big tech, for content, ads, search, etc).
- upbeat_general 3y agoAre they price-competitive for those applications though? My understanding is that the main focus is Memory/GPU-interconnect for large models. Are recommendation/ranking models large enough to take advantage of this? I don't think these cards are generally competitive in throughput/$.
- refulgentis 3y agoIt's not really clear what you mean, maybe there's gaps in my ML knowledge, but generally the answer depends on "what throughput do you need for your use case?" Generally I agree that if you only need to deliver mail once daily on a mile long route, a car that takes 24 hours to travel a mile is fine.
- upbeat_general 3y agoLots of ML applications don't generally care about latency, and throughput only matters per dollar (as you can just scale horizontally). To my knowledge, most inferencing (at least for simpler models) happens on cheaper, slower GPUs that have better throughput/dollar.
- ShamelessC 3y agoDon’t take this the wrong way but have you been drinking?
- refulgentis 3y ago? Have you? (Note I'm not op)
- justapassenger 3y agoDo you have any idea of the scale of ranking infrastructure at big tech and how it’s been all ML for about a decade across the industry?
- ShamelessC 3y agoSorry, it was a rude comment to make based on some strange grammar I noticed. I compared with some of your prior comments and noticed you didn’t have any such issues before. I wasn’t being facetious or judgemental - genuinely curious. There’s not a non rude way to ask that, should have kept it to myself.