3 ms·
Performances of local models are pretty bad compared to what AI vendors offer, token generation is just too slow to be that useful. And you need to allocate GBs
by dgellow 5mo ago
Performances of local models are pretty bad compared to what AI vendors offer, token generation is just too slow to be that useful. And you need to allocate GBs of memories, something that will stay very expensive to buy for a long time.
Running local models will stay niche for a while, unless we see breakthroughs
- jqpabc123 5mo agoDumb idea --- how about if we limit local models to specific domains --- medicine for example. Most doctors don't care much about engineering or accounting or software development or 10000 other things that big vendor models address. This area is yet to be really explored. Nvidia aims to provide the hardware to do so.
- CamperBob2 5mo agoThat's a fairly obvious idea, not dumb at all, but unfortunately it doesn't seem to pan out. Trying to specialize an LLM in one area harms its 'cognition' in all areas. For instance, if you train a coding model without all the Shakespeare and soap operas and Wikipedia and pirated Stephen King books and ancient Roman history and whatever, you end up with a worse coding model. I'm not sure anyone really understands why.
- jqpabc123 5mo agohttps://www.ibm.com/think/topics/domain-specific-llm https://www.ibm.com/think/topics/domain-specific-llm
- deleted 5mo ago[deleted]
- CamperBob2 5mo agoThe article is not backed up by reality. Why would use anything but a domain-specific LLM, if they actually worked? The author is probably confusing RAG with pretraining. You can RAG on PubMed but you can't arrive at a competitive model by pretraining solely on it.