3 ms·
I don't know a single person in real life that has any desire to run local LLMs. Even amongst my colleagues and tech friends, not very many use LLMs period. It'
by bongodongobob 2y ago
I don't know a single person in real life that has any desire to run local LLMs. Even amongst my colleagues and tech friends, not very many use LLMs period. It's still very niche outside AI enthusiasts. GPT is better than anything I can run locally anyway. It's not as popular as you think it is.
- dimensi0nal 2y agoThe only consumer demand for local AI models is for generating pornography
- treprinum 2y agoHow about running your intelligent home with a voice assistant on your own computer? In privacy-oriented countries (Germany) that would be massive.
- magicalhippo 2y agoThis is what I'm fiddling with. My 2080Ti is not quite enough to make it viable. I find the small models fail too often, so need larger Whisper and LLM models. Like the 4060 Ti would have been a nice fit if it hadn't been for the narrow memory bus, which makes it slower than my 2080 Ti for LLM inference. A more expensive card has the downside of not being cheap enough to justify idling in my server, and my gaming card is at times busy gaming.
- serf 2y agoabsolutely wrong -- if you're not clever enough to think of any other reason to run an LLM locally then don't condemn the rest of the world to "well they're just using it for porno!"
- knowitnone 2y agoso you're saying that a huge market?!
- throwaway48476 2y agoI want local copilot. I would pay for this.
- rubatuga 2y agoI run a 12GB model on my 3060 and use it to help answer healthcare questions. I'm currently doing a medical residency. (No I don't use it to diagnose). It helps comply with any HIPAA style regulations. I sometimes use it to fix up my emails. Not sure why people are longing for a 128GB card, just download a quantized model and run with LM Studio (https://lmstudio.ai/ https://lmstudio.ai/). At least two of my colleagues are using ChatGPT on a regular basis. LLMs are being used in the ER department. LLMs and speech models are being used in psychiatry visits.
- ryao 2y agoA 128GB card could run llama 3.1B 70B with FP8. It is a huge increase in quality over what the current 24GB cards can do.