5 ms·
Running models at home seems like a waste of money while at the same time they are currently heavily subsidized in the cloud by dumb money.
by emsign 2y ago
Running models at home seems like a waste of money while at the same time they are currently heavily subsidized in the cloud by dumb money.
- sterlind 2y agoit's nice to be able to have private conversations.
- borisk 2y agoHow is it a waste of money if a lot of people, including the author of the article, already have an nVidia GPU in their PC? Running locally has a lot of advantages - privacy, getting to learn how to run LLMs, not having to deal with quotas, logins, outages.
- hu3 2y agoHome electricity bill of a decent GPU alone is usually more expensive than renting GPUs on demand.
- borisk 2y agoDo you have any numbers to back that statement up? E.g. I have solar panels and a home battery and pay less than a $100 a year for electricity.
- hu3 2y agoSolar panels aren't free. And aren't hassle-free either. I wouldn't even bring that into the equasion since a large part of the population can't use solar panels. Not to mention GPU value depreciation.
- chpatrick 2y agoI use ChatGPT for most practical stuff but I really enjoy running local models. I find it really interesting and I think it's important for people to know how to run these without being beholden to big tech. If you have a used 3090 you can already run some really strong models. There are some really interesting local models as well, like the abliterated ones.
- calaphos 2y agoThere's also big efficiency increases when batching multiple requests, making clouds inherently more cost effective for normal use cases. Way better utilization of expensive hardware as well ofc.
- hnfong 2y agoSometimes you don't want your workflow to be subject to the whims of some LLM API provider.
- cma 2y agoHeavily subsidized on heavily price discriminated hardware (nvidia datacenter licensed hardware) kind of cancels out.