4 ms·
I'd love to host my own LLMs but I keep getting held back from the quality and affordability of Cloud LLMs. Why go local unless there's private data involved?
by imranq 1y ago
I'd love to host my own LLMs but I keep getting held back from the quality and affordability of Cloud LLMs. Why go local unless there's private data involved?
- mycall 1y agoOffline is another use case.
- seanmcdirmid 1y agoNothing like playing around with LLMs on an airplane without an internet connection.
- asteroidburger 1y agoIf I can afford a seat above economy with room to actually, comfortably work on a laptop, I can afford the couple bucks for wifi for the flight.
- seanmcdirmid 1y agoIf you are assuming that your Hainan airlines flight has wifi that isn't behind the GFW, even outside of cattle class, I have some news for you...
- sach1 1y agoGetting around the GFW is trivially easy.
- seanmcdirmid 1y agoya ya, just buy a VPN, pay the yearly subscription, and then have them disappear the week after you paid. Super trivially frustrating.
- vntok 1y agoVPN providers are first and foremost trust businesses. Why would you choose and pay one that is not well established and trusted? Mine have been there for more than a decade by now. Alternatively, you could just set up your own (cheaper?) VPN relay on the tiniest VPS you can rent on AWS or IBM Cloud, right?
- seanmcdirmid 1y agoThe VPN providers that get you to jump the cloud in China are Chinese, and China is not yet a high trust society, just like how they’ll take your payment for one year of gym fees and then disappear the next week (sigh). If AWS or IBM cloud find out you are using them as a VPN to jump the GFW, they will ban you for life, Microsoft, IBM, Amazon, aren’t interested in having their whole cloud added to the GFW block list. Many people have tried this (including Microsfties in China with free Azure credits) and they’ve all been dealt with harshly by the cloud providers.
- MangoToupe 1y agoWoah there Mr Money, slow down with these assumptions. A computer is worth the investment. But paying a cent extra to airlines? Unacceptable.
- seanmcdirmid 1y agoThe $3000 that a MBP M3 Max with 64GB of RAM costs might cover a round trip business class ticket for a trans pacific…if it is on sale (a Chinese carrier probably with GFW internet).
- diggan 1y agoSome of us don't have the most reliable ISPs or even network infrastructure, and I say that as someone who lives in Spain :) I live outside a huge metropolitan area and Vodafone fiber went down twice this year, not even counting the time the country's electricity grid was down for like 24 hours.
- PeterStuer 1y agoSame. For 'sovereignty ' reasons I eventually will move to local processing, but for now in development/prototyping the gap with hosted LLM's seems too wide.
- diggan 1y agoThere are some use cases I use LLMs for where I don't care a lot about the data being private (although that's a plus) but I don't want to pay XXX€ for classifying some data and I particularly don't want to worry about having to pay that again if I want to redo it with some changes. Using local LLMs for this I don't worry about the price at all, I can leave it doing three tries per "task" without tripling the cost if I wanted to. It's true that there is an upfront cost but way easier to get over that hump than on-demand/per-token costs, at least for me.