3 ms·
I never have seen the point of running locally. Not cost effective, worse model, etc.
by amazingamazing 2y ago
I never have seen the point of running locally. Not cost effective, worse model, etc.
- ogogmad 2y agoPrivacy?
- splintercell 2y agoEven if it was not cost-effective, or you're just running worse models, it's learning an important skill. Take for instance, self-hosting your website may have all these considerations, but you're getting information from the LLMs. It would be helpful to know that the LLM is in your control.
- Oras 2y agoSelf hosting website as local server running in your room? what’s the point? Same with LLMs, you can use providers who don’t log requests and SOC2 compliant. Small models that run locally is a waste of time as they don’t have adequate value compared to larger models.
- troyvit 2y ago> Self hosting website as local server running in your room? what’s the point? It depends on how important the web site is and what the purpose is. Personal blog (Ghost, WordPress)? File sharing (Nextcloud)? Document collaboration (Etherpad)? Media (Jellyfin)? Why _not_ run it from your room with a reverse proxy? You're paying for internet anyway. > Same with LLMs, you can use providers who don’t log requests and SOC2 compliant. Sure. Until they change their minds and decide they want to, or until they go belly-up because they didn't monetize you. > Small models that run locally is a waste of time as they don’t have adequate value compared to larger models. A small model won't get you GPT-4o value, but for coding, simple image generation, story prompts, "how long should I boil an egg" questions, etc. it'll do just fine, and it's yours. As a bonus, while a lot of energy went into creating the models you'd use, you're saving a lot of energy using them compared to asking the giant models simple questions.
- KetoManx64 2y agoMy self hosted website that has a bunch of articles about my self hosted setup and documentation, that i have running from a server in my living room got me my 6 figure devOps job.
- babyshake 2y agoCan you elaborate on the not cost effective part? That seems surprising unless the API providers are running at a loss.
- amazingamazing 2y agoA 4090 is $1000 at least. That’s years of subscription for the latest models, which don’t run locally anyway.
- homarp 2y agounless you have one already for playing games
- wongarsu 2y agoI run LLMs locally on a 2080TI I bought used years ago for another deep learning project. It's not the fastest thing in the world, but adequate for running 8B models. 70B models technically work but are too slow to realistically use them.
- IOT_Apprentice 2y agoLocal is private. You are not handing over your data to An AI for training.
- jiggawatts 2y agoMost major providers have EULAs that specify that they don’t keep your data and don’t use it for training.
- troyvit 2y agoConsidering how most major providers got their data to begin with these are EULAs that are probably impossible to enforce, and none of the major players have given much reason to believe they would stick to it anyway. And why bother when local models don't have nearly the same complications?
- jiggawatts 2y agoThis is FUD. Millions of businesses trust Google and Microsoft with their spreadsheets, emails, documents, etc… The EULAs are very similar. If any major provider was caught violating the terms of these agreements they’d lose a massive amount of business! It’s just not worth it.
- slowmovintarget 2y agoRight. And the former CEO of Google basically explained their operating procedure. You go and steal everything, you break the rules, "then you have your lawyers clean all that up" when you get caught. https://youtu.be/pvcK9X4XEF8?si=9j2ELkbXxHGoZEnZ&t=1157 https://youtu.be/pvcK9X4XEF8?si=9j2ELkbXxHGoZEnZ&t=1157 > ...if you're a Silicon Valley entrepreneur, which hopefully all of you will be, is if it took off then you'd hire a whole bunch of lawyers to go clean the mess up, right. But if nobody uses your product it doesn't matter that you stole all the content.
- troyvit 2y ago> This is FUD. Yeah. Because I have tons and tons of fear, uncertainty and doubt about how these giga-entities control so many aspects of my life. Maybe I shouldn't be spreading it but on the other hand they are so massively huge that any FUD I spread will have absolutely no effect on how they operate. If you want to smell real FUD go sniff Microsoft[1]. [1] https://en.wikipedia.org/wiki/Fear,_uncertainty,_and_doubt#Microsoft https://en.wikipedia.org/wiki/Fear,_uncertainty,_and_doubt#M...
- deleted 2y ago[deleted]