4 ms·
For the people who self-host LLMs at home: what use cases do you have? Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM su
by taosx 2y ago
For the people who self-host LLMs at home: what use cases do you have?
Personally, I have some notes and bookmarks that I'd like to scrape, then have an LLM summarize, generate hierarchical tags, and store in a database. For the notes part at least, I wouldn't want to give them to another provider; even for the bookmarks, I wouldn't be comfortable passing my reading profile to anyone.
- segalord 2y agoI use it exclusively for users on my personal website to chat with my data. I've given the setup tools to have read access my files and data
- netdevnet 2y agoIs this not something that you can with non-hosted LLMs like ChatGPT? If you expose your data, it should be able to access it iirc
- worldsayshi 2y agoYou can absolutely do that but then you pay by the token instead of a big upfront hardware cost. It feels different I suppose. Sunk cost and all that.
- xyc 2y agollama3.2 1b & 3b is really useful for quick tasks like creating some quick scripts from some text, then pasting them to execute as it's super fast & replaces a lot of temporary automation needs. If you don't feel like invest time into automation, sometimes you can just feed into an LLM. This is one of the reason why recently I added floating chat to https://recurse.chat/ https://recurse.chat/ to quickly access local LLM. Here's a demo: https://x.com/recursechat/status/1846309980091330815 https://x.com/recursechat/status/1846309980091330815
- taosx 2y agoLooks very nice, saved it for later. Last week, I worked on implementing always-on speech-to-text functionality for automating tasks. I've made significant progress, achieving decent accuracy, but I imposed some self-imposed constraints to implement certain parts from scratch to deliver a single binary deployable solution, which means I still have work to do (audio processing is new territory for me). However, I'm optimistic about its potential. That being said, I think the more straightforward approach would be to utilize an existing library like https://github.com/collabora/WhisperLive/ https://github.com/collabora/WhisperLive/ within a Docker container. This way, you can call it via WebSocket and integrate it with my LLM, which could also serve as a nice feature in your product.
- xyc 2y agoThanks! lmk when/if you wanna give it a spin as free trial hasn't been updated with the latest but I'll try to do it this week. I've actually been playing around with speech to text recently. Thank you for the pointer, docker is a bit too heavy to deploy for desktop app use case but it's good to know about the repo. Building binaries with Pyinstaller could be an option though. Real time transcription seems a bit complicated as it involves VAD so a feasible path for me is to first ship simple transcription with whisper.cpp. large-v3-turbo looks fast enough :D
- taosx 2y agoYes it's fast enough, especially if you don't need something live.
- afro88 2y agoCan you list some real temporary automation needs you've fulfilled? The demo shows asking for facts about space. Lower param models seem to be not great as raw chat models, so I'm interested in what they are doing well for you in this context
- xyc 2y agoThings like grab some markdown text and ask to make a pip/npm install one liner, or quick js scripts to paste in the console (which I didn't bother to open an editor), a fun use case was random drawing some lucky winners for the app giveaway from reddit usernames. Mostly it's converting unstructured text to short/one-liner executable scripts & doesn't require much intelligence. For more complex automation/scripts that I'll save for later, I do resort to providers (cursor w sonnet 3.5 mostly).
- TechDebtDevin 2y agoI keep an 8b running with ollama/openwebui to ask it to format things, summarization, and to generate SQL/simple bash commands and what not.
- worldsayshi 2y agoSo 8b is really smart enough to write scripts for you? How often does it fail?
- wokwokwok 2y ago> So 8b is really smart enough to write scripts for you? Depends on the model, but in general, no. ...but it's fine for simple 1 liner commands like "how do I revert my commit?" or "rename these files to camelcase". > How often does it fail? Immediately and constantly if you ask anything hard. An 8b model is not chat-gpt. The 3B model in the OP post is not chat-gpt. The capability compared to sonnet/4o is like a potato and a car. Search for 'LLM Leaderboard' and you can see for yourself. The 8b models do not even rank. They're generally not capable enough to use as a self hosted assistant.
- worldsayshi 2y agoI really hope we can get sonnet like performance down to single consumer level GPU someone soon. Maybe the hardware will get there before the models.
- TechDebtDevin 2y agoWell considering it probably takes several hundred GBs of VRAM to run inference for Claude its going to be a while. But yes, like the guy above said it's really only helpful for one line commands. Like if I forgot some sort flag thats available for a certain type of command. Or random things I don't work with often enough to memorize their little build commands etc. It's not helpful for programming just simple commands. It also can help with unstructured or messy data to make it more readable, although there's potential to hallucinate if the context is at all large.
- laniakean 2y agoI mostly use it to write some quick scripts or generate texts if it follows some pattern. Also, getting it up running with LM studio is pretty straightforward.
- ein0p 2y agoI run Mistral Large on 2xA6000. 9 times out of 10 the response is the same quality as GPT 4o. My employer does not allow the use of GPT for privacy related reasons. So I just use a private Mistral for that
- archerx 2y agoFor me at least the biggest feature of some self hosted LLMs is that you can get it then to be “uncensored”, you can get them to tell you dirty jokes or have the bias removed with controversial and politically incorrect subjects. Basically you have a freedom you won’t get from most of the main providers.
- ndheebebe 2y agoAnd reliability. When Azure sends you the "censored output" status code it had basically failed and no retry is gonna help. And unless you are some corp you wont get approved for lifting the censoring.
- williamcotton 2y agoI work with a lot of attorney’s eyes only documents and most protective orders do not allow for shipping off these files to a third-party.
- Rick76 2y agoI essentially use it like everyone else. I use it to search through my personal documents because I can control the token size and file embedding
- cma 2y agoThey are much more flexible, you can e.g. edit the system's own responses rather than waste context on telling it a correction.
- theodric 2y agoI've been enjoying fine-tuning various models with various data, for example 17 years of my own tweets, and then just cranking up the temperature and letting the model generate random crap that cracks me up. Is that practical? Is joy practical? I think there's a place for it.
- m0wer 2y agoTabbyML! Autocompletion like GitHub Copilot, using qwen-2.5-coder 7B.