3 ms·

> Let’s be clear. It’s going to be a long time before running a local LLM will produce the type of results that you can get from querying ChatGPT or Claude. (Yo

> Let’s be clear. It’s going to be a long time before running a local LLM will produce the type of results that you can get from querying ChatGPT or Claude. (You would need an insanely powerful homelab to produce that kind of results). Anyone experimented with local LLMs and compared the output from ChatGPT or Claude? The article mentions that they use local LLMs when they're not overly concerned with the quality or response time, but what are some other limitations or differences to running these models locally?

jdelsman 2y ago
You will simply need a lot of GPU cores/VRAM. On my $4,000 Mac Studio M2 Ultra with 64GB of RAM, I can comfortably run deepseek-r1:32b, but a) load times can be annoying (i.e. if you are switching models for different tasks, or let them idle out) and b) you can certainly tell that it requires tuning of the context length, temperature, etc. based on what you need to do. Compare that with the commercial models where a lot of that is done on a large scale for you.
- smjburton 2y ago
  Yeah that makes sense. Once the model is loaded though, does it work well in comparison to the commercial models? Do you find that the local models hallucinate more, or don't give the same response quality?
mercer 2y ago
I've been using phi4 a lot, in most cases to process incoming transcripts of recordings. based on keywords or keyphrases, these transcripts get sent with a custom prompt, or the prompt is in the transcript itself. I'd say it's suprisingly good at exacting reminders and todos from natural language into json 'action objects', or turning what is essentially a run-on sentence of a transcript into a markdown formatted text. What I've found most fun to play with is to get it to extract metadata like an 'anxiety score' and tags. Overall, it's clearly 'dumber' than the hosted big models, and in my case I have to deal with a small context window. In general my 'vibe' is that I have to be clearer and more explicit, and it's usually better to just do multiple passes over the same text with very targeted questions. Oh, and in my case I definitely notice my laptop screeching to a halt when it's processing a big transcript, but in my case I can specifically delay those jobs to a time where I'm not at my computer.
- smjburton 2y ago
  Nice, that's a cool use case. I like that the local model gives you more privacy when sharing potentially sensitive data like voice recordings with an LLM. I've been interested in hosting one locally, but I was curious what I would be giving up compared to the commercial models. It sounds like its still possible to get a reasonable result with some caveats. Thanks.

jdelsman 2y ago

You will simply need a lot of GPU cores/VRAM. On my $4,000 Mac Studio M2 Ultra with 64GB of RAM, I can comfortably run deepseek-r1:32b, but a) load times can be annoying (i.e. if you are switching models for different tasks, or let them idle out) and b) you can certainly tell that it requires tuning of the context length, temperature, etc. based on what you need to do. Compare that with the commercial models where a lot of that is done on a large scale for you.

smjburton 2y ago
Yeah that makes sense. Once the model is loaded though, does it work well in comparison to the commercial models? Do you find that the local models hallucinate more, or don't give the same response quality?

mercer 2y ago

I've been using phi4 a lot, in most cases to process incoming transcripts of recordings. based on keywords or keyphrases, these transcripts get sent with a custom prompt, or the prompt is in the transcript itself. I'd say it's suprisingly good at exacting reminders and todos from natural language into json 'action objects', or turning what is essentially a run-on sentence of a transcript into a markdown formatted text. What I've found most fun to play with is to get it to extract metadata like an 'anxiety score' and tags. Overall, it's clearly 'dumber' than the hosted big models, and in my case I have to deal with a small context window. In general my 'vibe' is that I have to be clearer and more explicit, and it's usually better to just do multiple passes over the same text with very targeted questions. Oh, and in my case I definitely notice my laptop screeching to a halt when it's processing a big transcript, but in my case I can specifically delay those jobs to a time where I'm not at my computer.

smjburton 2y ago
Nice, that's a cool use case. I like that the local model gives you more privacy when sharing potentially sensitive data like voice recordings with an LLM. I've been interested in hosting one locally, but I was curious what I would be giving up compared to the commercial models. It sounds like its still possible to get a reasonable result with some caveats. Thanks.