7 ms·

Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that yo

by rs186 2y ago

Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need a glimpse of the response, copy & paste and get things done. Whereas on local LLM, I watch it painstakingly prints preambles that I don't care about, and get what I actually need after 20 seconds (on a fast GPU). And I am not yet talking about context window etc. I have been researching about how people integrate local LLMs in their workflows. My finding is that most people play with it for a short time and that's about it, and most people are much better off spending money on OpenAI credits (which can last a very long time with typical usage) than getting a beefed up Mac Studio or building a machine with 4090.

simonw 2y ago
My tooling doesn't measure TPS yet. It feels snappy to me on MLX. I agree that hosted models are usually a better option for most people - much faster, higher quality, handle longer inputs, really cheap. I enjoy local models for research and for the occasional offline scenario. I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources.
- freeamz 2y ago
  >I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources. Think it is NOT just you. Most company with decent management also would not want their data going to anything outside the physical server they have in control of. But yeah for most people just use an app and hosted server. But this is HN,there are ppl here hosting their own email servers, so shouldn't be too hard to run llm locally.
  - simonw 2y ago
    "Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.
    - terhechte 2y ago
      Or GitHub. I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub. All big llm providers offer no-training-on-your-data business plans.
      - tarruda 2y ago
        > I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub What amuses me even more is people thinking their code is too unique and precious, and that GitHub/Microsoft wants to steal it.
        AlexCoventry 2y ago
        Concern about platform risk in regard to Microsoft is historically justified.
        Terretta 2y ago
        Unlikely they think Microsoft or GitHub wants to steal it. With LLMs, they're thinking of examples that regurgitated proprietary code, and contrary to everyday general observation, valuable proprietary code does exist. But with GitHub, the thinking is generally the opposite: the worry is that the code is terrible, and seeing it would be like giant blinkenlights* indicating the way in. * https://en.wikipedia.org/wiki/Blinkenlights https://en.wikipedia.org/wiki/Blinkenlights
        vikarti 2y ago
        Regulations sometimes matter. Stupid "security" rules sometimes matter too.
- triyambakam 2y ago
  > specifically for dealing with extremely sensitive data like leaked information from confidential sources. Can you explain this further? It seems in contrast to your previous comment about trusting Anthropic with your data
  - simonw 2y ago
    I trust Anthropic not to train on my data. If they get hit by a government subpoena because a journalist has been using them to analyze leaked corporate or government secret files I also trust them to honor that subpoena. Sometimes journalists deal with material that they cannot risk leaving their own machine. "News is what somebody somewhere wants to suppress"
overfeed 2y ago
> Whereas on local LLM, I watch it painstakingly prints preambles that I don't care about, and get what I actually need after 20 seconds. You may need to "right-size" the models you use to match your hardware, model, and TPS expectations, which may involve using a smaller version of the model with faster TPS, upgrading your jardware, or paying for hosted models. Alternatively, if you can use agentic workflows or tools like Aider, you don't have to watch the model work slowly with large modles locally. Instead you queue work for it, go to sleep, or eat, or do other work, and then much later look over the Pull Requests whenever it completes them.
- rs186 2y ago
  I have a 4070 super for gaming, and used it to play with LLM a few times. It is by no means a bad card, but I realize that unless I want to get 4090 or new Macs that I don't have any other use for, I can only use it to run smaller models. However, most smaller models aren't satisfactory and are still slower than hosted LLMs. I haven't found a model that I am happy with for my hardware. Regarding agentic workflows -- sounds nice but I am too scared to try it out, based on my experience with standard LLMs like GPT or Claude for writing code. Small snippets or filling in missing unit tests, fine, anything more complicated? Has been a disaster for me.
  - taneq 2y ago
    As I understand it, these models are limited on GPU memory far more than GPU compute. You’d be better off with dual 4070s than with a single 4090 unless the 4090 has more RAM than the other two combined.
- adastra22 2y ago
  I have never found any agent able to put together sensible pull requests without constant hand holding. I shudder to think of what those repositories must look like.
otabdeveloper4 2y ago
The only actually useful application of LLM's is processing large amounts of data for classification and/or summarizing purposes. That's not the stuff you want to send to a public API, this is something you want as a 24/7 locally running batch job. ("AI assistant" is an evolutionary dead end, and Star Trek be damned.)
DJHenk 2y ago
> More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. There is another aspect to consider, aside from privacy. These models are trained by downloading every scrap of information from the internet, including the works of many, many authors who have never consented to that. And they for sure are not going to get a share of the profits, if there is every going to be any. If you use a cloud provider, you are basically saying that is all fine. You are happy to pay them, and make yourself dependent on their service, based on work that wasn't theirs to use. However, if you use a local model, the authors still did not give consent, but one could argue that the company that made the model is at least giving back to the community. They don't get any money out of it, and you are not becoming dependent on their hyper capitalist service. No rent-seeking. The benefits of the work are free to use for everyone. This makes using AI a little more acceptable from a moral standpoint.
- deleted 2y ago
  [deleted]
ein0p 2y ago
Sometimes TPS doesn't matter. I've generated textual descriptions for 100K or so images in my photo archive, some of which I have absolutely no interest in uploading to someone else's computer. This works pretty well with Gemma. I use local LLMs all the time for things where privacy is even remotely important. I estimate this constitutes easily a quarter of my LLM usage.
- lodovic 2y ago
  This is a really cool idea. Do you pretrain the model so it can tag people? I have so many photo's that it seems impossible to ever categorize them,using a workflow like yours might help a lot
  - ein0p 2y ago
    No, tagging of people is already handled by another model. Gemma just describes what's in the image, and produces a comma separated list of keywords. No additional training is required besides a few tweaks to the prompt so that it outputs just the description, without any "fluff". E.g. it normally prepends such outputs with "Here's a description of the image:" unless you really insist that it should output only the description. I suppose I could use constrained decoding into JSON or something to achieve the same, but I didn't mess with that. On some images where Gemma3 struggles Mistral Small produces better descriptions, BTW. But it seems harder to make it follow my instructions exactly. I'm looking forward to the day when I can also do this with videos, a lot of which I also have no interest in uploading to someone else's computer.
    - fer 2y ago
      How do you use the keywords after? I have Immich running which does some analysis, but the querying is a bit of a hit and miss.
      - ein0p 2y ago
        Search is indeed hit and miss. Immich, for instance, currently does absolutely nothing with the EXIF "description" field, so I store textual descriptions on the side as well. I have found Immich's search by image embeddings to be pretty weak at recall, and even weaker at ranking. IIRC Lightroom Classic (which I also use, but haven't found a way to automate this for without writing an extension) does search that field, but ranking is a bit of a dumpster fire, so your best bet is searching uncommon terms or constraining search by metadata (e.g. not just "black kitten" but "black kitten AND 2025"). I expect this to improve significantly over time - it's a fairly obvious thing to add given the available tech.
k__ 2y ago
The local LLM is your project manager, the big remote ones are the engineers and designers :D
trees101 2y ago
Not sure how accurate my stats are. I used ollama with the --verbose flag. Using a 4090 and all default settings, I get 40TPS for Gemma 29B model `ollama run gemma3:27b --verbose` gives me 42.5 TPS +-0.3TPS `ollama run gemma3:27b-it-qat --verbose` gives me 41.5 TPS +-0.3TPS Strange results; the full model gives me slightly more TPS.
- orangecat 2y ago
  ollama's `gemma3:27b` is also 4-bit quantized, you need `27b-it-q8_0` for 8 bit or `27b-it-fp16` for FP16. See https://ollama.com/library/gemma3/tags https://ollama.com/library/gemma3/tags.
starik36 2y ago
On an A5000 with 24GB, this model typically gets between 20 to 25 tps.
a_e_k 2y ago
I'm seeing ~38--42 tps on a 4090 in a fresh build of llama.cpp under Fedora 42 on my personal machine. (-t 32 -ngl 100 -c 8192 -fa -ctk q8_0 -ctv q8_0 -m models/gemma-3-27b-it-qat-q4_0.gguf)
pantulis 2y ago
> Can you quote tps? LLM Studio running on a Mac Studio M4 Max with 128GB, gemma-3-27B-it-QAT-Q4_0.gguf with a 4096 token context I get 8.89 tps.
- jychang 2y ago
  That's pretty terrible. I'm getting 18tok/sec Gemma 3 27b QAT on a M1 Max 32gb macbook.
  - pantulis 2y ago
    Yeah, I know. Not sure if this due to something in LLM Studio or whatever.
  - pantulis 2y ago
    I was running the mac in low power mode!!! Getting 24tps now.
- kristianp 2y ago
  Is QAT a different quantisation format to Q4_0? Can you try "gemma-3-27b-it-qat" for a model: https://lmstudio.ai/model/gemma-3-27b-it-qat https://lmstudio.ai/model/gemma-3-27b-it-qat
  - pantulis 2y ago
    Thanks for your suggestion! I think it was just the filename. I tried the model you suggested by opening it in LLMStudio and keep getting 8.3 tps.
    - pantulis 2y ago
      Now this is raising my curiosity, is there anything else I could try and tweak to achieve better tps? Can it be related to being GGUF instead of MLX?
- pantulis 2y ago
  Gah, turns out I was running the Mac in low power mode! I get 24tps in LM Studio now with gemma-3-27b-it-qat.
jonaustin 2y ago
On a M4 Max 128GB via LM Studio: query: "make me a snake game in python with pygame" (mlx 4 bit quant) mlx-community/gemma-3-27b-it-qat@4bit: 26.39 tok/sec • 1681 tokens 0.63s to first token (gguf 4 bit quant) lmstudio-community/gemma-3-27b-it-qat: 22.72 tok/sec • 1866 tokens 0.49s to first token using Unsloth's settings: https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-tune-gemma-3 https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-...
- yencabulator 2y ago
  I genuinely would have expected a $3,500+ setup to do better than just 10x pure-CPU on a AMD Ryzen 9 8945HS.
  - jonaustin 2y ago
    Find another laptop that does that well.

by rs186 2y ago

simonw 2y ago

My tooling doesn't measure TPS yet. It feels snappy to me on MLX. I agree that hosted models are usually a better option for most people - much faster, higher quality, handle longer inputs, really cheap. I enjoy local models for research and for the occasional offline scenario. I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources.

freeamz 2y ago
>I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources. Think it is NOT just you. Most company with decent management also would not want their data going to anything outside the physical server they have in control of. But yeah for most people just use an app and hosted server. But this is HN,there are ppl here hosting their own email servers, so shouldn't be too hard to run llm locally.
- simonw 2y ago
  "Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.
  - terhechte 2y ago
    Or GitHub. I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub. All big llm providers offer no-training-on-your-data business plans.
    - tarruda 2y ago
      > I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub What amuses me even more is people thinking their code is too unique and precious, and that GitHub/Microsoft wants to steal it.
      - AlexCoventry 2y ago
        Concern about platform risk in regard to Microsoft is historically justified.
      - Terretta 2y ago
        Unlikely they think Microsoft or GitHub wants to steal it. With LLMs, they're thinking of examples that regurgitated proprietary code, and contrary to everyday general observation, valuable proprietary code does exist. But with GitHub, the thinking is generally the opposite: the worry is that the code is terrible, and seeing it would be like giant blinkenlights* indicating the way in. * https://en.wikipedia.org/wiki/Blinkenlights https://en.wikipedia.org/wiki/Blinkenlights
      - vikarti 2y ago
        Regulations sometimes matter. Stupid "security" rules sometimes matter too.
triyambakam 2y ago
> specifically for dealing with extremely sensitive data like leaked information from confidential sources. Can you explain this further? It seems in contrast to your previous comment about trusting Anthropic with your data
- simonw 2y ago
  I trust Anthropic not to train on my data. If they get hit by a government subpoena because a journalist has been using them to analyze leaked corporate or government secret files I also trust them to honor that subpoena. Sometimes journalists deal with material that they cannot risk leaving their own machine. "News is what somebody somewhere wants to suppress"

overfeed 2y ago

> Whereas on local LLM, I watch it painstakingly prints preambles that I don't care about, and get what I actually need after 20 seconds. You may need to "right-size" the models you use to match your hardware, model, and TPS expectations, which may involve using a smaller version of the model with faster TPS, upgrading your jardware, or paying for hosted models. Alternatively, if you can use agentic workflows or tools like Aider, you don't have to watch the model work slowly with large modles locally. Instead you queue work for it, go to sleep, or eat, or do other work, and then much later look over the Pull Requests whenever it completes them.

rs186 2y ago
I have a 4070 super for gaming, and used it to play with LLM a few times. It is by no means a bad card, but I realize that unless I want to get 4090 or new Macs that I don't have any other use for, I can only use it to run smaller models. However, most smaller models aren't satisfactory and are still slower than hosted LLMs. I haven't found a model that I am happy with for my hardware. Regarding agentic workflows -- sounds nice but I am too scared to try it out, based on my experience with standard LLMs like GPT or Claude for writing code. Small snippets or filling in missing unit tests, fine, anything more complicated? Has been a disaster for me.
- taneq 2y ago
  As I understand it, these models are limited on GPU memory far more than GPU compute. You’d be better off with dual 4070s than with a single 4090 unless the 4090 has more RAM than the other two combined.
adastra22 2y ago
I have never found any agent able to put together sensible pull requests without constant hand holding. I shudder to think of what those repositories must look like.

otabdeveloper4 2y ago

The only actually useful application of LLM's is processing large amounts of data for classification and/or summarizing purposes. That's not the stuff you want to send to a public API, this is something you want as a 24/7 locally running batch job. ("AI assistant" is an evolutionary dead end, and Star Trek be damned.)

DJHenk 2y ago

> More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. There is another aspect to consider, aside from privacy. These models are trained by downloading every scrap of information from the internet, including the works of many, many authors who have never consented to that. And they for sure are not going to get a share of the profits, if there is every going to be any. If you use a cloud provider, you are basically saying that is all fine. You are happy to pay them, and make yourself dependent on their service, based on work that wasn't theirs to use. However, if you use a local model, the authors still did not give consent, but one could argue that the company that made the model is at least giving back to the community. They don't get any money out of it, and you are not becoming dependent on their hyper capitalist service. No rent-seeking. The benefits of the work are free to use for everyone. This makes using AI a little more acceptable from a moral standpoint.

deleted 2y ago
[deleted]

ein0p 2y ago

Sometimes TPS doesn't matter. I've generated textual descriptions for 100K or so images in my photo archive, some of which I have absolutely no interest in uploading to someone else's computer. This works pretty well with Gemma. I use local LLMs all the time for things where privacy is even remotely important. I estimate this constitutes easily a quarter of my LLM usage.

lodovic 2y ago
This is a really cool idea. Do you pretrain the model so it can tag people? I have so many photo's that it seems impossible to ever categorize them,using a workflow like yours might help a lot
- ein0p 2y ago
  No, tagging of people is already handled by another model. Gemma just describes what's in the image, and produces a comma separated list of keywords. No additional training is required besides a few tweaks to the prompt so that it outputs just the description, without any "fluff". E.g. it normally prepends such outputs with "Here's a description of the image:" unless you really insist that it should output only the description. I suppose I could use constrained decoding into JSON or something to achieve the same, but I didn't mess with that. On some images where Gemma3 struggles Mistral Small produces better descriptions, BTW. But it seems harder to make it follow my instructions exactly. I'm looking forward to the day when I can also do this with videos, a lot of which I also have no interest in uploading to someone else's computer.
  - fer 2y ago
    How do you use the keywords after? I have Immich running which does some analysis, but the querying is a bit of a hit and miss.
    - ein0p 2y ago
      Search is indeed hit and miss. Immich, for instance, currently does absolutely nothing with the EXIF "description" field, so I store textual descriptions on the side as well. I have found Immich's search by image embeddings to be pretty weak at recall, and even weaker at ranking. IIRC Lightroom Classic (which I also use, but haven't found a way to automate this for without writing an extension) does search that field, but ranking is a bit of a dumpster fire, so your best bet is searching uncommon terms or constraining search by metadata (e.g. not just "black kitten" but "black kitten AND 2025"). I expect this to improve significantly over time - it's a fairly obvious thing to add given the available tech.

k__ 2y ago

The local LLM is your project manager, the big remote ones are the engineers and designers :D

trees101 2y ago

Not sure how accurate my stats are. I used ollama with the --verbose flag. Using a 4090 and all default settings, I get 40TPS for Gemma 29B model `ollama run gemma3:27b --verbose` gives me 42.5 TPS +-0.3TPS `ollama run gemma3:27b-it-qat --verbose` gives me 41.5 TPS +-0.3TPS Strange results; the full model gives me slightly more TPS.

orangecat 2y ago
ollama's `gemma3:27b` is also 4-bit quantized, you need `27b-it-q8_0` for 8 bit or `27b-it-fp16` for FP16. See https://ollama.com/library/gemma3/tags https://ollama.com/library/gemma3/tags.

starik36 2y ago

On an A5000 with 24GB, this model typically gets between 20 to 25 tps.

a_e_k 2y ago

I'm seeing ~38--42 tps on a 4090 in a fresh build of llama.cpp under Fedora 42 on my personal machine. (-t 32 -ngl 100 -c 8192 -fa -ctk q8_0 -ctv q8_0 -m models/gemma-3-27b-it-qat-q4_0.gguf)

pantulis 2y ago

> Can you quote tps? LLM Studio running on a Mac Studio M4 Max with 128GB, gemma-3-27B-it-QAT-Q4_0.gguf with a 4096 token context I get 8.89 tps.

jychang 2y ago
That's pretty terrible. I'm getting 18tok/sec Gemma 3 27b QAT on a M1 Max 32gb macbook.
- pantulis 2y ago
  Yeah, I know. Not sure if this due to something in LLM Studio or whatever.
- pantulis 2y ago
  I was running the mac in low power mode!!! Getting 24tps now.
kristianp 2y ago
Is QAT a different quantisation format to Q4_0? Can you try "gemma-3-27b-it-qat" for a model: https://lmstudio.ai/model/gemma-3-27b-it-qat https://lmstudio.ai/model/gemma-3-27b-it-qat
- pantulis 2y ago
  Thanks for your suggestion! I think it was just the filename. I tried the model you suggested by opening it in LLMStudio and keep getting 8.3 tps.
  - pantulis 2y ago
    Now this is raising my curiosity, is there anything else I could try and tweak to achieve better tps? Can it be related to being GGUF instead of MLX?
pantulis 2y ago
Gah, turns out I was running the Mac in low power mode! I get 24tps in LM Studio now with gemma-3-27b-it-qat.

jonaustin 2y ago

On a M4 Max 128GB via LM Studio: query: "make me a snake game in python with pygame" (mlx 4 bit quant) mlx-community/gemma-3-27b-it-qat@4bit: 26.39 tok/sec • 1681 tokens 0.63s to first token (gguf 4 bit quant) lmstudio-community/gemma-3-27b-it-qat: 22.72 tok/sec • 1866 tokens 0.49s to first token using Unsloth's settings: https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-tune-gemma-3 https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-...

yencabulator 2y ago
I genuinely would have expected a $3,500+ setup to do better than just 10x pure-CPU on a AMD Ryzen 9 8945HS.
- jonaustin 2y ago
  Find another laptop that does that well.