4 ms·

Once you get used to using claude as an abstraction layer you start getting pretty reckless with it. My organization has the concept of "premium models" where

Once you get used to using claude as an abstraction layer you start getting pretty reckless with it. My organization has the concept of "premium models" where our limits reset every month. I hit my limit pretty quickly last month because I was burning tokens doing things that would have been a simple bash loop in the past - all because I was used to interfacing with Claude at the chat layer for all my automation needs and not thinking any more about it.

devmor 6mo ago
This is a real danger that I think a lot of people will run into as prices go up more and more in the future. Completely outside of the productivity debate, offloading cognitive tasks to LLMs leaves you less practiced in them and less ready to do them when the LLM isn't available. When you have to delegate only certain tasks to the LLM for financial reasons, you may find yourself very frustrated.
- johntash 6mo ago
  I'm really hoping locally hosted llms get to the point of competing with current-day frontier models so that we all have "unlimited" usage.
  - totalmarkdown 6mo ago
    [flagged]
- 3abiton 6mo ago
  This is the bet of many of the big AI companies, and why they're subsidizing majorly the calls. With the latest cracks by the US gov, it seems Anthropic is starting to reduce those subsidies given their edge in the game. I am starting to consider local models more seriously beside just testing, but nowadays the ram/gpu market is bloated.
  - devmor 6mo ago
    Local models just don't seem that useful for me for these particular tasks yet - the most recent versions of Codex and Claude Opus are the first time I've found them to be particularly useful in a "real engineering" context that isn't just vibe coding. Google's TurboQuant might help address this, but it also might just widen the gap even further. I am far on the skeptic edge when it comes to the generative AI side of ML tools though, so do take my opinion with that weight.
    - 3abiton 6mo ago
      Turboquant is totally irrelevant compared to current quantization methods. It has been thoroughly test by people who build inferencing engines for local models. It's all talk no actual meat to it.
      - devmor 6mo ago
        Do you have any reading on this? I find it hard to believe something announced a week ago has been “thoroughly tested”.
        3abiton 6mo ago
        Their paper TurboQuant (TQ) is not new per say. It's released last year, and heavily rehash of old ideas that were released a year prior (RabitQ). There is also [a bit of drama](https://openreview.net/forum?id=tO3ASKZlok https://openreview.net/forum?id=tO3ASKZlok) there that boils down to what it seems a bit of malpractice for google's researchers. TQ does few things: it claims better compression quality and speed, and better KV cache handling. Currently KV cache takes a load of resources beside that of the model itself. Many people applied different quantization strategy for it, but the quality degradation is a too apparent. Enter Attention Rotation. This seems to have genuinely helped KV cache compression as per [llama.cpp latest tests](https://github.com/ggml-org/llama.cpp/pull/21038 https://github.com/ggml-org/llama.cpp/pull/21038). On the other hand, [ik_llama.cpp](https://www.reddit.com/r/LocalLLaMA/comments/1s7nq6b/technical_clarification_on_turboquant_rabitq_for/odaxvf6/ https://www.reddit.com/r/LocalLLaMA/comments/1s7nq6b/technic...) did tests on the quality of turboquant-3 compared to IQ4 quantized models, and yhe quality degradation is much worse. So it's 2 things: KV compression -> good. Turboquant quantazation -> not good.
- cyanydeez 6mo ago
  Seriously, who isnt planning a local first strategy?
  - devmor 6mo ago
    I am sure a lot of people and orgs are - but realistically the majority of users need to understand and prepare not for local-first, but for the fact that they will never have that option for the models they know are the most useful to them.
  - therobots927 6mo ago
    Every series A-C startup
- flu_bar 6mo ago
  do you think we're already seeing mental atrophy play out? or do you think model inference/training will get cheap that we won't reach the point of "high prices"?

devmor 6mo ago

This is a real danger that I think a lot of people will run into as prices go up more and more in the future. Completely outside of the productivity debate, offloading cognitive tasks to LLMs leaves you less practiced in them and less ready to do them when the LLM isn't available. When you have to delegate only certain tasks to the LLM for financial reasons, you may find yourself very frustrated.

johntash 6mo ago
I'm really hoping locally hosted llms get to the point of competing with current-day frontier models so that we all have "unlimited" usage.
- totalmarkdown 6mo ago
  [flagged]
3abiton 6mo ago
This is the bet of many of the big AI companies, and why they're subsidizing majorly the calls. With the latest cracks by the US gov, it seems Anthropic is starting to reduce those subsidies given their edge in the game. I am starting to consider local models more seriously beside just testing, but nowadays the ram/gpu market is bloated.
- devmor 6mo ago
  Local models just don't seem that useful for me for these particular tasks yet - the most recent versions of Codex and Claude Opus are the first time I've found them to be particularly useful in a "real engineering" context that isn't just vibe coding. Google's TurboQuant might help address this, but it also might just widen the gap even further. I am far on the skeptic edge when it comes to the generative AI side of ML tools though, so do take my opinion with that weight.
  - 3abiton 6mo ago
    Turboquant is totally irrelevant compared to current quantization methods. It has been thoroughly test by people who build inferencing engines for local models. It's all talk no actual meat to it.
    - devmor 6mo ago
      Do you have any reading on this? I find it hard to believe something announced a week ago has been “thoroughly tested”.
      - 3abiton 6mo ago
        Their paper TurboQuant (TQ) is not new per say. It's released last year, and heavily rehash of old ideas that were released a year prior (RabitQ). There is also [a bit of drama](https://openreview.net/forum?id=tO3ASKZlok https://openreview.net/forum?id=tO3ASKZlok) there that boils down to what it seems a bit of malpractice for google's researchers. TQ does few things: it claims better compression quality and speed, and better KV cache handling. Currently KV cache takes a load of resources beside that of the model itself. Many people applied different quantization strategy for it, but the quality degradation is a too apparent. Enter Attention Rotation. This seems to have genuinely helped KV cache compression as per [llama.cpp latest tests](https://github.com/ggml-org/llama.cpp/pull/21038 https://github.com/ggml-org/llama.cpp/pull/21038). On the other hand, [ik_llama.cpp](https://www.reddit.com/r/LocalLLaMA/comments/1s7nq6b/technical_clarification_on_turboquant_rabitq_for/odaxvf6/ https://www.reddit.com/r/LocalLLaMA/comments/1s7nq6b/technic...) did tests on the quality of turboquant-3 compared to IQ4 quantized models, and yhe quality degradation is much worse. So it's 2 things: KV compression -> good. Turboquant quantazation -> not good.
cyanydeez 6mo ago
Seriously, who isnt planning a local first strategy?
- devmor 6mo ago
  I am sure a lot of people and orgs are - but realistically the majority of users need to understand and prepare not for local-first, but for the fact that they will never have that option for the models they know are the most useful to them.
- therobots927 6mo ago
  Every series A-C startup
flu_bar 6mo ago
do you think we're already seeing mental atrophy play out? or do you think model inference/training will get cheap that we won't reach the point of "high prices"?