5 ms·

We need to improve the waster and energy usage and this method doesn't. Most are not reinventing the wheel, a shared AI repository, communicated between online

We need to improve the waster and energy usage and this method doesn't. Most are not reinventing the wheel, a shared AI repository, communicated between online local computers would save a lot of need for these large models.

echelon 4mo ago
NO! This is the wrong approach that will turn us into serfs. We need big honking models that do what the leading foundation hyperscaler models do to within a few percentage points of measured performance. The small-scale models are not productive, and the duct tape solutions built on top of them are hobbyist-tier "year of Linux on desktop" toys. I imagine fedora-wearing, crypto-shilling, coupon-cutting boffins every time I see small weights thing lauded as the future. This is the Pine Phone F-Droid of AI. "SMS works most of the time on my phone, I swear! I don't really need my banking app!" That is not big model energy. Nothing outside of the top ten is worth spending any time on, and we need to focus on models that bridge the gap. You're talking about impractical toys for highly technical people wasting their own time. That doesn't move the needle or have any economic impact on the competitive landscape. We need sharp teeth that bite at the legs of the top-tier foundation labs and hold them back from running away with the prize. We've been through this time and time again over the last thirty years. It's the same shaped problem as before. We don't need toys - we need real infra for real people paying money to do work. Not freeware for freeloaders who don't spend and invest in the problem space. Large models fit that precisely, because it forces investment into a wide variety of open infra, routers, inference engines, etc. Not to mention the weights ecosystem itself.
- ktallett 4mo ago
  Firstly, unless you are the leader of any of the faangs, you are a serf on the whole, if you believe in that philosophy as being relevant. We need the right tool for the job. Certain models have minimum energy expense no matter what the task is and that's often wasted, both on the scale of some tasks and also repetition. There is a place and a need for large models, local models, and single purpose models. The same way there is a need for HPC and single board.
  - echelon 4mo ago
    > if you believe in that philosophy as being relevant. If you tell the open weights labs that you want tiny models, you'll get more shitty tiny models. If tell them you don't want tiny models and you want big models, they'll change their focus. > We need the right tool for the job. The world has enough shitty tiny models. You can frankly distill your own without wasting the time of foundation labs. > energy expense You are so far removed from the discourse that I think I'm wasting my time talking to you. This is not about energy in the slightest. > There is a place and a need for large models, local models, and single purpose models. This is where 99.999% of the demand is. And if you look, you'll notice that most of the token usage is accruing to hyperscalers because the market isn't meeting this demand with enough alternatives. The robotics industry will produce energy efficient models. Let them be the ones to press for that. Stop demanding cheapo LLMs, image, and video models. Big is all that matters in this domain.
- robwwilliams 4mo ago
  I initial thought this was a great tongue-in-cheek comment. I still think it is a joke.
  - ktallett 4mo ago
    Oh no, he is just so into focusing on profit and "REAL WORK" that this is serious
simonw 4mo ago
I'd love to see credible numbers on the energy usage of thousands of people running models on their own devices compared to sharing data center resources to run big models that serve many different people at the same time. My hunch is that the energy/water usage of the data centers is a whole lot more efficient than everyone running at home, but I'd be interested in seeing real data on that.
- cold_harbor 4mo ago
  the comparison misses that local LLM usage covers tasks you'd never send to an API — private code, offline work, medical notes. the baseline is 'local vs not-doing-it', not 'local vs cloud'
  - simonw 4mo ago
    I have bad news about my private code and medical notes...
    - Chu4eeno 4mo ago
      Looking forward to some pre/non-finetuned frontier model to leak, and people to start completing medical notes.
      - echelon 4mo ago
        > Looking forward to some pre/non-finetuned frontier model to leak I'm looking forward to this too. It'll be incredibly useful.
- verdverm 4mo ago
  With hardware like the Spark and Strix, the water usage is known to be zero, yea? On the energy front, I assume less efficient, but I also think there is a tradeoff in efficiency versus freedom, that's why I have my own hardware.
  - 4mo ago

echelon 4mo ago

NO! This is the wrong approach that will turn us into serfs. We need big honking models that do what the leading foundation hyperscaler models do to within a few percentage points of measured performance. The small-scale models are not productive, and the duct tape solutions built on top of them are hobbyist-tier "year of Linux on desktop" toys. I imagine fedora-wearing, crypto-shilling, coupon-cutting boffins every time I see small weights thing lauded as the future. This is the Pine Phone F-Droid of AI. "SMS works most of the time on my phone, I swear! I don't really need my banking app!" That is not big model energy. Nothing outside of the top ten is worth spending any time on, and we need to focus on models that bridge the gap. You're talking about impractical toys for highly technical people wasting their own time. That doesn't move the needle or have any economic impact on the competitive landscape. We need sharp teeth that bite at the legs of the top-tier foundation labs and hold them back from running away with the prize. We've been through this time and time again over the last thirty years. It's the same shaped problem as before. We don't need toys - we need real infra for real people paying money to do work. Not freeware for freeloaders who don't spend and invest in the problem space. Large models fit that precisely, because it forces investment into a wide variety of open infra, routers, inference engines, etc. Not to mention the weights ecosystem itself.

ktallett 4mo ago
Firstly, unless you are the leader of any of the faangs, you are a serf on the whole, if you believe in that philosophy as being relevant. We need the right tool for the job. Certain models have minimum energy expense no matter what the task is and that's often wasted, both on the scale of some tasks and also repetition. There is a place and a need for large models, local models, and single purpose models. The same way there is a need for HPC and single board.
- echelon 4mo ago
  > if you believe in that philosophy as being relevant. If you tell the open weights labs that you want tiny models, you'll get more shitty tiny models. If tell them you don't want tiny models and you want big models, they'll change their focus. > We need the right tool for the job. The world has enough shitty tiny models. You can frankly distill your own without wasting the time of foundation labs. > energy expense You are so far removed from the discourse that I think I'm wasting my time talking to you. This is not about energy in the slightest. > There is a place and a need for large models, local models, and single purpose models. This is where 99.999% of the demand is. And if you look, you'll notice that most of the token usage is accruing to hyperscalers because the market isn't meeting this demand with enough alternatives. The robotics industry will produce energy efficient models. Let them be the ones to press for that. Stop demanding cheapo LLMs, image, and video models. Big is all that matters in this domain.
robwwilliams 4mo ago
I initial thought this was a great tongue-in-cheek comment. I still think it is a joke.
- ktallett 4mo ago
  Oh no, he is just so into focusing on profit and "REAL WORK" that this is serious

simonw 4mo ago

I'd love to see credible numbers on the energy usage of thousands of people running models on their own devices compared to sharing data center resources to run big models that serve many different people at the same time. My hunch is that the energy/water usage of the data centers is a whole lot more efficient than everyone running at home, but I'd be interested in seeing real data on that.

cold_harbor 4mo ago
the comparison misses that local LLM usage covers tasks you'd never send to an API — private code, offline work, medical notes. the baseline is 'local vs not-doing-it', not 'local vs cloud'
- simonw 4mo ago
  I have bad news about my private code and medical notes...
  - Chu4eeno 4mo ago
    Looking forward to some pre/non-finetuned frontier model to leak, and people to start completing medical notes.
    - echelon 4mo ago
      > Looking forward to some pre/non-finetuned frontier model to leak I'm looking forward to this too. It'll be incredibly useful.
verdverm 4mo ago
With hardware like the Spark and Strix, the water usage is known to be zero, yea? On the energy front, I assume less efficient, but I also think there is a tradeoff in efficiency versus freedom, that's why I have my own hardware.
- 4mo ago