3 ms·

Transformer models are quickly becoming a commodity, and I suspect in time we'll all be running them locally. Even now, you can run something pretty useful on

Transformer models are quickly becoming a commodity, and I suspect in time we'll all be running them locally. Even now, you can run something pretty useful on a 16 GB graphics card, and I suspect a decade into the future, entry level hardware will be running better models than high-end graphics cards can run now, as entry-level hardware gets better and models get more efficient. It doesn't mean hosted frontier models wont exist, they'll just be rare. It's no different than any other commodity market, for example most cars are cheap commodity models, with rare individuals buying expensive luxury cars and businesses buying expensive trucks and specialized equipment.

chr15m 2mo ago
Yep local models will be good enough for most things you need to do, in the same way as most people need a laptop not a supercomputer.
- DiscourseFan 2mo ago
  Well nobody thought you could write software that helps tightly coordinate processes happening simultaneously from millions upon millions of nodes on nearly every corner of the world, but here we are. When industry expands further off earth, we will need more complex and intricate software to coordinate its movements, why wouldn’t our systems become more powerful. If you are hopeful for humanity than you must expect the scale of industrial necessity to only ever increase alongside the imagination and capacity of its people’s.
  - gritzko 2mo ago
    Also, pairing a wireless mouse to a laptop is not getting easier as years pass.
    - dlcarrier 2mo ago
      That's a Bluetooth thing: https://xkcd.com/2055/ https://xkcd.com/2055/ My theory is that it's because the Bluetooth protocol is named in honor of the Viking king Harald "Bluetooth" Gormsson, who conquered several disparate peoples into a single kingdom, so as an act of solidarity toward those peoples, the Bluetooth protocol is written in a way where a devices are as likely to rebel against consolidation as they are to pair together.
  - chr15m 2mo ago
    I think the opposite can be true. Billions of humans could be happy on the Earth we have now for a billion years.
rustcleaner 2mo ago
>and I suspect a decade into the future, entry level hardware will be running better models than high-end graphics cards can run now, as entry-level hardware gets better and models get more efficient. I doubt it. The play seems to be: lock what was once commodity compute up into datacenters depriving us regular folk of it, then sell it back to us on subscription. Even if my #NeverSubscribe movement succeeds, all that misdirected hardware [into datacenters] won't likely be practical for home use.
- curious_cat_163 2mo ago
  > The play seems to be: lock what was once commodity compute up into datacenters depriving us regular folk of it, then sell it back to us on subscription. I had not considered that as a possibility. It is somewhat dark and unlikely, imho, but a real possibility. I am more inclined to believe that the fab capacity will grow over time and "commodity" compute will be available to us all again. However, I can also imagine going back to the 60s era "hyper verticalized" mainframes, in which case, the frontier labs might not just be producing models, but chips and an ecosystem around themselves and their suppliers/customers.
- dlcarrier 2mo ago
  I had an internship at HP in the early 2000's, and my computer was a terminal that connected to an X server in a data center. Doing everything as a service is not a new idea, and suppliers have been trying to push it for decades. The only time it really succeeds is when regulations force it. I can store files on my own computer or a home NAS just fine, but if I start a medical practice, I have pretty much no hope of being HIPAA compliant without signing up for a data hosting service. The same goes for tax preparation, banking, and several other fields. It may very well be the case in China, Europe, or Australia that regulatory restrictions force users into models-as-a-service, but in the US, regulations censoring models, even if they frame that censorship as a safety measure, won't pass constitutional muster.
- queenkjuul 2mo ago
  But the Chinese models exist, are competitive, and have locally useful quantized versions. If you're actually willing to pick up old DC inference hardware on the cheap you can already do a lot at home. The new hardware not so much admittedly but if the Chinese models keep getting better and running on less hardware... I don't see a reason to be quite so pessimistic.
FeepingCreature 2mo ago
This won't happen so long as the bandwidth bottleneck is a thing. Local models are just several orders less efficient at scale, and this limitation is inherent to the architecture and won't go away unless both hardware and model structure change enormously. There are really only two usecases I can see for local models going forward: - 10-1000 person groups such as corporations where you can amortize serious hardware with parallel use - Porn.

chr15m 2mo ago

Yep local models will be good enough for most things you need to do, in the same way as most people need a laptop not a supercomputer.

DiscourseFan 2mo ago
Well nobody thought you could write software that helps tightly coordinate processes happening simultaneously from millions upon millions of nodes on nearly every corner of the world, but here we are. When industry expands further off earth, we will need more complex and intricate software to coordinate its movements, why wouldn’t our systems become more powerful. If you are hopeful for humanity than you must expect the scale of industrial necessity to only ever increase alongside the imagination and capacity of its people’s.
- gritzko 2mo ago
  Also, pairing a wireless mouse to a laptop is not getting easier as years pass.
  - dlcarrier 2mo ago
    That's a Bluetooth thing: https://xkcd.com/2055/ https://xkcd.com/2055/ My theory is that it's because the Bluetooth protocol is named in honor of the Viking king Harald "Bluetooth" Gormsson, who conquered several disparate peoples into a single kingdom, so as an act of solidarity toward those peoples, the Bluetooth protocol is written in a way where a devices are as likely to rebel against consolidation as they are to pair together.
- chr15m 2mo ago
  I think the opposite can be true. Billions of humans could be happy on the Earth we have now for a billion years.

rustcleaner 2mo ago

>and I suspect a decade into the future, entry level hardware will be running better models than high-end graphics cards can run now, as entry-level hardware gets better and models get more efficient. I doubt it. The play seems to be: lock what was once commodity compute up into datacenters depriving us regular folk of it, then sell it back to us on subscription. Even if my #NeverSubscribe movement succeeds, all that misdirected hardware [into datacenters] won't likely be practical for home use.

curious_cat_163 2mo ago
> The play seems to be: lock what was once commodity compute up into datacenters depriving us regular folk of it, then sell it back to us on subscription. I had not considered that as a possibility. It is somewhat dark and unlikely, imho, but a real possibility. I am more inclined to believe that the fab capacity will grow over time and "commodity" compute will be available to us all again. However, I can also imagine going back to the 60s era "hyper verticalized" mainframes, in which case, the frontier labs might not just be producing models, but chips and an ecosystem around themselves and their suppliers/customers.
dlcarrier 2mo ago
I had an internship at HP in the early 2000's, and my computer was a terminal that connected to an X server in a data center. Doing everything as a service is not a new idea, and suppliers have been trying to push it for decades. The only time it really succeeds is when regulations force it. I can store files on my own computer or a home NAS just fine, but if I start a medical practice, I have pretty much no hope of being HIPAA compliant without signing up for a data hosting service. The same goes for tax preparation, banking, and several other fields. It may very well be the case in China, Europe, or Australia that regulatory restrictions force users into models-as-a-service, but in the US, regulations censoring models, even if they frame that censorship as a safety measure, won't pass constitutional muster.
queenkjuul 2mo ago
But the Chinese models exist, are competitive, and have locally useful quantized versions. If you're actually willing to pick up old DC inference hardware on the cheap you can already do a lot at home. The new hardware not so much admittedly but if the Chinese models keep getting better and running on less hardware... I don't see a reason to be quite so pessimistic.

FeepingCreature 2mo ago

This won't happen so long as the bandwidth bottleneck is a thing. Local models are just several orders less efficient at scale, and this limitation is inherent to the architecture and won't go away unless both hardware and model structure change enormously. There are really only two usecases I can see for local models going forward: - 10-1000 person groups such as corporations where you can amortize serious hardware with parallel use - Porn.