3 ms·

I was wondering today if we would start to see the reverse of this. Small ASICS or some kind of optimized for LLM Gpu for desktop / or maybe even laptops of mob

I was wondering today if we would start to see the reverse of this. Small ASICS or some kind of optimized for LLM Gpu for desktop / or maybe even laptops of mobile. It is evident I think that LLM are here to stay and will be a major part of computing for a while. Getting this local, so we aren't reliant on clouds would be a huge boon for personal computing. Even if its a "worse" experience, being able to load up an LLM into our computer, tell it to only look at this directory and help out would be cool.

01100011 4y ago
A couple of the big players are already looking at developing their own chips.
- JonChesterfield 4y ago
  Have been for years. Maybe lots of years. It's expensive to have a go (many engineers plus cost of making the things) and it's difficult to beat the established players unless you see something they're doing wrong or your particular niche really cares about something the off the shelf hardware doesn't.
ethbr0 4y ago
Software/hardware co-evolution. Wouldn't be the first time we went down that road to good effect. For anything that can be run remotely, it'll always be deployed and optimized server-side first. Higher utilization means more economy. Then trickle down to local and end user devices if it makes sense.
wyldfire 4y ago
In fact, Qualcomm has announced a "Cloud AI" PCIe card designed for inference (as opposed to training & inference) [1, 2]. It's populated with NSPs like the ones in mobile SoCs. [1] https://www.qualcomm.com/products/technology/processors/cloud-artificial-intelligence/cloud-ai-100 https://www.qualcomm.com/products/technology/processors/clou... [2] https://github.com/quic/software-kit-for-qualcomm-cloud-ai-100-cc https://github.com/quic/software-kit-for-qualcomm-cloud-ai-1...
Sol- 4y ago
Why is the sentiment here so much that LLMs will somehow be decentralized and run locally at some point? Has the story of the internet so far not been that centralization has pretty much always won?
- throwaway743 4y ago
  Sure, for big business, but torrents are still alive and well.
- kaoD 4y ago
  I think it's because it feels more similar to Google Stadia than to Facebook.
- wmf 4y ago
  Hackers want to run LLMs locally just because. It's not a mainstream thing.
  - capableweb 4y ago
    It makes business sense as well. It doesn't make much sense to build an entire company around the idea that OpenAI's APIs are always available and you won't eventually get screwed. "Be careful of basing your business on top of another" and all that yadda yadda. If you want to build a business around LLMs, it makes a lot of sense to be able to run the core service of what you want to offer on your own infrastructure instead of rely on a 3rd party that most likely doesn't give more than 1% care about you.
    - wmf 4y ago
      Running LLMs on your own servers doesn't mean PCs which is what this thread is about. A100/H100 is fine for a business but people can't justify them for personal use.
- jacquesm 4y ago
  Because that is pretty much the pendulum swinging in the IT world. Right now it is solidly in 'centralization' territory, hopefully it will go back towards decentralization again in the future. The whole PC revolution was an excellent datapoint for decentralization, now we're back to 'dumb terminals' but as local compute strengthens the things that you need a whole farm of servers for today can probably fit in your pocket tomorrow, or at the latest in a few years.
wmf 4y ago
Apple, Intel, AMD, Qualcomm, Samsung, etc. already have "neural engines" in their SoCs. These engines continue to evolve to better support common types of models.

01100011 4y ago

A couple of the big players are already looking at developing their own chips.

JonChesterfield 4y ago
Have been for years. Maybe lots of years. It's expensive to have a go (many engineers plus cost of making the things) and it's difficult to beat the established players unless you see something they're doing wrong or your particular niche really cares about something the off the shelf hardware doesn't.

ethbr0 4y ago

Software/hardware co-evolution. Wouldn't be the first time we went down that road to good effect. For anything that can be run remotely, it'll always be deployed and optimized server-side first. Higher utilization means more economy. Then trickle down to local and end user devices if it makes sense.

wyldfire 4y ago

In fact, Qualcomm has announced a "Cloud AI" PCIe card designed for inference (as opposed to training & inference) [1, 2]. It's populated with NSPs like the ones in mobile SoCs. [1] https://www.qualcomm.com/products/technology/processors/cloud-artificial-intelligence/cloud-ai-100 https://www.qualcomm.com/products/technology/processors/clou... [2] https://github.com/quic/software-kit-for-qualcomm-cloud-ai-100-cc https://github.com/quic/software-kit-for-qualcomm-cloud-ai-1...

Sol- 4y ago

Why is the sentiment here so much that LLMs will somehow be decentralized and run locally at some point? Has the story of the internet so far not been that centralization has pretty much always won?

throwaway743 4y ago
Sure, for big business, but torrents are still alive and well.
kaoD 4y ago
I think it's because it feels more similar to Google Stadia than to Facebook.
wmf 4y ago
Hackers want to run LLMs locally just because. It's not a mainstream thing.
- capableweb 4y ago
  It makes business sense as well. It doesn't make much sense to build an entire company around the idea that OpenAI's APIs are always available and you won't eventually get screwed. "Be careful of basing your business on top of another" and all that yadda yadda. If you want to build a business around LLMs, it makes a lot of sense to be able to run the core service of what you want to offer on your own infrastructure instead of rely on a 3rd party that most likely doesn't give more than 1% care about you.
  - wmf 4y ago
    Running LLMs on your own servers doesn't mean PCs which is what this thread is about. A100/H100 is fine for a business but people can't justify them for personal use.
jacquesm 4y ago
Because that is pretty much the pendulum swinging in the IT world. Right now it is solidly in 'centralization' territory, hopefully it will go back towards decentralization again in the future. The whole PC revolution was an excellent datapoint for decentralization, now we're back to 'dumb terminals' but as local compute strengthens the things that you need a whole farm of servers for today can probably fit in your pocket tomorrow, or at the latest in a few years.

wmf 4y ago

Apple, Intel, AMD, Qualcomm, Samsung, etc. already have "neural engines" in their SoCs. These engines continue to evolve to better support common types of models.