4 ms·

A long time ago, in order to get computers to do anything useful, the hardware required huge rooms. Today, a little cell phone is significantly more powerful t

A long time ago, in order to get computers to do anything useful, the hardware required huge rooms. Today, a little cell phone is significantly more powerful than a room sized computer back in the days. LLMs shouldn't be any different. Today, they require giant server farms to run. Tomorrow, they will run on little robots/cell phones. A few things will allow this: 1. Chip makers like Apple, Qualcomm, AMD, Intel, Nvidia, will heavily emphasize AI performance in their future chips. Expect accelerators like the Neural Engine to get significantly bigger. In the future, I expect that most of the transistors in a SoC will be dedicated to AI acceleration - and not CPUs/GPUs. 2. Moore's law is not dead. We will continue to get more transistors in a given area. For example, TSMC has plans for 1nm which will likely have 4-5x more transistors per area than the 5nm Apple M2 or around 100 billion transistors. For comparison, the 4090 has 76 billion transistors. A 1nm Apple Silicon Max chip could theoretically have around 300 billion transistors, which makes it 4x more than a 4090. If GPT-4 requires 4x 4090 to run, then a single 1nm M Max might be able to do it in the future. 3. Larger models will optimize to require less resources. 4. Smaller models will become more capable. These forces will converge and we will have local LLMs that are top of the line.

yyyk 3y ago
We started from thin clients connecting to room-sized mainframes made by giant corporations, and we ended at small cellphone-size devices mostly acting as thin clients connecting to datacenter-building-sized services run by giant corporations, even though the small devices could theoretically do much more than the original thin clients. Being able to run locally won't change much, if the local LLMs are owned by Microsoft, Google and Apple. Or if the 'local' LLM is actually an API call to a web service because the programmer/company decided to delegate the issue and not deal with all the software deployment issues. Or because Microsoft decided to hoover up all the data, and programmers decided it's so much easier to just call the OS API and not care whether it's running locally or remotely. Yeah, you'll have a local LLM that is pretty good. There'll always be a bit better LLM running remotely, because most people won't run giant servers and don't care to update often. How much does any of that matter, when the AI is completely controlled by GiantCorp?
- senttoschool 3y ago
  It will honestly depend on just how much better the cloud versions are relative to the local versions. If the cloud versions are significantly better (like they are now) than local LLMs, then cloud will continue to be the way. If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. In 3-4 years? I'm willing to bet that local LLMs will have a sizable market. I basically expect the neural engine inside an Apple Silicon M7 to be 80% of the SoC - instead of 10% like it is today. We're going to be buying NPUs (Neural Processor Unit) with a CPU and GPU attached to it. Right now, we're buying a CPU with a GPU and NPU attached to it. I don't expect giant corporations to control LLMs. I expect smaller companies to be able to compete and that techniques for training and deploying LLMs will eventually look similar to how software is built today - where anyone can train and deploy LLMs.
  - yyyk 3y ago
    >It will honestly depend on just how much better the cloud versions are relative to the local versions. 'Better' has many meanings. Ultimately LLM performance depends not only on processor performance, but also on data. One could imagine real-time self-training, where the real-time availability of (allegedly anonymized) data from other users leads to a stable advantage over local LLM. Another issue is deployment. Imagine this: You are a software developer using an LLM in your commercial product. You have two/three choices. A) Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers. B) Include a local LLM, and deal with all the lovely deployment issues, especially when the guys running a 8th gen Intel laptop with an iGPU and 4GB memory call and ask why they can't run your product. Perhaps there will be a third option: C) Call a local API provided by the OS, which may or may not call a cloud API in the background. One can see how even if the performance is totally equivalent, Cloud API may end up with an advantage. IMHO, this is not a desirable future, but to avoid it we need to start by cutting down on the triumphalism. [EDIT: >If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. Cost for whom? The local user will pay less if the call is remote. The cloud AIs can make up for their costs in data. Unfortunately, users often ignore their privacy. ]
TheOtherHobbes 3y ago
Today a little cell phone spends much of its time accessing hardware in huge rooms. Even if that weren't true and you look exclusively at on-prem and in-hand software, it took 25-30 years to shrink supercomputers to pocket size. Moore's Law is certainly struggling. Modern developments are at least as much about architectural refinement as transistor density. That won't change unless there's a completely new game changer technology. (Optical? Quantum?) Whatever it is, it hasn't been invented yet. So it's unlikely to be productised within the next ten years. There's no huge technical benefit to running models locally. Aside from subscription costs - which would be balanced by more expensive hardware - and maybe response time, if the cycles are there, there's not a lot you can do with a local model that you can't do with a hosted model and an API. The real issue with models is training. If you want to train locally you need access to a huge dataset. BigCo has that. You don't.
- senttoschool 3y ago
  >Today a little cell phone spends much of its time accessing hardware in huge rooms. The vast majority of this behavior are basic CRUD operations. The iPhone's single thread speed is likely faster than the server it's calling. For example, the Apple A16 contains 16 billion transistors - which is significantly more than a Zen4 CCD. >Even if that weren't true and you look exclusively at on-prem and in-hand software, it took 25-30 years to shrink supercomputers to pocket size. True but technology moves at a faster pace now. Chips get exponentially faster, not linearly. Do I expect a cell phone in 2027 to be as fast as a server room in 2023? No. But I expect the forces I outlined above to make it so that local LLMs will be useful enough for people to keep on their phone or computer? Yes. That's my bet. >The real issue with models is training. If you want to train locally you need access to a huge dataset. The assumption is that we will be downloading and updating models similar to how apps are currently deployed.

yyyk 3y ago

We started from thin clients connecting to room-sized mainframes made by giant corporations, and we ended at small cellphone-size devices mostly acting as thin clients connecting to datacenter-building-sized services run by giant corporations, even though the small devices could theoretically do much more than the original thin clients. Being able to run locally won't change much, if the local LLMs are owned by Microsoft, Google and Apple. Or if the 'local' LLM is actually an API call to a web service because the programmer/company decided to delegate the issue and not deal with all the software deployment issues. Or because Microsoft decided to hoover up all the data, and programmers decided it's so much easier to just call the OS API and not care whether it's running locally or remotely. Yeah, you'll have a local LLM that is pretty good. There'll always be a bit better LLM running remotely, because most people won't run giant servers and don't care to update often. How much does any of that matter, when the AI is completely controlled by GiantCorp?

senttoschool 3y ago
It will honestly depend on just how much better the cloud versions are relative to the local versions. If the cloud versions are significantly better (like they are now) than local LLMs, then cloud will continue to be the way. If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. In 3-4 years? I'm willing to bet that local LLMs will have a sizable market. I basically expect the neural engine inside an Apple Silicon M7 to be 80% of the SoC - instead of 10% like it is today. We're going to be buying NPUs (Neural Processor Unit) with a CPU and GPU attached to it. Right now, we're buying a CPU with a GPU and NPU attached to it. I don't expect giant corporations to control LLMs. I expect smaller companies to be able to compete and that techniques for training and deploying LLMs will eventually look similar to how software is built today - where anyone can train and deploy LLMs.
- yyyk 3y ago
  >It will honestly depend on just how much better the cloud versions are relative to the local versions. 'Better' has many meanings. Ultimately LLM performance depends not only on processor performance, but also on data. One could imagine real-time self-training, where the real-time availability of (allegedly anonymized) data from other users leads to a stable advantage over local LLM. Another issue is deployment. Imagine this: You are a software developer using an LLM in your commercial product. You have two/three choices. A) Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers. B) Include a local LLM, and deal with all the lovely deployment issues, especially when the guys running a 8th gen Intel laptop with an iGPU and 4GB memory call and ask why they can't run your product. Perhaps there will be a third option: C) Call a local API provided by the OS, which may or may not call a cloud API in the background. One can see how even if the performance is totally equivalent, Cloud API may end up with an advantage. IMHO, this is not a desirable future, but to avoid it we need to start by cutting down on the triumphalism. [EDIT: >If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. Cost for whom? The local user will pay less if the call is remote. The cloud AIs can make up for their costs in data. Unfortunately, users often ignore their privacy. ]

TheOtherHobbes 3y ago

Today a little cell phone spends much of its time accessing hardware in huge rooms. Even if that weren't true and you look exclusively at on-prem and in-hand software, it took 25-30 years to shrink supercomputers to pocket size. Moore's Law is certainly struggling. Modern developments are at least as much about architectural refinement as transistor density. That won't change unless there's a completely new game changer technology. (Optical? Quantum?) Whatever it is, it hasn't been invented yet. So it's unlikely to be productised within the next ten years. There's no huge technical benefit to running models locally. Aside from subscription costs - which would be balanced by more expensive hardware - and maybe response time, if the cycles are there, there's not a lot you can do with a local model that you can't do with a hosted model and an API. The real issue with models is training. If you want to train locally you need access to a huge dataset. BigCo has that. You don't.

senttoschool 3y ago
>Today a little cell phone spends much of its time accessing hardware in huge rooms. The vast majority of this behavior are basic CRUD operations. The iPhone's single thread speed is likely faster than the server it's calling. For example, the Apple A16 contains 16 billion transistors - which is significantly more than a Zen4 CCD. >Even if that weren't true and you look exclusively at on-prem and in-hand software, it took 25-30 years to shrink supercomputers to pocket size. True but technology moves at a faster pace now. Chips get exponentially faster, not linearly. Do I expect a cell phone in 2027 to be as fast as a server room in 2023? No. But I expect the forces I outlined above to make it so that local LLMs will be useful enough for people to keep on their phone or computer? Yes. That's my bet. >The real issue with models is training. If you want to train locally you need access to a huge dataset. The assumption is that we will be downloading and updating models similar to how apps are currently deployed.