3 ms·

It will honestly depend on just how much better the cloud versions are relative to the local versions. If the cloud versions are significantly better (like the

It will honestly depend on just how much better the cloud versions are relative to the local versions. If the cloud versions are significantly better (like they are now) than local LLMs, then cloud will continue to be the way. If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. In 3-4 years? I'm willing to bet that local LLMs will have a sizable market. I basically expect the neural engine inside an Apple Silicon M7 to be 80% of the SoC - instead of 10% like it is today. We're going to be buying NPUs (Neural Processor Unit) with a CPU and GPU attached to it. Right now, we're buying a CPU with a GPU and NPU attached to it. I don't expect giant corporations to control LLMs. I expect smaller companies to be able to compete and that techniques for training and deploying LLMs will eventually look similar to how software is built today - where anyone can train and deploy LLMs.

yyyk 3y ago
>It will honestly depend on just how much better the cloud versions are relative to the local versions. 'Better' has many meanings. Ultimately LLM performance depends not only on processor performance, but also on data. One could imagine real-time self-training, where the real-time availability of (allegedly anonymized) data from other users leads to a stable advantage over local LLM. Another issue is deployment. Imagine this: You are a software developer using an LLM in your commercial product. You have two/three choices. A) Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers. B) Include a local LLM, and deal with all the lovely deployment issues, especially when the guys running a 8th gen Intel laptop with an iGPU and 4GB memory call and ask why they can't run your product. Perhaps there will be a third option: C) Call a local API provided by the OS, which may or may not call a cloud API in the background. One can see how even if the performance is totally equivalent, Cloud API may end up with an advantage. IMHO, this is not a desirable future, but to avoid it we need to start by cutting down on the triumphalism. [EDIT: >If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. Cost for whom? The local user will pay less if the call is remote. The cloud AIs can make up for their costs in data. Unfortunately, users often ignore their privacy. ]
- senttoschool 3y ago
  >Another issue is deployment. Imagine this: You are a software developer using an LLM in your commercial product. You have two/three choices. >Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers In the not too distant future, I expect most applications to be replaced by LLM assistants. So the above scenario won't really happen. You will interact with one LLM, maybe it's a local flavor or maybe it's a cloud flavor. The LLM will then call other services for you. So I don't expect services to call LLMs but I do expect LLMs to call other services. Yes, we are getting a little ahead of ourselves.
  - yyyk 3y ago
    >>Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers. >In the not too distant future, I expect most applications to be replaced by LLM assistants. So the above scenario won't really happen. Regardless we have the question of how to deploy the local LLM. If the frontend will be the OS, than we can be reliably sure that Microsoft and Google will have motivation to use their cloud hosted AI and you may have to fight the OS to get it to install your model. We may need regulation to ensure local LLMs are possible.
    - usrbinbash 3y ago
      > Regardless we have the question of how to deploy the local LLM. The same as we do any other software? There is nothing inherently more difficult in deploying an ML model than there is, say, in setting up a redudant high traffic RDBMS. > If the frontend will be the OS Why would the frontend be the OS? The OS isn't the frontend to my database. The OS isn't my IDE. The OS isn't the games I play. The OS isn't even my browser. All these are Userspace programs.
      - yyyk 3y ago
        >Why would the frontend be the OS? I answered this above. https://news.ycombinator.com/item?id=36194304 https://news.ycombinator.com/item?id=36194304

yyyk 3y ago

>It will honestly depend on just how much better the cloud versions are relative to the local versions. 'Better' has many meanings. Ultimately LLM performance depends not only on processor performance, but also on data. One could imagine real-time self-training, where the real-time availability of (allegedly anonymized) data from other users leads to a stable advantage over local LLM. Another issue is deployment. Imagine this: You are a software developer using an LLM in your commercial product. You have two/three choices. A) Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers. B) Include a local LLM, and deal with all the lovely deployment issues, especially when the guys running a 8th gen Intel laptop with an iGPU and 4GB memory call and ask why they can't run your product. Perhaps there will be a third option: C) Call a local API provided by the OS, which may or may not call a cloud API in the background. One can see how even if the performance is totally equivalent, Cloud API may end up with an advantage. IMHO, this is not a desirable future, but to avoid it we need to start by cutting down on the triumphalism. [EDIT: >If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. Cost for whom? The local user will pay less if the call is remote. The cloud AIs can make up for their costs in data. Unfortunately, users often ignore their privacy. ]

senttoschool 3y ago
>Another issue is deployment. Imagine this: You are a software developer using an LLM in your commercial product. You have two/three choices. >Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers In the not too distant future, I expect most applications to be replaced by LLM assistants. So the above scenario won't really happen. You will interact with one LLM, maybe it's a local flavor or maybe it's a cloud flavor. The LLM will then call other services for you. So I don't expect services to call LLMs but I do expect LLMs to call other services. Yes, we are getting a little ahead of ourselves.
- yyyk 3y ago
  >>Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers. >In the not too distant future, I expect most applications to be replaced by LLM assistants. So the above scenario won't really happen. Regardless we have the question of how to deploy the local LLM. If the frontend will be the OS, than we can be reliably sure that Microsoft and Google will have motivation to use their cloud hosted AI and you may have to fight the OS to get it to install your model. We may need regulation to ensure local LLMs are possible.
  - usrbinbash 3y ago
    > Regardless we have the question of how to deploy the local LLM. The same as we do any other software? There is nothing inherently more difficult in deploying an ML model than there is, say, in setting up a redudant high traffic RDBMS. > If the frontend will be the OS Why would the frontend be the OS? The OS isn't the frontend to my database. The OS isn't my IDE. The OS isn't the games I play. The OS isn't even my browser. All these are Userspace programs.
    - yyyk 3y ago
      >Why would the frontend be the OS? I answered this above. https://news.ycombinator.com/item?id=36194304 https://news.ycombinator.com/item?id=36194304