8 ms·

1) Giant serverfarms are expensive, but Microsoft has money and savings on scale. This can compare favourably to running locally. 2) People use what LLMs softw

• senttoschool 3y ago
  A long time ago, in order to get computers to do anything useful, the hardware required huge rooms. Today, a little cell phone is significantly more powerful than a room sized computer back in the days. LLMs shouldn't be any different. Today, they require giant server farms to run. Tomorrow, they will run on little robots/cell phones. A few things will allow this: 1. Chip makers like Apple, Qualcomm, AMD, Intel, Nvidia, will heavily emphasize AI performance in their future chips. Expect accelerators like the Neural Engine to get significantly bigger. In the future, I expect that most of the transistors in a SoC will be dedicated to AI acceleration - and not CPUs/GPUs. 2. Moore's law is not dead. We will continue to get more transistors in a given area. For example, TSMC has plans for 1nm which will likely have 4-5x more transistors per area than the 5nm Apple M2 or around 100 billion transistors. For comparison, the 4090 has 76 billion transistors. A 1nm Apple Silicon Max chip could theoretically have around 300 billion transistors, which makes it 4x more than a 4090. If GPT-4 requires 4x 4090 to run, then a single 1nm M Max might be able to do it in the future. 3. Larger models will optimize to require less resources. 4. Smaller models will become more capable. These forces will converge and we will have local LLMs that are top of the line.
  • yyyk 3y ago
    We started from thin clients connecting to room-sized mainframes made by giant corporations, and we ended at small cellphone-size devices mostly acting as thin clients connecting to datacenter-building-sized services run by giant corporations, even though the small devices could theoretically do much more than the original thin clients. Being able to run locally won't change much, if the local LLMs are owned by Microsoft, Google and Apple. Or if the 'local' LLM is actually an API call to a web service because the programmer/company decided to delegate the issue and not deal with all the software deployment issues. Or because Microsoft decided to hoover up all the data, and programmers decided it's so much easier to just call the OS API and not care whether it's running locally or remotely. Yeah, you'll have a local LLM that is pretty good. There'll always be a bit better LLM running remotely, because most people won't run giant servers and don't care to update often. How much does any of that matter, when the AI is completely controlled by GiantCorp?
    • senttoschool 3y ago
      It will honestly depend on just how much better the cloud versions are relative to the local versions. If the cloud versions are significantly better (like they are now) than local LLMs, then cloud will continue to be the way. If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. In 3-4 years? I'm willing to bet that local LLMs will have a sizable market. I basically expect the neural engine inside an Apple Silicon M7 to be 80% of the SoC - instead of 10% like it is today. We're going to be buying NPUs (Neural Processor Unit) with a CPU and GPU attached to it. Right now, we're buying a CPU with a GPU and NPU attached to it. I don't expect giant corporations to control LLMs. I expect smaller companies to be able to compete and that techniques for training and deploying LLMs will eventually look similar to how software is built today - where anyone can train and deploy LLMs.
      • yyyk 3y ago
        >It will honestly depend on just how much better the cloud versions are relative to the local versions. 'Better' has many meanings. Ultimately LLM performance depends not only on processor performance, but also on data. One could imagine real-time self-training, where the real-time availability of (allegedly anonymized) data from other users leads to a stable advantage over local LLM. Another issue is deployment. Imagine this: You are a software developer using an LLM in your commercial product. You have two/three choices. A) Call a cloud API using a simple cross-platform HTTP call. This will reliably work. While at it, you may gain useful data on your customers. B) Include a local LLM, and deal with all the lovely deployment issues, especially when the guys running a 8th gen Intel laptop with an iGPU and 4GB memory call and ask why they can't run your product. Perhaps there will be a third option: C) Call a local API provided by the OS, which may or may not call a cloud API in the background. One can see how even if the performance is totally equivalent, Cloud API may end up with an advantage. IMHO, this is not a desirable future, but to avoid it we need to start by cutting down on the triumphalism. [EDIT: >If local LLMs reach 95% of what cloud versions can do, then I think local ones might win out because the cost will be smaller, it will be faster (in latency), and it will have more privacy. Cost for whom? The local user will pay less if the call is remote. The cloud AIs can make up for their costs in data. Unfortunately, users often ignore their privacy. ]
• mtlmtlmtlmtl 3y ago
  1) Did you do the math? Makes no intuitive sense to me. 2) did you do a survey? Because I see no reason to just assume no one wants or needs on-prem. 3) you're extrapolating the current state of affairs into the future with no logical justification for doing so. 4) This just relies on 1-3 and is also only speculation. I can easily see scenarious where big tech fail to pivot into AI properly and go down.
  • yyyk 3y ago
    1) I can't do the complete math since I don't know the exact costs, but basically: A) Running a remote cloud LLM costs about nothing in dollars to the user. B) Running a remote cloud LLM does cost for the provider. Note however Microsoft and Google are the cloud, so they don't have to pay the absurd profit margins. They already have specialized hardware and can bring their costs down. C) Microsoft and Google can probably make money from your data, even if it's 'anonymized' LLM requests. D) Data for convenience is a typical bargain, e.g. in the search space. There's good reason to assume it can work here too. 2-3) I'm not arguing no one needs on prem. I am extrapolating the current state of affairs into the future, because, well, we don't have much else to go on. The current state of affairs isn't good. 4) Yes, this is called a logical extrapolation. I don't think you have much to base scenarios where big tech goes down upon, but I'll be glad to hear an argument. Hopefully being based on more than predictions of local LLM performance.
    • usrbinbash 3y ago
      > Running a remote cloud LLM costs about nothing in dollars to the user. The user doesn't care where it runs, because the user interacts with my product, not my backend. He also doesn't pay my cloud provider, he pays me. I think we don't need to argue the fact that an on premise solution is cheaper than a cloud solution for a lot of tasks, especially when talking about bounded resources. There is some convenience in setup, and some maintenance tasks are easier, but this comes at significant costs, especially as projects get larger. > Hopefully being based on more than predictions of local LLM performance. What else should they be based on, pray, given the fact that smaller models suitable for on-premise and even on-machine use are improving rapidly? https://arxiv.org/pdf/2303.16199.pdf https://arxiv.org/pdf/2303.16199.pdf Are they at the performance levels of cloud based very large LLMs? Not yet. But their turnover times are measured in weeks, not months. And it's not a question if there will be a higher quality base model, only when that will happen.
• usrbinbash 3y ago
  > 4) but doing product is icky for OSS people for some reason I am not talking about OSS people. I am talking about integrators. And there is already demand for on premise LLM solutions. Integrators don't ask customers to `git clone` some repo, same as they don't ask them to familiarize themselves with some cloud providers API. They ask them to buy/use the already integrated product. > Note that the Stable Diffusion area is in way better shape. Yes, because integrators got to work wrapping up the OSS into viable products. The same is going to happen with LLMs.
  • yyyk 3y ago
    >there is already demand for on premise LLM solutions. There's also demand for on premise office suits and on premise internal company communication and private smartphones. Guess who controls these markets and how often it's on premise? Unfortunately, experience suggests users would drop privacy for convenience... >Integrators Here the requirements are far more complex than for the SD case. SD can run standalone, or as plugin. Its users are adapt, tend to have good GPUs, and there isn't that much relevant software - one can do a bespoke solution for each (a PS plugin is enough to keep many happy). For LLMs, either you supply a cloud API, or deal with a significant deployment burden nobody AFAIK has even started handling. Not to mention the licenses... Current best hope for on-premise is that one of the OS makers decides to do the work and does it well enough we could trust it.
    • usrbinbash 3y ago
      > Guess who controls these markets These markets benefit from deep OS integration, interconnectivity between users products and a long history of "we have always used Word". None of that is the case for LLMs. > For LLMs, either you supply a cloud API, or deal with a significant deployment burden nobody AFAIK has even started handling. Why would there be a deployment burden? The actual setup isn't hard, the rest is wrapping the thingamabob into an API the product can use...same btw. as I have to do for cloud-LLM APIs, because the product ideally wants to be able to use different providers, so I have to abstract the API away anyway. Sure, I have to worry about infrastructure, maintenance, and performance characteristics, but that's hardly different from running our databases or web-backends. > Current best hope for on-premise is that one of the OS makers decides to do the work and does it well enough we could trust it. Why? I don't have to trust the OS providers to integrate, say, postgres or nginx or redis either. Why would I need their help to run another piece of software that lives entirely in userspace?