4 ms·

> 4) but doing product is icky for OSS people for some reason I am not talking about OSS people. I am talking about integrators. And there is already demand fo

> 4) but doing product is icky for OSS people for some reason I am not talking about OSS people. I am talking about integrators. And there is already demand for on premise LLM solutions. Integrators don't ask customers to `git clone` some repo, same as they don't ask them to familiarize themselves with some cloud providers API. They ask them to buy/use the already integrated product. > Note that the Stable Diffusion area is in way better shape. Yes, because integrators got to work wrapping up the OSS into viable products. The same is going to happen with LLMs.

yyyk 3y ago
>there is already demand for on premise LLM solutions. There's also demand for on premise office suits and on premise internal company communication and private smartphones. Guess who controls these markets and how often it's on premise? Unfortunately, experience suggests users would drop privacy for convenience... >Integrators Here the requirements are far more complex than for the SD case. SD can run standalone, or as plugin. Its users are adapt, tend to have good GPUs, and there isn't that much relevant software - one can do a bespoke solution for each (a PS plugin is enough to keep many happy). For LLMs, either you supply a cloud API, or deal with a significant deployment burden nobody AFAIK has even started handling. Not to mention the licenses... Current best hope for on-premise is that one of the OS makers decides to do the work and does it well enough we could trust it.
- usrbinbash 3y ago
  > Guess who controls these markets These markets benefit from deep OS integration, interconnectivity between users products and a long history of "we have always used Word". None of that is the case for LLMs. > For LLMs, either you supply a cloud API, or deal with a significant deployment burden nobody AFAIK has even started handling. Why would there be a deployment burden? The actual setup isn't hard, the rest is wrapping the thingamabob into an API the product can use...same btw. as I have to do for cloud-LLM APIs, because the product ideally wants to be able to use different providers, so I have to abstract the API away anyway. Sure, I have to worry about infrastructure, maintenance, and performance characteristics, but that's hardly different from running our databases or web-backends. > Current best hope for on-premise is that one of the OS makers decides to do the work and does it well enough we could trust it. Why? I don't have to trust the OS providers to integrate, say, postgres or nginx or redis either. Why would I need their help to run another piece of software that lives entirely in userspace?
  - yyyk 3y ago
    >These markets benefit from deep OS integration, interconnectivity between users products and a long history of "we have always used Word". None of that is the case for LLMs. The old moats did not stop existing because LLMs came along. e.g. a big use case for LLMs is running over one's own documents. We can guess who very likely already has access (MS and Google), and who is going to get a scary permission prompt at best (everyone else). OS integration is obviously coming, MS announced Copilot and Google can just silently upgrade Assistant. We may not be far from a repeat of the IE saga, where MS declares Copilot as essential part of Windows. My point is that commercial demand for privacy is not enough, and may not prevent Cloud APIs from dominating. >>For LLMs, either you supply a cloud API, or deal with a significant deployment burden >Why would there be a deployment burden? The actual setup isn't hard, the rest is wrapping the thingamabob into an API the product can use... Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs, and many many hardware variations (like the many Intel AVX variants, low memory configurations, etc.). Just take a look at the recent moves to flatpack/snap everything, and that's merely due to software variations on a single OS. >Why? I don't have to trust the OS providers to integrate, say, postgres or nginx or redis either. Why would I need their help to run another piece of software that lives entirely in userspace? I expect LLMs to be a ubiquitous UI paradigm. We do expect OS providers to bundle a UI kit, and not everyone to roll their own, right? Besides, you are comparing to server software. But non-cloud LLMs are expected to run locally on clients. That's a different world. Clients are a lot more diverse, weaker in processing power, and most can't handle upgrading postgres. All that said, perhaps we should look at legislation as well as the possibility of a OS provider playing nice. We're talking about a lot of data/power in the hands of a few corps.
    - usrbinbash 3y ago
      > Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs, You mean, like a runtime that can be a single, statically linked binary, that requires a c-compiler? With bindings for several other popular languages? https://github.com/ggerganov/llama.cpp/ https://github.com/ggerganov/llama.cpp/ Yeah, maybe I am an optimist, but somehow I don't see that being a big problem... > I expect LLMs to be a ubiquitous UI paradigm. I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software. Software that already bundles large amounts of libraries. But what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter. > But non-cloud LLMs are expected to run locally on clients. They don't have to. I can already see many useful applications for LLMs that run in on-premises servers. I never said that open source LLMs have to be restricted to client-devices. > All that said, perhaps we should look at legislation as well as the possibility of a OS provider playing nice. Legislation is another good argument for open source LLMs taking over. Looking at the EU AI Act, and the recent ideas coming up in the US, it looks like model training data being open for audits is gonna be a thing in the future, especially if the model is used in a context a lot of peoples livelihoods could rely on.

yyyk 3y ago

>there is already demand for on premise LLM solutions. There's also demand for on premise office suits and on premise internal company communication and private smartphones. Guess who controls these markets and how often it's on premise? Unfortunately, experience suggests users would drop privacy for convenience... >Integrators Here the requirements are far more complex than for the SD case. SD can run standalone, or as plugin. Its users are adapt, tend to have good GPUs, and there isn't that much relevant software - one can do a bespoke solution for each (a PS plugin is enough to keep many happy). For LLMs, either you supply a cloud API, or deal with a significant deployment burden nobody AFAIK has even started handling. Not to mention the licenses... Current best hope for on-premise is that one of the OS makers decides to do the work and does it well enough we could trust it.

usrbinbash 3y ago
> Guess who controls these markets These markets benefit from deep OS integration, interconnectivity between users products and a long history of "we have always used Word". None of that is the case for LLMs. > For LLMs, either you supply a cloud API, or deal with a significant deployment burden nobody AFAIK has even started handling. Why would there be a deployment burden? The actual setup isn't hard, the rest is wrapping the thingamabob into an API the product can use...same btw. as I have to do for cloud-LLM APIs, because the product ideally wants to be able to use different providers, so I have to abstract the API away anyway. Sure, I have to worry about infrastructure, maintenance, and performance characteristics, but that's hardly different from running our databases or web-backends. > Current best hope for on-premise is that one of the OS makers decides to do the work and does it well enough we could trust it. Why? I don't have to trust the OS providers to integrate, say, postgres or nginx or redis either. Why would I need their help to run another piece of software that lives entirely in userspace?
- yyyk 3y ago
  >These markets benefit from deep OS integration, interconnectivity between users products and a long history of "we have always used Word". None of that is the case for LLMs. The old moats did not stop existing because LLMs came along. e.g. a big use case for LLMs is running over one's own documents. We can guess who very likely already has access (MS and Google), and who is going to get a scary permission prompt at best (everyone else). OS integration is obviously coming, MS announced Copilot and Google can just silently upgrade Assistant. We may not be far from a repeat of the IE saga, where MS declares Copilot as essential part of Windows. My point is that commercial demand for privacy is not enough, and may not prevent Cloud APIs from dominating. >>For LLMs, either you supply a cloud API, or deal with a significant deployment burden >Why would there be a deployment burden? The actual setup isn't hard, the rest is wrapping the thingamabob into an API the product can use... Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs, and many many hardware variations (like the many Intel AVX variants, low memory configurations, etc.). Just take a look at the recent moves to flatpack/snap everything, and that's merely due to software variations on a single OS. >Why? I don't have to trust the OS providers to integrate, say, postgres or nginx or redis either. Why would I need their help to run another piece of software that lives entirely in userspace? I expect LLMs to be a ubiquitous UI paradigm. We do expect OS providers to bundle a UI kit, and not everyone to roll their own, right? Besides, you are comparing to server software. But non-cloud LLMs are expected to run locally on clients. That's a different world. Clients are a lot more diverse, weaker in processing power, and most can't handle upgrading postgres. All that said, perhaps we should look at legislation as well as the possibility of a OS provider playing nice. We're talking about a lot of data/power in the hands of a few corps.
  - usrbinbash 3y ago
    > Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs, You mean, like a runtime that can be a single, statically linked binary, that requires a c-compiler? With bindings for several other popular languages? https://github.com/ggerganov/llama.cpp/ https://github.com/ggerganov/llama.cpp/ Yeah, maybe I am an optimist, but somehow I don't see that being a big problem... > I expect LLMs to be a ubiquitous UI paradigm. I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software. Software that already bundles large amounts of libraries. But what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter. > But non-cloud LLMs are expected to run locally on clients. They don't have to. I can already see many useful applications for LLMs that run in on-premises servers. I never said that open source LLMs have to be restricted to client-devices. > All that said, perhaps we should look at legislation as well as the possibility of a OS provider playing nice. Legislation is another good argument for open source LLMs taking over. Looking at the EU AI Act, and the recent ideas coming up in the US, it looks like model training data being open for audits is gonna be a thing in the future, especially if the model is used in a context a lot of peoples livelihoods could rely on.