4 ms·

> Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs, You mean, like a runtime that can

> Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs, You mean, like a runtime that can be a single, statically linked binary, that requires a c-compiler? With bindings for several other popular languages? https://github.com/ggerganov/llama.cpp/ https://github.com/ggerganov/llama.cpp/ Yeah, maybe I am an optimist, but somehow I don't see that being a big problem... > I expect LLMs to be a ubiquitous UI paradigm. I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software. Software that already bundles large amounts of libraries. But what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter. > But non-cloud LLMs are expected to run locally on clients. They don't have to. I can already see many useful applications for LLMs that run in on-premises servers. I never said that open source LLMs have to be restricted to client-devices. > All that said, perhaps we should look at legislation as well as the possibility of a OS provider playing nice. Legislation is another good argument for open source LLMs taking over. Looking at the EU AI Act, and the recent ideas coming up in the US, it looks like model training data being open for audits is gonna be a thing in the future, especially if the model is used in a context a lot of peoples livelihoods could rely on.

yyyk 3y ago
>>> Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs, >llama.cpp I'm well aware of llama.cpp, and run it myself. I'm also aware of the recent breaking change to the quantization format because 'everyone can just rebuild from the ggml'. And its difficulties with base level AVX or AVX512 (in fairness it focuses on M2 and has support for AVX2). And how you need to work to add additional non-Llama models to it (necessary due to the commercial clause). It's not quite ready yet. Even when it's ready, there's the work for actual deployment, and the mess of diverse user systems beyond what devs tend to run. I'm not saying the challenge is impossible - but it's an extra hurdle for OSS. >> I expect LLMs to be a ubiquitous UI paradigm. > I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software Look at the recent photoshop generative fill feature for what could be done. >what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter. Good ideas. I think for now BigCorp can avoid it by pushing LLMs as an addon UI and keeping the base UI functional. It's possible they'd ship smaller models on device too. >Legislation is another good argument for open source LLMs taking over. That I can buy. The legal details are however a bit beyond me right now, and I don't know what the current status is. It could push OSS or stop it.

yyyk 3y ago

>>> Well, the entire idea is to run locally, right? Now you have to worry about getting the runtime to work on 3 different OSs, >llama.cpp I'm well aware of llama.cpp, and run it myself. I'm also aware of the recent breaking change to the quantization format because 'everyone can just rebuild from the ggml'. And its difficulties with base level AVX or AVX512 (in fairness it focuses on M2 and has support for AVX2). And how you need to work to add additional non-Llama models to it (necessary due to the commercial clause). It's not quite ready yet. Even when it's ready, there's the work for actual deployment, and the mess of diverse user systems beyond what devs tend to run. I'm not saying the challenge is impossible - but it's an extra hurdle for OSS. >> I expect LLMs to be a ubiquitous UI paradigm. > I don't expect that to be the case. LLMs are useful in text wrangling tasks, for which we have specialized software Look at the recent photoshop generative fill feature for what could be done. >what I do expect from an UI: a) that it is fully functional even when the machine is offline, and b) that it's responsiveness is independent from the load on someones datacenter. Good ideas. I think for now BigCorp can avoid it by pushing LLMs as an addon UI and keeping the base UI functional. It's possible they'd ship smaller models on device too. >Legislation is another good argument for open source LLMs taking over. That I can buy. The legal details are however a bit beyond me right now, and I don't know what the current status is. It could push OSS or stop it.