3 ms·
LLMs are the death of open source. Labor might be free, but training resources seldom are. They won't be able to keep up.
by echelon 3y ago
LLMs are the death of open source. Labor might be free, but training resources seldom are. They won't be able to keep up.
- jrm4 3y agoThat people actually entertain these ideas is SO WEIRD to me. The last sentence is just obviously false, precisely because the models are software that anyone can trade around. LoRA's, in other words. The fact that original models "might" -- and even that's a big might -- be hard to produce from scratch, so what? Operating systems are hard to produce from scratch, but anyone can grab Linux and do anything.
- echelon 3y agoYour opinion doesn't make any sense to me. If it takes a hundred million dollars to train a SOTA LLM, who is going to pay for and put that in open source? Even Stability is starting to put restrictions on their releases (and is rumored to be searching for an acquirer).
- jrm4 3y agoTwo possibilities: 1) Why wouldn't Facebook (who looks like they're kind of doing that now) or anyone trying to compete with whoever the big monolith is threatening to be (which looks like OpenAI/Microsoft?) 2) Do they even need to, or is/can LLM development be incremental? I haven't heard anything about Google Fuchsia (sp?) lately either.
- alerighi 3y agoThe problem to me is another. Yes, there are people willingly to spend time and computing resources to train a model and improve it. The thing is, with software it's easy to verify (or, quite easy) that a contribution actually improves the code and thus gets accepted in the project. A modification to an AI model, for how models are made these day, it's completely opaque, a black box. How we can evaluate the modification to ensure that it actually improves the model and does not harm it, or worse introduce malicious behavior (since AI are also used to write code, for example train the AI to write malicious code). This is the real problem to solve, and to this day it's not solved. Probably the solution is to NOT have LARGE language model but rather have a multitude of small models (small enough that we can retrain them in hours with a normal PC) that are trained and tested individually, maintained by a group of persons in the open source community, and then merged together in a big model, just like a kernel is made up of thousands of individual modules that are then assembled together in a single software.
- jrm4 3y agoBut at this stage: why a big model AT ALL? Which is to say, a useful LLM is probably not going to be complex like an Operating System. (Now, an LLM that looks fancy and can fool dumb investors, sure, maybe -- but an actually useful one?)
- flohofwoe 3y agoThey can only remix what already exists, not create new things, no?
- jzombie 3y agoTechnically, anything you could ever write in any [programmed or spoken] language is some sort of remix of existing tokens.
- biofunsf 3y agoI think by most definitions LLMs can definitely create new things. As much as I can create new things by cobbling together open source projects and answers on stack overflow at least.
- Der_Einzige 3y agoThere are papers including one by Yann LeCunn which shows that the overwhelming majority of data found within high dimensional spaces is within decision boundries created by AI models. This implies that the overwhelming majority of things we think as humans are "novel" or "extrapolative" are in fact not novel and are nothing more than fancy interpolation. Basically, we humans cannot tell when something is "new to us" or "new to the universe" and it honestly hardly matters in practical reality. Finally, LLMs can totally "extrapolate" with techniques which encourage it to avoid its traditional training loss functions. A high temperature would do this, but it'll mostly just make your model crazy.