4 ms·
If they open sourced the training data and code but you had to train the network yourself, would that be open source?
by WithinReason 2mo ago
If they open sourced the training data and code but you had to train the network yourself, would that be open source?
- croemer 2mo agoYes, obviously.
- swiftcoder 2mo agoEven though none of us could actually afford to train it?
- croemer 2mo agoSome of us might be rich, or get funding, or in the future it could be useful when training is cheaper. Open source has benefits even if you can run yourself. You can read the code for understanding/insights. Other labs could replicate/build on it.
- gpugreg 2mo ago> Some of us might be rich I sure wish I had a few 100M of disposable income to train a frontier model. > or in the future it could be useful when training is cheaper. I do not think that physics will allow hardware getting that much faster. But maybe we will have different, cheaper architectures by then.
- Muromec 2mo agoIf you would tell people at the start of 20th century how much energy we consume, they may not believe you or think it is wasteful.
- gpugreg 2mo agoLooks like global energy consumption has risen by an order of magnitude from 1900 to 2000: https://www.encyclopedie-energie.org/en/world-energy-consumption-1800-2000-results/#h3-1 https://www.encyclopedie-energie.org/en/world-energy-consump... Unfortunately, electricity prices did not fall by the same factor, so I fear that training a frontier model will still cause a an unsustainable dent in my monthly budget.
- croemer 2mo agoPrices have dropped by a factor of around 6 in real terms since 1925 for US household electricity. https://chatgpt.com/share/6a6f3f3b-7624-83ed-a11a-248cb39728d2 https://chatgpt.com/share/6a6f3f3b-7624-83ed-a11a-248cb39728...
- Muromec 2mo agoOnly an order of magnitude? I'm both surprised by both the fact it's so low and the fact 1900 is 3x of 1800.
- halJordan 2mo agoYou say that but for like 2 years there was a guy on huggingface releasing quants as TheBloke. No identity no nothing except he likely had a grant or a university job that let him do it. Quants aren't the same as training but it was beyond 99% of people at the time (as you say this is) Cant stand when you guys try and force something as impossible on the rest of us simply because you could never accomplish it.
- gpugreg 2mo agoI did not say that it is impossible. I just think that we need architectural improvements, or maybe even a fundamentally different approach to get something like Kimi K3 for cheap. The point I was trying to make was that we shouldn't just laze about and hope that hardware improvements will get us there. (Also, I know Tom Jobbins (TheBloke), and have personally contributed to increase the adoption of GGUF, e.g. in the transformers and ktransformers libraries, so I find the personal dig quite amusing.)
- eptcyka 2mo agoMaybe not now, but what about 10 years down the line?
- croes 2mo agoYet, I couldn’t afford a PC to run the original Crysis when it came out, now the hardware isn’t an issue anymore
- solarkraft 2mo agoYes.
- WithinReason 2mo agoAs an example, Grok 4 took $388M to train
- HPsquared 2mo agoCompiling source code also takes computing resources, only the scale is different. It's a very close analogy: source code with training data, and compiled binaries with model weights. The weights are literally a binary blob.
- WithinReason 2mo agoThe weights are the modifiable representation. You modify them with gradient updates.
- HPsquared 2mo agoThey are a cooked stew; the ingredients have already been chopped and mixed together. You can add things, yes, but you can't inspect the ingredients.
- ungovernableCat 2mo agoIf we start with the same ingredients and independently make our stews they're likely still going to taste a bit different due to the non deterministic nature of the process. Isn't it similar with training LLMs at scale? Do you get identical weights if you do multiple runs?
- HPsquared 2mo agoThey don't even say what the ingredients are.
- ungovernableCat 2mo agoYeah I understand that. I mean just the methods they use to acquire the training data has already had a shit ton of drama, there's no way the western labs are ever going to be public about the ingredients.
- HarHarVeryFunny 2mo agoNo, obviously! These models are a combination of a small amount of code, a ton of training data, and a ton of expertise in how to train them (which I suspect includes how best design/curate training sets for different model improvement goals). The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data. You don't need the source code - just read their architecture docs and implement it yourself. The Chinese have actually been very open about training, starting perhaps with the DeepSeek-R1 paper which told the world in detail how to train a reasoning model. The Kimi 3 paper also gives a lot of training details. There really isn't much of a comparison to be had between building a traditional software project where all you need is the (maybe open source) source code and the Makefile that automates the build process, and building a machine learning system where it's primarily about data not source code, and even with a road map of what may be very complex training (cf build) process, you'd probably still have a hard time building it since AFAIK the training process may still involve expert knowledge and intervention - I don't think training has been reduced to a hands-off "Makefile" or build script. So, basically lack of source code is the least of the issues in being able to build one of these models - it's mostly the training data and training processes/expertise that you would need.
- throw10920 2mo ago> The Chinese models are mostly very well documented in terms of architecture and training processes/flows, with what is missing to recreate them being the training data. ...and because that training data is missing, they can't be replicated. Which means that you cannot assert that the Chinese are being open in their LLM development, because there's no way to verify that the techniques they describe are actually the ones being used. The reason that the training data is missing is that they're trained on a large amount of American copyrighted data and distilled on American models, which is where a lot of their performance comes from.
- HarHarVeryFunny 2mo agoYou can replicate the architectural innovations, and try them for yourself with your own dataset. It seems some of them are certainly being used by western companies, such as DeepSeek Sparse Attention, now supported by NVIDIA cuDNN. Ditto for training algorithms and procedures such as Slime or DeepSeek's details instructions on how to build a reasoning model. This is the exact value of openly shared details - others CAN copy and try them and modify them themselves. Yes, the training data specifically has not been released for any model, American or Chinese, but that doesn't detract from what has been shared, and the reason the Chinese are not sharing data are no more nefarious than why the American companies are not sharing - because they are all using data from sources they don't want you to know about, and at the end of the day the data is the closest thing any of them do have to a moat.
- solarkraft 2mo agoYes
- girvo 2mo agoWhich some companies have done! Nvidia, I believe, among others.