7 ms·
Didn't they only "opensource" weights like others?
by feverzsj 2y ago
Didn't they only "opensource" weights like others?
- helpfulclippy 2y agoIt's amazing to me that "open source" has been so diluted that it is now used to mean "we will give you an opaque binary and permission to run it on your own computer."
- paxys 2y agoHaha yup. Going by the current definition of "open source" in AI 100% of software created before the cloud era would have been considered open source.
- mFixman 2y agoI can't believe Microsoft finally made Windows open source.
- zelphirkalt 2y agoYay, let's "fine-tune" and share the result with everyone!
- desdenova 2y agoEvery binary is open source if you can read assembly.
- dartos 2y agoYeah blame the crowds of newbies calling llama open source bc it was free after being leaked.
- hexomancer 2y agoIf I publish some c++ code that has some hard-coded magic values in it, can the code not be considered open source until I also publish how I came up with those magic values?
- z3c0 2y agoI don't know if that compares to an AI model, where the most significant portions are the data preparation and training. The code DeepSeek released only demonstrates how to use the given weights for inferencing with Torch/Triton. I wouldn't consider that an open-source model, just wrapper code for publicly available weights. I think a closer comparison would be Android and GApps, where if you remove the latter, most would deem the phone unusable.
- mohsen1 2y agoif you publish only the binary it's not open source if open the source then it is open source if you write a book/blog about how you came up with the ideas but didn't publish the source it's not open source, even if you publish the blog+binaries
- mistercheph 2y agomodel weights != binaries
- fragmede 2y agowhy not?
- jay_kyburz 2y agoIts like the image you generated in Photoshop released as creative commons, not the Photoshop source code.
- fragmede 2y agothat adds to model weights == binaries tho
- bityard 2y agoIt depends on what those magic numbers are for. If they represent pure data, and it's obvious what the data is (perhaps a bitmap image), then sure, it's open source. If the magic values are some kind of microcode or firmware, or something else that is executed in some way, then no, it is not really open source. Even algorithms can be open source in spirit but closed source in practice. See ECDSA. The NSA has never revealed in any verifiable way how they came up with the specific curves used in the algorithm, so there is room for doubt that they weren't specifically chosen due to some inherent (but hard to find) weakness. I don't know a ton about AI, but I gather there are lots of areas in the process of producing a model where they can claim everything is "open source" as a marketing gimmick but in reality, there is no explanation for how certain results were achieved. (Trade secrets, in other words.)
- blackeyeblitzar 2y agoIt’s because prominent people with large followings are confusing the terms on purpose. Yann LeCun of Meta and Clem Delangue of Hugging Face constantly use the wrong terms for models that only release weights, and market them to their huge audiences as “open source”. This is a willful open washing campaign to benefit from the positivity that label generates.
- seberino 2y agoI agree it would be nice to have the training specifics. Nevertheless everything DeepSeek released is under the MIT license right? So you can go set up a cloud LLM, fine tune it, and, do whatever else you wish with it right? That is pretty significant no?
- fragmede 2y agoIt is, but words mean things. If I said I got you a puppy and gave you a million dollars instead, that'd be nice, but what about the puppy?
- ogrisel 2y agoIt's better to be specific: - open-source inference code - open weights (for inference and fine-tuning) - open pretraining recipe (code + data) - open fine-tuning recipe (code + data) Very few entities publish the later two items (https://huggingface.co/blog/smollm https://huggingface.co/blog/smollm and https://allenai.org/olmo https://allenai.org/olmo come to mind). Arguably, publishing curated large scale pretraining data is very costly but publishing code to automatically curate pretraining data from uncurated sources is already very valuable.
- Palmik 2y agoAlso open-weights comes in several flavors -- there is "restricted" open-weights like Mistral's research license that prohibits most use cases (most importantly, commercial applications), then there are licenses like Llama's or DeepSeek's with some limitations, and then there are some Apache 2.0 or MIT licensed model weights.
- seberino 2y agoWait timeout. I thought DeepSeek's stuff was all MIT licensed too no? What limitations are you thinking of that DeepSeek still has?
- Palmik 2y agoI am referring to this one: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/LICENSE-MODEL https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/LIC... It is a bit more permissive than Llama's it seems (no MAU threshold it seems).
- seberino 2y agoWow. Your link is frustrating because I thought everything was under the MIT license. Why did people claim it is MIT licensed if they sneaked in this additional license?
- 2y ago
- cedws 2y agoEven with the training material what good is it? The model isn’t reproducible, and even if it were you’re not going to spend the money to verify the output.
- deegles 2y agoI guess something like a kickstarter campaign would be needed to get together the millions of dollars needed per training run
- Gigachad 2y agoWho would fund that? What would be the point?
- mistercheph 2y agoFrontier models will never be reproducible in the freedom-loving countries that enforce intellectual property law, since they all depend on copyrighted content in their training data.
- barnabee 2y ago> The model isn’t reproducible Not necessarily[0], it's a WIP, but: https://github.com/huggingface/open-r1 https://github.com/huggingface/open-r1 [0] Surely they won't end up with the exact same weights, but it should be possible to verify something about the model and approach
- fragmede 2y agowhy not? if we could get a version of ChatGPT that wasn't censored and would tell me how to make meth, or an censored version of deepseek that wanted to talk about tank man, you don't think the Internet would come together and make that happen?
- mcbuilder 2y agoSurely the architecture released as a HF transformers python file counts as "open source". https://huggingface.co/deepseek-ai/DeepSeek-R1/raw/main/modeling_deepseek.py https://huggingface.co/deepseek-ai/DeepSeek-R1/raw/main/mode... Yes training is left as an exercise to the user, but it's outlined in the paper, and a good ML engineer should be able to get started with it, cluster of GPUs not included
- cma 2y agoThere was an article saying they used hand-tuned PMX instead of CUDA so it might be a bit hard to match just from the paper without some good performance experts.
- LiamPowell 2y agoCUDA isn't so bad that hand writing PTX will give you a huge performance improvement, but when you're spending a few million dollars on training it makes sense to chase even a single digit percentage improvement, maybe more in a very hot code-path. Also these articles are based on a single mention of PTX in a paper.
- cma 2y agoThe mention is here: "3.2.2. Efficient Implementation of Cross-Node All-to-All Communication In order to ensure sufficient computational performance for DualPipe, we customize efficient cross-node all-to-all communication kernels (including dispatching and combining) to conserve the number of SMs dedicated to communication. The implementation of the kernels is codesigned with the MoE gating algorithm and the network topology of our cluster. To be specific, in our cluster, cross-node GPUs are fully interconnected with IB, and intra-node communications are handled via NVLink. NVLink offers a bandwidth of 160 GB/s, roughly 3.2 times that of IB (50 GB/s). To effectively leverage the different bandwidths of IB and NVLink, we limit each token to be dispatched to at most 4 nodes, thereby reducing IB traffic. For each token, when its routing decision is made, it will first be transmitted via IB to the GPUs with the same in-node index on its target nodes. Once it reaches the target nodes, we will endeavor to ensure that it is instantaneously forwarded via NVLink to specific GPUs that host their target experts, without being blocked by subsequently arriving tokens. In this way, communications via IB and NVLink are fully overlapped, and each token can efficiently select an average of 3.2 experts per node without incurring additional overhead from NVLink. This implies that, although DeepSeek-V3 13 selects only 8 routed experts in practice, it can scale up this number to a maximum of 13 experts (4 nodes × 3.2 experts/node) while preserving the same communication cost. Overall, under such a communication strategy, only 20 SMs are sufficient to fully utilize the bandwidths of IB and NVLink. In detail, we employ the warp specialization technique (Bauer et al., 2014) and partition 20 SMs into 10 communication channels. During the dispatching process, (1) IB sending, (2) IB-to-NVLink forwarding, and (3) NVLink receiving are handled by respective warps. The number of warps allocated to each communication task is dynamically adjusted according to the actual workload across all SMs. Similarly, during the combining process, (1) NVLink sending, (2) NVLink-to-IB forwarding and accumulation, and (3) IB receiving and accumulation are also handled by dynamically adjusted warps. In addition, both dispatching and combining kernels overlap with the computation stream, so we also consider their impact on other SM computation kernels. Specifically, we employ customized PTX (Parallel Thread Execution) instructions and auto-tune the communication chunk size, which significantly reduces the use of the L2 cache and the interference to other SMs." It's definitely not the full model written in PTX or anything, but still some significant engineering effort to replicate, from people commanding 7-figure salaries in this wave, since the training code isn't open.
- JumpCrisscross 2y ago> amazing to me that "open source" has been so diluted It’s not and I called it [1]. We had three options: (A) Open weights (favoured by Altman et al); (B) Open training data (favoured by some FOSS advocates); and (C) Open weights and model, which doesn’t provide the training data, but would let you derive the weights if you had it. OSI settled on (C) [2], but it did so late. FOSS argued for (B), but it’s impractical. So the world, for a while, had a choice between impractical (B) and the useful-if-flawed (A). The public, predictably, went with the pragmatic. This was Betamax vs VHS, except in natural linguistics. There is still hope for (C). But it relies on (A) being rendered impractical. Unfortunately, the path to that flows through institutionalising OpenAI et al’s TOS-based fair use paradigm. Which means while we may get a definition (not exactly (B), but (A) absent use restrictions) we’ll also get restrictions on even using Chinese AI. [1] https://news.ycombinator.com/item?id=41047269 https://news.ycombinator.com/item?id=41047269 [2] https://opensource.org/ai/open-source-ai-definition https://opensource.org/ai/open-source-ai-definition
- sho_hn 2y agoWe absolutely had a choice (D), in that no one was forced to call it "open source" at all, which was arguably done to unfaithfully communicate benefits that don't exist. This is the part that riles people up, and that furthermore is causing collateral damage outside the AI bubble, and is nothing like Betamax vs. VHS. If you want to prioritize pragmatism, that every discussion of this includes a lengthy "so what open source do you mean, exactly?" subthread proves this was a poor choice. It causes uncertainly that also makes it harder for the folks releasing these models to make their case and be taken seriously for their approach. We should probably call them "free to run", if the "it's cheap" connotation of "freeware" needs to be avoided. Or maybe "open architecture" to appreciate the Python file that utilizes the weights more.
- JumpCrisscross 2y ago> We absolutely had a choice (D), in that no one was forced to call it "open source" at all Technically yes, practically no. You’re describing a prisoner’s dilemma. The term was available, there was (and remains) genuine ambiguity over what it meant in this context, and there are first-mover advantages in branding. (Exhibit A: how we label charges). > causing collateral damage outside the AI bubble, and is nothing like Betamax vs. VHS Standards wars have collateral damage. > We should probably call them "free to run", if the "it's cheap" connotation of "freeware" needs to be avoided. Or maybe "open architecture" Language is parsimonious. A neologism will never win when a semantic shift will do.
- Palmik 2y agoExcept the "binary" is not really opaque, and can be "edited" in exactly the same way it was produced in the first place (continued pre-training / fine-tuning).
- seberino 2y agoI'm not an expert but didn't they release the weights under MIT license? So you can make your own LLM with complete control right? I agree it would nice to know the details of their training, but, simply calling this drop an "opaque binary" is seriously underselling it no?
- mritchie712 2y agoyes, all the training code is still closed and doesn't seem it will ever be released. Here's[0] a comment from a dev that worked at deepseek. tldr: we're already on to the next model, don't expect anything else to get open sourced. > I was just told that the amount of people there are too limited, and open-sourcing needs another layer of hard work beyond making the training framework brrr on their own infra. So their priority has been to open-source everything that is MINIMUM + NECESSARY to the community while pushing most efforts on iterating to the next generation of models I think. They have been write everything clearly in technical reports and encourage the community to engage in reproduction , which is the unique insight of the team as well I think. 0 - https://x.com/wzihanw/status/1884374329334387017 https://x.com/wzihanw/status/1884374329334387017
- ComputerGuru 2y agoThey “open sourced” it enough (via the whitepaper) that huggingface is trying to reproduce their training now.
- dartos 2y agoHow is it open source at all with no source? Paxos isn’t open source just because you can read the paxos paper.
- famouswaffles 2y agoThey released the weights
- ComputerGuru 2y agoAs an actual FOSS developer: they didn’t open source it. But I was merely adding the missing context using the (sorry) lingua Franca of AI.
- nexus_six 2y agoFor people who have the disciplinary background in neural networks and machine learning I imagine that replicating that paper into some type of framework would be straight forward right? Or am I mistaken?
- jampekka 2y agoThe model itself yes. The changes from previous architectures are often quite small code-wise. Quite often just adding/changing few lines in a torch model. Things like tweaking all the hyperparameters to make the training process actually work may be more tricky though.
- ru552 2y agoWith an LLM, the actual 0s and 1s of the model are fairly standard, common, and freely available to anyone that wants to use them. The "source code" for an LLM, is the process used to create the outcome, and to an extent, the data used to train with. DeepSeek released a highly detailed paper that describes the process used to create the outcome. People/Companies are actively trying to reproduce the work of DeepSeek to confirm the findings. It's more akin to scientific research where everyone is using the same molecules, but depending on the process you put the molecules through, you get a different outcome.
- Buttons840 2y agoWeights are kind of like a compiled binary, because they are an incomprehensible blob of bits. But they are also unlike a compiled binary, because they can be fine-tuned.
- dartos 2y agoYou can decompile binaries. You can also edit binaries by hand.
- behrlich 2y agoComparing fine tuning to editing binaries by hand is not a fair comparison. If I could show the decompiler some output I liked and it edited the binary for me to make the output match, then the comparison would be closer.
- dartos 2y agoWith regard to the argument about open source, it’s pretty much the same. Especially with dynamically linked binaries like many games.
- carom 2y agoIt is a fair comparison. Normal programming takes inputs and a function and produces outputs. Deep learning takes inputs and outputs and derives a functions. Of course the decompilers for traditional programs do not work on inputs and outputs, it is a different paradigm!
- _ea1k 2y agoI feel like a lot of people in this thread have never done continued training on an LLM and it shows. Seriously, a set of weights that already works really well is basically the ideal basis for a _lot_ of ML tasks.
- zelphirkalt 2y agoThe question is not, whether it is ideal to do some ML tasks with it, the question is, whether you can do the things you could typically do with open sourced software, including looking at the source and build it, or modify the source and build it. If you don't have the original training data, or mechanism of getting the training data, the compiled result is not reproducible, like normal code would be, and you cannot make a version saying for example: "I want just the same, but without it ever learning from CCP prop."
- badgersnake 2y agoYeah, it’s not opensource. It’s just not SaaS. We need to call out these AI companies more on this.
- otterley 2y agoWe have a term for this: "freeware."
- badgersnake 2y agoSure, call it that then. Cut the open source bollocks.
- culi 2y agoNo its fully open sourced. Even Janus is. https://github.com/deepseek-ai https://github.com/deepseek-ai More importantly, they spelled out their methodology in depth in a paper (the code/implementation is trivial in comparison to the methodology)
- aldanor 2y agoThat's just inference code.
- Philpax 2y agoIf it's fully open source, where's the code for training it? The implementation - at least, theirs - is also not trivial as they've mentioned optimising below the CUDA level to get maximum throughout out of their cluster. I'm very appreciative of what they've done, but it's open weights and methodology, not open source.
- tarsinge 2y agoOthers? Do OpenAI, Google or Anthropic release weights?
- marcosdumay 2y agoWeights are actually all you have. The "Open Source" name never applies to LLMs because they don't have a source. But China did distribute them with sharing-friendly terms, what is completely different from others, like Meta, and makes the name way less misleading this time.
- frontfor 2y agoStop referring to them as “China”. It’s as ridiculous as referring to Meta as “America”.
- tim333 2y agoNo. Basically everything. It's not even GPL, it's MIT license. See https://news.ycombinator.com/item?id=42768547 https://news.ycombinator.com/item?id=42768547