7 ms·

OSI readies controversial open-source AI definition

117 points by rettichschnidi 2y ago

swyx 2y ago
i like this style of article with extensive citing of original sources. previously on: https://news.ycombinator.com/item?id=41791426 https://news.ycombinator.com/item?id=41791426 its really interesting to contrast this "outsider" definition of open ai with people with real money at stake https://news.ycombinator.com/item?id=41046773 https://news.ycombinator.com/item?id=41046773
- didibus 2y ago
  > its really interesting to contrast this "outsider" definition of open ai with people with real money at stake I guess this is a question of what we want out of "open source". Companies want to make money. Their asset is data, access to customers, hardware and integration. They want to "open source" models, so that other people improve their models for free, and then they can take them back, and sell them, or build something profitable using them. The idea is that, like with other software, eventually, the open source version becomes the best, or just as good as the commercial ones, and companies that build on top no longer have to pay for those, and can use the open source ones. But if what you want out of "open source" is open knowledge, peeking at how something is built, and being able to take that and fork it for your own. Well, you kind of need the data. And your goal in this case is more freedom, using things that you have full access to inspect, alter, repair, modify, etc. To me, both are valid, we just need a name for one and a name for the other, and then we can clearly filter for what we are looking for.
chrisfosterelli 2y ago
I imagine that Open AI (the company) must really not like this.
- talldayo 2y ago
  I hate OpenAI but Sam Altman is probably giddy with excitement watching the Open Source pundits fight about weights being "good enough". He's suffered the criticism over his brand for years but they own the trademark and openly have no fucks to give about the matter. Founding OpenAI more than 5 years before Open AI was defined is probably another perverse laurel he wears. At the end of the day, what threatens OpenAI is falling apart before they hit the runway. They can't lose the Microsoft deal, they can't lose more founders (almost literally at this point) and they can't afford to let their big-ticket partnerships collapse. They are financially unstable even by Valley standards - one year in a down market could decimate them.
JumpCrisscross 2y ago
> After long deliberation and co-design sessions we have concluded that defining training data as a benefit, not a requirement, is the best way to go Huh, then this will be a useful definition. The FSF position is untenable. Sure, it’s philosophically pure. But given a choice between a practical definition and a pedantically-correct but useless one, people will use the former. Irrespective of what some organisation claims. > would have been better, he said, if the OSI had not tried to "bend and reshape a decades old definition" and instead had tried to craft something from a clean slate Not how language works.
- tourmalinetaco 2y ago
  It is in no way useful for the advancement of MLMs. Training data is literally the closest thing to source code MLMs have and to say it’s a “benefit” rather than a requirement only allows for the moat to be maintained. The OSI doesn’t care about the creation of truly free models, only what benefits companies like Facebook or IBM who release model weights but don’t open up the training data.
- blackeyeblitzar 2y ago
  I don’t understand why the “practical” reality requires using the phrase “open source” then. It’s not open source. That label is false and fraudulent if you can’t produce the same artifact or approximately the same artifact. The data is part of the source for models.
  - JumpCrisscross 2y ago
    > don’t understand why the “practical” reality requires using the phrase “open source” then. It’s not open source. That label is false and fraudulent Natural languages are parsimonious; they reuse related words. In this case, the closest practical analogy to open-source software has the lower barrier to entry. Hence, it will win. There is no place for defining open source as data available. In software, too, this problem is solved by using “free software” for the extreme definition. The practical competition is between the Facebook model available with restrictions definition and this.
- 2y ago
andrewmcwatters 2y ago
I’m sure this will be controversial for some reason, but I think we should mostly reject the OSI’s definitions of “open” anything and leave that to the engineering public. I don’t need a board to tell me what’s open. And in the case of AI, if I can’t train the model from source materials with public source code and end up with the same weights, then it’s not open. I don’t need people to tell me that. OSI approved this and that has turned into a Ministry of Magic approved thinking situation that feels gross to me.
- JumpCrisscross 2y ago
  > if I can’t train the model from source materials with public source code and end up with the same weights, then it’s not open This is the new cracker/hacker, GIF pronunciation, crypto(currency)/crypto(graphy) mole hill. Like sure, nobody forces you to recognise any word. But the common usage already precludes open training data—that will only get more ensconced as more contracts and jurisdictions embrace it.
- strangecasts 2y ago
  > And in the case of AI, if I can’t train the model from source materials with public source code and end up with the same weights, then it’s not open. Making training exactly reproducible locks off a lot of optimizations, you are practically not going to get bit-for-bit reproducibility for nontrivial models
  - didibus 2y ago
    That's kind of true for normal programs as well, depending on the compiler you use, and if it has non-deterministic processes in it's compilation. But still, it's about being able to reproduce the same build process, and get a true realization of the program, even if not bit-for-bit, it's the same intended program.
  - samj 2y ago
    Nobody's asking for exact reproducibility — if the source code produces the software and it's appropriately licensed then it's Open Source. Similarly, if you run the scripts and it produces the model then it's Open Source that happens to be AI. To quote Bruce Perens (definition author): the training data IS the source code. Not a perfect analogy but better than a recipe calling for unicorn horns (e.g., FB/IG social graphs) and other toxic candy (e.g., NYT articles that will get users sued).
blogmxc 2y ago
OSI sponsors include Meta, Microsoft, Salesforce and many others. It would seem unlikely that they'd demand the training data to be free and available. Well, another org is getting directors' salaries while open source writers get nothing.
- dokyun 2y ago
  This is why I'd wait for the FSF to deliver their statement before taking anything OSI comes out with seriously.
  - JoshTriplett 2y ago
    The FSF delivering a statement on AI will have zero effect, no matter what position they take.
    - dokyun 2y ago
      As programs that utilize AI continue to become more prevalent, the concern for their freedom is going to become very important. It might require a new license, like a new version or variant of the GPL. In any case I believe the FSF is going to continue to campaign for the ethical freedom of these new classes of software, even if it requires new insight into what it means for them to be free, as they have done before. The FSF is also a much larger and more vocal organization than OSI is, even without the latter's corporate monarc--I mean, monetizers.
      - pxc 2y ago
        > In any case I believe the FSF is going to continue to campaign for the ethical freedom of these new classes of software, even if it requires new insight into what it means for them to be free, as they have done before. And going the quotations in TFA, it seems the FSF's thinking about this is clear and nuanced, as usual: > [T]he FSF makes a distinction between non-free and unethical in this case: > > It may be that some nonfree ML have valid moral reasons for not releasing training data, such as personal medical data. In that case, we would describe the application as a whole as nonfree. But using it could be ethically excusable if it helps you do a specialized job that is vital for society, such as diagnosing disease or injury. If they end up needing new terminology to describe this case, I'm sure they will devise some-- and it will be more explicit than a moniker like 'shared source'.
didibus 2y ago
> Maybe the supporter of the definition could demonstrate practically modifying a ML model without using the original training data, and show that it is just as easy as with the original data and it does not limit what you can do with it (e.g. demonstrate it can unlearn any parts of the original data as if they were not used). I quite like that comment that was left on the article. I know some models you can tweak the weights, without the source data, but it does seem like you are more restricted without the actual dataset. Personally, the data seems to be part of the source to me, in this case. I mean, the code is derived from the data itself, the weights are the artifact of training. If anything, they should provide the data, the training methodology, the model architecture, the code to train and infer, and the weights could be optional. I mean, the weights basically are equivalent to a built artifact, like the compiled software. And that means commercially, people would pay for the cost of training. I might not have the resources to "compile" it myself, aka, run the training, so maybe I pay a subscription to a service that did.
- a2128 2y ago
  The usefulness of data here is that you can retrain the model after making changes to its architecture, e.g. seeing if it works better with a different activation function. Of course this is most useful for models small enough that you could train it within a few days on a consumer GPU. When it comes to LLMs only the richest companies would have the adequate resources to retrain.
- nextaccountic 2y ago
  The source is really the training data plus all code required to train the model. I might not have resources to "compile", and also "compilation" is not deterministic, but those are technical details You could have a programming language whose compiler is a superoptimizer that's very slow and is also stochastic, and it would amount to the same thing in practice.
- lolinder 2y ago
  A lot of people get hung up on `weights = compiled-artifact` because both are binary representations, but there are major limitations to this comparison. When we're dealing with source code, the cost of getting from source -> binary is minimal. The entire Linux kernel builds in two hours on one modest machine. Since it's cheap to compile and the source code is itself legible, the source code is the preferred form for making modifications. This doesn't work when we try to apply the same reasoning to `training data -> weights`. "Compilation" in this world costs hundreds of millions of dollars per compilation run. Cost of "compilation" alone means that the preferred form for making modifications can't possibly be the training data, even for the company that built the thing in the first place. As for the data itself, it's a far cry from source code—we're talking tens of terrabytes of data at a minimum, which is likewise infeasible to work with on a regular basis. The weights must be the preferred form for making modifications for simple logistics reasons. Importantly, the weights are the preferred form for modifications even for the companies that built them. I think a far more reasonable analogy, to the extent that any are reasonable, is that the training data is all the stuff that the developers of the FOSS software ever learned, and the thousands of computer-hours spent on training are the thousands of man-hours spent coding. The entire point of FOSS is for a few experts to do all that work once and then we all can share and modify the output of those years of work and millions of dollars invested as we see fit, without having to waste all that time and money doing it over again. We don't expect the authors of the Linux kernel to document their every waking thought so we could recreate the process they used to produce the kernel code... we just thank them for the kernel code and contribute to it as best we can.
Legend2440 2y ago
Does "open-source" even make sense as a category for AI models? There isn't really a source code in the traditional sense.
- mistrial9 2y ago
  I have heard government people talk about "the data is open-source" meaning it has public, no cost copy points to get data files e.g. csv or other.
- paulddraper 2y ago
  Yeah, it's like an open-source jacket. I don't really know what you're referring to....
  - echoangle 2y ago
    An Open Source jacket actually makes more sense to me than an open source LLM. I generally understand hardware to be open source when all design files are available (for example CAD models of a case and KiCad files for a PCB). If the patterns of a jacket were available in an editable standard-format file, you could argue that’s an open source jacket.
- Barrin92 2y ago
  I had the same thought. "Source Code" is a human readable and modifiable set of instructions that describe the execution of a program. There's obviously parts of an AI system that include literal code, usually a bunch of python scripts or whatever, to interact and build the thing, but most of it is on the one hand data, and on the other an artifact, the AI model and neither is source code really. If you want to talk about the openness and accessibility of these systems I'd just ditch the "source" part and create some new criteria for what makes an AI model open.
- atq2119 2y ago
  There's code for training and inference that could be open-source. For the weights, I agree that open-source doesn't make sense as a category. They're really a kind of database. Perhaps a better way to think about it is in terms of "commons". Consider how creative commons licenses are explicit about requirements like attribution, noncommercial, share-alike, etc.; that feels like a useful model for talking about weights.
aithrowawaycomm 2y ago
What I find frustrating is that this isn't just about pedantry - you can't meaningfully audit an "open-source" model for security or reliability problems if you don't know what's in the training data. I believe that should be the "know it when I see it" test for open-source: has enough information been released for a competent programmer (or team) to understand the how the software actually works? I understand the analogy to other types of critical data often not included in open-source distros (e.g Quake III's source is GPL but its resources like textures are not, as mentioned in the article). The distinction is in these cases the data does not clarify anything about the functioning of the engine, nor does its absence obscure anything. So by my earlier smell test it makes sense to say Quake III is open source. But open-sourcing a transformer ANN without the training data tells us almost nothing about the internal functioning of the software. The exact same source code might be a medical diagnosis machine, or a simple translator. It does not pass my smell test to say this counts as "open source." It makes more sense to say that ANNs are data-as-code programming paradigms, glued together by a bit of Python. An analogy would be if id released its build scripts and announced Quake III was open-source, but claimed the .cpp and .h files were proprietary data. The batch scripts tell you a lot of useful info - maybe even that Q3 has a client-server architecture - but they don't tell you that the game is an FPS, let alone the tricks and foibles in its renderer.
- lolinder 2y ago
  > I believe that should be the "know it when I see it" test for open-source: has enough information been released for a competent programmer (or team) to understand the how the software actually works? Training data simply does not help you here. Our existing architectures are not explainable or auditable in any meaningful way, training data or no training data.
  - samj 2y ago
    That's why Open Source analyst Redmonk now "do not believe the term open source can or should be extended into the AI world." https://redmonk.com/sogrady/2024/10/22/from-open-source-to-ai/ https://redmonk.com/sogrady/2024/10/22/from-open-source-to-a... I don't necessarily agree and suggest the Open Source Definition could be extended to cover data in general (media, databases, and yes, models) with a single sentence, but the lowest risk option is to not touch something that has worked well for a quarter century. The community is starting to regroup and discuss possible next steps over at https://discuss.opensourcedefinition.org https://discuss.opensourcedefinition.org
lolinder 2y ago
> Training data is valuable to study AI systems: to understand the biases that have been learned, which can impact system behavior. But training data is not part of the preferred form for making modifications to an existing AI system. The insights and correlations in that data have already been learned. This makes sense. What the OSI gets right here is that the artifact that is made open source is the weights. Making modifications to the weights is called fine tuning and does not require the original training data any more than making modifications to a piece of source code requires the brain of the original developer. Tens of thousands of people have fine-tuned these models for as long as they've been around. Years ago I trained GPT-2 to produce text resembling Shakespeare. For that I needed Shakespeare, not GPT-2's training data. The training data is properly part of the development process of the open source artifact, not part of the artifact itself. Some open source companies (GitLab) make their development process fully open. Most don't, but we don't call IntelliJ Community closed source on the grounds that they don't record their meetings and stream them for everyone to watch their planning process. Edit: Downvotes are fine, but please at least deign to respond and engage. I realize that I'm expressing a controversial opinion here, but in all the times I've posted similar no one's yet given me a good reason why I'm wrong.
- deleted 2y ago
  [deleted]
- tourmalinetaco 2y ago
  By your logic, many video games have been “open source” for decades because tools were accessible to modify the binary files in certain ways. We lacked the source code, but that’s just “part of the development process”, and maybe parts like comments were lost during the compiling process, but really, why isn’t it open source? Tens of thousands have modified the binaries as long as they’ve been around, and for that I needed community tools, not the source code. In short, your argument doesn’t work because source code is to binaries as training data is to MLMs. Source code is the closest comparison we have with training data, and the useless OSI claims that’s only a “benefit” not a “requirement”. This isn’t a stance meant for long term growth but for maintaining a moat of training data for “AI” companies.
abecedarius 2y ago
The side note on hidden backdoors links to a paper that apparently goes beyond the usual ordinary point that reverse engineering is harder without source: > We show how a malicious learner can plant an undetectable backdoor into a classifier. On the surface, such a backdoored classifier behaves normally, but in reality, the learner maintains a mechanism for changing the classification of any input, with only a slight perturbation. Importantly, without the appropriate "backdoor key", the mechanism is hidden and cannot be detected by any computationally-bounded observer. (I didn't read the paper. The ordinary version of this point is already compelling imo, given the current state of the art of reverse-engineering large models.)
- Terr_ 2y ago
  Reminds me of a saying usually about "bugs" but adapted from this bit from Tony Hoare: > I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. My impression is that LLMs are very much the latter-case, with respect to unwanted behaviors. You can't audit them, you can't secure them against malicious inputs, and whatever limited steering we have over the LSD-trip-generator involves a lot of arbitrary trial and error and hoping our luck holds.
AlienRobot 2y ago
If I remember correctly, Stallman's whole point about FLOSS was that consumers were beholden to developers who monopolized the means to produce binaries. If I can't reproduce the model, I'm beholden to whoever trained it. >"If you're explaining, you're losing." That is an interesting point, but isn't this the same organization that makes "open source" vs. "source available" a topic? e.g. why Winamp wouldn't be open source? I don't think you can even call a trained AI model "source available." To me the "source" is the training data. The model is as much of a binary as machine code. It doesn't even feel right to have it GPL licensed like code. I think it should get the same license you would give to a fractal art released to the public, e.g. CC.
- alwayslikethis 2y ago
  It's not clear that copyright applies to model weights at all, given they are generated by a computer and isn't really a creative work. It is closer to a quantitative description of the underlying data, like a dictionary or word frequency list.
  - AlienRobot 2y ago
    That's interesting. I wonder what will protect these models then, if anything? NDAs? Or maybe the model can be a trade secret or patented? I think dictionaries are copyrightable, however?
- klabb3 2y ago
  I think this makes the most sense. The only meaningful part of the term is whether or not you can hack on it, without permission from (or even coordination with) owners, founders or creators. Heck, a regular old binary is much less opaque than “open” weights. You can at least run it through a disassembler and slowly, dreadfully, figure out how it works. Just look at the game emulator community. For open weight AI models, is there anything close to that?
  - AlienRobot 2y ago
    It's a bit impressive that AI managed to produce something blobbier than a binary blob. AI is the blobbiest blob, so blobby that it's a black box to even its own authors. I wonder how could anyone be an open source enthusiast, distrusting source code they can't verify, and yet a LLM enthusiast, trusting a huge configuration file that can't be debugged. Granted I don't have a lot of knowledge about LLMs. From what I know, there are some tools that can tell you the confidence/stickiness of certain parts of the generate output, e.g. "for a prompt like this, this word WILL appear almost every time, while this other word will almost never appear." I think there was something similar for image generation that could tell what areas of an image stemmed from what terms in the prompt. I have no idea how this information is derived, but it doesn't feel like there are many end-user tools for this. Maybe the AI researchers have access to more powerful tooling. For source code I can just open a file in notepad.exe to inspect it. I think that's the standard. If, for example, a computer program was programmed using an esoteric language that read used image files instead of text files as source code, I don't think you would be able to consider that program "open source" unless the image format it used was also open source, e.g. PNG. If it was some proprietary format, people can't create tools for it, so they can't actually do anything the image blob, which restricts their freedoms.
samj 2y ago
The OSI apparently doesn't have the mandate from its members to even work on this, let alone approve it. The community is starting to regroup at https://discuss.opensourcedefinition.org https://discuss.opensourcedefinition.org because the OSI's own forums are now heavily censored. I encourage you to join the discussion about the future of Open Source, the first option being to keep everything as is.
- justinclift 2y ago
  For reference, this is the OSI Forum mentioned: https://discuss.opensource.org https://discuss.opensource.org Didn't personally know they even had one. ;)
- jart 2y ago
  OSI must defend the open source trademark. Otherwise the community loses everything. The legal system in the US doesn't provide them any other options but to act.
  - tzs 2y ago
    They don’t have a US trademark on “open source”. Their trademarks are on “open source initiative” and “open source initiative approved license”.
- scrollaway 2y ago
  Heh... HN has always been full of massive proponents of the OSI, with people staunchly claiming any software under a license that isn't OSI-approved isn't 'real open source'. Now we're seeing that maybe putting all that trust and responsibility in one entity wasn't such a great idea.
  - opan 2y ago
    We still have the FSF and free software, both predating "open source" and the OSI.
  - seba_dos1 2y ago
    OSD is widely accepted in the community and I don't expect that to change regardless of what happens with AI definitions. Plus we still have FSF's definition and DFSG.
wmf 2y ago
On one hand if you require people to provide data they just won't. People will never provide the data because it's full of smoking guns. On the other hand if the data isn't open you should probably use the term open weights not open source. They're so close.
- samj 2y ago
  Yes, and Open Source started out with a much smaller set of software that has since grown exponentially thanks to the meaningful Open Source Definition. We risk giving AI the same opportunity to grow in an open direction, and by our own hand. Massive own goal.
  - bjornsing 2y ago
    > Yes, and Open Source started out with a much smaller set of software that has since grown exponentially thanks to the meaningful Open Source Definition. I thought it was thanks to a lot of software developers’ uncompensated labor. Silly me.
- mistrial9 2y ago
  > ... require people to provide data they just won't. People will never provide the data ... the word "people" is so striking here... teams and companies, corporations and governments.. how can the cast of characters be so completely missed. An extreme opposite to a far previous era where one person could only be their group member. Vocabulary has to evolve in deliberations.
- skissane 2y ago
  > On one hand if you require people to provide data they just won't. People will never provide the data because it's full of smoking guns. Tangential, but I wonder how well an AI performs when trained on genuine human data, versus a synthetic data set of AI-generated texts. If performance when trained on the synthetic data set is close to that when trained on the original human dataset – this could be a good way to "launder" the original training data and reduce any potential legal issues with it.
koolala 2y ago
"sufficiently detailed information about the data used to train the system so that a skilled person can build a substantially equivalent system"". So a URL to the data? To download the data? Or what? Someone says "Just scrape the data from the web yourself." And a skilled person doesn't need a URL to the source data? No source? Source: The entire WWW?
mensetmanusman 2y ago
The 1000 lines of code is open source, the $100,000,000 in electricity costs to train is not.
- echelon 2y ago
  Training costs will come down. We already have hacks for switching mathematical operators and precision. We originally used to program machines on room-sized computers, yet we now all have access. "Open source" should include the training code and the data. Anything you need to train from scratch or fine tune. Otherwise it's just a binary artifact.
- JoshTriplett 2y ago
  In the early days of Open Source, many people didn't have access to a computer, and those who had access to a computer often didn't have access to development tools. The aspirations of early Open Source became more and more feasible as more people had access to technology, but the definitions still targeted developers.
a-dub 2y ago
the term "open source" means that all of the materials that were used to create a distribution are available to inspect and modify. anything else is closed source. it's as simple as that.
eadwu 2y ago
If only they kept their "Debian Free Software" name instead of hijacking another word ...
looneysquash 2y ago
The trained model is object code. Think of it as Java byte code. You have some sort of engine that runs the model. That's like the JVM, and the JIT. And you have the program that takes the training data and trains the model. That's your compiler, your javac, your Makefile and your make. And you have the training data itself, that's your source code. Each of the above pieces has its own source code. And the training set is also source code. All those pieces have to be open to have a fully open system. If only the training data is open, that's like having the source, but the compiler is proprietary. If everything but the training set is open, well, that's like giving me gcc and calling it Microsoft Word.
pabs3 2y ago
I prefer the Debian policy about this: https://salsa.debian.org/deeplearning-team/ml-policy https://salsa.debian.org/deeplearning-team/ml-policy
metalman 2y ago
call it what it is a search engine,feeding back extracts from real human interaction,useing targeted advertising data to refine the responses and since, what humans say is more horrible than good, the whole thing is a garbage mine go talk to the crews ,who have been maintaining the consise oxford for the last number of centuries,or the French government and the department in charge of regulating the french language,remembering that the french, all but worship there language there you will find,perhaps insight,or terror of the idea of creating a standard,consistant,concise,and useable,LLM
rdsubhas 2y ago
There are already hundreds of OSI licenses for source code. Just create a couple more for AI, one with training data, one without. Holy grail thinking, finding "the one and only open" license instead of "an open" license, is in a sense anti-open.
judithhayes8 2y ago
[dead]
scarlettbrown 2y ago
[dead]
wilbertburke1 2y ago
[dead]

swyx 2y ago

i like this style of article with extensive citing of original sources. previously on: https://news.ycombinator.com/item?id=41791426 https://news.ycombinator.com/item?id=41791426 its really interesting to contrast this "outsider" definition of open ai with people with real money at stake https://news.ycombinator.com/item?id=41046773 https://news.ycombinator.com/item?id=41046773

didibus 2y ago
> its really interesting to contrast this "outsider" definition of open ai with people with real money at stake I guess this is a question of what we want out of "open source". Companies want to make money. Their asset is data, access to customers, hardware and integration. They want to "open source" models, so that other people improve their models for free, and then they can take them back, and sell them, or build something profitable using them. The idea is that, like with other software, eventually, the open source version becomes the best, or just as good as the commercial ones, and companies that build on top no longer have to pay for those, and can use the open source ones. But if what you want out of "open source" is open knowledge, peeking at how something is built, and being able to take that and fork it for your own. Well, you kind of need the data. And your goal in this case is more freedom, using things that you have full access to inspect, alter, repair, modify, etc. To me, both are valid, we just need a name for one and a name for the other, and then we can clearly filter for what we are looking for.

chrisfosterelli 2y ago

I imagine that Open AI (the company) must really not like this.

talldayo 2y ago
I hate OpenAI but Sam Altman is probably giddy with excitement watching the Open Source pundits fight about weights being "good enough". He's suffered the criticism over his brand for years but they own the trademark and openly have no fucks to give about the matter. Founding OpenAI more than 5 years before Open AI was defined is probably another perverse laurel he wears. At the end of the day, what threatens OpenAI is falling apart before they hit the runway. They can't lose the Microsoft deal, they can't lose more founders (almost literally at this point) and they can't afford to let their big-ticket partnerships collapse. They are financially unstable even by Valley standards - one year in a down market could decimate them.

JumpCrisscross 2y ago

> After long deliberation and co-design sessions we have concluded that defining training data as a benefit, not a requirement, is the best way to go Huh, then this will be a useful definition. The FSF position is untenable. Sure, it’s philosophically pure. But given a choice between a practical definition and a pedantically-correct but useless one, people will use the former. Irrespective of what some organisation claims. > would have been better, he said, if the OSI had not tried to "bend and reshape a decades old definition" and instead had tried to craft something from a clean slate Not how language works.

tourmalinetaco 2y ago
It is in no way useful for the advancement of MLMs. Training data is literally the closest thing to source code MLMs have and to say it’s a “benefit” rather than a requirement only allows for the moat to be maintained. The OSI doesn’t care about the creation of truly free models, only what benefits companies like Facebook or IBM who release model weights but don’t open up the training data.
blackeyeblitzar 2y ago
I don’t understand why the “practical” reality requires using the phrase “open source” then. It’s not open source. That label is false and fraudulent if you can’t produce the same artifact or approximately the same artifact. The data is part of the source for models.
- JumpCrisscross 2y ago
  > don’t understand why the “practical” reality requires using the phrase “open source” then. It’s not open source. That label is false and fraudulent Natural languages are parsimonious; they reuse related words. In this case, the closest practical analogy to open-source software has the lower barrier to entry. Hence, it will win. There is no place for defining open source as data available. In software, too, this problem is solved by using “free software” for the extreme definition. The practical competition is between the Facebook model available with restrictions definition and this.
2y ago

andrewmcwatters 2y ago

I’m sure this will be controversial for some reason, but I think we should mostly reject the OSI’s definitions of “open” anything and leave that to the engineering public. I don’t need a board to tell me what’s open. And in the case of AI, if I can’t train the model from source materials with public source code and end up with the same weights, then it’s not open. I don’t need people to tell me that. OSI approved this and that has turned into a Ministry of Magic approved thinking situation that feels gross to me.

JumpCrisscross 2y ago
> if I can’t train the model from source materials with public source code and end up with the same weights, then it’s not open This is the new cracker/hacker, GIF pronunciation, crypto(currency)/crypto(graphy) mole hill. Like sure, nobody forces you to recognise any word. But the common usage already precludes open training data—that will only get more ensconced as more contracts and jurisdictions embrace it.
strangecasts 2y ago
> And in the case of AI, if I can’t train the model from source materials with public source code and end up with the same weights, then it’s not open. Making training exactly reproducible locks off a lot of optimizations, you are practically not going to get bit-for-bit reproducibility for nontrivial models
- didibus 2y ago
  That's kind of true for normal programs as well, depending on the compiler you use, and if it has non-deterministic processes in it's compilation. But still, it's about being able to reproduce the same build process, and get a true realization of the program, even if not bit-for-bit, it's the same intended program.
- samj 2y ago
  Nobody's asking for exact reproducibility — if the source code produces the software and it's appropriately licensed then it's Open Source. Similarly, if you run the scripts and it produces the model then it's Open Source that happens to be AI. To quote Bruce Perens (definition author): the training data IS the source code. Not a perfect analogy but better than a recipe calling for unicorn horns (e.g., FB/IG social graphs) and other toxic candy (e.g., NYT articles that will get users sued).

blogmxc 2y ago

OSI sponsors include Meta, Microsoft, Salesforce and many others. It would seem unlikely that they'd demand the training data to be free and available. Well, another org is getting directors' salaries while open source writers get nothing.

dokyun 2y ago
This is why I'd wait for the FSF to deliver their statement before taking anything OSI comes out with seriously.
- JoshTriplett 2y ago
  The FSF delivering a statement on AI will have zero effect, no matter what position they take.
  - dokyun 2y ago
    As programs that utilize AI continue to become more prevalent, the concern for their freedom is going to become very important. It might require a new license, like a new version or variant of the GPL. In any case I believe the FSF is going to continue to campaign for the ethical freedom of these new classes of software, even if it requires new insight into what it means for them to be free, as they have done before. The FSF is also a much larger and more vocal organization than OSI is, even without the latter's corporate monarc--I mean, monetizers.
    - pxc 2y ago
      > In any case I believe the FSF is going to continue to campaign for the ethical freedom of these new classes of software, even if it requires new insight into what it means for them to be free, as they have done before. And going the quotations in TFA, it seems the FSF's thinking about this is clear and nuanced, as usual: > [T]he FSF makes a distinction between non-free and unethical in this case: > > It may be that some nonfree ML have valid moral reasons for not releasing training data, such as personal medical data. In that case, we would describe the application as a whole as nonfree. But using it could be ethically excusable if it helps you do a specialized job that is vital for society, such as diagnosing disease or injury. If they end up needing new terminology to describe this case, I'm sure they will devise some-- and it will be more explicit than a moniker like 'shared source'.

didibus 2y ago

> Maybe the supporter of the definition could demonstrate practically modifying a ML model without using the original training data, and show that it is just as easy as with the original data and it does not limit what you can do with it (e.g. demonstrate it can unlearn any parts of the original data as if they were not used). I quite like that comment that was left on the article. I know some models you can tweak the weights, without the source data, but it does seem like you are more restricted without the actual dataset. Personally, the data seems to be part of the source to me, in this case. I mean, the code is derived from the data itself, the weights are the artifact of training. If anything, they should provide the data, the training methodology, the model architecture, the code to train and infer, and the weights could be optional. I mean, the weights basically are equivalent to a built artifact, like the compiled software. And that means commercially, people would pay for the cost of training. I might not have the resources to "compile" it myself, aka, run the training, so maybe I pay a subscription to a service that did.

a2128 2y ago
The usefulness of data here is that you can retrain the model after making changes to its architecture, e.g. seeing if it works better with a different activation function. Of course this is most useful for models small enough that you could train it within a few days on a consumer GPU. When it comes to LLMs only the richest companies would have the adequate resources to retrain.
nextaccountic 2y ago
The source is really the training data plus all code required to train the model. I might not have resources to "compile", and also "compilation" is not deterministic, but those are technical details You could have a programming language whose compiler is a superoptimizer that's very slow and is also stochastic, and it would amount to the same thing in practice.
lolinder 2y ago
A lot of people get hung up on `weights = compiled-artifact` because both are binary representations, but there are major limitations to this comparison. When we're dealing with source code, the cost of getting from source -> binary is minimal. The entire Linux kernel builds in two hours on one modest machine. Since it's cheap to compile and the source code is itself legible, the source code is the preferred form for making modifications. This doesn't work when we try to apply the same reasoning to `training data -> weights`. "Compilation" in this world costs hundreds of millions of dollars per compilation run. Cost of "compilation" alone means that the preferred form for making modifications can't possibly be the training data, even for the company that built the thing in the first place. As for the data itself, it's a far cry from source code—we're talking tens of terrabytes of data at a minimum, which is likewise infeasible to work with on a regular basis. The weights must be the preferred form for making modifications for simple logistics reasons. Importantly, the weights are the preferred form for modifications even for the companies that built them. I think a far more reasonable analogy, to the extent that any are reasonable, is that the training data is all the stuff that the developers of the FOSS software ever learned, and the thousands of computer-hours spent on training are the thousands of man-hours spent coding. The entire point of FOSS is for a few experts to do all that work once and then we all can share and modify the output of those years of work and millions of dollars invested as we see fit, without having to waste all that time and money doing it over again. We don't expect the authors of the Linux kernel to document their every waking thought so we could recreate the process they used to produce the kernel code... we just thank them for the kernel code and contribute to it as best we can.

Legend2440 2y ago

Does "open-source" even make sense as a category for AI models? There isn't really a source code in the traditional sense.

mistrial9 2y ago
I have heard government people talk about "the data is open-source" meaning it has public, no cost copy points to get data files e.g. csv or other.
paulddraper 2y ago
Yeah, it's like an open-source jacket. I don't really know what you're referring to....
- echoangle 2y ago
  An Open Source jacket actually makes more sense to me than an open source LLM. I generally understand hardware to be open source when all design files are available (for example CAD models of a case and KiCad files for a PCB). If the patterns of a jacket were available in an editable standard-format file, you could argue that’s an open source jacket.
Barrin92 2y ago
I had the same thought. "Source Code" is a human readable and modifiable set of instructions that describe the execution of a program. There's obviously parts of an AI system that include literal code, usually a bunch of python scripts or whatever, to interact and build the thing, but most of it is on the one hand data, and on the other an artifact, the AI model and neither is source code really. If you want to talk about the openness and accessibility of these systems I'd just ditch the "source" part and create some new criteria for what makes an AI model open.
atq2119 2y ago
There's code for training and inference that could be open-source. For the weights, I agree that open-source doesn't make sense as a category. They're really a kind of database. Perhaps a better way to think about it is in terms of "commons". Consider how creative commons licenses are explicit about requirements like attribution, noncommercial, share-alike, etc.; that feels like a useful model for talking about weights.

aithrowawaycomm 2y ago

What I find frustrating is that this isn't just about pedantry - you can't meaningfully audit an "open-source" model for security or reliability problems if you don't know what's in the training data. I believe that should be the "know it when I see it" test for open-source: has enough information been released for a competent programmer (or team) to understand the how the software actually works? I understand the analogy to other types of critical data often not included in open-source distros (e.g Quake III's source is GPL but its resources like textures are not, as mentioned in the article). The distinction is in these cases the data does not clarify anything about the functioning of the engine, nor does its absence obscure anything. So by my earlier smell test it makes sense to say Quake III is open source. But open-sourcing a transformer ANN without the training data tells us almost nothing about the internal functioning of the software. The exact same source code might be a medical diagnosis machine, or a simple translator. It does not pass my smell test to say this counts as "open source." It makes more sense to say that ANNs are data-as-code programming paradigms, glued together by a bit of Python. An analogy would be if id released its build scripts and announced Quake III was open-source, but claimed the .cpp and .h files were proprietary data. The batch scripts tell you a lot of useful info - maybe even that Q3 has a client-server architecture - but they don't tell you that the game is an FPS, let alone the tricks and foibles in its renderer.

lolinder 2y ago
> I believe that should be the "know it when I see it" test for open-source: has enough information been released for a competent programmer (or team) to understand the how the software actually works? Training data simply does not help you here. Our existing architectures are not explainable or auditable in any meaningful way, training data or no training data.
- samj 2y ago
  That's why Open Source analyst Redmonk now "do not believe the term open source can or should be extended into the AI world." https://redmonk.com/sogrady/2024/10/22/from-open-source-to-ai/ https://redmonk.com/sogrady/2024/10/22/from-open-source-to-a... I don't necessarily agree and suggest the Open Source Definition could be extended to cover data in general (media, databases, and yes, models) with a single sentence, but the lowest risk option is to not touch something that has worked well for a quarter century. The community is starting to regroup and discuss possible next steps over at https://discuss.opensourcedefinition.org https://discuss.opensourcedefinition.org

lolinder 2y ago

> Training data is valuable to study AI systems: to understand the biases that have been learned, which can impact system behavior. But training data is not part of the preferred form for making modifications to an existing AI system. The insights and correlations in that data have already been learned. This makes sense. What the OSI gets right here is that the artifact that is made open source is the weights. Making modifications to the weights is called fine tuning and does not require the original training data any more than making modifications to a piece of source code requires the brain of the original developer. Tens of thousands of people have fine-tuned these models for as long as they've been around. Years ago I trained GPT-2 to produce text resembling Shakespeare. For that I needed Shakespeare, not GPT-2's training data. The training data is properly part of the development process of the open source artifact, not part of the artifact itself. Some open source companies (GitLab) make their development process fully open. Most don't, but we don't call IntelliJ Community closed source on the grounds that they don't record their meetings and stream them for everyone to watch their planning process. Edit: Downvotes are fine, but please at least deign to respond and engage. I realize that I'm expressing a controversial opinion here, but in all the times I've posted similar no one's yet given me a good reason why I'm wrong.

deleted 2y ago
[deleted]
tourmalinetaco 2y ago
By your logic, many video games have been “open source” for decades because tools were accessible to modify the binary files in certain ways. We lacked the source code, but that’s just “part of the development process”, and maybe parts like comments were lost during the compiling process, but really, why isn’t it open source? Tens of thousands have modified the binaries as long as they’ve been around, and for that I needed community tools, not the source code. In short, your argument doesn’t work because source code is to binaries as training data is to MLMs. Source code is the closest comparison we have with training data, and the useless OSI claims that’s only a “benefit” not a “requirement”. This isn’t a stance meant for long term growth but for maintaining a moat of training data for “AI” companies.

abecedarius 2y ago

The side note on hidden backdoors links to a paper that apparently goes beyond the usual ordinary point that reverse engineering is harder without source: > We show how a malicious learner can plant an undetectable backdoor into a classifier. On the surface, such a backdoored classifier behaves normally, but in reality, the learner maintains a mechanism for changing the classification of any input, with only a slight perturbation. Importantly, without the appropriate "backdoor key", the mechanism is hidden and cannot be detected by any computationally-bounded observer. (I didn't read the paper. The ordinary version of this point is already compelling imo, given the current state of the art of reverse-engineering large models.)

Terr_ 2y ago
Reminds me of a saying usually about "bugs" but adapted from this bit from Tony Hoare: > I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. My impression is that LLMs are very much the latter-case, with respect to unwanted behaviors. You can't audit them, you can't secure them against malicious inputs, and whatever limited steering we have over the LSD-trip-generator involves a lot of arbitrary trial and error and hoping our luck holds.

AlienRobot 2y ago

If I remember correctly, Stallman's whole point about FLOSS was that consumers were beholden to developers who monopolized the means to produce binaries. If I can't reproduce the model, I'm beholden to whoever trained it. >"If you're explaining, you're losing." That is an interesting point, but isn't this the same organization that makes "open source" vs. "source available" a topic? e.g. why Winamp wouldn't be open source? I don't think you can even call a trained AI model "source available." To me the "source" is the training data. The model is as much of a binary as machine code. It doesn't even feel right to have it GPL licensed like code. I think it should get the same license you would give to a fractal art released to the public, e.g. CC.

alwayslikethis 2y ago
It's not clear that copyright applies to model weights at all, given they are generated by a computer and isn't really a creative work. It is closer to a quantitative description of the underlying data, like a dictionary or word frequency list.
- AlienRobot 2y ago
  That's interesting. I wonder what will protect these models then, if anything? NDAs? Or maybe the model can be a trade secret or patented? I think dictionaries are copyrightable, however?
klabb3 2y ago
I think this makes the most sense. The only meaningful part of the term is whether or not you can hack on it, without permission from (or even coordination with) owners, founders or creators. Heck, a regular old binary is much less opaque than “open” weights. You can at least run it through a disassembler and slowly, dreadfully, figure out how it works. Just look at the game emulator community. For open weight AI models, is there anything close to that?
- AlienRobot 2y ago
  It's a bit impressive that AI managed to produce something blobbier than a binary blob. AI is the blobbiest blob, so blobby that it's a black box to even its own authors. I wonder how could anyone be an open source enthusiast, distrusting source code they can't verify, and yet a LLM enthusiast, trusting a huge configuration file that can't be debugged. Granted I don't have a lot of knowledge about LLMs. From what I know, there are some tools that can tell you the confidence/stickiness of certain parts of the generate output, e.g. "for a prompt like this, this word WILL appear almost every time, while this other word will almost never appear." I think there was something similar for image generation that could tell what areas of an image stemmed from what terms in the prompt. I have no idea how this information is derived, but it doesn't feel like there are many end-user tools for this. Maybe the AI researchers have access to more powerful tooling. For source code I can just open a file in notepad.exe to inspect it. I think that's the standard. If, for example, a computer program was programmed using an esoteric language that read used image files instead of text files as source code, I don't think you would be able to consider that program "open source" unless the image format it used was also open source, e.g. PNG. If it was some proprietary format, people can't create tools for it, so they can't actually do anything the image blob, which restricts their freedoms.

samj 2y ago

The OSI apparently doesn't have the mandate from its members to even work on this, let alone approve it. The community is starting to regroup at https://discuss.opensourcedefinition.org https://discuss.opensourcedefinition.org because the OSI's own forums are now heavily censored. I encourage you to join the discussion about the future of Open Source, the first option being to keep everything as is.

justinclift 2y ago
For reference, this is the OSI Forum mentioned: https://discuss.opensource.org https://discuss.opensource.org Didn't personally know they even had one. ;)
jart 2y ago
OSI must defend the open source trademark. Otherwise the community loses everything. The legal system in the US doesn't provide them any other options but to act.
- tzs 2y ago
  They don’t have a US trademark on “open source”. Their trademarks are on “open source initiative” and “open source initiative approved license”.
scrollaway 2y ago
Heh... HN has always been full of massive proponents of the OSI, with people staunchly claiming any software under a license that isn't OSI-approved isn't 'real open source'. Now we're seeing that maybe putting all that trust and responsibility in one entity wasn't such a great idea.
- opan 2y ago
  We still have the FSF and free software, both predating "open source" and the OSI.
- seba_dos1 2y ago
  OSD is widely accepted in the community and I don't expect that to change regardless of what happens with AI definitions. Plus we still have FSF's definition and DFSG.

wmf 2y ago

On one hand if you require people to provide data they just won't. People will never provide the data because it's full of smoking guns. On the other hand if the data isn't open you should probably use the term open weights not open source. They're so close.

samj 2y ago
Yes, and Open Source started out with a much smaller set of software that has since grown exponentially thanks to the meaningful Open Source Definition. We risk giving AI the same opportunity to grow in an open direction, and by our own hand. Massive own goal.
- bjornsing 2y ago
  > Yes, and Open Source started out with a much smaller set of software that has since grown exponentially thanks to the meaningful Open Source Definition. I thought it was thanks to a lot of software developers’ uncompensated labor. Silly me.
mistrial9 2y ago
> ... require people to provide data they just won't. People will never provide the data ... the word "people" is so striking here... teams and companies, corporations and governments.. how can the cast of characters be so completely missed. An extreme opposite to a far previous era where one person could only be their group member. Vocabulary has to evolve in deliberations.
skissane 2y ago
> On one hand if you require people to provide data they just won't. People will never provide the data because it's full of smoking guns. Tangential, but I wonder how well an AI performs when trained on genuine human data, versus a synthetic data set of AI-generated texts. If performance when trained on the synthetic data set is close to that when trained on the original human dataset – this could be a good way to "launder" the original training data and reduce any potential legal issues with it.

koolala 2y ago

"sufficiently detailed information about the data used to train the system so that a skilled person can build a substantially equivalent system"". So a URL to the data? To download the data? Or what? Someone says "Just scrape the data from the web yourself." And a skilled person doesn't need a URL to the source data? No source? Source: The entire WWW?

mensetmanusman 2y ago

The 1000 lines of code is open source, the $100,000,000 in electricity costs to train is not.

echelon 2y ago
Training costs will come down. We already have hacks for switching mathematical operators and precision. We originally used to program machines on room-sized computers, yet we now all have access. "Open source" should include the training code and the data. Anything you need to train from scratch or fine tune. Otherwise it's just a binary artifact.
JoshTriplett 2y ago
In the early days of Open Source, many people didn't have access to a computer, and those who had access to a computer often didn't have access to development tools. The aspirations of early Open Source became more and more feasible as more people had access to technology, but the definitions still targeted developers.

a-dub 2y ago

the term "open source" means that all of the materials that were used to create a distribution are available to inspect and modify. anything else is closed source. it's as simple as that.

eadwu 2y ago

If only they kept their "Debian Free Software" name instead of hijacking another word ...

looneysquash 2y ago

The trained model is object code. Think of it as Java byte code. You have some sort of engine that runs the model. That's like the JVM, and the JIT. And you have the program that takes the training data and trains the model. That's your compiler, your javac, your Makefile and your make. And you have the training data itself, that's your source code. Each of the above pieces has its own source code. And the training set is also source code. All those pieces have to be open to have a fully open system. If only the training data is open, that's like having the source, but the compiler is proprietary. If everything but the training set is open, well, that's like giving me gcc and calling it Microsoft Word.

pabs3 2y ago

I prefer the Debian policy about this: https://salsa.debian.org/deeplearning-team/ml-policy https://salsa.debian.org/deeplearning-team/ml-policy

metalman 2y ago

call it what it is a search engine,feeding back extracts from real human interaction,useing targeted advertising data to refine the responses and since, what humans say is more horrible than good, the whole thing is a garbage mine go talk to the crews ,who have been maintaining the consise oxford for the last number of centuries,or the French government and the department in charge of regulating the french language,remembering that the french, all but worship there language there you will find,perhaps insight,or terror of the idea of creating a standard,consistant,concise,and useable,LLM

rdsubhas 2y ago

There are already hundreds of OSI licenses for source code. Just create a couple more for AI, one with training data, one without. Holy grail thinking, finding "the one and only open" license instead of "an open" license, is in a sense anti-open.

judithhayes8 2y ago

[dead]

scarlettbrown 2y ago

[dead]

wilbertburke1 2y ago

[dead]