6 ms·
There is no definition of what an "open source AI model" is. Too bad the AI Act was rushed, without waiting for a proper definition and consensus around what i
by zoobab 2y ago
There is no definition of what an "open source AI model" is.
Too bad the AI Act was rushed, without waiting for a proper definition and consensus around what it means.
- stavros 2y agoAn open source AI model should be "if I run what you give me, for the amount of cycles you told me, on the data you give me, I get byte-for-byte your weights".
- DebtDeflation 2y agoI would go a step further than that. A true open source AI model provides: - Training source code - Inferencing source code - Weights - Full training data set #4 makes #3 possibly redundant (if you have the HW and time), but it is also problematic due to copyright, etc.
- blackeyeblitzar 2y agoI would go even further and say that any data set must be available with the original raw data and a fully transparent log of everything filtered out, with reasoning. AI2’s Olmo is the most open LLM but relies on a dataset that they’ve editorialized without transparency. Another consideration is licenses. Everything (code, data) must be released under an OSI approved license, not some made up proprietary one.
- jononor 2y agoByte for byte reproducibility is an extremely high bar. We don't even have that for most open source software. The reproducibility standard that is practice is perform within -+ epsilon on the test sets. That would be sufficient, and more practical, in my opinion.
- stavros 2y agoWe don't have that for software because compiler introduce a lot of randomness. Maybe training introduces randomness as well, I don't know (floating point errors accumulating?). If it does, your metric sounds good, if not, why not go for byte-for-byte...
- pabs3 2y agoReproducible Builds are quite common now, there is even a movement about them: https://reproducible-builds.org/ https://reproducible-builds.org/
- Zambyte 2y agoWhen does a compiler produce randomness? I think non reproducible builds are usually due to environmental inputs (for example: build a date macro) or an undeespecified build process (such as: what compiler version, what build optimizations are used, etc). All of that can be constrained / defined, it's just work that people often don't do.
- theshaper 2y agoOSI is in the stage of gathering opinions to find an appropriate definition: https://thenewstack.io/open-source-ai-osi-wrestles-with-a-definition/ https://thenewstack.io/open-source-ai-osi-wrestles-with-a-de...