4 ms·

Author here. I agree with you. LLaMA2 isn't open source (as my title says, the HN one was modified). My point is that the average person will still call it "ope

Author here. I agree with you. LLaMA2 isn't open source (as my title says, the HN one was modified). My point is that the average person will still call it "open source" because they don't know any better, and it's hard to fix that. Rather than just saying "this isn't open source", we should try to come up with better terminology. Also, while weights usage might be restricted, it's a very big compute investment shared with the public. They use a 285:1 training tokens to params ratio, and the loss graphs show the model wasn't yet saturated. This is valuable information for other teams looking to train their own models. LLaMA1 was highly restrictive, but the data mix mentioned in the paper led to the creation of RedPajama, which was used in the training of MPT. There's still plenty of value in this work that will flow to open source, even if it doesn't fit in the traditional labels.

danShumway 3y ago
Thanks for replying! And agreed on the title change; I think your original title is much, much better phrased and I'm sorry that I glossed over it when reading the article (although I'm not sure "doesn't matter" fully captures the distinction you're making here) -- mods probably shouldn't have changed it. > There's still plenty of value in this work that will flow to open source, even if it doesn't fit in the traditional labels. That is a good point; the fight over what is open source and what is source available can get heated, and part of that is a defense against the erosion of the term. But... in general source available is better than closed source software. And LLaMA 2 is a significant improvement over LLaMA 1 in that regard, it really is. So I don't necessarily want to be down on it, in some ways it's just backlash of being tired of companies stretching definitions. But they're doing a thing that will absolutely help improve open access to LLMs. I'm always a little bit torn about how to go about this kind of criticism of terminology, and I'm not trying to say that people shouldn't be excited about LLaMA 2. But the way it works out I'm often playing word police because the erosion of the term does make it harder to refer to models with actual open weights like StableLM. Facebook deserves real praise for releasing a model with weights that can be used commercially. It doesn't deserve to be treated as if what it's doing is equivalent to what StabilityAI or RedPanda is doing. I do like your terminology of "open weights" and "restricted weights", and I wouldn't be opposed to even breaking that down even further, I think there's a clear difference between LLaMA 1 and 2 in terms of user freedom, so I'm not opposed to people trying to distinguish, just... it's not hitting the bar of being open weights. It's a bit like if the word vegetarian didn't exist, and if everyone argued about how it's unhelpful to say that drinking milk isn't vegan because it's still tangibly different from eating meat. On one hand I agree, but on the other hand it's better to have another category for it that means "not vegan, but still not eating meat." There is an actual danger in blurring a line so much that the line doesn't mean anything anymore, and where people who mean something more rigorous no longer have a term to communicate amongst themselves. If average people get bothered by throwing LLaMA 2 into the "restricted weights" category, it's better to introduce another category between restricted and open that means "restricted but not commercially". Beyond that though... yeah, I agree. I don't really have a problem with people calling open weights open source, my only objection to that is kind of technical and pedantic, but I don't think it causes any actual harm if someone wants to call StableLM open source.
StackOverlord 3y ago
As I said last week, compiling source-code does not cost millions of dollars. How much does it cost to gather training data ? Training llama costed around 30 millions in infrastructure + 50k in power costs (source: https://news.ycombinator.com/item?id=35008694 https://news.ycombinator.com/item?id=35008694).

danShumway 3y ago

Thanks for replying! And agreed on the title change; I think your original title is much, much better phrased and I'm sorry that I glossed over it when reading the article (although I'm not sure "doesn't matter" fully captures the distinction you're making here) -- mods probably shouldn't have changed it. > There's still plenty of value in this work that will flow to open source, even if it doesn't fit in the traditional labels. That is a good point; the fight over what is open source and what is source available can get heated, and part of that is a defense against the erosion of the term. But... in general source available is better than closed source software. And LLaMA 2 is a significant improvement over LLaMA 1 in that regard, it really is. So I don't necessarily want to be down on it, in some ways it's just backlash of being tired of companies stretching definitions. But they're doing a thing that will absolutely help improve open access to LLMs. I'm always a little bit torn about how to go about this kind of criticism of terminology, and I'm not trying to say that people shouldn't be excited about LLaMA 2. But the way it works out I'm often playing word police because the erosion of the term does make it harder to refer to models with actual open weights like StableLM. Facebook deserves real praise for releasing a model with weights that can be used commercially. It doesn't deserve to be treated as if what it's doing is equivalent to what StabilityAI or RedPanda is doing. I do like your terminology of "open weights" and "restricted weights", and I wouldn't be opposed to even breaking that down even further, I think there's a clear difference between LLaMA 1 and 2 in terms of user freedom, so I'm not opposed to people trying to distinguish, just... it's not hitting the bar of being open weights. It's a bit like if the word vegetarian didn't exist, and if everyone argued about how it's unhelpful to say that drinking milk isn't vegan because it's still tangibly different from eating meat. On one hand I agree, but on the other hand it's better to have another category for it that means "not vegan, but still not eating meat." There is an actual danger in blurring a line so much that the line doesn't mean anything anymore, and where people who mean something more rigorous no longer have a term to communicate amongst themselves. If average people get bothered by throwing LLaMA 2 into the "restricted weights" category, it's better to introduce another category between restricted and open that means "restricted but not commercially". Beyond that though... yeah, I agree. I don't really have a problem with people calling open weights open source, my only objection to that is kind of technical and pedantic, but I don't think it causes any actual harm if someone wants to call StableLM open source.

StackOverlord 3y ago

As I said last week, compiling source-code does not cost millions of dollars. How much does it cost to gather training data ? Training llama costed around 30 millions in infrastructure + 50k in power costs (source: https://news.ycombinator.com/item?id=35008694 https://news.ycombinator.com/item?id=35008694).