3 ms·
LLaMA is the large language model published by Facebook (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-lan
by Majromax 4y ago
LLaMA is the large language model published by Facebook (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta...). In theory the model is private, but the model weights were shared with researchers and quickly leaked to the wider Internet. This is one of the first large language models available to ordinary people, much like Stable Diffusion is an image generation model available to ordinary people in contrast to DALL-E or MidJourney.
With the model's weights open to people, people can do interesting generative stuff. However, it's still hard to train the model to do new things: training large language models is famously expensive because of both their raw size and their structure. Enter...
LoRA is a "low rank adaptation" technique for training large language models, fairly recently published by Microsoft (https://github.com/microsoft/LoRA https://github.com/microsoft/LoRA). In brief, the technique assumes that fine-tuning a model really just involves tweaks to the model parameters that are "small" in some sense, and through math this algorithm confines the fine-tuning to just the small adjustment weights. Rather than asking an ordinary person to re-train 7 billion or 11 billion or 65 billion parameters, LoRA lets users fine-tune a model with about three orders of magnitude fewer adjustment parameters.
Combine these two – publicly-available language model weights and a way to fine tune it – and you get work like the story here, where the language model is turned into something a lot like ChatGPT that can run on a consumer-grade laptop.
- chronicler 4y agoThanks for sharing. How do you know this? Can you recommend any papers to read to start learning about LLMs? I have very limited ML/AI knowledge.
- xmonkee 4y agoThanks, very helpful. Are llama and chatGPT essentially the same “program”, just with different weights? And is one better than the other (for the same number of parameters) just because it has better weights?
- lbotos 4y agoMy understanding is they are both "LLM" (Large language models). That's the generic term you are looking for. I don't think you can compare one LLMs weights to another directly, because the weights are a product of the LLM. In theory (I don't know actually) llama and chatGPT may be using different source datasets so you can't compare them like for like.
- manojlds 4y agoLLaMA and GPT are like Pepsi and Coke.
- FooBarWidget 4y agoHow are the llama weights usable by the public? Even if leaked, doesn't using it count as piracy and thus a violation of either copyright or database laws?
- jsnell 4y agoIt's not at all clear whether weights are copyrightable.
- Radim 4y agoThere's some irony in BigCos using everyone's actual IP freely to train their models, no qualms whatsoever. And then people being scared to even download said models because of "OMG IP!" The asymmetry of power (and dare I say, domestication) is astounding.
- FooBarWidget 4y agoI'm pretty sure they are. If not copyrightable, then at least the database law should apply. One can easily make the case in front of a judge that the situation is similar to databases: the value of weights lies in the amount of work needed to gather the training data, thus weights should be considered a sort of crystallization of a database.
- muyuu 4y agoyea I do wonder about this, but even Meta are acting as if their releasing it means in effect that the cat is out of the bag at this point, their not even complaining about it must mean that they accept the data is public now
- jsnell 4y agoBut the entire business model of the companies making the models seems to be including copyrighted data into the training set under the guise of fair use. If the weights are considered to be a derived work of the training data as a whole, it seems the weights would also have to be a derived work of the individual items in the training data. So I doubt any of them will be making that argument. (Except maybe companies that have access to vast amounts of training data with an explicit license, e.g. because the content is created by their users rather than just scraped from the web?)