5 ms·

A Lora is a layer on top of a model, the big deal isn’t that this exists (it’s a Lora for the weakest llama), but the fact they shared their dataset. The strong

A Lora is a layer on top of a model, the big deal isn’t that this exists (it’s a Lora for the weakest llama), but the fact they shared their dataset. The stronger llamas trained with this data will produce even better Lora’s and better results.

axlee 4y ago
What is a lora or llama? Google gives me nothing.
- bestcoder69 4y ago
  llama: gpt-3 alternative that you can download and run on a toaster lora: efficient way of fine-tuning a model like llama, where instead of recreating an entire model, you're keeping the base model and generating a fine-tunings file to apply on top of it. toaster: any machine with like 4GB of RAM available to fit the model
- h11h 4y ago
  LLaMA is Facebook's LLM (large language model, comparable with GPT). It's publicly available (anyone can download the weights and run it themselves), so it's popular here. LoRA, or Low-Rank Adaptation of Large Language Models, lets people fine tune a LLM (making it perform better for a particular application) using vastly less resources. Paper: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf
- dpiers 4y ago
  LoRA: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf LLaMA: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta... Both have been the subjects of numerous HN posts in the last month.
  - corford 4y ago
    And the LLaMA paper from Meta.ai team is here: https://arxiv.org/pdf/2302.13971v1 https://arxiv.org/pdf/2302.13971v1
- Majromax 4y ago
  LLaMA is the large language model published by Facebook (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta...). In theory the model is private, but the model weights were shared with researchers and quickly leaked to the wider Internet. This is one of the first large language models available to ordinary people, much like Stable Diffusion is an image generation model available to ordinary people in contrast to DALL-E or MidJourney. With the model's weights open to people, people can do interesting generative stuff. However, it's still hard to train the model to do new things: training large language models is famously expensive because of both their raw size and their structure. Enter... LoRA is a "low rank adaptation" technique for training large language models, fairly recently published by Microsoft (https://github.com/microsoft/LoRA https://github.com/microsoft/LoRA). In brief, the technique assumes that fine-tuning a model really just involves tweaks to the model parameters that are "small" in some sense, and through math this algorithm confines the fine-tuning to just the small adjustment weights. Rather than asking an ordinary person to re-train 7 billion or 11 billion or 65 billion parameters, LoRA lets users fine-tune a model with about three orders of magnitude fewer adjustment parameters. Combine these two – publicly-available language model weights and a way to fine tune it – and you get work like the story here, where the language model is turned into something a lot like ChatGPT that can run on a consumer-grade laptop.
  - chronicler 4y ago
    Thanks for sharing. How do you know this? Can you recommend any papers to read to start learning about LLMs? I have very limited ML/AI knowledge.
  - xmonkee 4y ago
    Thanks, very helpful. Are llama and chatGPT essentially the same “program”, just with different weights? And is one better than the other (for the same number of parameters) just because it has better weights?
    - lbotos 4y ago
      My understanding is they are both "LLM" (Large language models). That's the generic term you are looking for. I don't think you can compare one LLMs weights to another directly, because the weights are a product of the LLM. In theory (I don't know actually) llama and chatGPT may be using different source datasets so you can't compare them like for like.
- physPop 4y ago
  silly back-ronyms made by academics
robwwilliams 4y ago
Here is a good start in “low-rank adaptation” or LoRA module. A way to train/adapt a general purpose LLM to efficiently and iteratively accommodate specialize data types and knowledge. A bolt-on. https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf Not “on top” but more “in parallel” if I understand correctly

axlee 4y ago

What is a lora or llama? Google gives me nothing.

bestcoder69 4y ago
llama: gpt-3 alternative that you can download and run on a toaster lora: efficient way of fine-tuning a model like llama, where instead of recreating an entire model, you're keeping the base model and generating a fine-tunings file to apply on top of it. toaster: any machine with like 4GB of RAM available to fit the model
h11h 4y ago
LLaMA is Facebook's LLM (large language model, comparable with GPT). It's publicly available (anyone can download the weights and run it themselves), so it's popular here. LoRA, or Low-Rank Adaptation of Large Language Models, lets people fine tune a LLM (making it perform better for a particular application) using vastly less resources. Paper: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf
dpiers 4y ago
LoRA: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf LLaMA: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta... Both have been the subjects of numerous HN posts in the last month.
- corford 4y ago
  And the LLaMA paper from Meta.ai team is here: https://arxiv.org/pdf/2302.13971v1 https://arxiv.org/pdf/2302.13971v1
Majromax 4y ago
LLaMA is the large language model published by Facebook (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta...). In theory the model is private, but the model weights were shared with researchers and quickly leaked to the wider Internet. This is one of the first large language models available to ordinary people, much like Stable Diffusion is an image generation model available to ordinary people in contrast to DALL-E or MidJourney. With the model's weights open to people, people can do interesting generative stuff. However, it's still hard to train the model to do new things: training large language models is famously expensive because of both their raw size and their structure. Enter... LoRA is a "low rank adaptation" technique for training large language models, fairly recently published by Microsoft (https://github.com/microsoft/LoRA https://github.com/microsoft/LoRA). In brief, the technique assumes that fine-tuning a model really just involves tweaks to the model parameters that are "small" in some sense, and through math this algorithm confines the fine-tuning to just the small adjustment weights. Rather than asking an ordinary person to re-train 7 billion or 11 billion or 65 billion parameters, LoRA lets users fine-tune a model with about three orders of magnitude fewer adjustment parameters. Combine these two – publicly-available language model weights and a way to fine tune it – and you get work like the story here, where the language model is turned into something a lot like ChatGPT that can run on a consumer-grade laptop.
- chronicler 4y ago
  Thanks for sharing. How do you know this? Can you recommend any papers to read to start learning about LLMs? I have very limited ML/AI knowledge.
- xmonkee 4y ago
  Thanks, very helpful. Are llama and chatGPT essentially the same “program”, just with different weights? And is one better than the other (for the same number of parameters) just because it has better weights?
  - lbotos 4y ago
    My understanding is they are both "LLM" (Large language models). That's the generic term you are looking for. I don't think you can compare one LLMs weights to another directly, because the weights are a product of the LLM. In theory (I don't know actually) llama and chatGPT may be using different source datasets so you can't compare them like for like.
physPop 4y ago
silly back-ronyms made by academics

robwwilliams 4y ago

Here is a good start in “low-rank adaptation” or LoRA module. A way to train/adapt a general purpose LLM to efficiently and iteratively accommodate specialize data types and knowledge. A bolt-on. https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf Not “on top” but more “in parallel” if I understand correctly