6 ms·

What is a lora or llama? Google gives me nothing.

• bestcoder69 4y ago
  llama: gpt-3 alternative that you can download and run on a toaster lora: efficient way of fine-tuning a model like llama, where instead of recreating an entire model, you're keeping the base model and generating a fine-tunings file to apply on top of it. toaster: any machine with like 4GB of RAM available to fit the model
• h11h 4y ago
  LLaMA is Facebook's LLM (large language model, comparable with GPT). It's publicly available (anyone can download the weights and run it themselves), so it's popular here. LoRA, or Low-Rank Adaptation of Large Language Models, lets people fine tune a LLM (making it perform better for a particular application) using vastly less resources. Paper: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf
• dpiers 4y ago
  LoRA: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf LLaMA: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta... Both have been the subjects of numerous HN posts in the last month.
  • corford 4y ago
    And the LLaMA paper from Meta.ai team is here: https://arxiv.org/pdf/2302.13971v1 https://arxiv.org/pdf/2302.13971v1
• Majromax 4y ago
  LLaMA is the large language model published by Facebook (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta...). In theory the model is private, but the model weights were shared with researchers and quickly leaked to the wider Internet. This is one of the first large language models available to ordinary people, much like Stable Diffusion is an image generation model available to ordinary people in contrast to DALL-E or MidJourney. With the model's weights open to people, people can do interesting generative stuff. However, it's still hard to train the model to do new things: training large language models is famously expensive because of both their raw size and their structure. Enter... LoRA is a "low rank adaptation" technique for training large language models, fairly recently published by Microsoft (https://github.com/microsoft/LoRA https://github.com/microsoft/LoRA). In brief, the technique assumes that fine-tuning a model really just involves tweaks to the model parameters that are "small" in some sense, and through math this algorithm confines the fine-tuning to just the small adjustment weights. Rather than asking an ordinary person to re-train 7 billion or 11 billion or 65 billion parameters, LoRA lets users fine-tune a model with about three orders of magnitude fewer adjustment parameters. Combine these two – publicly-available language model weights and a way to fine tune it – and you get work like the story here, where the language model is turned into something a lot like ChatGPT that can run on a consumer-grade laptop.
  • chronicler 4y ago
    Thanks for sharing. How do you know this? Can you recommend any papers to read to start learning about LLMs? I have very limited ML/AI knowledge.
  • xmonkee 4y ago
    Thanks, very helpful. Are llama and chatGPT essentially the same “program”, just with different weights? And is one better than the other (for the same number of parameters) just because it has better weights?
    • lbotos 4y ago
      My understanding is they are both "LLM" (Large language models). That's the generic term you are looking for. I don't think you can compare one LLMs weights to another directly, because the weights are a product of the LLM. In theory (I don't know actually) llama and chatGPT may be using different source datasets so you can't compare them like for like.
    • manojlds 4y ago
      LLaMA and GPT are like Pepsi and Coke.
  • FooBarWidget 4y ago
    How are the llama weights usable by the public? Even if leaked, doesn't using it count as piracy and thus a violation of either copyright or database laws?
    • jsnell 4y ago
      It's not at all clear whether weights are copyrightable.
      • Radim 4y ago
        There's some irony in BigCos using everyone's actual IP freely to train their models, no qualms whatsoever. And then people being scared to even download said models because of "OMG IP!" The asymmetry of power (and dare I say, domestication) is astounding.
• physPop 4y ago
  silly back-ronyms made by academics