8 ms·

This sentence defies lay people: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data,

This sentence defies lay people: The biggest deal with this isn't the published lora adapter (which seems limited to llama 7b), but the cleaned training data, which is likely better than the previous data sets used to train the alpaca-inspired loras that have been publicly released so far.

refulgentis 4y ago
A Lora is a layer on top of a model, the big deal isn’t that this exists (it’s a Lora for the weakest llama), but the fact they shared their dataset. The stronger llamas trained with this data will produce even better Lora’s and better results.
- axlee 4y ago
  What is a lora or llama? Google gives me nothing.
  - bestcoder69 4y ago
    llama: gpt-3 alternative that you can download and run on a toaster lora: efficient way of fine-tuning a model like llama, where instead of recreating an entire model, you're keeping the base model and generating a fine-tunings file to apply on top of it. toaster: any machine with like 4GB of RAM available to fit the model
  - h11h 4y ago
    LLaMA is Facebook's LLM (large language model, comparable with GPT). It's publicly available (anyone can download the weights and run it themselves), so it's popular here. LoRA, or Low-Rank Adaptation of Large Language Models, lets people fine tune a LLM (making it perform better for a particular application) using vastly less resources. Paper: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf
  - dpiers 4y ago
    LoRA: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf LLaMA: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta... Both have been the subjects of numerous HN posts in the last month.
    - corford 4y ago
      And the LLaMA paper from Meta.ai team is here: https://arxiv.org/pdf/2302.13971v1 https://arxiv.org/pdf/2302.13971v1
  - Majromax 4y ago
    LLaMA is the large language model published by Facebook (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta...). In theory the model is private, but the model weights were shared with researchers and quickly leaked to the wider Internet. This is one of the first large language models available to ordinary people, much like Stable Diffusion is an image generation model available to ordinary people in contrast to DALL-E or MidJourney. With the model's weights open to people, people can do interesting generative stuff. However, it's still hard to train the model to do new things: training large language models is famously expensive because of both their raw size and their structure. Enter... LoRA is a "low rank adaptation" technique for training large language models, fairly recently published by Microsoft (https://github.com/microsoft/LoRA https://github.com/microsoft/LoRA). In brief, the technique assumes that fine-tuning a model really just involves tweaks to the model parameters that are "small" in some sense, and through math this algorithm confines the fine-tuning to just the small adjustment weights. Rather than asking an ordinary person to re-train 7 billion or 11 billion or 65 billion parameters, LoRA lets users fine-tune a model with about three orders of magnitude fewer adjustment parameters. Combine these two – publicly-available language model weights and a way to fine tune it – and you get work like the story here, where the language model is turned into something a lot like ChatGPT that can run on a consumer-grade laptop.
    - chronicler 4y ago
      Thanks for sharing. How do you know this? Can you recommend any papers to read to start learning about LLMs? I have very limited ML/AI knowledge.
    - xmonkee 4y ago
      Thanks, very helpful. Are llama and chatGPT essentially the same “program”, just with different weights? And is one better than the other (for the same number of parameters) just because it has better weights?
      - lbotos 4y ago
        My understanding is they are both "LLM" (Large language models). That's the generic term you are looking for. I don't think you can compare one LLMs weights to another directly, because the weights are a product of the LLM. In theory (I don't know actually) llama and chatGPT may be using different source datasets so you can't compare them like for like.
  - physPop 4y ago
    silly back-ronyms made by academics
- robwwilliams 4y ago
  Here is a good start in “low-rank adaptation” or LoRA module. A way to train/adapt a general purpose LLM to efficiently and iteratively accommodate specialize data types and knowledge. A bolt-on. https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf Not “on top” but more “in parallel” if I understand correctly
slg 4y ago
I have casually followed countless different news cycles on various complicated tech topics over my decades long career. I can't recall a single one that has consistently made me feel like an idiot more than how people talk about this recent AI wave. There just seems to be so much more jargon involved in this subject that makes casual perusing of the latest developments impenetrable.
- akiselev 4y ago
  Ah, my fellow citizen of the interwebs, fear not! Your intellectual frustrations are but a natural reaction to the tsunami of technological jargon. You see, the AI wave is the epitome of obfuscation, a testament to the labyrinthine lexicon of the digital age. It's as if a group of caffeinated, sleep-deprived tech enthusiasts assembled in the dark of night and decided to create an impenetrable fortress of vernacular, just to keep the uninitiated at bay.
  - Grothendank 4y ago
    Should jackasses on HN use plain language instead of jargon? Surely. But AI workers mainly develop and use jargon because it is an easy and natural way to consolidate concepts. Sure, there is a kind of conspiracy caused by publish or perish. Researchers may use jargon to make their work harder to reject on review; laborious speech and jargon can make statements sound more profound. However, no technical field is immune to this. We'll need to systematically change science before we can eliminate that problem. Until we manage that, if you care about the concepts enough to want to understand them before there are good plain speech descriptions, just pop the jargon into google scholar and skim read a few papers, and you're good to go. If you don't care about the concepts that much, then don't worry about the jargon. The important concepts will get their own non-technical explanations in time. As it stands, AI jargon is not that bad. It tends to be pretty fair and easy to understand, compared to jargon in, say, biochemistry or higher math.
    - kordlessagain 4y ago
      With chatots, explaining everything clearly is an option.
jhbadger 4y ago
As someone who is following this technology while not really an expert (I'm a computational biologist in my day job) LoRA is a way of reducing the number of parameters in a large language model (LLM, the technology behind all these new chatbots) so that it can be run on less powerful hardware (say a laptop or even a phone). The OP is saying that the improvements this chatbot provides isn't so much being more clever about reducing parameters but being trained on text that has been cleaned up rather than the rather messy training sets used in other small LLMs.
- f_devd 4y ago
  LoRA (Low-Rank Adapter) is way to customize/finetune the LLM to a new datasets without needing to retrain the entire network (which makes it better (and in theory easier to do). It doesn't not change the speed significantly afaik
throwaway2865 4y ago
Less garbage in, less garbage out?

refulgentis 4y ago

A Lora is a layer on top of a model, the big deal isn’t that this exists (it’s a Lora for the weakest llama), but the fact they shared their dataset. The stronger llamas trained with this data will produce even better Lora’s and better results.

axlee 4y ago
What is a lora or llama? Google gives me nothing.
- bestcoder69 4y ago
  llama: gpt-3 alternative that you can download and run on a toaster lora: efficient way of fine-tuning a model like llama, where instead of recreating an entire model, you're keeping the base model and generating a fine-tunings file to apply on top of it. toaster: any machine with like 4GB of RAM available to fit the model
- h11h 4y ago
  LLaMA is Facebook's LLM (large language model, comparable with GPT). It's publicly available (anyone can download the weights and run it themselves), so it's popular here. LoRA, or Low-Rank Adaptation of Large Language Models, lets people fine tune a LLM (making it perform better for a particular application) using vastly less resources. Paper: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf
- dpiers 4y ago
  LoRA: https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf LLaMA: https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta... Both have been the subjects of numerous HN posts in the last month.
  - corford 4y ago
    And the LLaMA paper from Meta.ai team is here: https://arxiv.org/pdf/2302.13971v1 https://arxiv.org/pdf/2302.13971v1
- Majromax 4y ago
  LLaMA is the large language model published by Facebook (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta...). In theory the model is private, but the model weights were shared with researchers and quickly leaked to the wider Internet. This is one of the first large language models available to ordinary people, much like Stable Diffusion is an image generation model available to ordinary people in contrast to DALL-E or MidJourney. With the model's weights open to people, people can do interesting generative stuff. However, it's still hard to train the model to do new things: training large language models is famously expensive because of both their raw size and their structure. Enter... LoRA is a "low rank adaptation" technique for training large language models, fairly recently published by Microsoft (https://github.com/microsoft/LoRA https://github.com/microsoft/LoRA). In brief, the technique assumes that fine-tuning a model really just involves tweaks to the model parameters that are "small" in some sense, and through math this algorithm confines the fine-tuning to just the small adjustment weights. Rather than asking an ordinary person to re-train 7 billion or 11 billion or 65 billion parameters, LoRA lets users fine-tune a model with about three orders of magnitude fewer adjustment parameters. Combine these two – publicly-available language model weights and a way to fine tune it – and you get work like the story here, where the language model is turned into something a lot like ChatGPT that can run on a consumer-grade laptop.
  - chronicler 4y ago
    Thanks for sharing. How do you know this? Can you recommend any papers to read to start learning about LLMs? I have very limited ML/AI knowledge.
  - xmonkee 4y ago
    Thanks, very helpful. Are llama and chatGPT essentially the same “program”, just with different weights? And is one better than the other (for the same number of parameters) just because it has better weights?
    - lbotos 4y ago
      My understanding is they are both "LLM" (Large language models). That's the generic term you are looking for. I don't think you can compare one LLMs weights to another directly, because the weights are a product of the LLM. In theory (I don't know actually) llama and chatGPT may be using different source datasets so you can't compare them like for like.
- physPop 4y ago
  silly back-ronyms made by academics
robwwilliams 4y ago
Here is a good start in “low-rank adaptation” or LoRA module. A way to train/adapt a general purpose LLM to efficiently and iteratively accommodate specialize data types and knowledge. A bolt-on. https://arxiv.org/pdf/2106.09685.pdf https://arxiv.org/pdf/2106.09685.pdf Not “on top” but more “in parallel” if I understand correctly

slg 4y ago

I have casually followed countless different news cycles on various complicated tech topics over my decades long career. I can't recall a single one that has consistently made me feel like an idiot more than how people talk about this recent AI wave. There just seems to be so much more jargon involved in this subject that makes casual perusing of the latest developments impenetrable.

akiselev 4y ago
Ah, my fellow citizen of the interwebs, fear not! Your intellectual frustrations are but a natural reaction to the tsunami of technological jargon. You see, the AI wave is the epitome of obfuscation, a testament to the labyrinthine lexicon of the digital age. It's as if a group of caffeinated, sleep-deprived tech enthusiasts assembled in the dark of night and decided to create an impenetrable fortress of vernacular, just to keep the uninitiated at bay.
- Grothendank 4y ago
  Should jackasses on HN use plain language instead of jargon? Surely. But AI workers mainly develop and use jargon because it is an easy and natural way to consolidate concepts. Sure, there is a kind of conspiracy caused by publish or perish. Researchers may use jargon to make their work harder to reject on review; laborious speech and jargon can make statements sound more profound. However, no technical field is immune to this. We'll need to systematically change science before we can eliminate that problem. Until we manage that, if you care about the concepts enough to want to understand them before there are good plain speech descriptions, just pop the jargon into google scholar and skim read a few papers, and you're good to go. If you don't care about the concepts that much, then don't worry about the jargon. The important concepts will get their own non-technical explanations in time. As it stands, AI jargon is not that bad. It tends to be pretty fair and easy to understand, compared to jargon in, say, biochemistry or higher math.
  - kordlessagain 4y ago
    With chatots, explaining everything clearly is an option.

jhbadger 4y ago

As someone who is following this technology while not really an expert (I'm a computational biologist in my day job) LoRA is a way of reducing the number of parameters in a large language model (LLM, the technology behind all these new chatbots) so that it can be run on less powerful hardware (say a laptop or even a phone). The OP is saying that the improvements this chatbot provides isn't so much being more clever about reducing parameters but being trained on text that has been cleaned up rather than the rather messy training sets used in other small LLMs.

f_devd 4y ago
LoRA (Low-Rank Adapter) is way to customize/finetune the LLM to a new datasets without needing to retrain the entire network (which makes it better (and in theory easier to do). It doesn't not change the speed significantly afaik

throwaway2865 4y ago

Less garbage in, less garbage out?