5 ms·

I feel like there has to be another shoe to drop here, this seems almost too good to be true. > Alpaca shows that you can apply fine-tuning with a feasible siz

by swyx 4y ago

I feel like there has to be another shoe to drop here, this seems almost too good to be true. > Alpaca shows that you can apply fine-tuning with a feasible sized set of examples (52,000) and cost ($600) such that even the smallest of the LLaMA models—the 7B one, which can compress down to a 4GB file with 4-bit quantization—provides results that compare well to cutting edge text-davinci-003 in initial human evaluation. this is the most exciting thing. the cost of finetuning is rapidly coming down which means everyone will be able to train their own models for their usecases. Looking for the contrarians on HN: what is being left unsaid here that people like myself and Simon might be getting too optimistic about? what are the known downsides that people in academia already know about?

Kye 4y ago
The big failure mode is they can hallucinate nonsense that isn't obviously nonsense. You have to check any facts against expert sources. At that point, you could just email an expert who can use their own LLM to whip up an answer and check the facts themselves.
- simonw 4y ago
  That's a big problem if you're using a language model as a search engine. The trick is to learn how to use them for the things that they're good for outside of that.
  - typest 4y ago
    ^^ this. For instance, LLMs are really good at turning natural language into SQL. And if you know SQL, you can read it and make sure it looks good. But, it's much faster and easier than writing SQL by hand.
    - flir 4y ago
      But that's still "You have to check any facts against expert sources"! You just have the advantage of being your own personal expert.
      - hesk 4y ago
        Verifying a solution is easier than coming up with it.
porcc 4y ago
We saw this happen with Stable Diffusion and it's not surprising we see this happening here. There is a lot of interest in taking these models that are in striking distance (single order of magnitude) from running inference and training on consumer level hardware and as such a lot of energy is going into making the optimizations that can get us there. Generally speaking, research is not usually done with consumer usage in mind, so what this is, and Dreambooth etc. for Stable Diffusion was, is that gap between researcher software and accessible software being bridged.
smoldesu 4y ago
> what is being left unsaid here that people like myself and Simon might be getting too optimistic about? The past week has felt like a wake-up call to enthusiasts. Running models locally has been available for a while (even small, fairly coherent ones), and the majority of "improvements" recently have come from implementing the leaked LLaMa model. The results from 7B are an improvement on what we had a year ago, but not by much. We're learning that there's room to optimize these models, but also that size matters. ChatGPT and 7B are both great at bullshitting, but you can feel the difference in model size during regular conversation. Adding insult to injury, it will almost always be faster to query an API for AI results than it will be to run it locally. Analysis: Things are moving at a clip right now, but people expecting competitive LLMs running locally on their smartphone will be disappointed for quite a while. As the technology improves, it's also safe to assume that we'll find ways to scale model intelligence with greater resources, and the status quo will look much different than it does today.
- atleastoptimal 4y ago
  >API for AI results than it will be to run it locally. True, and remotely called APIs will always be the mover for the AI craze. Only niche hobbyists will be running them locally. There is no company on the planet that would benefit from providing people local means to run LLMs. As a result only hacks and leaks will be how individuals can manage to run LLMs outside of heavily monitored remote API calls.
  - mikek 4y ago
    Apple comes to mind.
    - BryantD 4y ago
      Agreed. Apple will run models remotely if necessary, but from a PR perspective they align with their stated intentions when they can run locally.
      - mustacheemperor 4y ago
atleastoptimal 4y ago
"initial human evaluation" is codeword for "cherrypicked prompts given to people who don't know how to trick an LLM"
- sebzim4500 4y ago
  I don't understand why that is a bad thing. If your goal is to make an AI assistant, then you should be optimizing for giving answers that real users find useful, not trying to impress other AI researchers.
- ChubbyGlasses 4y ago
  i always found this to be a strange pov to have on LLMs. imo, it's not humans tricking/gaming the ai, but rather chatgpt has tricked you into believing it's smarter than it actually is. (in human terms, chatgpt is just more articulate than llama) it's a subtle distinction, but i think it shapes and reflects how you view ai as a tool for humans or as a replacement.
- bitL 4y ago
  There aren't any good automated metrics for generative LLM quality, so it's all up to humans to evaluate.
blueblimp 4y ago
It seems still unclear how much quality loss there is compared to the best models. What's really needed is systematic evaluation of the output quality, but that's tricky and relatively expensive (compared to automated benchmarks), so I understand why it hasn't happened yet. Edit: I just tried it with a single task of my own (that I've successfully used with ChatGPT and Bing) and it flubbed it horribly, so this model at least is noticeably inferior to the SOTA, which is not surprising given how small it is.
- yunyu 4y ago
  I assume you haven't tried Alpaca (which hasn't been released), only Llama. See the instruction fine tuning section in the article.
  - karmasimida 4y ago
    They currently only supports a single input/response format of input right? Multi turns will be more challenging to handle. I am optimistic for 30B or 66B to catch up with OpenAI, but 7B is unlikely to have the same quality.
    - enlyth 4y ago
      Here is a sample chatbot style conversation I had locally with LLAMA 30B 4bit quantization on a 3090 RTX (generation speed 5 tokens per second): https://i.imgur.com/V4lzLz7.png https://i.imgur.com/V4lzLz7.png It does quite well at simple reasoning. More complex stuff, it does struggle sometimes, but I am impressed by the output and the fact that it even runs on my computer. I used the oobabooga repository on Ubuntu 22.
- plaba417 4y ago
  I have had a few questions where it gave a completely nonsensical answer, but by fiddling with the parameters, I've gotten more usable output. Have you tried lowering the noise and increasing top_p?
flir 4y ago
At a guess: assuming quality scales with size, the model in the data centre is always going to outcompete the model on the device. So in any situation where you've got bandwidth >4800bps, why would you choose the model on the device?
- CuriouslyC 4y ago
  Your own fine tuning, no restrictions on output, privacy/security, and if you have a reason to produce a lot of output it'll be cheaper. Use ChatGPT if you only want to use it occasionally, you don't care about privacy/security in this context, the output restrictions don't bother you and having the best possible language model is the most important thing to you.
  - zone411 4y ago
    Right, not sending out your text to third parties is absolutely huge for certain industries. ChatGPT has been already banned by some companies, so the issue is whether or not the language model can be used at all. I'll add the possibility of network-based service downtime to the list.
dougmwne 4y ago
First catch is that someone needed to spend the enormous up front cost to train the base model, then release it under a flexible enough license for your use case. The second catch is that you would get much higher quality out of the 65b model, but would need to lay out a few thousand for the hardware. The third catch is that you need the fine tuning data, but that seems easier than ever to create out of more capable LMMs.
- recuter 4y ago
  The 65b model now runs on the 14" MBP with 64GB of RAM for $3,500. The base config is $2500 to begin with. So not thousands plural, it is one grand more than the base config and a lot of devs/pros go for the 32GB anyhow. Apple laptops also have atypically high resale value, I usually sell mine after a couple of years for 50%+ of what I paid for it. By the by, Apple have a two week return policy. ;) I think only your first catch applies. But if you can come up with the right mvp even that might fall away.
  - Taek 4y ago
    Don't need a macbook anymore, any AVX2 CPU can do more than one token per second. You could spend $600 and build your own PC that runs LLaMA 65b.
    - recuter 4y ago
      It is still much faster/cost-effective thanks to the unified memory. Would be interested in building out a PC as soon as something comparable becomes available.
      - Taek 4y ago
        If you are going to spend $3500 on macbook because its more cost effective, might as well go all the way to buying an a6000 for $4095 which is iirc 5-10x faster than macbook
        4y ago
a_bonobo 4y ago
>what are the known downsides that people in academia already know about? I work in biology with some interface to wildlife managers/'end users' of our biological insights. I was hoping we could use these models as advanced chatbots so that our end users could ask biological questions from our data. However, even the replies to the most basic biological questions in LLAMA/ChatGPT/OpenChatKit are hallucinated, even when setting temperature to 0. We simply can't trust any of these answers; the bot needs to be as truthful as humans in its replies, but none of them are so far. So what good are these models then?
- int_19h 4y ago
  This is the wrong way to go about it. Instead of trying to force that informaiton into the model via training, you need to set it up such that it "thinks out loud" as it answers the question, generating intermediate queries against your data sources in the process. Take a look at this chat log from Bing. Note that the user only sees the stuff that is marked #message and #suggestions. https://www.make-safe-ai.com/is-bing-chat-safe/Prompts_Conversations.txt https://www.make-safe-ai.com/is-bing-chat-safe/Prompts_Conve... Then take a look at https://langchain.readthedocs.io/en/latest/ https://langchain.readthedocs.io/en/latest/, specifically the Document Loaders and the Indexes sections to begin with.
  - a_bonobo 4y ago
    I've been playing a lot with the langchain Document Loaders - their examples make the results look more impressive than they are, or perhaps that's because our documents are 'special' (heavy in taxonomy/biology). I've been priming it with 'if you don't know, say that you don't know, don't invent anything' and it usually ends up saying 'I don't know' to questions it can clearly answer. But yeah, working on that part!
Havoc 4y ago
The licensing still seems like a significant hurdle for commercial
zitterbewegung 4y ago
Nothing much unsaid but huggingface has been doing this for smaller models. What is happening is they are distilling Davinci by using llama and it’s been done before . But this performance will still page behind OpenAI for a bit but within a month or a few months we should have something almost comparable it’s just frustrating that Stanford is blocking the code and model release when it’s just a matter of time for a full copy to come out.

by swyx 4y ago

Kye 4y ago

The big failure mode is they can hallucinate nonsense that isn't obviously nonsense. You have to check any facts against expert sources. At that point, you could just email an expert who can use their own LLM to whip up an answer and check the facts themselves.

simonw 4y ago
That's a big problem if you're using a language model as a search engine. The trick is to learn how to use them for the things that they're good for outside of that.
- typest 4y ago
  ^^ this. For instance, LLMs are really good at turning natural language into SQL. And if you know SQL, you can read it and make sure it looks good. But, it's much faster and easier than writing SQL by hand.
  - flir 4y ago
    But that's still "You have to check any facts against expert sources"! You just have the advantage of being your own personal expert.
    - hesk 4y ago
      Verifying a solution is easier than coming up with it.

porcc 4y ago

We saw this happen with Stable Diffusion and it's not surprising we see this happening here. There is a lot of interest in taking these models that are in striking distance (single order of magnitude) from running inference and training on consumer level hardware and as such a lot of energy is going into making the optimizations that can get us there. Generally speaking, research is not usually done with consumer usage in mind, so what this is, and Dreambooth etc. for Stable Diffusion was, is that gap between researcher software and accessible software being bridged.

smoldesu 4y ago

> what is being left unsaid here that people like myself and Simon might be getting too optimistic about? The past week has felt like a wake-up call to enthusiasts. Running models locally has been available for a while (even small, fairly coherent ones), and the majority of "improvements" recently have come from implementing the leaked LLaMa model. The results from 7B are an improvement on what we had a year ago, but not by much. We're learning that there's room to optimize these models, but also that size matters. ChatGPT and 7B are both great at bullshitting, but you can feel the difference in model size during regular conversation. Adding insult to injury, it will almost always be faster to query an API for AI results than it will be to run it locally. Analysis: Things are moving at a clip right now, but people expecting competitive LLMs running locally on their smartphone will be disappointed for quite a while. As the technology improves, it's also safe to assume that we'll find ways to scale model intelligence with greater resources, and the status quo will look much different than it does today.

atleastoptimal 4y ago
>API for AI results than it will be to run it locally. True, and remotely called APIs will always be the mover for the AI craze. Only niche hobbyists will be running them locally. There is no company on the planet that would benefit from providing people local means to run LLMs. As a result only hacks and leaks will be how individuals can manage to run LLMs outside of heavily monitored remote API calls.
- mikek 4y ago
  Apple comes to mind.
  - BryantD 4y ago
    Agreed. Apple will run models remotely if necessary, but from a PR perspective they align with their stated intentions when they can run locally.
    - mustacheemperor 4y ago

atleastoptimal 4y ago

"initial human evaluation" is codeword for "cherrypicked prompts given to people who don't know how to trick an LLM"

sebzim4500 4y ago
I don't understand why that is a bad thing. If your goal is to make an AI assistant, then you should be optimizing for giving answers that real users find useful, not trying to impress other AI researchers.
ChubbyGlasses 4y ago
i always found this to be a strange pov to have on LLMs. imo, it's not humans tricking/gaming the ai, but rather chatgpt has tricked you into believing it's smarter than it actually is. (in human terms, chatgpt is just more articulate than llama) it's a subtle distinction, but i think it shapes and reflects how you view ai as a tool for humans or as a replacement.
bitL 4y ago
There aren't any good automated metrics for generative LLM quality, so it's all up to humans to evaluate.

blueblimp 4y ago

It seems still unclear how much quality loss there is compared to the best models. What's really needed is systematic evaluation of the output quality, but that's tricky and relatively expensive (compared to automated benchmarks), so I understand why it hasn't happened yet. Edit: I just tried it with a single task of my own (that I've successfully used with ChatGPT and Bing) and it flubbed it horribly, so this model at least is noticeably inferior to the SOTA, which is not surprising given how small it is.

yunyu 4y ago
I assume you haven't tried Alpaca (which hasn't been released), only Llama. See the instruction fine tuning section in the article.
- karmasimida 4y ago
  They currently only supports a single input/response format of input right? Multi turns will be more challenging to handle. I am optimistic for 30B or 66B to catch up with OpenAI, but 7B is unlikely to have the same quality.
  - enlyth 4y ago
    Here is a sample chatbot style conversation I had locally with LLAMA 30B 4bit quantization on a 3090 RTX (generation speed 5 tokens per second): https://i.imgur.com/V4lzLz7.png https://i.imgur.com/V4lzLz7.png It does quite well at simple reasoning. More complex stuff, it does struggle sometimes, but I am impressed by the output and the fact that it even runs on my computer. I used the oobabooga repository on Ubuntu 22.
plaba417 4y ago
I have had a few questions where it gave a completely nonsensical answer, but by fiddling with the parameters, I've gotten more usable output. Have you tried lowering the noise and increasing top_p?

flir 4y ago

At a guess: assuming quality scales with size, the model in the data centre is always going to outcompete the model on the device. So in any situation where you've got bandwidth >4800bps, why would you choose the model on the device?

CuriouslyC 4y ago
Your own fine tuning, no restrictions on output, privacy/security, and if you have a reason to produce a lot of output it'll be cheaper. Use ChatGPT if you only want to use it occasionally, you don't care about privacy/security in this context, the output restrictions don't bother you and having the best possible language model is the most important thing to you.
- zone411 4y ago
  Right, not sending out your text to third parties is absolutely huge for certain industries. ChatGPT has been already banned by some companies, so the issue is whether or not the language model can be used at all. I'll add the possibility of network-based service downtime to the list.

dougmwne 4y ago

First catch is that someone needed to spend the enormous up front cost to train the base model, then release it under a flexible enough license for your use case. The second catch is that you would get much higher quality out of the 65b model, but would need to lay out a few thousand for the hardware. The third catch is that you need the fine tuning data, but that seems easier than ever to create out of more capable LMMs.

recuter 4y ago
The 65b model now runs on the 14" MBP with 64GB of RAM for $3,500. The base config is $2500 to begin with. So not thousands plural, it is one grand more than the base config and a lot of devs/pros go for the 32GB anyhow. Apple laptops also have atypically high resale value, I usually sell mine after a couple of years for 50%+ of what I paid for it. By the by, Apple have a two week return policy. ;) I think only your first catch applies. But if you can come up with the right mvp even that might fall away.
- Taek 4y ago
  Don't need a macbook anymore, any AVX2 CPU can do more than one token per second. You could spend $600 and build your own PC that runs LLaMA 65b.
  - recuter 4y ago
    It is still much faster/cost-effective thanks to the unified memory. Would be interested in building out a PC as soon as something comparable becomes available.
    - Taek 4y ago
      If you are going to spend $3500 on macbook because its more cost effective, might as well go all the way to buying an a6000 for $4095 which is iirc 5-10x faster than macbook
      - 4y ago

a_bonobo 4y ago

>what are the known downsides that people in academia already know about? I work in biology with some interface to wildlife managers/'end users' of our biological insights. I was hoping we could use these models as advanced chatbots so that our end users could ask biological questions from our data. However, even the replies to the most basic biological questions in LLAMA/ChatGPT/OpenChatKit are hallucinated, even when setting temperature to 0. We simply can't trust any of these answers; the bot needs to be as truthful as humans in its replies, but none of them are so far. So what good are these models then?

int_19h 4y ago
This is the wrong way to go about it. Instead of trying to force that informaiton into the model via training, you need to set it up such that it "thinks out loud" as it answers the question, generating intermediate queries against your data sources in the process. Take a look at this chat log from Bing. Note that the user only sees the stuff that is marked #message and #suggestions. https://www.make-safe-ai.com/is-bing-chat-safe/Prompts_Conversations.txt https://www.make-safe-ai.com/is-bing-chat-safe/Prompts_Conve... Then take a look at https://langchain.readthedocs.io/en/latest/ https://langchain.readthedocs.io/en/latest/, specifically the Document Loaders and the Indexes sections to begin with.
- a_bonobo 4y ago
  I've been playing a lot with the langchain Document Loaders - their examples make the results look more impressive than they are, or perhaps that's because our documents are 'special' (heavy in taxonomy/biology). I've been priming it with 'if you don't know, say that you don't know, don't invent anything' and it usually ends up saying 'I don't know' to questions it can clearly answer. But yeah, working on that part!

Havoc 4y ago

The licensing still seems like a significant hurdle for commercial

zitterbewegung 4y ago

Nothing much unsaid but huggingface has been doing this for smaller models. What is happening is they are distilling Davinci by using llama and it’s been done before . But this performance will still page behind OpenAI for a bit but within a month or a few months we should have something almost comparable it’s just frustrating that Stanford is blocking the code and model release when it’s just a matter of time for a full copy to come out.