4 ms·

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being

Hardly. I've played a lot with the 7,13, and 30B llamas as well as the 7 and 13B alpacas fine tuned by Stanford. They do not have emergent abilities like being able to generate rhymes or, say, represent a movie plot as emoji. Even openai's old text-davinci-003 (gpt3.5, but text completion, not the chat ones) far outperforms them. That said, I have hopes for a 65B 3-bit quantized alpaca-fine tuned. We'll see when someone spends the money to do the (more costly) 65B training. The alpacas are also much more likely to go off rails and start regurgitating their fine-tuning inputs. Either that or openai is doing a lot of post processing on their end to hide the same problems in their LLM. For now my IRC bots run the alpaca 7B 4-bit. 13B was not a significant improvement for twice the computational time. But it's best to learn them now because as soon as openai gets sued for the first time all the turing test passing older models without the legal-butt-covering bolted on will be removed.

nickthegreek 4y ago
Where does one find the 13B alpaca model?
- superkuh 4y ago
  Be aware this file is a single ~8GB 4-bit model (ggml-alpaca-13b-q4.bin) instead of the 2x ~4GB models (ggml-model-q4_0.bin, ggml-model-q4_0.bin.1) that most llama.cpp style inference running programs expect. You'll probably have to edit the line, n_parts = LLAMA_N_PARTS.at(hparams.n_embd); in chat.cpp (or main.cpp) to hard code it to treat this 1 file model properly like, n_parts = 1; Or re-write the parameter config subroutine to recognize and handle non-standard weights file. magnet: magnet:?xt=urn:btih:053b3d54d2e77ff020ebddf51dad681f2a651071&dn=ggml-alpaca-13b-q4.bin&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A6969%2Fannounce&tr=udp%3A%2F%2F9.rarbg.com%3A2810%2Fannounce torrent: https://btcache.me/torrent/053B3D54D2E77FF020EBDDF51DAD681F2A651071 https://btcache.me/torrent/053B3D54D2E77FF020EBDDF51DAD681F2... torrent: https://torrage.info/torrent.php?h=053b3d54d2e77ff020ebddf51dad681f2a651071 https://torrage.info/torrent.php?h=053b3d54d2e77ff020ebddf51... via: https://github.com/antimatter15/alpaca.cpp https://github.com/antimatter15/alpaca.cpp
LASR 4y ago
In my personal testing, I throw some sophisticated use cases at LLMs - particularly chain of thought reasoning. None of the models out there are able to do this this well, except for the OG GPT-3 Davinci-003. Even the newer turbo models are not as good. I am playing around with GPT-4 this week though. Let’s see how that goes.
- stavros 4y ago
  The newer turbo models are the ChatGPT models, and are worse than text-davinci-003, in my experience. The gpt-4 model is also not as good as the GPT-4 chat version, which is very odd.
  - LASR 4y ago
    Yeah I think it lines up with Davinci being 10x as expensive as Turbo.
crooked-v 4y ago
For me the easiest comparison between models is to give it an absurd but entirely possible request, like "Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain". So far nothing else I've tried has done even as well as GPT 3.5 yet, let alone GPT 4.
- starik36 4y ago
  That is my experience as well. I've tried various models but nothing comes even close to the current ChatGPT implementation (when it manages to stay up).
- TedDoesntTalk 4y ago
  > Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain Whoa. I want to read this! Duel - what a great film. Twain - amazing writer. Animorphs - published after my teen years but sounds like a great story!
  - Larrikin 4y ago
    Highly recommend Animorphs. As an adult the books can be read through pretty quickly, but the story really holds up. Deals with some heavy topics of genocide, enslavement (even exploring the justifications of the enslavers), PTSD among other topics but was still considered a kids book. It becomes obvious in the middle when some of the books were written by ghost writers, but the books are so easy to read I don't really recommend skipping them. If you must you could probably get away with reading the first ten, last ten, but should definitely read all of the Chronicle books.
    - crooked-v 4y ago
      Maybe the most striking bit of its exploration of the morality of war is that the series spends something like 45 books building up the alien Andalites as beneficent do-gooders who the team of protagonists expect to swoop in and fix everything (including one of the protagonists who is a young Andalite stranded on Earth), and then it turns out that they've been as morally worn down by the ongoing interstellar war as the protagonists have in their guerrilla conflicts on Earth and are effectively as hostile to humans as the original invaders.
deleted 4y ago
[deleted]
thomasahle 4y ago
3 bits? Is that for all weights in the network?
- deleted 4y ago
  [deleted]
- superkuh 4y ago
  As far as I know, yes. https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 "Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline." This would be 175 billion 3 bit weights instead of 175 billion 16 (or 32!) bit weights. It massively reduces the size of the model. It makes loading it in ram on consumer computers feasible. The number of parameters stays the same.
  - rnosov 4y ago
    > https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 I've read the paper and to be honest I'm not sure what to make of it. Their headline benchmark is perplexity on WikiText2 which would not be particularly relevant to most users. If you look at the tables in the appendix A.4 with some more relevant benchmarks you'll sometimes find that straight RTN 4 bit quantisation beats both GPTQ and even full 16 bit original! No explanation of it is given in the paper.
    - sebzim4500 4y ago
      Some of those benchmarks have a pretty small sample size IIRC, might just be coincidence that the noise introduced by RTN just happens to slightly improve them. GPTQ beats RTN on almost every benchmark at almost every size, though.
  - rcme 4y ago
    The number of parameters stays the same, but the amount of information encodable by those parameters is not the same.
ryoshu 4y ago
So what you're saying is it's a matter of time?
UncleOxidant 4y ago
> the alpaca 7B _4-bit_ [and presumably also 4bit for the 13B, 30B and larger parameter sets] This is the wild card here, though, isn't it? OpenAI's chatGPT likely uses more than 4 bits for it's parameters. IIRC the original LLaMA params were 16bit floats and they were quantitized down to 4bit - considering that large amount of compression, they sill do pretty OK, but not as good as chatGPT. I wonder how the alpaca/LLaMA models would do with 16bit floating point params (as they were originally trained)? What if they would have gone with 8 bits for the params as a compromise? EDIT: Come to think of it, unless you're using vectorized ops on a CPU, 4 bit and 8 bit math is going to run at the same speed (for most popular CPUs), is it not? So why did they go all the way down to 4 bits instead of stopping at 8 bits (other than to make the param files 1/2 the size)? EDIT2: looking through the alpacca.cpp code and there is mention of AVX, AVX2, AVX512 (and NEON on ARM) so it probably is taking advantage of vectorized ops where that's possible.
- leodriesch 4y ago
  Not an expert on the matter so take this with a grain of salt, but I’d say the compression is also about VRAM/RAM, which seems to be the more limiting factor over inference speed.
genericacct 4y ago
fwiw 7B is totaly useless for the subset of non english languages i've used, 13B a bit less so, but nowhere near as good as gpt. GPT's performance in non-trivial translation tasks is unbelievable. all those articles mentioning jobs that are going to be replaced fail to mention translators are probably going to be the first.
- user_named 4y ago
  You need the translators to QA the output from GPT. It's less work but not much less, and more types of translation work becomes feasible when leveraging GPT. I'm guessing the job market for translators will grow, not decline.
onlyrealcuzzo 4y ago
It's interesting that when ChatGPT 3.5 came out - everyone said, this is it! It's ready for primetime. And now that there's a few competitors in the same league - 3.5 quality is suddenly garbage and only 4.0 is good enough. Was it good enough before or wasn't it?
- bioemerl 4y ago
  It's really not good enough yet, it's impressive for what it is in our current time. But we're looking at the 1980s computers. They are neat, they are useful, but they can do so much more.
- coeneedell 4y ago
  The problem is that you’ve identified two distinct and non-overlapping sets of people as “everyone”. Everyone who was applauding 3.5 when it came out were industry hype people. Even the critical voices were industry hype people, paid to assume the AI is powerful and write about the possible negative consequences of that assumption. Now we’ve all gotten familiar with 3.5, and we’ve come to understand its limitations, so the public knows it’s not a “godlike” AI. Luckily there’s a fresh new model, not technically different from the earlier one but it cost more money to build. The hype group can start again, citing the publicly known limitations of 3.5. But in 6 months we’ll understand what’s wrong with it, and the public will be talking about the limitations, just in time for 4.5.
- jonplackett 4y ago
  I think it’s going to be like movie special effects. When Jurassic park first came out, or even something like Star Trek next gen. It looked AMAZING. So so realistic. But then…. As time goes on new things showed us what realistic could be. I think we actually got better at seeing. Same thing here. The more time you spend with it the more you notice things that don’t quite work. And then the new thing solves those problems, but we’ll find more wrongness
codetrotter 4y ago
> represent a movie plot as emoji This sounded like a really cool idea but I asked ChatGPT to do this for the plot of the movie The Shawashank Redemption and there is no way that I would ever have been able to guess that movie from the emojis it gave me. Perhaps GPT-4 does a better job at it.
throwaway1851 4y ago
Hm. I haven’t tried the local installs yet. However, when the Alpaca web demo was live, I did find it to be comparable (though not quite as capable) to davinci-003. It answered arbitrary factual questions about pop culture references, law, medicine, and programming. It generated rhymes and poems. (I didn’t try asking for the emoji thing, so can’t say anything about that.) It performed natural language tasks such as information extraction and summarization. And it did all of it coherently.
satvikpendem 4y ago
You might need to fix your parameters. From the text-generation-gui guide: > For a more creative chat, use: temp 0.72, rep pen 1.1, top_k 0, and top_p 0.73 > For a more precise chat, use temp 0.7, repetition_penalty 1.1764705882352942 (1/0.85), top_k 40, and top_p 0.1 https://old.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/ https://old.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_... https://old.reddit.com/r/singularity/comments/11vsvro/in_case_you_missed_it_open_assistant_recently/jcwbvi7/ https://old.reddit.com/r/singularity/comments/11vsvro/in_cas... https://twitter.com/theshawwn/status/1632569215348531201 https://twitter.com/theshawwn/status/1632569215348531201 --- That being said, I found the OpenAssistant model much better: https://huggingface.co/spaces/olivierdehaene/chat-llm-streaming https://huggingface.co/spaces/olivierdehaene/chat-llm-stream... It's also completely OSS, Apache 2.0, unlike LLaMA and Alpaca which are non-commercial.
- circuit10 4y ago
  I’m impressed by how ChatGPT-like it is but also it’s saying things like “No, OpenAI does not have an API for dogs. They do, however, have an API for other animals, such as cats. To retrieve an image of a cat, you can use the OpenAI API for Dogs API and select the cat breed or type.”
- superkuh 4y ago
  I've tried all sorts of parameters including those exact ones. As for the huggingface stuff, it's not exactly clear how to use it without going down the python dependency rabbit hole. I am not confident I could get the correct python packages all together on Debian 11 to support running it. The llama.cpp stuff is very simple to compile and run comparatively.

nickthegreek 4y ago

Where does one find the 13B alpaca model?

superkuh 4y ago
Be aware this file is a single ~8GB 4-bit model (ggml-alpaca-13b-q4.bin) instead of the 2x ~4GB models (ggml-model-q4_0.bin, ggml-model-q4_0.bin.1) that most llama.cpp style inference running programs expect. You'll probably have to edit the line, n_parts = LLAMA_N_PARTS.at(hparams.n_embd); in chat.cpp (or main.cpp) to hard code it to treat this 1 file model properly like, n_parts = 1; Or re-write the parameter config subroutine to recognize and handle non-standard weights file. magnet: magnet:?xt=urn:btih:053b3d54d2e77ff020ebddf51dad681f2a651071&dn=ggml-alpaca-13b-q4.bin&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A6969%2Fannounce&tr=udp%3A%2F%2F9.rarbg.com%3A2810%2Fannounce torrent: https://btcache.me/torrent/053B3D54D2E77FF020EBDDF51DAD681F2A651071 https://btcache.me/torrent/053B3D54D2E77FF020EBDDF51DAD681F2... torrent: https://torrage.info/torrent.php?h=053b3d54d2e77ff020ebddf51dad681f2a651071 https://torrage.info/torrent.php?h=053b3d54d2e77ff020ebddf51... via: https://github.com/antimatter15/alpaca.cpp https://github.com/antimatter15/alpaca.cpp

LASR 4y ago

In my personal testing, I throw some sophisticated use cases at LLMs - particularly chain of thought reasoning. None of the models out there are able to do this this well, except for the OG GPT-3 Davinci-003. Even the newer turbo models are not as good. I am playing around with GPT-4 this week though. Let’s see how that goes.

stavros 4y ago
The newer turbo models are the ChatGPT models, and are worse than text-davinci-003, in my experience. The gpt-4 model is also not as good as the GPT-4 chat version, which is very odd.
- LASR 4y ago
  Yeah I think it lines up with Davinci being 10x as expensive as Turbo.

crooked-v 4y ago

For me the easiest comparison between models is to give it an absurd but entirely possible request, like "Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain". So far nothing else I've tried has done even as well as GPT 3.5 yet, let alone GPT 4.

starik36 4y ago
That is my experience as well. I've tried various models but nothing comes even close to the current ChatGPT implementation (when it manages to stay up).
TedDoesntTalk 4y ago
> Write me a fanfic where the Animorphs battle the truck from Duel, but in the style of Mark Twain Whoa. I want to read this! Duel - what a great film. Twain - amazing writer. Animorphs - published after my teen years but sounds like a great story!
- Larrikin 4y ago
  Highly recommend Animorphs. As an adult the books can be read through pretty quickly, but the story really holds up. Deals with some heavy topics of genocide, enslavement (even exploring the justifications of the enslavers), PTSD among other topics but was still considered a kids book. It becomes obvious in the middle when some of the books were written by ghost writers, but the books are so easy to read I don't really recommend skipping them. If you must you could probably get away with reading the first ten, last ten, but should definitely read all of the Chronicle books.
  - crooked-v 4y ago
    Maybe the most striking bit of its exploration of the morality of war is that the series spends something like 45 books building up the alien Andalites as beneficent do-gooders who the team of protagonists expect to swoop in and fix everything (including one of the protagonists who is a young Andalite stranded on Earth), and then it turns out that they've been as morally worn down by the ongoing interstellar war as the protagonists have in their guerrilla conflicts on Earth and are effectively as hostile to humans as the original invaders.

deleted 4y ago

[deleted]

thomasahle 4y ago

3 bits? Is that for all weights in the network?

deleted 4y ago
[deleted]
superkuh 4y ago
As far as I know, yes. https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 "Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline." This would be 175 billion 3 bit weights instead of 175 billion 16 (or 32!) bit weights. It massively reduces the size of the model. It makes loading it in ram on consumer computers feasible. The number of parameters stays the same.
- rnosov 4y ago
  > https://arxiv.org/abs/2210.17323 https://arxiv.org/abs/2210.17323 I've read the paper and to be honest I'm not sure what to make of it. Their headline benchmark is perplexity on WikiText2 which would not be particularly relevant to most users. If you look at the tables in the appendix A.4 with some more relevant benchmarks you'll sometimes find that straight RTN 4 bit quantisation beats both GPTQ and even full 16 bit original! No explanation of it is given in the paper.
  - sebzim4500 4y ago
    Some of those benchmarks have a pretty small sample size IIRC, might just be coincidence that the noise introduced by RTN just happens to slightly improve them. GPTQ beats RTN on almost every benchmark at almost every size, though.
- rcme 4y ago
  The number of parameters stays the same, but the amount of information encodable by those parameters is not the same.

ryoshu 4y ago

So what you're saying is it's a matter of time?

UncleOxidant 4y ago

> the alpaca 7B _4-bit_ [and presumably also 4bit for the 13B, 30B and larger parameter sets] This is the wild card here, though, isn't it? OpenAI's chatGPT likely uses more than 4 bits for it's parameters. IIRC the original LLaMA params were 16bit floats and they were quantitized down to 4bit - considering that large amount of compression, they sill do pretty OK, but not as good as chatGPT. I wonder how the alpaca/LLaMA models would do with 16bit floating point params (as they were originally trained)? What if they would have gone with 8 bits for the params as a compromise? EDIT: Come to think of it, unless you're using vectorized ops on a CPU, 4 bit and 8 bit math is going to run at the same speed (for most popular CPUs), is it not? So why did they go all the way down to 4 bits instead of stopping at 8 bits (other than to make the param files 1/2 the size)? EDIT2: looking through the alpacca.cpp code and there is mention of AVX, AVX2, AVX512 (and NEON on ARM) so it probably is taking advantage of vectorized ops where that's possible.

leodriesch 4y ago
Not an expert on the matter so take this with a grain of salt, but I’d say the compression is also about VRAM/RAM, which seems to be the more limiting factor over inference speed.

genericacct 4y ago

fwiw 7B is totaly useless for the subset of non english languages i've used, 13B a bit less so, but nowhere near as good as gpt. GPT's performance in non-trivial translation tasks is unbelievable. all those articles mentioning jobs that are going to be replaced fail to mention translators are probably going to be the first.

user_named 4y ago
You need the translators to QA the output from GPT. It's less work but not much less, and more types of translation work becomes feasible when leveraging GPT. I'm guessing the job market for translators will grow, not decline.

onlyrealcuzzo 4y ago

It's interesting that when ChatGPT 3.5 came out - everyone said, this is it! It's ready for primetime. And now that there's a few competitors in the same league - 3.5 quality is suddenly garbage and only 4.0 is good enough. Was it good enough before or wasn't it?

bioemerl 4y ago
It's really not good enough yet, it's impressive for what it is in our current time. But we're looking at the 1980s computers. They are neat, they are useful, but they can do so much more.
coeneedell 4y ago
The problem is that you’ve identified two distinct and non-overlapping sets of people as “everyone”. Everyone who was applauding 3.5 when it came out were industry hype people. Even the critical voices were industry hype people, paid to assume the AI is powerful and write about the possible negative consequences of that assumption. Now we’ve all gotten familiar with 3.5, and we’ve come to understand its limitations, so the public knows it’s not a “godlike” AI. Luckily there’s a fresh new model, not technically different from the earlier one but it cost more money to build. The hype group can start again, citing the publicly known limitations of 3.5. But in 6 months we’ll understand what’s wrong with it, and the public will be talking about the limitations, just in time for 4.5.
jonplackett 4y ago
I think it’s going to be like movie special effects. When Jurassic park first came out, or even something like Star Trek next gen. It looked AMAZING. So so realistic. But then…. As time goes on new things showed us what realistic could be. I think we actually got better at seeing. Same thing here. The more time you spend with it the more you notice things that don’t quite work. And then the new thing solves those problems, but we’ll find more wrongness

codetrotter 4y ago

> represent a movie plot as emoji This sounded like a really cool idea but I asked ChatGPT to do this for the plot of the movie The Shawashank Redemption and there is no way that I would ever have been able to guess that movie from the emojis it gave me. Perhaps GPT-4 does a better job at it.

throwaway1851 4y ago

Hm. I haven’t tried the local installs yet. However, when the Alpaca web demo was live, I did find it to be comparable (though not quite as capable) to davinci-003. It answered arbitrary factual questions about pop culture references, law, medicine, and programming. It generated rhymes and poems. (I didn’t try asking for the emoji thing, so can’t say anything about that.) It performed natural language tasks such as information extraction and summarization. And it did all of it coherently.

satvikpendem 4y ago

You might need to fix your parameters. From the text-generation-gui guide: > For a more creative chat, use: temp 0.72, rep pen 1.1, top_k 0, and top_p 0.73 > For a more precise chat, use temp 0.7, repetition_penalty 1.1764705882352942 (1/0.85), top_k 40, and top_p 0.1 https://old.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/ https://old.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_... https://old.reddit.com/r/singularity/comments/11vsvro/in_case_you_missed_it_open_assistant_recently/jcwbvi7/ https://old.reddit.com/r/singularity/comments/11vsvro/in_cas... https://twitter.com/theshawwn/status/1632569215348531201 https://twitter.com/theshawwn/status/1632569215348531201 --- That being said, I found the OpenAssistant model much better: https://huggingface.co/spaces/olivierdehaene/chat-llm-streaming https://huggingface.co/spaces/olivierdehaene/chat-llm-stream... It's also completely OSS, Apache 2.0, unlike LLaMA and Alpaca which are non-commercial.

circuit10 4y ago
I’m impressed by how ChatGPT-like it is but also it’s saying things like “No, OpenAI does not have an API for dogs. They do, however, have an API for other animals, such as cats. To retrieve an image of a cat, you can use the OpenAI API for Dogs API and select the cat breed or type.”
superkuh 4y ago
I've tried all sorts of parameters including those exact ones. As for the huggingface stuff, it's not exactly clear how to use it without going down the python dependency rabbit hole. I am not confident I could get the correct python packages all together on Debian 11 to support running it. The llama.cpp stuff is very simple to compile and run comparatively.