5 ms·

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

by MacsHeadroom 4y ago

deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
deleted 4y ago
[deleted]
linearalgebra45 4y ago
What instance are you using?
sillysaurusx 4y ago
Careful though — we need to evaluate llama on its own merits. It’s easy to mess up the quantization in subtle ways, then conclude that the outputs aren’t great. So if you’re seeing poor results vs gpt-3, hold off judgement till people have had time to really make sure the quantized models are >97% the effectiveness of the original weights. That said, this is awesome — please share some outputs! What’s it like?
- MacsHeadroom 4y ago
  The output is at least as good as davinci. I think some early results are using bad repetition penalty and/or temperature settings. I had to set both fairly high to get the best results. (Some people are also incorrectly comparing it to chatGPT/ChatGPT API which is not a good comparison. But that's a different problem.) I've had it translate, write poems, tell jokes, banter, write executable code. It does it all-- and all on a single card.
  - data_maan 4y ago
    Is it just the RLHF training for the prompting that makes a difference, or are there also other, more tangible differences?
  - akreal 4y ago
    Which prompt did you use for translation? I'd be curious to try it for my task too.
  - sillysaurusx 4y ago
    That's great to hear. Thank you very much, both for reporting this, and especially for the crucial note about temperature. In fact, sampling settings are so important and so easily underestimated that I should just pester you to post your exact settings. If you get a moment, would you mind sharing your temperature, repetition penalty, top-k, and anything else? I'll be experimenting with those today, but having some known working defaults would be wonderful. (You're also the first person I've seen that got excellent outputs from llama; whatever you did, no one else seems to have noticed yet.) If you're busy or don't feel like it, no worries though. I'm just grateful you gave us some hope that llama might be really good. There were so many tweet chains showing universally awful outputs that I wasn't sure. EDIT: I added your comments to the top of the README and credited you. Thanks again.
    - linearalgebra45 4y ago
      Would you mind publishing your notes/learnings once you gain enough understanding of this model?
      - sillysaurusx 4y ago
        Absolutely! I'll make sure to leave a comment here for you whenever something gets written up so you don't miss it. Getting "as good as davinci" on a single A100 is groundbreaking work. Facebook and the community should both be credited here -- maybe llama-int8 would've been created even if the model hadn't leaked, but I don't think it would've happened so quickly. Everyone is doing phenomenal work, and it's so amazing to see it all come together. But, we'll see. Going to try it myself soon. Long ago, I cloned OpenAI's API: https://github.com/shawwn/openai-server https://github.com/shawwn/openai-server -- my plan is, once I get it running, I'll try to host it somewhere so that anyone can play with it. I assume it'll be quickly swamped, but it's still an interesting challenge; some basic load balancing should make it scalable across several A100 instances, so there's no reason we can't just roll our own OpenAI API.
        rnosov 4y ago
        Seconded. Do write it up. I see vast.ai listing interruptible instance with a single A100 80GB at $1/hour which is pretty reasonable. ChatGPT plus is $20/month which would be roughly 20 hours of use and I wont't be lectured like I'm in a kindergarten or something. A bonus point would be to make the writeup accessible for AI challenged developers. Asking for a friend.
  - v64 4y ago
    Note that unlike ChatGPT, these models are pure text completers and have not been trained to be prompted. The llama FAQ [1] mentions this and gives tips for how to get out of the ChatGPT mindset and prompt llama better. [1] https://github.com/facebookresearch/llama/blob/main/FAQ.md#2 https://github.com/facebookresearch/llama/blob/main/FAQ.md#2
youssefabdelm 4y ago
What's the speed like? How many tokens per second? / Is it as fast as say ChatGPT?
- MacsHeadroom 4y ago
  It's about as fast as chatGPT when chatGPT first launched. Not as fast as the new "Turbo" version of chatGPT, but much faster than you or anyone can read (so I'm not sure the difference matters).
  - youssefabdelm 4y ago
    That's awesome! thanks!
dangoodmanUT 4y ago
Did you modify the lambda.cpp repo to move frmo 4-bit to 8-bit quantization? Or did you write something custom?

I'm running LLaMA-65B on a single A100 80GB with 8bit quantization. $1.5/hr on vast.ai

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

deleted 4y ago

[deleted]

linearalgebra45 4y ago

What instance are you using?

sillysaurusx 4y ago

Careful though — we need to evaluate llama on its own merits. It’s easy to mess up the quantization in subtle ways, then conclude that the outputs aren’t great. So if you’re seeing poor results vs gpt-3, hold off judgement till people have had time to really make sure the quantized models are >97% the effectiveness of the original weights. That said, this is awesome — please share some outputs! What’s it like?

MacsHeadroom 4y ago
The output is at least as good as davinci. I think some early results are using bad repetition penalty and/or temperature settings. I had to set both fairly high to get the best results. (Some people are also incorrectly comparing it to chatGPT/ChatGPT API which is not a good comparison. But that's a different problem.) I've had it translate, write poems, tell jokes, banter, write executable code. It does it all-- and all on a single card.
- data_maan 4y ago
  Is it just the RLHF training for the prompting that makes a difference, or are there also other, more tangible differences?
- akreal 4y ago
  Which prompt did you use for translation? I'd be curious to try it for my task too.
- sillysaurusx 4y ago
  That's great to hear. Thank you very much, both for reporting this, and especially for the crucial note about temperature. In fact, sampling settings are so important and so easily underestimated that I should just pester you to post your exact settings. If you get a moment, would you mind sharing your temperature, repetition penalty, top-k, and anything else? I'll be experimenting with those today, but having some known working defaults would be wonderful. (You're also the first person I've seen that got excellent outputs from llama; whatever you did, no one else seems to have noticed yet.) If you're busy or don't feel like it, no worries though. I'm just grateful you gave us some hope that llama might be really good. There were so many tweet chains showing universally awful outputs that I wasn't sure. EDIT: I added your comments to the top of the README and credited you. Thanks again.
  - linearalgebra45 4y ago
    Would you mind publishing your notes/learnings once you gain enough understanding of this model?
    - sillysaurusx 4y ago
      Absolutely! I'll make sure to leave a comment here for you whenever something gets written up so you don't miss it. Getting "as good as davinci" on a single A100 is groundbreaking work. Facebook and the community should both be credited here -- maybe llama-int8 would've been created even if the model hadn't leaked, but I don't think it would've happened so quickly. Everyone is doing phenomenal work, and it's so amazing to see it all come together. But, we'll see. Going to try it myself soon. Long ago, I cloned OpenAI's API: https://github.com/shawwn/openai-server https://github.com/shawwn/openai-server -- my plan is, once I get it running, I'll try to host it somewhere so that anyone can play with it. I assume it'll be quickly swamped, but it's still an interesting challenge; some basic load balancing should make it scalable across several A100 instances, so there's no reason we can't just roll our own OpenAI API.
      - rnosov 4y ago
        Seconded. Do write it up. I see vast.ai listing interruptible instance with a single A100 80GB at $1/hour which is pretty reasonable. ChatGPT plus is $20/month which would be roughly 20 hours of use and I wont't be lectured like I'm in a kindergarten or something. A bonus point would be to make the writeup accessible for AI challenged developers. Asking for a friend.
- v64 4y ago
  Note that unlike ChatGPT, these models are pure text completers and have not been trained to be prompted. The llama FAQ [1] mentions this and gives tips for how to get out of the ChatGPT mindset and prompt llama better. [1] https://github.com/facebookresearch/llama/blob/main/FAQ.md#2 https://github.com/facebookresearch/llama/blob/main/FAQ.md#2

youssefabdelm 4y ago

What's the speed like? How many tokens per second? / Is it as fast as say ChatGPT?

MacsHeadroom 4y ago
It's about as fast as chatGPT when chatGPT first launched. Not as fast as the new "Turbo" version of chatGPT, but much faster than you or anyone can read (so I'm not sure the difference matters).
- youssefabdelm 4y ago
  That's awesome! thanks!

dangoodmanUT 4y ago

Did you modify the lambda.cpp repo to move frmo 4-bit to 8-bit quantization? Or did you write something custom?