4 ms·

That's great to hear. Thank you very much, both for reporting this, and especially for the crucial note about temperature. In fact, sampling settings are so im

That's great to hear. Thank you very much, both for reporting this, and especially for the crucial note about temperature. In fact, sampling settings are so important and so easily underestimated that I should just pester you to post your exact settings. If you get a moment, would you mind sharing your temperature, repetition penalty, top-k, and anything else? I'll be experimenting with those today, but having some known working defaults would be wonderful. (You're also the first person I've seen that got excellent outputs from llama; whatever you did, no one else seems to have noticed yet.) If you're busy or don't feel like it, no worries though. I'm just grateful you gave us some hope that llama might be really good. There were so many tweet chains showing universally awful outputs that I wasn't sure. EDIT: I added your comments to the top of the README and credited you. Thanks again.

linearalgebra45 4y ago
Would you mind publishing your notes/learnings once you gain enough understanding of this model?
- sillysaurusx 4y ago
  Absolutely! I'll make sure to leave a comment here for you whenever something gets written up so you don't miss it. Getting "as good as davinci" on a single A100 is groundbreaking work. Facebook and the community should both be credited here -- maybe llama-int8 would've been created even if the model hadn't leaked, but I don't think it would've happened so quickly. Everyone is doing phenomenal work, and it's so amazing to see it all come together. But, we'll see. Going to try it myself soon. Long ago, I cloned OpenAI's API: https://github.com/shawwn/openai-server https://github.com/shawwn/openai-server -- my plan is, once I get it running, I'll try to host it somewhere so that anyone can play with it. I assume it'll be quickly swamped, but it's still an interesting challenge; some basic load balancing should make it scalable across several A100 instances, so there's no reason we can't just roll our own OpenAI API.
  - rnosov 4y ago
    Seconded. Do write it up. I see vast.ai listing interruptible instance with a single A100 80GB at $1/hour which is pretty reasonable. ChatGPT plus is $20/month which would be roughly 20 hours of use and I wont't be lectured like I'm in a kindergarten or something. A bonus point would be to make the writeup accessible for AI challenged developers. Asking for a friend.
    - davrosthedalek 4y ago
      I would like to support this request for AI challenged developers :) For things like these, I always wonder: How much slower would it be to run such a model on a CPU? I mean, clearly a lot less interactive, but is it possible at all? Could it be chopped up and "streamed" to a GPU with less memory halfway efficiently? What is the bottleneck currently on GPUs, memory bw or compute?
      - nl 4y ago
        On a CPU I'd estimate it would get a maximum of around 5 tokens per second (a token being a sub-word token, so generally a couple of letters). I suspect it'd be more like 1 token per second on the large model without additional optimisation. Yes models can be split up. See eg Hugging Face Accelerate.
        davrosthedalek 4y ago
        That's actually a lot better than I would have thought. Almost usable, and a good exercise in patience.
        nl 4y ago
        I'd expect significant performance improvements over the next few months are more people work on this in the same way the stable diffusion is now fairly usable on a CPU. It's always going to be slow on a CPU, but the smaller models might be usable for experimentation at some point.

linearalgebra45 4y ago

Would you mind publishing your notes/learnings once you gain enough understanding of this model?

sillysaurusx 4y ago
Absolutely! I'll make sure to leave a comment here for you whenever something gets written up so you don't miss it. Getting "as good as davinci" on a single A100 is groundbreaking work. Facebook and the community should both be credited here -- maybe llama-int8 would've been created even if the model hadn't leaked, but I don't think it would've happened so quickly. Everyone is doing phenomenal work, and it's so amazing to see it all come together. But, we'll see. Going to try it myself soon. Long ago, I cloned OpenAI's API: https://github.com/shawwn/openai-server https://github.com/shawwn/openai-server -- my plan is, once I get it running, I'll try to host it somewhere so that anyone can play with it. I assume it'll be quickly swamped, but it's still an interesting challenge; some basic load balancing should make it scalable across several A100 instances, so there's no reason we can't just roll our own OpenAI API.
- rnosov 4y ago
  Seconded. Do write it up. I see vast.ai listing interruptible instance with a single A100 80GB at $1/hour which is pretty reasonable. ChatGPT plus is $20/month which would be roughly 20 hours of use and I wont't be lectured like I'm in a kindergarten or something. A bonus point would be to make the writeup accessible for AI challenged developers. Asking for a friend.
  - davrosthedalek 4y ago
    I would like to support this request for AI challenged developers :) For things like these, I always wonder: How much slower would it be to run such a model on a CPU? I mean, clearly a lot less interactive, but is it possible at all? Could it be chopped up and "streamed" to a GPU with less memory halfway efficiently? What is the bottleneck currently on GPUs, memory bw or compute?
    - nl 4y ago
      On a CPU I'd estimate it would get a maximum of around 5 tokens per second (a token being a sub-word token, so generally a couple of letters). I suspect it'd be more like 1 token per second on the large model without additional optimisation. Yes models can be split up. See eg Hugging Face Accelerate.
      - davrosthedalek 4y ago
        That's actually a lot better than I would have thought. Almost usable, and a good exercise in patience.
        nl 4y ago
        I'd expect significant performance improvements over the next few months are more people work on this in the same way the stable diffusion is now fairly usable on a CPU. It's always going to be slow on a CPU, but the smaller models might be usable for experimentation at some point.