3 ms·

Thanks be to open-source https://huggingface.co/models?sort=trending&search=thebloke%2FLlama-2 https://huggingface.co/models?sort=trending&search=thebloke%...

by lumost 3y ago

Thanks be to open-source https://huggingface.co/models?sort=trending&search=thebloke%2FLlama-2 https://huggingface.co/models?sort=trending&search=thebloke%... Has the quantized weights, available to download now. I tried out the Llama-2-7B-GPTQ on an A100 hosted at runpod.io. Llama-2 is anecdotally much better at instruction following for langchain compared to Falcon-7b-GPTQ - but worse than GPT-3.5 and much worse than GPT-4. Specifically, the Llama-2 model is actually capable of using langchain without hitting parse errors. Something that Falcon wasn't capable of. Would love to hear folks inference setups, the A100 was... not fast - but I didn't spend any time trying to make it fast.

LoganDark 3y ago
> Would love to hear folks inference setups, the A100 was... not fast - but I didn't spend any time trying to make it fast. What do you mean? I get something like 25 tokens per second on an RTX 3060 12G. Try using quantized weights, the full-size ones are only for training.
- lumost 3y ago
  Aye was on quantized weights using gptq.
  - LoganDark 3y ago
    Try GGML, llama.cpp is pretty fast
    - lumost 3y ago
      makes sense - I ultimately need to train the weights so was focusing on GPTQ, I'll try out ggml and see if the latency is better. I have some flexibility on whether I run inference/training on the same model instance. What context length were you using? I was maxing ~2048 tokens, which may also explain the apparent latency.
      - LoganDark 3y ago
        llama.cpp builds a prefix cache so the only latency is on the first generation :)