5 ms·

What's the best way to download and get setup with this stuff atm? Ie lets say i want to run currently available variations of LLaMA -- 7B, 13B, and 30B [1] --

What's the best way to download and get setup with this stuff atm? Ie lets say i want to run currently available variations of LLaMA -- 7B, 13B, and 30B [1] -- is there a current summary of how to acquire them, possibly quantize them, etc? Would i download a quantized version or do it myself? etc I ran Alpaca 7B Q4 almost instantly because they provided Curl's to download it. Super simple. But it seems most aren't doing that because it's prone to getting Facebook's gaze. So.. what's recommended? I happened to find this[2], but i think that's the non-quantized raw models? Not sure yet. [1]: Won't bother with 65B, can't fit in memory i believe? [2]: https://github.com/shawwn/llama-dl/blob/main/llama.sh https://github.com/shawwn/llama-dl/blob/main/llama.sh edit: I forgot about https://github.com/cocktailpeanut/dalai https://github.com/cocktailpeanut/dalai - i suspect this is best in breed atm? Though a Docker container would be nice to wrangle all the dependencies

NameError 4y ago
The shell script you linked is indeed the raw weights, but if you have the storage space you can download them and use the llama-cpp repo posted here a few days ago (https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp) to compress them, the 'Usage' section of the readme worked for me. I'm using a not-that-new Macbook Pro (intel, 16GB memory) and was able to run the 7B and 13B models that way, tried 30B but it seemed to hang.
- unshavedyak 4y ago
  I'm in the process of using a PR'd Dockerfile for Dalai, which i think will handle everything hands free. Ie download the raw files, quantize them, pick which you want (7, 13, etc). We'll see how it goes. A bit heavy handed perhaps to use NPM/etc, but the Dockerfile really helps me ignore all the dependencies i'm adding hah.
  - zamnos 4y ago
    I might be being dense, but why not distribute the quantized files?
    - turbo_fart 4y ago
      With alpaca you get just the quantized file, for llama you need to do it all yourself. Why? No idea
volodia 4y ago
Try this: it installs with a simple python command if you have an NVIDIA GPU: https://github.com/kuleshov/minillm https://github.com/kuleshov/minillm

NameError 4y ago

The shell script you linked is indeed the raw weights, but if you have the storage space you can download them and use the llama-cpp repo posted here a few days ago (https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp) to compress them, the 'Usage' section of the readme worked for me. I'm using a not-that-new Macbook Pro (intel, 16GB memory) and was able to run the 7B and 13B models that way, tried 30B but it seemed to hang.

unshavedyak 4y ago
I'm in the process of using a PR'd Dockerfile for Dalai, which i think will handle everything hands free. Ie download the raw files, quantize them, pick which you want (7, 13, etc). We'll see how it goes. A bit heavy handed perhaps to use NPM/etc, but the Dockerfile really helps me ignore all the dependencies i'm adding hah.
- zamnos 4y ago
  I might be being dense, but why not distribute the quantized files?
  - turbo_fart 4y ago
    With alpaca you get just the quantized file, for llama you need to do it all yourself. Why? No idea

volodia 4y ago

Try this: it installs with a simple python command if you have an NVIDIA GPU: https://github.com/kuleshov/minillm https://github.com/kuleshov/minillm