7 ms·

Load LLaMA Models Instantly

208 points by polyrand 4y ago

superkuh 4y ago
Woo! This sounds like it will make it easier to run in normal mode (ie, not interactive) and manage the chat history yourself if there's less penalty for a full program reload. Currently my perl irc bot wrapper for llama.cpp just open2's the program in interactive mode (-i) and reads/writes to stdout/stdin of llama.cpp to get the loading time savings of having it manage history and keep state. In one-shot there'd still be the "extra" inference time of processing the full history each time instead of saving state like interactive does but the memory load time matters just as much. For me personally this matters most because right now when llama.cpp runs out of the 2048 tokens it segfaults and this causes difficulties. In interactive mode if it goes off rails and generates 1000 tokens of nonsense then that nonsense is taking up tokens for the next line from chat. In normal mode where it just runs once and all history has to be manually supplied this can be avoided.
- jart 4y ago
  Indeed. I'm working on persuading the author of this project to make it shell scriptable too. The default behavior of the command should only print to stdout the response to a prompt.
dougmwne 4y ago
Can someone break this down? Since this seems to be inferencing without having the entire model loaded into memory, is it possible this could be a way to relax memory requirements of the 65b model?
- layer8 4y ago
  I don’t think so: > the gains here are mostly due to not copying memory anymore, and better cooperation with the kernel's page manager. We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the model file needs to be loaded. What this means is that first runs that load from spinning disk are still going to be slow, even though the average case has greatly improved. If I understand correctly, this only provides a speed up when the model is already in the OS’s file system cache, and in any case you still have to load the entire model into memory.
  - jart 4y ago
    Author here. It does speed things up. It's just that a 2 second improvement doesn't mean much if the disk reads take 60 seconds. When it comes to disk though there's something far more important happening here. When you read() or memcpy() loaded memory from a 12GB file, then you need at least 24GB of RAM, because the copied memory pages are competing with the kernel's file cache pages. Once you go over that, the kernel is going to delete the file cache, thereby ensuring you have to do a full disk read every time. Using mmap() ensures the kernel knows they're both the same thing. That means you should be able to run models that are 2x larger than before, without compromising system stability.
    - ithkuil 4y ago
      Would reading without the cache (e.g. O_DIRECT) can be used to achieve a similar effect?
      - jart 4y ago
        Well I think that would guarantee your always hit disk for sure. The goal here is to be able to run the llama.cpp command repeatedly, possibly in shell scripts, and rely on the file caches being available so that the command loads instantly (we're talking like 1 second rather than 60 seconds to run, because disk is 100x slower).
      - dekhn 4y ago
        O_DIRECT is absurdly slow in my my experience.
        the8472 4y ago
        O_DIRECT is basically telling the OS to load a block of data straight from disk into your process memory, skipping all the caching, sharing, scheduling, coalescing and readaheads the OS normally does for you. Which means you now either must do all that yourself in userspace or you can use io_uring to have it happen in the background and get notified when it's done. But if the intent is to have multiple processes share the samedata then mmap likely is the better choice.
- adultSwim 4y ago
  "We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the model file needs to be loaded. What this means is that first runs that load from spinning disk are still going to be slow, even though the average case has greatly improved" https://github.com/ggerganov/llama.cpp/issues/91#issuecomment-1473271638 https://github.com/ggerganov/llama.cpp/issues/91#issuecommen...
  - jart 4y ago
    You still get the gain of zero loading time. The issue with loading a model is it ensures 100% of the file needs to be copied into memory. This change means we don't have to copy at all. The file cache pages can be made directly accessible to the matmul ops. You could have ten of these processes running at once, and they'd all share the same memory. However those pages still have to be pulled into memory. What I was hoping for, when I said that, is it's possible in some cases for mmap() to make things even faster than this change managed to accomplish. Some scientific computing applications use sparse datasets. If you skip loading, and use mmap(), then the system will only load pages as-needed on a 4096-byte basis. If the data were actually sparsely used, then with mmap() you could for instance load a 1TB file into memory on a system with 32GB of RAM, and no file cache, only touch a small portion, and it'll go fast and barely use any memory at all. That's not the case here sadly. It's still however a big improvement.
dekhn 4y ago
There are some cool ideas in here. I've long been curious why people don't use mmap to re-use all those wonderful pages that got loaded (without reparsing the disk data).
- liuliu 4y ago
  That's actually the point of safetensors format, although it needs to take care of alignment. I actually use this technique to mmap directly to GPU buffer on Apple systems (although it seems to segfault on iOS 15.x systems, only supported on 16.x and above).
- arthurcolle 4y ago
  Can you explain how this is possible to do? Sorry I haven't gotten this low level with any of these models before and I'd really appreciate how to understand this.
  - delusional 4y ago
    This has nothing to do with the models, just standard *nix stuff. If you mmap the file readonly the pages can be shared by multiple processes without duplication since they are guaranteed to be the same.
  - dekhn 4y ago
    To run inference, you need to load a model from disk into RAM. Usually the model is written in a disk format that is convenient but needs to be parsed at runtime into a RAM-resident C application data structure. In this case, it looks like jart@ modified malloc to capture the memory generated by the loading process and serialized that to disk. When you run, the application calls mmap to make a virtual memory association with the bytes on disk- so any time you access RAM and it's not yet loaded, ti gets loaded from disk. At that point it gets saved by the kernel in a page cache and since the files on disk don't change, those pages can stay in memory longer than the process. So when the process restarts, all those RAM requests are immediately mapped to already-cached virtual memory, rather than reading from disk. The inference library here supports a data pointer that would point to the memory mapped location. This is faster than relying on the kernel's disk read cache; in that case, you'd still need to convert the data from the disk format to the in-memory format. Normally the data build process is run as an external program that writes the mmap-ready structure to disk (an example is the BLAST program which writes the DNA sequence data into an index structure that is mmapped at runtime). But in this case ti looks like using an instrumented malloc() helps simplify the process of building the disk structure.
eternalban 4y ago
If you want to avoid twitter this discusses the changes: https://github.com/ggerganov/llama.cpp/issues/91 https://github.com/ggerganov/llama.cpp/issues/91
0cf8612b2e1e 4y ago
How does jart find the time? Just a brilliant engineer who is seemingly cranking out amazing projects on a regular basis.
- ipaddr 4y ago
  Works at facebook and can't find work?
- Cyph0n 4y ago
  A 100x developer if I’ve ever seen one.
- jart 4y ago
  You can thank Mozilla's MIECO program, my GitHub sponsors, and my Patreon supporters. See my recent blog post: https://justine.lol/rusage/#funding https://justine.lol/rusage/#funding It's thanks to them that I'm able to keep doing what I'm doing.
  - deleted 4y ago
    [deleted]
meghan_rain 4y ago
I welcome all progress but I don't see why these models aren't simply run on a thin Python server that loads the model into memory once and then you can curl it instantly whenever you want?
- mungoman2 4y ago
  Yes tbh this is the right answer
- kkielhofner 4y ago
  Because it’s a waste for anything other than proof of concept/handful of users. It’s really simple to take some python inference code and wrap FastAPI around it. However, inference servers exist for a reason. You’ll quickly find that performance, VRAM usage, model management, etc isn’t practical with the FastAPI approach. Speaking personally inference server implementations like Nvidia Triton bring a model to performance metrics that are absolutely night and day vs the FastAPI approach - in many cases orders of magnitude higher performance in terms of response time and requests per second.
  - sankha93 4y ago
    Can you list the concrete problems a FastAPI approach will have, and what tools like Nvidia Triton do differently to get around it? I have no idea about running such models at scale.
    - woodson 4y ago
      Not GP, but what NVidia Triton can do includes - Dynamic batching while limiting latency to a set threshold - Running multiple instances of a model, effectively load-balancing inference requests. - Loading/unloading/running multiple versions of models dynamically, which is useful if you want to update (or roll back) your model while not interfering with existing inference requests. Its client provides async based inference APIs, so you can easily put a FastAPI-based API server in front and don't necessarily need a queue (like Celery).
    - 4y ago
kir-gadjello 4y ago
This is cool, but SSD read bandwidth is still the bottleneck. On my non-mac machine it still takes several seconds to load the model.
- Tepix 4y ago
  This assumes (and makes it possible) that it's already in the kernel cache
lxe 4y ago
I wonder if something like this is possible for CUDA/pytorch loading?
- jart 4y ago
  I don't know about Cuda, but with PyTorch it's quite possible. The solution should generalize to any code that loads into memory created by malloc(). You'd obviously be capturing a lot of superfluous Python objects in your mappable model file, but given the size of these things it should be comparatively minor.
  - liuliu 4y ago
    PyTorch saves tensor data and metadata separately (metadata in Pickle format a.k.a. pickle VM, which is famously not mmap friendly because it builds up Python object through the VM execution). These then zipped together in a zip file. Whether you can mmap depends on two things: if the zip file has compression enabled, you are out of luck. If not, CUDA's unified memory requires I think 64-byte alignment, and that may at odds with the zip archive, otherwise CPU depending on optimization may require some other alignment too (typically 16 bytes or 4 bytes).
    - jart 4y ago
      The same was true with llama.cpp. The trick I used was basically to identify which part of the program is performing the transformation, where the unwieldy file from disk is being turned into the aligned data structures that actually get used. In the llama.cpp codebase, that function was called llama_model_load(). So I basically just wrapped that code in a magic transaction: main() { magic_init(); // replaces malloc() with ./magic.dat file if (magic->was_committed) { model = magic->model; // use the heap built from last run } else { // create a new heap model = new model; llama_model_load(model); // pre-existing model loading code magic->model = model; magic_commit(); // mark memory transaction complete } // rest of pre-existing code } You could write a Python extension for this code https://github.com/ggerganov/llama.cpp/commit/5b8023d935401072b73b63ea995aaae040d57b87 https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010... and see if it works for your use case.
turnsout 4y ago
"It's a ~200 LOC change that only took me a few hours and it worked the first time." jart is on another plane of existence—and 100% earned this flex
- MacsHeadroom 4y ago
  I've been producing about 300 LOC/hr with the help of GPT-4. Generated code practically always runs correctly the first time. I'm not a particularly proficient and before this I would probably be happy with averaging 30 LOC/hr.
  - turnsout 4y ago
    It's all about quality, not quantity—call me when you can do what jart does with a LLM. Actually, don't call me, I'll be in a bunker by then
- mhh__ 4y ago
  For knowing mmap exists (i.e. soft page faults are intensely useful explicitly for this kind of thing)? Or am I reading this wrong?
  - v64 4y ago
    Due to nuances of how data is split across the various model files, the implementation involved some non-trivial steps in order for everything to be allocated correctly. The changes are here [1]. [1] https://github.com/ggerganov/llama.cpp/commit/5b8023d935401072b73b63ea995aaae040d57b87 https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010...
adultSwim 4y ago
I'm seeing a lot of interest in generative use cases. Has anyone tried LLaMA or GPT for classification?
- inawarminister 4y ago
  FWIW I was using ChatGPT with few-shot prompting to classify topics, extracting keywords, and sentiment analysis. Perhaps it might be interesting to make a comparison between Alpaca-7B and ChatGPT, hmm.
unshavedyak 4y ago
What's the best way to download and get setup with this stuff atm? Ie lets say i want to run currently available variations of LLaMA -- 7B, 13B, and 30B [1] -- is there a current summary of how to acquire them, possibly quantize them, etc? Would i download a quantized version or do it myself? etc I ran Alpaca 7B Q4 almost instantly because they provided Curl's to download it. Super simple. But it seems most aren't doing that because it's prone to getting Facebook's gaze. So.. what's recommended? I happened to find this[2], but i think that's the non-quantized raw models? Not sure yet. [1]: Won't bother with 65B, can't fit in memory i believe? [2]: https://github.com/shawwn/llama-dl/blob/main/llama.sh https://github.com/shawwn/llama-dl/blob/main/llama.sh edit: I forgot about https://github.com/cocktailpeanut/dalai https://github.com/cocktailpeanut/dalai - i suspect this is best in breed atm? Though a Docker container would be nice to wrangle all the dependencies
- NameError 4y ago
  The shell script you linked is indeed the raw weights, but if you have the storage space you can download them and use the llama-cpp repo posted here a few days ago (https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp) to compress them, the 'Usage' section of the readme worked for me. I'm using a not-that-new Macbook Pro (intel, 16GB memory) and was able to run the 7B and 13B models that way, tried 30B but it seemed to hang.
  - unshavedyak 4y ago
    I'm in the process of using a PR'd Dockerfile for Dalai, which i think will handle everything hands free. Ie download the raw files, quantize them, pick which you want (7, 13, etc). We'll see how it goes. A bit heavy handed perhaps to use NPM/etc, but the Dockerfile really helps me ignore all the dependencies i'm adding hah.
    - zamnos 4y ago
      I might be being dense, but why not distribute the quantized files?

208 points by polyrand 4y ago

Woo! This sounds like it will make it easier to run in normal mode (ie, not interactive) and manage the chat history yourself if there's less penalty for a full program reload. Currently my perl irc bot wrapper for llama.cpp just open2's the program in interactive mode (-i) and reads/writes to stdout/stdin of llama.cpp to get the loading time savings of having it manage history and keep state. In one-shot there'd still be the "extra" inference time of processing the full history each time instead of saving state like interactive does but the memory load time matters just as much. For me personally this matters most because right now when llama.cpp runs out of the 2048 tokens it segfaults and this causes difficulties. In interactive mode if it goes off rails and generates 1000 tokens of nonsense then that nonsense is taking up tokens for the next line from chat. In normal mode where it just runs once and all history has to be manually supplied this can be avoided.

jart 4y ago
Indeed. I'm working on persuading the author of this project to make it shell scriptable too. The default behavior of the command should only print to stdout the response to a prompt.

dougmwne 4y ago

Can someone break this down? Since this seems to be inferencing without having the entire model loaded into memory, is it possible this could be a way to relax memory requirements of the 65b model?

layer8 4y ago
I don’t think so: > the gains here are mostly due to not copying memory anymore, and better cooperation with the kernel's page manager. We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the model file needs to be loaded. What this means is that first runs that load from spinning disk are still going to be slow, even though the average case has greatly improved. If I understand correctly, this only provides a speed up when the model is already in the OS’s file system cache, and in any case you still have to load the entire model into memory.
- jart 4y ago
  Author here. It does speed things up. It's just that a 2 second improvement doesn't mean much if the disk reads take 60 seconds. When it comes to disk though there's something far more important happening here. When you read() or memcpy() loaded memory from a 12GB file, then you need at least 24GB of RAM, because the copied memory pages are competing with the kernel's file cache pages. Once you go over that, the kernel is going to delete the file cache, thereby ensuring you have to do a full disk read every time. Using mmap() ensures the kernel knows they're both the same thing. That means you should be able to run models that are 2x larger than before, without compromising system stability.
  - ithkuil 4y ago
    Would reading without the cache (e.g. O_DIRECT) can be used to achieve a similar effect?
    - jart 4y ago
      Well I think that would guarantee your always hit disk for sure. The goal here is to be able to run the llama.cpp command repeatedly, possibly in shell scripts, and rely on the file caches being available so that the command loads instantly (we're talking like 1 second rather than 60 seconds to run, because disk is 100x slower).
    - dekhn 4y ago
      O_DIRECT is absurdly slow in my my experience.
      - the8472 4y ago
        O_DIRECT is basically telling the OS to load a block of data straight from disk into your process memory, skipping all the caching, sharing, scheduling, coalescing and readaheads the OS normally does for you. Which means you now either must do all that yourself in userspace or you can use io_uring to have it happen in the background and get notified when it's done. But if the intent is to have multiple processes share the samedata then mmap likely is the better choice.
adultSwim 4y ago
"We unfortunately aren't getting any additional gains from lazy page loading, since this is a dense model. To generate a single token, every single page in the model file needs to be loaded. What this means is that first runs that load from spinning disk are still going to be slow, even though the average case has greatly improved" https://github.com/ggerganov/llama.cpp/issues/91#issuecomment-1473271638 https://github.com/ggerganov/llama.cpp/issues/91#issuecommen...
- jart 4y ago
  You still get the gain of zero loading time. The issue with loading a model is it ensures 100% of the file needs to be copied into memory. This change means we don't have to copy at all. The file cache pages can be made directly accessible to the matmul ops. You could have ten of these processes running at once, and they'd all share the same memory. However those pages still have to be pulled into memory. What I was hoping for, when I said that, is it's possible in some cases for mmap() to make things even faster than this change managed to accomplish. Some scientific computing applications use sparse datasets. If you skip loading, and use mmap(), then the system will only load pages as-needed on a 4096-byte basis. If the data were actually sparsely used, then with mmap() you could for instance load a 1TB file into memory on a system with 32GB of RAM, and no file cache, only touch a small portion, and it'll go fast and barely use any memory at all. That's not the case here sadly. It's still however a big improvement.

dekhn 4y ago

There are some cool ideas in here. I've long been curious why people don't use mmap to re-use all those wonderful pages that got loaded (without reparsing the disk data).

liuliu 4y ago
That's actually the point of safetensors format, although it needs to take care of alignment. I actually use this technique to mmap directly to GPU buffer on Apple systems (although it seems to segfault on iOS 15.x systems, only supported on 16.x and above).
arthurcolle 4y ago
Can you explain how this is possible to do? Sorry I haven't gotten this low level with any of these models before and I'd really appreciate how to understand this.
- delusional 4y ago
  This has nothing to do with the models, just standard *nix stuff. If you mmap the file readonly the pages can be shared by multiple processes without duplication since they are guaranteed to be the same.
- dekhn 4y ago
  To run inference, you need to load a model from disk into RAM. Usually the model is written in a disk format that is convenient but needs to be parsed at runtime into a RAM-resident C application data structure. In this case, it looks like jart@ modified malloc to capture the memory generated by the loading process and serialized that to disk. When you run, the application calls mmap to make a virtual memory association with the bytes on disk- so any time you access RAM and it's not yet loaded, ti gets loaded from disk. At that point it gets saved by the kernel in a page cache and since the files on disk don't change, those pages can stay in memory longer than the process. So when the process restarts, all those RAM requests are immediately mapped to already-cached virtual memory, rather than reading from disk. The inference library here supports a data pointer that would point to the memory mapped location. This is faster than relying on the kernel's disk read cache; in that case, you'd still need to convert the data from the disk format to the in-memory format. Normally the data build process is run as an external program that writes the mmap-ready structure to disk (an example is the BLAST program which writes the DNA sequence data into an index structure that is mmapped at runtime). But in this case ti looks like using an instrumented malloc() helps simplify the process of building the disk structure.

eternalban 4y ago

If you want to avoid twitter this discusses the changes: https://github.com/ggerganov/llama.cpp/issues/91 https://github.com/ggerganov/llama.cpp/issues/91

0cf8612b2e1e 4y ago

How does jart find the time? Just a brilliant engineer who is seemingly cranking out amazing projects on a regular basis.

ipaddr 4y ago
Works at facebook and can't find work?
Cyph0n 4y ago
A 100x developer if I’ve ever seen one.
jart 4y ago
You can thank Mozilla's MIECO program, my GitHub sponsors, and my Patreon supporters. See my recent blog post: https://justine.lol/rusage/#funding https://justine.lol/rusage/#funding It's thanks to them that I'm able to keep doing what I'm doing.
- deleted 4y ago
  [deleted]

meghan_rain 4y ago

I welcome all progress but I don't see why these models aren't simply run on a thin Python server that loads the model into memory once and then you can curl it instantly whenever you want?

mungoman2 4y ago
Yes tbh this is the right answer
kkielhofner 4y ago
Because it’s a waste for anything other than proof of concept/handful of users. It’s really simple to take some python inference code and wrap FastAPI around it. However, inference servers exist for a reason. You’ll quickly find that performance, VRAM usage, model management, etc isn’t practical with the FastAPI approach. Speaking personally inference server implementations like Nvidia Triton bring a model to performance metrics that are absolutely night and day vs the FastAPI approach - in many cases orders of magnitude higher performance in terms of response time and requests per second.
- sankha93 4y ago
  Can you list the concrete problems a FastAPI approach will have, and what tools like Nvidia Triton do differently to get around it? I have no idea about running such models at scale.
  - woodson 4y ago
    Not GP, but what NVidia Triton can do includes - Dynamic batching while limiting latency to a set threshold - Running multiple instances of a model, effectively load-balancing inference requests. - Loading/unloading/running multiple versions of models dynamically, which is useful if you want to update (or roll back) your model while not interfering with existing inference requests. Its client provides async based inference APIs, so you can easily put a FastAPI-based API server in front and don't necessarily need a queue (like Celery).
  - 4y ago

kir-gadjello 4y ago

This is cool, but SSD read bandwidth is still the bottleneck. On my non-mac machine it still takes several seconds to load the model.

Tepix 4y ago
This assumes (and makes it possible) that it's already in the kernel cache

lxe 4y ago

I wonder if something like this is possible for CUDA/pytorch loading?

jart 4y ago
I don't know about Cuda, but with PyTorch it's quite possible. The solution should generalize to any code that loads into memory created by malloc(). You'd obviously be capturing a lot of superfluous Python objects in your mappable model file, but given the size of these things it should be comparatively minor.
- liuliu 4y ago
  PyTorch saves tensor data and metadata separately (metadata in Pickle format a.k.a. pickle VM, which is famously not mmap friendly because it builds up Python object through the VM execution). These then zipped together in a zip file. Whether you can mmap depends on two things: if the zip file has compression enabled, you are out of luck. If not, CUDA's unified memory requires I think 64-byte alignment, and that may at odds with the zip archive, otherwise CPU depending on optimization may require some other alignment too (typically 16 bytes or 4 bytes).
  - jart 4y ago
    The same was true with llama.cpp. The trick I used was basically to identify which part of the program is performing the transformation, where the unwieldy file from disk is being turned into the aligned data structures that actually get used. In the llama.cpp codebase, that function was called llama_model_load(). So I basically just wrapped that code in a magic transaction: main() { magic_init(); // replaces malloc() with ./magic.dat file if (magic->was_committed) { model = magic->model; // use the heap built from last run } else { // create a new heap model = new model; llama_model_load(model); // pre-existing model loading code magic->model = model; magic_commit(); // mark memory transaction complete } // rest of pre-existing code } You could write a Python extension for this code https://github.com/ggerganov/llama.cpp/commit/5b8023d935401072b73b63ea995aaae040d57b87 https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010... and see if it works for your use case.

turnsout 4y ago

"It's a ~200 LOC change that only took me a few hours and it worked the first time." jart is on another plane of existence—and 100% earned this flex

MacsHeadroom 4y ago
I've been producing about 300 LOC/hr with the help of GPT-4. Generated code practically always runs correctly the first time. I'm not a particularly proficient and before this I would probably be happy with averaging 30 LOC/hr.
- turnsout 4y ago
  It's all about quality, not quantity—call me when you can do what jart does with a LLM. Actually, don't call me, I'll be in a bunker by then
mhh__ 4y ago
For knowing mmap exists (i.e. soft page faults are intensely useful explicitly for this kind of thing)? Or am I reading this wrong?
- v64 4y ago
  Due to nuances of how data is split across the various model files, the implementation involved some non-trivial steps in order for everything to be allocated correctly. The changes are here [1]. [1] https://github.com/ggerganov/llama.cpp/commit/5b8023d935401072b73b63ea995aaae040d57b87 https://github.com/ggerganov/llama.cpp/commit/5b8023d9354010...

adultSwim 4y ago

I'm seeing a lot of interest in generative use cases. Has anyone tried LLaMA or GPT for classification?

inawarminister 4y ago
FWIW I was using ChatGPT with few-shot prompting to classify topics, extracting keywords, and sentiment analysis. Perhaps it might be interesting to make a comparison between Alpaca-7B and ChatGPT, hmm.

unshavedyak 4y ago

What's the best way to download and get setup with this stuff atm? Ie lets say i want to run currently available variations of LLaMA -- 7B, 13B, and 30B [1] -- is there a current summary of how to acquire them, possibly quantize them, etc? Would i download a quantized version or do it myself? etc I ran Alpaca 7B Q4 almost instantly because they provided Curl's to download it. Super simple. But it seems most aren't doing that because it's prone to getting Facebook's gaze. So.. what's recommended? I happened to find this[2], but i think that's the non-quantized raw models? Not sure yet. [1]: Won't bother with 65B, can't fit in memory i believe? [2]: https://github.com/shawwn/llama-dl/blob/main/llama.sh https://github.com/shawwn/llama-dl/blob/main/llama.sh edit: I forgot about https://github.com/cocktailpeanut/dalai https://github.com/cocktailpeanut/dalai - i suspect this is best in breed atm? Though a Docker container would be nice to wrangle all the dependencies

NameError 4y ago
The shell script you linked is indeed the raw weights, but if you have the storage space you can download them and use the llama-cpp repo posted here a few days ago (https://github.com/ggerganov/llama.cpp https://github.com/ggerganov/llama.cpp) to compress them, the 'Usage' section of the readme worked for me. I'm using a not-that-new Macbook Pro (intel, 16GB memory) and was able to run the 7B and 13B models that way, tried 30B but it seemed to hang.
- unshavedyak 4y ago
  I'm in the process of using a PR'd Dockerfile for Dalai, which i think will handle everything hands free. Ie download the raw files, quantize them, pick which you want (7, 13, etc). We'll see how it goes. A bit heavy handed perhaps to use NPM/etc, but the Dockerfile really helps me ignore all the dependencies i'm adding hah.
  - zamnos 4y ago
    I might be being dense, but why not distribute the quantized files?