13 ms·
Same, performance of the quantised 30B model on my m1 16GB air is absolutely terrible. A couple of things I noticed on activity monitor: 1. "memory used" + "cac
by freehorse 4y ago
Same, performance of the quantised 30B model on my m1 16GB air is absolutely terrible. A couple of things I noticed on activity monitor:
1. "memory used" + "cached files" == 16GB (while swap is zero)
2. Disk reading is 500-600MB/s
3. it seems that every token is computed exactly _after every ~20GB read from disk_ which actually points that for calculating each token it actually re-reads the weights file again (instead of caching it). I actually suspect that swapping may have been more efficient.
The last part (3) that it rereads the whole file again is an assumption and it could just be a coincidence that the new token is computed at every ~20GB read from disk, but it makes sense, as I do not think swapping would have been that inefficient.
- muyuu 4y agoCan you share the intermediate files? They're taking ages to process on my 16GB-RAM laptop
- freehorse 4y agoWhich files are you referring to exactly?
- muyuu 4y agoggml-model-f16.bin and ggml-model-q4_0.bin those are the output of convert-pth-to-ggml.py and quantize respectively I had to cancel 30B as I needed to use the computer after some 12 hours, now I have to fix the ext4 filesystem of the drive where I was doing it, fun times for the weekend guess I'll settle for 13B, I was using 7B but the results are pretty lousy compared to GPT4all's Lora, let alone GPT3.5-turbo or better I'll give a shot to quantising 13B, I'm on 16GB of RAM locally
- dekhn 4y agoYeah, the first time I ran the 30B model, it crashed my machine and I had to reinstall from scratch (linux).