4 ms·
Yes on arm. Sorry I meant the cpu (x86_64) not a cpu. :-D
by Roark66 3y ago
Yes on arm. Sorry I meant the cpu (x86_64) not a cpu. :-D
- brucethemoose2 3y agoIts quite fast on x86. llama.cpp looks really good on Mac ARM CPUs because: - they have tons of memory bandwidth - they have a really good proprietary acceleration library (accelerate) But I don't think it would be so fast on, say, an Ampere Altra compared to a similarly priced EPYC cpu.
- summarity 3y agoAmpere has native acceleration (Ampere Optimised Frameworks) for AI workloads
- brucethemoose2 3y agoI dont think llama.cpp integrates this though. Accelerate is all but depreciated because of the Metal backend anyway.
- api 3y agoThese things are quite memory bandwidth bound, so if you are getting any kind of machine to run them go for the absolute fastest RAM you can.
- deleted 3y ago[deleted]
- gliptic 3y agoI told you a while ago that llama.cpp supports 4/8-bit (and other) quantization just fine (and fast) on x86_64.
- Roark66 3y agoYes, you did. Back then I went to their github page and I found no mention of x86. Now I see it is there. Great. I'll give it a try. Also I see something much better. The same guy behind llama.cpp authors a universal ml library that does cpu quantization. I wasn't aware of that and it is far more impressive than running a couple of chosen models.