3 ms·
Its quite fast on x86. llama.cpp looks really good on Mac ARM CPUs because: - they have tons of memory bandwidth - they have a really good proprietary accele
by brucethemoose2 3y ago
Its quite fast on x86.
llama.cpp looks really good on Mac ARM CPUs because:
- they have tons of memory bandwidth
- they have a really good proprietary acceleration library (accelerate)
But I don't think it would be so fast on, say, an Ampere Altra compared to a similarly priced EPYC cpu.
- summarity 3y agoAmpere has native acceleration (Ampere Optimised Frameworks) for AI workloads
- brucethemoose2 3y agoI dont think llama.cpp integrates this though. Accelerate is all but depreciated because of the Metal backend anyway.
- api 3y agoThese things are quite memory bandwidth bound, so if you are getting any kind of machine to run them go for the absolute fastest RAM you can.
- deleted 3y ago[deleted]