3 ms·
exactly. i have a hard enough time running a 70b model at fp16. no way i attempt 180 without a significant performance gain.
by natsucks 3y ago
exactly. i have a hard enough time running a 70b model at fp16. no way i attempt 180 without a significant performance gain.
- cypress66 3y ago8bit is virtually lossless. Not much point in running fp16