What's the point of running unquantized models? If you want to run 70b knock yourself out, but you're wasting electricity and life-hours by using 16-bit float a

What's the point of running unquantized models? If you want to run 70b knock yourself out, but you're wasting electricity and life-hours by using 16-bit float at full precision. Even if you could accelerate such a model well, it's not going to perform any better than a decently trained finetune. The line seems pretty clearly drawn, to me. Datacenter customers flat-out get better treatment, and the gaming segment is going to have to cope with that. Mellanox networking is not coming to gaming hardware, and Grace CPUs are probably not going to get crammed on an ATX motherboard. They aren't the same market, full stop. Maybe Nvidia will have something for you when their desktop chips launch.