3 ms·
Right, especially when it's not clear if such a machine will be needed a year or two from now to run a model with the equivalent performance.
by natsucks 3y ago
Right, especially when it's not clear if such a machine will be needed a year or two from now to run a model with the equivalent performance.
- hereonout2 3y agoAt this rate it may not be needed in a few months to run a model of equivalent performance. At work I've been successfully running llama2 variants on Mac M2 systems with more than adequate throughput for small experiments and examples. Can also boot a 4x Nvidia T4 instance on AWS (g4dn.12xlarge) to larger models. I feel you'd need to be doing inference on a 24/7 basis for at least a year to justify the price of this machine.