3 ms·
To run a 7b model? Not much. To run the 405b model mentioned, at interactive speeds, at reasonable quantisation? A lot.
by taneq 2y ago
To run a 7b model? Not much. To run the 405b model mentioned, at interactive speeds, at reasonable quantisation? A lot.