4 ms·
I bought a used dual-socket xeon workstation with 768GB of RAM for ~$3k and can run the 405B model at ~0.3 tokens/sec.
by mechagodzilla 2y ago
I bought a used dual-socket xeon workstation with 768GB of RAM for ~$3k and can run the 405B model at ~0.3 tokens/sec.