4 ms·
I would guess that's the minimum needed to run an 'instance' of the model for inference not sure how many concurrent users that could support, but I'd imagine
by anentropic 4y ago
I would guess that's the minimum needed to run an 'instance' of the model for inference
not sure how many concurrent users that could support, but I'd imagine the public service has a whole bunch of these racks
from what I read... GPT-J at 6B params is too big to load on any consumer GPU (needs ~25GB)
and then GPT-3 is 175B params or ~29x larger, so maybe needing 7-800GB RAM
8x 80GB A100s would give 640GB RAM, so its approx the right ballpark
- erichocean 4y ago> GPT-J at 6B params is too big to load on any consumer GPU (needs ~25GB) You can load that on M1/M2 Macs, which have unified GPU memory up to 128GB.