4 ms·
Dumb question: does 175B parameters mean the number of bytes (or floats?) in the model? Does that also mean you need the whole model in memory to do inference
by timmg 4y ago
Dumb question: does 175B parameters mean the number of bytes (or floats?) in the model? Does that also mean you need the whole model in memory to do inference (in practice)?
If so, not many machines have that much RAM. Makes it hard to "play" with.
- lostmsu 4y agofloat16s or bfloat16s so 2x of that for storage You can infer using DeepSpeed.