3 ms·
The model is huge and is currently run in the cloud on many machines.
by kordlessagain 6y ago
The model is huge and is currently run in the cloud on many machines.
- mortehu 6y agoIt's only 175 billion parameters, so presumably it can fit on a single computer with 1024 GB RAM.
- Voloskaya 6y agoOn CPU the latency would be absolutly prohibitive to the point of being useless.
- typon 6y agoFor training yes, but not for inference.
- leogao 6y agoThe inference latency would also be prohibitive.
- Voloskaya 6y agoInference on GPU is already very slow on the full-scale non-distilled model (in the 1-2 sec range iirc), on CPU it would be an order of magnitude more.
- kordlessagain 6y agoFrom 2019: https://heartbeat.fritz.ai/deep-learning-has-a-size-problem-ea601304cd8 https://heartbeat.fritz.ai/deep-learning-has-a-size-problem-... > Earlier this year, researchers at NVIDIA announced MegatronLM, a massive transformer model with 8.3 billion parameters (24 times larger than BERT) > The parameters alone weigh in at just over 33 GB on disk. Training the final model took 512 V100 GPUs running continuously for 9.2 days. Running this model on a "regular" machine at some useful rate is probably not possible at this time.
- woko 6y agoSorry, but I don't see the link between the quote and your sentence.
- stingraycharles 6y agoWouldn't you need this model to be in GPU RAM instead of regular RAM, though?