7 ms·
On CPU the latency would be absolutly prohibitive to the point of being useless.
by Voloskaya 6y ago
On CPU the latency would be absolutly prohibitive to the point of being useless.
- typon 6y agoFor training yes, but not for inference.
- leogao 6y agoThe inference latency would also be prohibitive.
- Voloskaya 6y agoInference on GPU is already very slow on the full-scale non-distilled model (in the 1-2 sec range iirc), on CPU it would be an order of magnitude more.
- kordlessagain 6y agoFrom 2019: https://heartbeat.fritz.ai/deep-learning-has-a-size-problem-ea601304cd8 https://heartbeat.fritz.ai/deep-learning-has-a-size-problem-... > Earlier this year, researchers at NVIDIA announced MegatronLM, a massive transformer model with 8.3 billion parameters (24 times larger than BERT) > The parameters alone weigh in at just over 33 GB on disk. Training the final model took 512 V100 GPUs running continuously for 9.2 days. Running this model on a "regular" machine at some useful rate is probably not possible at this time.
- woko 6y agoSorry, but I don't see the link between the quote and your sentence.