3 ms·
Does this imply that to tune 11b params your basement needs to have 16 machines with 4 GPUs each (64 GPUs in all)?
by zhanxw 11y ago
Does this imply that to tune 11b params your basement needs to have 16 machines with 4 GPUs each (64 GPUs in all)?
- m-i-l 11y agoSlides 28 and 35 suggest it was 3 servers, each with 4 GPUs, i.e. 12 GPUs total. If that's the case, then you can probably build a Google scale network (1 billion parameter, 9 layer neural network, which needed 1,000 machines and 16,000 cores running for a week in 2012) at home for around £4K (US$6.2K).