3 ms·
> The training setup follows the classic data parallelism paradigm. There are 12 replicas running concurrently on separate machines. Every replica updates the s
by visarga 10y ago
> The training setup follows the classic data parallelism paradigm. There are 12 replicas running concurrently on separate machines. Every replica updates the shared parameters asynchronously.
So it's actually 96 GPUs for a model. And they have many languages to train, and probably don't use cheap GPUs, either.
- TulliusCicero 10y agoIf it's just expensive to train the model, I wouldn't expect that to stop Google. They don't have to do the actual training that often, after all. Right?
- visarga 10y agoThey would have to do that for each language (or language pair), but they can afford it. I think the training the network is cheap compared to running it for user queries, at Google scale.