3 ms·
There is no elaboration of your statement there. It says it is hard to reproduce results on single 16GB GPU, but nor they used single TPU in their result. They
by riku_iki 7y ago
There is no elaboration of your statement there.
It says it is hard to reproduce results on single 16GB GPU, but nor they used single TPU in their result.
They explicitly state that large number of GPU required: "Therefore, a large number (ranging from 32 to 128, equal to batch_size) of GPUs are required to reproduce many results in the paper.", and they used around 200 TPUs themself according to their publication.
- m0zg 7y agoYes, it boils down to using either a single machine with multiple TPUs (and therefore doing things "the easy way" and relatively quickly) or having to use 128 GPUs (up to 8 per machine) and working with a single sample per GPU, really, really slowly. Given that a single model often requires dozens, if not hundreds of training runs to figure out hyperparameters that achieve a SOTA result, this means that with TPU you can do this, and with GPU you aren't even going to bother training something this big because it will take forever, and someone with a TPU will figure out something better by the time you're done. Which is what happened in this case, it looks like.