Hmm? Section 4 has the details of their setup (below). It’s 32 V100s, or just under $100/hr including reading the data out of BQ. They don’t even mention how lo

Hmm? Section 4 has the details of their setup (below). It’s 32 V100s, or just under $100/hr including reading the data out of BQ. They don’t even mention how long they trained for, because presumably it wasn’t material (if it were, they’d have probably used more than 32 GPUs so they wouldn’t have to wait). The details section: > 4.1 Training details We use a transformer with 6 layers, 8 attention heads, and set the dimensionality of the model to 1024. We use a single encoder and a single decoder for all programming languages. During XLM pretraining, we alternate between batches of C++, Java, and Python, composed of 32 sequences of source code of 512 tokens. At training time, we alternate between the denoising auto-encoding and back-translation objectives, and use batches of around 6000 tokens. We optimize TransCoder with the Adam optimizer [25], a learning rate of 10−4, and use the same learning rate scheduler as Vaswani et al. [45]. We implement our models in PyTorch [39] and train them on 32 V100 GPUs. We use float16 operations to speed up training and to reduce the memory usage of our models. > 4.2 Training data We download the GitHub public dataset available on Google BigQuery . It contains more than 2.8 million open source GitHub repositories. We filter projects whose license explicitly permits the re-distribution of parts of the project, and select the C++, Java, and Python files within those projects. Ideally, a transcompiler should be able to translate whole projects. In this work, we decide to translate at function level.