3 ms·

This paper is very interesting, but as with many recent company-funded DNN-related publications, I am missing some technical information here. Publications by

by qayxc 6y ago

This paper is very interesting, but as with many recent company-funded DNN-related publications, I am missing some technical information here. Publications by universities will almost always include details like dataset sizes, hardware used, and durations for pre-processing, training, and inference. As-is, this paper leaves me puzzled as to whether this novel solution is even practical in any way shape or form. If I essentially need the computational resources of a medium-sized data centre to prepare the data, and train the system for adding a new language-pair or fine-tuning an existing one, it'd not be a practical method outside of the Four Horsemen (GAFA) and their closest competitors... I applaud the use of computational accuracy as the primary metric for evaluating the results, though.

zitterbewegung 6y ago
Companies / researchers in general have no strong requirement to show you any artifacts to reproduce their work. There are incentives to not even provide that information at all. What we usually see is that someone else goes through the time to reproduce the results when reproducing it. Or internally from companies you can see projects moving from one company to another like MapReduce (Google) to Yahoo (Hadoop). And then if enough companies don't want to manage the whole project the software gets donated to the Apache Software foundation.
- qayxc 6y ago
  > Companies / researchers in general have no strong requirement to show you any artifacts to reproduce their work. That's quite literally the opposite of what science is all about! If they don't want others to reproduce their results, they might just as well end each paper with "You can take our word for it!" and skip the details altogether... What I'm rather interested in are points of comparison. Performance in terms of a chosen metric is one thing, but research gets more useful if it can easily be reproduced. This is the norm in all other sciences - why not in AI research? If I can see that their approach is 20% better than SOTA, but they require 1M LoC plus 3 weeks of total computation time on a 100 machine cluster with 8 V100 per node, I can safely say - sod it! - use the inferior commercial product instead and add 20% manual effort (since I need to add manual work anyways as the accuracy isn't 100%).
  - deleted 6y ago
    [deleted]
  - zitterbewegung 6y ago
    Yes, it is and I agree with you. I worked in a lab where I had to reproduce others peoples code in Java. I never finished any of those projects. For Example GPT-2 would need around ~$50k to reproduce from scratch. GPT-3 is probably a few orders of magnitude than that. How would anyone reproduce it unless they are a company? I've seen NVIDIA reproduce some results. Also, most of the issues are you don't have the datasets and after the PhD students graduate and the professor gets a job your access of the datasets go away like bitrot.
  - visarga 6y ago
    > If I can see that their approach is 20% better than SOTA, but they require 1M LoC plus 3 weeks of total computation time on a 100 machine cluster with 8 V100 per node, I can safely say - sod it! 8 V100's cost about $20/h, 100 machines for 2 weeks (allowing for a long training time) will cost $638K. This is the salary of three to five engineers for a year. If your model reduces more than that amount of time it is worth it. It's just a matter of how much use you can get out of it. Of course a model can be reused by different teams and companies, so it could easily be worth the price. I expect the number I calculated to be exaggerated for this task, though, you don't need that much compute for this model. GPT-3 cost $1.2M per run and it is the largest model in existence.
boulos 6y ago
Hmm? Section 4 has the details of their setup (below). It’s 32 V100s, or just under $100/hr including reading the data out of BQ. They don’t even mention how long they trained for, because presumably it wasn’t material (if it were, they’d have probably used more than 32 GPUs so they wouldn’t have to wait). The details section: > 4.1 Training details We use a transformer with 6 layers, 8 attention heads, and set the dimensionality of the model to 1024. We use a single encoder and a single decoder for all programming languages. During XLM pretraining, we alternate between batches of C++, Java, and Python, composed of 32 sequences of source code of 512 tokens. At training time, we alternate between the denoising auto-encoding and back-translation objectives, and use batches of around 6000 tokens. We optimize TransCoder with the Adam optimizer [25], a learning rate of 10−4, and use the same learning rate scheduler as Vaswani et al. [45]. We implement our models in PyTorch [39] and train them on 32 V100 GPUs. We use float16 operations to speed up training and to reduce the memory usage of our models. > 4.2 Training data We download the GitHub public dataset available on Google BigQuery . It contains more than 2.8 million open source GitHub repositories. We filter projects whose license explicitly permits the re-distribution of parts of the project, and select the C++, Java, and Python files within those projects. Ideally, a transcompiler should be able to translate whole projects. In this work, we decide to translate at function level.
visarga 6y ago
To add a new language you can simply reuse an old model that was trained on lots of languages and fine-tune it for your needs. The same practice exists for huge natural-language models such as BERT and GPT.

by qayxc 6y ago

zitterbewegung 6y ago

Companies / researchers in general have no strong requirement to show you any artifacts to reproduce their work. There are incentives to not even provide that information at all. What we usually see is that someone else goes through the time to reproduce the results when reproducing it. Or internally from companies you can see projects moving from one company to another like MapReduce (Google) to Yahoo (Hadoop). And then if enough companies don't want to manage the whole project the software gets donated to the Apache Software foundation.

qayxc 6y ago
> Companies / researchers in general have no strong requirement to show you any artifacts to reproduce their work. That's quite literally the opposite of what science is all about! If they don't want others to reproduce their results, they might just as well end each paper with "You can take our word for it!" and skip the details altogether... What I'm rather interested in are points of comparison. Performance in terms of a chosen metric is one thing, but research gets more useful if it can easily be reproduced. This is the norm in all other sciences - why not in AI research? If I can see that their approach is 20% better than SOTA, but they require 1M LoC plus 3 weeks of total computation time on a 100 machine cluster with 8 V100 per node, I can safely say - sod it! - use the inferior commercial product instead and add 20% manual effort (since I need to add manual work anyways as the accuracy isn't 100%).
- deleted 6y ago
  [deleted]
- zitterbewegung 6y ago
  Yes, it is and I agree with you. I worked in a lab where I had to reproduce others peoples code in Java. I never finished any of those projects. For Example GPT-2 would need around ~$50k to reproduce from scratch. GPT-3 is probably a few orders of magnitude than that. How would anyone reproduce it unless they are a company? I've seen NVIDIA reproduce some results. Also, most of the issues are you don't have the datasets and after the PhD students graduate and the professor gets a job your access of the datasets go away like bitrot.
- visarga 6y ago
  > If I can see that their approach is 20% better than SOTA, but they require 1M LoC plus 3 weeks of total computation time on a 100 machine cluster with 8 V100 per node, I can safely say - sod it! 8 V100's cost about $20/h, 100 machines for 2 weeks (allowing for a long training time) will cost $638K. This is the salary of three to five engineers for a year. If your model reduces more than that amount of time it is worth it. It's just a matter of how much use you can get out of it. Of course a model can be reused by different teams and companies, so it could easily be worth the price. I expect the number I calculated to be exaggerated for this task, though, you don't need that much compute for this model. GPT-3 cost $1.2M per run and it is the largest model in existence.

boulos 6y ago

Hmm? Section 4 has the details of their setup (below). It’s 32 V100s, or just under $100/hr including reading the data out of BQ. They don’t even mention how long they trained for, because presumably it wasn’t material (if it were, they’d have probably used more than 32 GPUs so they wouldn’t have to wait). The details section: > 4.1 Training details We use a transformer with 6 layers, 8 attention heads, and set the dimensionality of the model to 1024. We use a single encoder and a single decoder for all programming languages. During XLM pretraining, we alternate between batches of C++, Java, and Python, composed of 32 sequences of source code of 512 tokens. At training time, we alternate between the denoising auto-encoding and back-translation objectives, and use batches of around 6000 tokens. We optimize TransCoder with the Adam optimizer [25], a learning rate of 10−4, and use the same learning rate scheduler as Vaswani et al. [45]. We implement our models in PyTorch [39] and train them on 32 V100 GPUs. We use float16 operations to speed up training and to reduce the memory usage of our models. > 4.2 Training data We download the GitHub public dataset available on Google BigQuery . It contains more than 2.8 million open source GitHub repositories. We filter projects whose license explicitly permits the re-distribution of parts of the project, and select the C++, Java, and Python files within those projects. Ideally, a transcompiler should be able to translate whole projects. In this work, we decide to translate at function level.

visarga 6y ago

To add a new language you can simply reuse an old model that was trained on lots of languages and fine-tune it for your needs. The same practice exists for huge natural-language models such as BERT and GPT.