6 ms·

Unsupervised translation of programming languages

166 points by elsewhen 6y ago

linux-hog 6y ago
To what degree must one consume the "prediction is more fundamental than understanding" koolaid before work like this looks reasonable?
- junipertea 6y ago
  In Table 2, we observe that TransCoder significantly outperforms both baselines in terms of computational accuracy, with 74.8% and 68.7% in the C++ → Java and Java → Python directions, compared to 61% and 38.3% It beats existing commercial products that try to do the same, so to me it seems it has some value right now.
  - qayxc 6y ago
    > It beats existing commercial products that try to do the same, so to me it seems it has some value right now. While I agree with your conclusion, a minor correction is in order here: it beats a (as in a single selected) commercial product and a single free OSS transpiler. The improvement for the commercial product is 61% vs. 75% accuracy (i.e. 23% better) which - while impressive given the unsupervised learning aspect - isn't a game changer (yet!).
- whymauri 6y ago
  This is a rude comment. The paper has no grander claim of AGI or 'fundamental' understanding. It's a novel algorithm that outperforms existing solutions. Not everything needs to be part of work towards 'understanding.'
  - linux-hog 6y ago
    It outperforms existing solutions on a publicly unavailable corpus with no clear way to reproduce the results? Hmm... The authors literally ran a series of experiments and published them without giving a clear way to reproduce the results. How is that useful to anyone except to self advertise? My comment about prediction vs understanding was simply meant to underscore this, albeit it might rub some people the wrong way. If you publish a paper purely about prediction (ie a set of experiments) be prepared to release all pertinent information to reproduce said experiments. If you choose to publish a paper that aims to improve mankind's understanding of the problem at hand, you are intrinsically required to provide all proof in your exposition. Otherwise we might as well just believe everything anyone ever says with no proof.
p4bl0 6y ago
It says that it is possible to train the translator for any pair of languages. But they did it with C++, Java, and Python, which are all object oriented imperative languages. Would it work as well for a C++ to OCaml translation, or a Prolog to Java translation? Changing of programming paradigm can be quite tricky…
- lostcolony 6y ago
  Yeah; I was far less impressed when I saw the languages in question when I ran across this. Yes, there are extra/fewer bits to consider when translating between these, but fundamentally there is a direct translation possible, that would still be reasonably idiomatic. Translate a Java program to Erlang, though...you have to completely change the design or it's not at all idiomatic. A task scheduler wouldn't rely on a priority queue; it would rely on spinning up a process per task that would sleep until it needs to execute, for instance.
- tanilama 6y ago
  It is possible but the quality will suffer. In NLP land, En-Fr unsupervised translation yield significantly better result than say for En-Zh translation.
vinceguidry 6y ago
It would be neat to see each language get a functionality subset defined that transpilation can smoothly translate from one language to another. The code itself wouldn't even have to be stored as one language, but rather in an intermediate representation. I like the idea of shared VMs like Truffle and LLVM, but they really don't fix the problem of allowing developers with disparate backgrounds and preferences collaborate.
qayxc 6y ago
This paper is very interesting, but as with many recent company-funded DNN-related publications, I am missing some technical information here. Publications by universities will almost always include details like dataset sizes, hardware used, and durations for pre-processing, training, and inference. As-is, this paper leaves me puzzled as to whether this novel solution is even practical in any way shape or form. If I essentially need the computational resources of a medium-sized data centre to prepare the data, and train the system for adding a new language-pair or fine-tuning an existing one, it'd not be a practical method outside of the Four Horsemen (GAFA) and their closest competitors... I applaud the use of computational accuracy as the primary metric for evaluating the results, though.
- zitterbewegung 6y ago
  Companies / researchers in general have no strong requirement to show you any artifacts to reproduce their work. There are incentives to not even provide that information at all. What we usually see is that someone else goes through the time to reproduce the results when reproducing it. Or internally from companies you can see projects moving from one company to another like MapReduce (Google) to Yahoo (Hadoop). And then if enough companies don't want to manage the whole project the software gets donated to the Apache Software foundation.
  - qayxc 6y ago
    > Companies / researchers in general have no strong requirement to show you any artifacts to reproduce their work. That's quite literally the opposite of what science is all about! If they don't want others to reproduce their results, they might just as well end each paper with "You can take our word for it!" and skip the details altogether... What I'm rather interested in are points of comparison. Performance in terms of a chosen metric is one thing, but research gets more useful if it can easily be reproduced. This is the norm in all other sciences - why not in AI research? If I can see that their approach is 20% better than SOTA, but they require 1M LoC plus 3 weeks of total computation time on a 100 machine cluster with 8 V100 per node, I can safely say - sod it! - use the inferior commercial product instead and add 20% manual effort (since I need to add manual work anyways as the accuracy isn't 100%).
Animats 6y ago
Now that's interesting. The examples are a bit cherry-picked, but there are some interesting results. Seeing C++ int allocation[n]; memset(allocation, -1, sizeof(allocation)); for(int i = 0; i < n; i ++) translated into Python: allocation = [-1] * n for i in range(n): is impressive. The translator picked up on two idioms of C++ and translated them to more concise forms in Python. Most "transpilers" just compile into the target language as if compiling to something like machine code, generating worst case wordy output. There's a C to Rust translator which translated C pointer indexing into Rust unsafe pointer arithmetic, which is not a gain in safety. It's also claimed that this system is good at guessing types from untyped languages. So a good way to use this technology might be to have something that looks at source code and tries to translate all the function signatures and type definitions. This includes looking at function calls and bodies to help guess ("infer" is a stretch for this approach) the types of ambiguous variables. Example is int foo(char* s); Is "s" a single character or a pointer to an array? You can't tell without context. This system might be able to do that. Or int getdata(int *buf, size_t bufl); A system like this should be able to recognize the intent there. Then try to translate the executable code with that information available. Bad guesses about types will usually result in translation failures or code that compiles with type errors, so someone will notice. So this is promising for modernizing code. I want to see one able to turn C pointer arithmetic into slice syntax. Now that would be a big step forward.
- userbinator 6y ago
  The translator picked up on two idioms of C++ and translated them to more concise forms in Python. It also either knew about the machine integer representation, or else made a combination of two mistakes that turned out to cancel each other out. This int allocation[n]; memset(allocation, 42, sizeof(allocation)); for(int i = 0; i < n; i ++) would NOT be equivalent to this: allocation = [42] * n for i in range(n):
  - Animats 6y ago
    Python 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0] on linux >>> n = 10 >>> allocation = [42] * n >>> allocation [42, 42, 42, 42, 42, 42, 42, 42, 42, 42] That's what "*" does on lists in Python. The idea is that "abc" + def" is "abcdef", and then they overgeneralized.
scotty79 6y ago
JavaScript > TypeScript translator might be a useful tool.
- yalok 6y ago
  And then TS to C#, to make it compilable through things like il2cpp
zild3d 6y ago
Curious about how dependencies could be handled. Translating every library used doesn't sound ideal, but replacing with an "equivalent" library for the language wouldn't always (or even usually) work either
dang 6y ago
There's a related video at https://www.youtube.com/watch?v=xTzFJIknh7E https://www.youtube.com/watch?v=xTzFJIknh7E. (via https://news.ycombinator.com/item?id=23470668 https://news.ycombinator.com/item?id=23470668, but no comments there)
avsteele 6y ago
Related: An IRS employee reportedly got 90% of their assembler (yes really) code translated to JAVA. IRS strangely abandoned the project. https://federalnewsnetwork.com/tom-temin-commentary/2020/01/irs-programming-mystery-continues/ https://federalnewsnetwork.com/tom-temin-commentary/2020/01/...
- cheschire 6y ago
  Any developer with more than a few years of experience knows the first 90% is the easy part though.
- coolgeek 6y ago
  This is code from the 60s/70s running on IBM mainframes. It should not be surprising that it is written in Assembler, given those factors and the performance advantages over the alternatives. I was writing new 370 Assembler in the 90s for high throughput electronic pre-press systems.
- WalterGR 6y ago
  Was it by hand or automated? I couldn’t penetrate the gossip and court intrigue on that site.
lostmsu 6y ago
This is very impressive, but it can not handle large contexts. E.g. in its current state it would not be able to translate large codebases, even more so using different libraries. For example, the translator consistently fails to convert Python max(arr) into proper Java. It generates Math.max(arr) instead, which is a non-existent overload of the function. Another problem is that unless you can port tests too, the code is not very reliable.
- peteretep 6y ago
  I wonder if there's value there for in-IDE snippets. I need to make some changes to some Java code, I know how I'd say it in Python, and this'll do the translation for me?
  - lostmsu 6y ago
    Overall I don't think it is a convenient way to program. Besides, you still need to test it.
- c3534l 6y ago
  How do you know this? Do you have access to the program somewhere?
  - lostmsu 6y ago
    The max example is listed in the very last paragraph of the paper as one of the failure modes.
pontusrehula 6y ago
Could I use this to clean up my code? First write some garbage in java then do a round java -> python -> java. Does it result in cleaner code? :)
- travisjungroth 6y ago
  Probably more like doing a round trip in a natural language translator.
latrot 6y ago
Next step would then be FP<->Imperative paradigms?
latrot 6y ago
Next step would then be FP<->Imperative.. Beginning with the static ones? For dynamic it would require introspection...
blackbear_ 6y ago
I don't know, the examples don't look too complicated. I wonder how it is possible that an approach based on rules would not work on those examples.
mister_hn 6y ago
what about template meta-programming C++ code? Can be converted in python or java?
dunefox 6y ago
It would be interesting compile languages into an intermediate form - like a lisp dialect/form or so - to see if that makes things easier. This could be used to have a common foundation on top of LLVM, for example.

166 points by elsewhen 6y ago

To what degree must one consume the "prediction is more fundamental than understanding" koolaid before work like this looks reasonable?

junipertea 6y ago
In Table 2, we observe that TransCoder significantly outperforms both baselines in terms of computational accuracy, with 74.8% and 68.7% in the C++ → Java and Java → Python directions, compared to 61% and 38.3% It beats existing commercial products that try to do the same, so to me it seems it has some value right now.
- qayxc 6y ago
  > It beats existing commercial products that try to do the same, so to me it seems it has some value right now. While I agree with your conclusion, a minor correction is in order here: it beats a (as in a single selected) commercial product and a single free OSS transpiler. The improvement for the commercial product is 61% vs. 75% accuracy (i.e. 23% better) which - while impressive given the unsupervised learning aspect - isn't a game changer (yet!).
whymauri 6y ago
This is a rude comment. The paper has no grander claim of AGI or 'fundamental' understanding. It's a novel algorithm that outperforms existing solutions. Not everything needs to be part of work towards 'understanding.'
- linux-hog 6y ago
  It outperforms existing solutions on a publicly unavailable corpus with no clear way to reproduce the results? Hmm... The authors literally ran a series of experiments and published them without giving a clear way to reproduce the results. How is that useful to anyone except to self advertise? My comment about prediction vs understanding was simply meant to underscore this, albeit it might rub some people the wrong way. If you publish a paper purely about prediction (ie a set of experiments) be prepared to release all pertinent information to reproduce said experiments. If you choose to publish a paper that aims to improve mankind's understanding of the problem at hand, you are intrinsically required to provide all proof in your exposition. Otherwise we might as well just believe everything anyone ever says with no proof.

p4bl0 6y ago

It says that it is possible to train the translator for any pair of languages. But they did it with C++, Java, and Python, which are all object oriented imperative languages. Would it work as well for a C++ to OCaml translation, or a Prolog to Java translation? Changing of programming paradigm can be quite tricky…

lostcolony 6y ago
Yeah; I was far less impressed when I saw the languages in question when I ran across this. Yes, there are extra/fewer bits to consider when translating between these, but fundamentally there is a direct translation possible, that would still be reasonably idiomatic. Translate a Java program to Erlang, though...you have to completely change the design or it's not at all idiomatic. A task scheduler wouldn't rely on a priority queue; it would rely on spinning up a process per task that would sleep until it needs to execute, for instance.
tanilama 6y ago
It is possible but the quality will suffer. In NLP land, En-Fr unsupervised translation yield significantly better result than say for En-Zh translation.

vinceguidry 6y ago

It would be neat to see each language get a functionality subset defined that transpilation can smoothly translate from one language to another. The code itself wouldn't even have to be stored as one language, but rather in an intermediate representation. I like the idea of shared VMs like Truffle and LLVM, but they really don't fix the problem of allowing developers with disparate backgrounds and preferences collaborate.

qayxc 6y ago

This paper is very interesting, but as with many recent company-funded DNN-related publications, I am missing some technical information here. Publications by universities will almost always include details like dataset sizes, hardware used, and durations for pre-processing, training, and inference. As-is, this paper leaves me puzzled as to whether this novel solution is even practical in any way shape or form. If I essentially need the computational resources of a medium-sized data centre to prepare the data, and train the system for adding a new language-pair or fine-tuning an existing one, it'd not be a practical method outside of the Four Horsemen (GAFA) and their closest competitors... I applaud the use of computational accuracy as the primary metric for evaluating the results, though.

zitterbewegung 6y ago
Companies / researchers in general have no strong requirement to show you any artifacts to reproduce their work. There are incentives to not even provide that information at all. What we usually see is that someone else goes through the time to reproduce the results when reproducing it. Or internally from companies you can see projects moving from one company to another like MapReduce (Google) to Yahoo (Hadoop). And then if enough companies don't want to manage the whole project the software gets donated to the Apache Software foundation.
- qayxc 6y ago
  > Companies / researchers in general have no strong requirement to show you any artifacts to reproduce their work. That's quite literally the opposite of what science is all about! If they don't want others to reproduce their results, they might just as well end each paper with "You can take our word for it!" and skip the details altogether... What I'm rather interested in are points of comparison. Performance in terms of a chosen metric is one thing, but research gets more useful if it can easily be reproduced. This is the norm in all other sciences - why not in AI research? If I can see that their approach is 20% better than SOTA, but they require 1M LoC plus 3 weeks of total computation time on a 100 machine cluster with 8 V100 per node, I can safely say - sod it! - use the inferior commercial product instead and add 20% manual effort (since I need to add manual work anyways as the accuracy isn't 100%).

Animats 6y ago

Now that's interesting. The examples are a bit cherry-picked, but there are some interesting results. Seeing C++ int allocation[n]; memset(allocation, -1, sizeof(allocation)); for(int i = 0; i < n; i ++) translated into Python: allocation = [-1] * n for i in range(n): is impressive. The translator picked up on two idioms of C++ and translated them to more concise forms in Python. Most "transpilers" just compile into the target language as if compiling to something like machine code, generating worst case wordy output. There's a C to Rust translator which translated C pointer indexing into Rust unsafe pointer arithmetic, which is not a gain in safety. It's also claimed that this system is good at guessing types from untyped languages. So a good way to use this technology might be to have something that looks at source code and tries to translate all the function signatures and type definitions. This includes looking at function calls and bodies to help guess ("infer" is a stretch for this approach) the types of ambiguous variables. Example is int foo(char* s); Is "s" a single character or a pointer to an array? You can't tell without context. This system might be able to do that. Or int getdata(int *buf, size_t bufl); A system like this should be able to recognize the intent there. Then try to translate the executable code with that information available. Bad guesses about types will usually result in translation failures or code that compiles with type errors, so someone will notice. So this is promising for modernizing code. I want to see one able to turn C pointer arithmetic into slice syntax. Now that would be a big step forward.

userbinator 6y ago
The translator picked up on two idioms of C++ and translated them to more concise forms in Python. It also either knew about the machine integer representation, or else made a combination of two mistakes that turned out to cancel each other out. This int allocation[n]; memset(allocation, 42, sizeof(allocation)); for(int i = 0; i < n; i ++) would NOT be equivalent to this: allocation = [42] * n for i in range(n):
- Animats 6y ago
  Python 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0] on linux >>> n = 10 >>> allocation = [42] * n >>> allocation [42, 42, 42, 42, 42, 42, 42, 42, 42, 42] That's what "*" does on lists in Python. The idea is that "abc" + def" is "abcdef", and then they overgeneralized.

scotty79 6y ago

JavaScript > TypeScript translator might be a useful tool.

yalok 6y ago
And then TS to C#, to make it compilable through things like il2cpp

zild3d 6y ago

Curious about how dependencies could be handled. Translating every library used doesn't sound ideal, but replacing with an "equivalent" library for the language wouldn't always (or even usually) work either

dang 6y ago

There's a related video at https://www.youtube.com/watch?v=xTzFJIknh7E https://www.youtube.com/watch?v=xTzFJIknh7E. (via https://news.ycombinator.com/item?id=23470668 https://news.ycombinator.com/item?id=23470668, but no comments there)

avsteele 6y ago

Related: An IRS employee reportedly got 90% of their assembler (yes really) code translated to JAVA. IRS strangely abandoned the project. https://federalnewsnetwork.com/tom-temin-commentary/2020/01/irs-programming-mystery-continues/ https://federalnewsnetwork.com/tom-temin-commentary/2020/01/...

cheschire 6y ago
Any developer with more than a few years of experience knows the first 90% is the easy part though.
coolgeek 6y ago
This is code from the 60s/70s running on IBM mainframes. It should not be surprising that it is written in Assembler, given those factors and the performance advantages over the alternatives. I was writing new 370 Assembler in the 90s for high throughput electronic pre-press systems.
WalterGR 6y ago
Was it by hand or automated? I couldn’t penetrate the gossip and court intrigue on that site.

lostmsu 6y ago

This is very impressive, but it can not handle large contexts. E.g. in its current state it would not be able to translate large codebases, even more so using different libraries. For example, the translator consistently fails to convert Python max(arr) into proper Java. It generates Math.max(arr) instead, which is a non-existent overload of the function. Another problem is that unless you can port tests too, the code is not very reliable.

peteretep 6y ago
I wonder if there's value there for in-IDE snippets. I need to make some changes to some Java code, I know how I'd say it in Python, and this'll do the translation for me?
- lostmsu 6y ago
  Overall I don't think it is a convenient way to program. Besides, you still need to test it.
c3534l 6y ago
How do you know this? Do you have access to the program somewhere?
- lostmsu 6y ago
  The max example is listed in the very last paragraph of the paper as one of the failure modes.

pontusrehula 6y ago

Could I use this to clean up my code? First write some garbage in java then do a round java -> python -> java. Does it result in cleaner code? :)

travisjungroth 6y ago
Probably more like doing a round trip in a natural language translator.

latrot 6y ago

Next step would then be FP<->Imperative paradigms?

latrot 6y ago

Next step would then be FP<->Imperative.. Beginning with the static ones? For dynamic it would require introspection...

blackbear_ 6y ago

I don't know, the examples don't look too complicated. I wonder how it is possible that an approach based on rules would not work on those examples.

mister_hn 6y ago

what about template meta-programming C++ code? Can be converted in python or java?

dunefox 6y ago

It would be interesting compile languages into an intermediate form - like a lisp dialect/form or so - to see if that makes things easier. This could be used to have a common foundation on top of LLVM, for example.