4 ms·

This sounds more impressive than it is. I kept an auto-translation going for quite a while from C# to Go just by creating some bash scripts. These languages are

by loopz 6y ago

This sounds more impressive than it is. I kept an auto-translation going for quite a while from C# to Go just by creating some bash scripts. These languages are so similar, that while not idiomatic, you can be up and running with quite few rules and some simplifications to the original code. While I believe this is a good arena for algorithms to provide solution, I'm sceptical that ML in its current incarnations is the right tool. The problem is the variations in the problem space: generic programs, need for idiomatic code, renewed design and lack of human insight and ownership. Sure for an easy and specialized solution, this will work and may be helpful as a one-off. That is, if one just wants a straight port and test alot.

ganafagol 6y ago
And it's also a quite nonsensical use of ML technology, to be honest. Neural nets are good for problem domains with fuzzy definitions of right and wrong as you find them in the physical world out there. Is this a tiger or a rock? Is this food or poison? Should I walk around this pond or swim through it? That's what the human brain is good at handling and neural nets are trying to approximate that. Programming languages, on the other hand, are domains that are entirely man-made. We have created the rules, they are known in its entirety (in contrast to knowing all objects over there in the shade below the tree), and they are optimized to be so precise that they can be followed mechanically by a comparatively simple machine. Sure there are a few "undefined" corners but that's not because we don't understand them but because they are deliberately left open. Even they are precisely defined. So, with two precise definitions of semantics for two programming languages, it should be possible to build a translation based on those rules. Inherently, this is not an "empirical" endeavour, it's a logical/mathematical well-defined one. Using neural nets for this is using the wrong hammer for the job. It's like using them for deriving the implementation of a sorting algorithm or to find the first 100k digits of pi. It just does not make sense. And I'm not talking about "idiomatic" translations. That's a less well-defined domain an maybe that would be more suitable, but the post explicitly excludes this kind of thing. In principle, without providing examples of idiomatic style to the learning machinery, this can't ever work.
- eklavya 6y ago
  So why does it work better than the commercially available tools at the moment as claimed in the article?
  - PaulHoule 6y ago
    Because the commercially available tools aren't that good and because they kept working on it until it was better. If they kept working on it I think they would run into an asymptote. Maybe they could get closer and closer to 90% accuracy on a task with real hardware, 92% boiling the oceans, and 93% with a Dyson sphere, 93.5% if you can harness a quasar. At that point it probably passes a whiteboard interview and the people who have to fix the bugs can console themselves that the last programmer had neither a brain nor a soul. That system has an approximate, not an exact model of the domain it works on and that is why it has an asymptote. Turning a graph-structured program into a vector is like mapping the curved surface of the Earth onto a flat map -- except instead of it being a 3-dimensional space it is more like a 1000-dimensional space. Information is destroyed in that process and forever lost so there will always be important characteristics of the problem that it will never "get". If the message recipient is a person they will meet you halfway and might even accept bullshit if it is presented with complete confidence and lack of shame. The computer will interpret exactly what you said and reveal that you're a dog. (e.g. mute animal)
  - sankha93 6y ago
    This article is a classic example of Facebook PR doing a wonderful job of selling the research and linked paper [1] claiming too much in the introduction. Please, please talk to actual researchers before you buy such claims. If you go through the paper - you have to check the evaluation section to see how they measured their success. They used some programs from GeeksforGeeks to evaluate their approach. Problems on GeeksforGeeks do not represent the vast majority of programming tasks encountered in daily life. This is very much in contrast to the overarching claims presented in the introduction of the paper. Second issue with the evaluation: they use BLEU scores to judge how good their translations are. BLEU makes sense for natural language translations (even that is widely debated in the NLP community these days). For a program there is no concept of an almost correct program (based on how things look similar), it is either correct or not. Eg. if I am asked to write a program to add two numbers and I write `x - y`, I am not almost correct, I am completely wrong. And in some ways that is what their model does, it optimizes for BLEU scores. Third, the correctness of the programs are tested based on 10 random inputs. Are 10 random inputs enough to cover the entire input space that can be accepted by a program? It is indeed a great advance in the application of ML technology, but it is nowhere close to the broader claims. One can even debate, ROI on time spent in gathering and curating data and then checking the correctness of translation from such system vs the ROI on writing rules for a rule based system since all programming languages are easily expressible that way. [1]: https://arxiv.org/pdf/2006.03511.pdf https://arxiv.org/pdf/2006.03511.pdf
KMag 6y ago
From my limited machine learning experience, it seems machine learning is great at pattern-matching/cargo-culting itself to 95% solutions. Most of the time, porting between two similar languages can be done by cargo-cult pattern matching, but there are some very tough corner cases for semantic mismatches between the languages: thread safety (std::vector vs. java.util.Vector), iterator invalidation differences, hashmap iteration order differences, etc., etc. Most of the time, you can ignore these minor differences between languages, but sometimes you need whole-program, or at least whole-module analysis to know if the semantic differences matter. Heck, you could probably mostly get away porting from a lexically scoped language to a dynamically scoped language without putting any effort into fixing up scoping issues. I hope our tools eventually get good enough to perform these sort of transformations, but at this point, it's far from being safe to trust automated tools for this sort of thing.
malux85 6y ago
Yeah I was thinking it might be better to use something like LLVM IR as the “data structure” and then train deep nets to go $(LANG) => IR and then IR => $(LANG), trained with a lot of idiomatic well written code for each Lang, then the nets might be able to glean the intent more than implementation details. Not sure though, I’m not an expert in LLVM IR, so I’m not sure how feasible that is...
- loopz 6y ago
  Good call on LLVM. Was thinking about that too, but had to look up IR. This one looked interesting: https://idea.popcount.org/2013-07-24-ir-is-better-than-assembly/ https://idea.popcount.org/2013-07-24-ir-is-better-than-assem... IR looks to be targetting machine code, so not sure it'll be the right format to embed high-level language concepts.
- londons_explore 6y ago
  $(LANG) => IR can be done by LLVM itself... And that conversion misses out important details like comments. Perhaps the ideal solution takes as input both the original code and the IR to generate output.
- antoinealb 6y ago
  The issue is that LANG -> IR is a lossy transform: some constructs are lost, some language specific optimization passes could be applied, etc. Going from LANG1 -> LANG2 starts with more information, so it could in theory work better.
  - webmaven 6y ago
    It would be interesting to see what neural translation systems (in general, not just for code) use as an implicit internal IR.
  - Certhas 6y ago
    It's lossy, the inverse is not well defined. But you could try to train a NN on reconstructing plausible (idiomatic) inputs leading to the current IR...
screye 6y ago
I have seen some ML methods that learn explicit structural mappings [1][2]. I think ML is great when used as such. Even RL can work, because it operates over discrete spaces. But casting any problem with a discrete correct answer into floating point calculations that mimic probability is a bad idea. [1] : Learning to Compose Neural Networks for Question Answering [2] : Inferring and Executing Programs for Visual Reasoning
tasogare 6y ago
How do you handle generics? Go doesn’t have them, right? C# has an open source compiler (Rosyln) with an API that exposes lot of information, so for advanced needs there is this road too.
- loopz 6y ago
  It was just a bunch of sed. It wouldn't handle generics in a general manner, so was just a script to fill a specific need in the span of a period. What was "amazing" is how much of C# didn't need any translation into Go code, for core business logic. It was up and running within say 20-30 minutes, though needed a tweak now and then (sort of like modern-day "AI" ;).
Veedrac 6y ago
An ML method is able to outperform hard-coded translators by large margins, supporting a wide mix of languages like Python → C++ and translation between varied native APIs, but it's not impressive impressive because you can do the same thing with `sed`? Maybe take a look at the translations in the paper and come back to give a second opinion?

by loopz 6y ago

ganafagol 6y ago

And it's also a quite nonsensical use of ML technology, to be honest. Neural nets are good for problem domains with fuzzy definitions of right and wrong as you find them in the physical world out there. Is this a tiger or a rock? Is this food or poison? Should I walk around this pond or swim through it? That's what the human brain is good at handling and neural nets are trying to approximate that. Programming languages, on the other hand, are domains that are entirely man-made. We have created the rules, they are known in its entirety (in contrast to knowing all objects over there in the shade below the tree), and they are optimized to be so precise that they can be followed mechanically by a comparatively simple machine. Sure there are a few "undefined" corners but that's not because we don't understand them but because they are deliberately left open. Even they are precisely defined. So, with two precise definitions of semantics for two programming languages, it should be possible to build a translation based on those rules. Inherently, this is not an "empirical" endeavour, it's a logical/mathematical well-defined one. Using neural nets for this is using the wrong hammer for the job. It's like using them for deriving the implementation of a sorting algorithm or to find the first 100k digits of pi. It just does not make sense. And I'm not talking about "idiomatic" translations. That's a less well-defined domain an maybe that would be more suitable, but the post explicitly excludes this kind of thing. In principle, without providing examples of idiomatic style to the learning machinery, this can't ever work.

eklavya 6y ago
So why does it work better than the commercially available tools at the moment as claimed in the article?
- PaulHoule 6y ago
  Because the commercially available tools aren't that good and because they kept working on it until it was better. If they kept working on it I think they would run into an asymptote. Maybe they could get closer and closer to 90% accuracy on a task with real hardware, 92% boiling the oceans, and 93% with a Dyson sphere, 93.5% if you can harness a quasar. At that point it probably passes a whiteboard interview and the people who have to fix the bugs can console themselves that the last programmer had neither a brain nor a soul. That system has an approximate, not an exact model of the domain it works on and that is why it has an asymptote. Turning a graph-structured program into a vector is like mapping the curved surface of the Earth onto a flat map -- except instead of it being a 3-dimensional space it is more like a 1000-dimensional space. Information is destroyed in that process and forever lost so there will always be important characteristics of the problem that it will never "get". If the message recipient is a person they will meet you halfway and might even accept bullshit if it is presented with complete confidence and lack of shame. The computer will interpret exactly what you said and reveal that you're a dog. (e.g. mute animal)
- sankha93 6y ago
  This article is a classic example of Facebook PR doing a wonderful job of selling the research and linked paper [1] claiming too much in the introduction. Please, please talk to actual researchers before you buy such claims. If you go through the paper - you have to check the evaluation section to see how they measured their success. They used some programs from GeeksforGeeks to evaluate their approach. Problems on GeeksforGeeks do not represent the vast majority of programming tasks encountered in daily life. This is very much in contrast to the overarching claims presented in the introduction of the paper. Second issue with the evaluation: they use BLEU scores to judge how good their translations are. BLEU makes sense for natural language translations (even that is widely debated in the NLP community these days). For a program there is no concept of an almost correct program (based on how things look similar), it is either correct or not. Eg. if I am asked to write a program to add two numbers and I write `x - y`, I am not almost correct, I am completely wrong. And in some ways that is what their model does, it optimizes for BLEU scores. Third, the correctness of the programs are tested based on 10 random inputs. Are 10 random inputs enough to cover the entire input space that can be accepted by a program? It is indeed a great advance in the application of ML technology, but it is nowhere close to the broader claims. One can even debate, ROI on time spent in gathering and curating data and then checking the correctness of translation from such system vs the ROI on writing rules for a rule based system since all programming languages are easily expressible that way. [1]: https://arxiv.org/pdf/2006.03511.pdf https://arxiv.org/pdf/2006.03511.pdf

KMag 6y ago

From my limited machine learning experience, it seems machine learning is great at pattern-matching/cargo-culting itself to 95% solutions. Most of the time, porting between two similar languages can be done by cargo-cult pattern matching, but there are some very tough corner cases for semantic mismatches between the languages: thread safety (std::vector vs. java.util.Vector), iterator invalidation differences, hashmap iteration order differences, etc., etc. Most of the time, you can ignore these minor differences between languages, but sometimes you need whole-program, or at least whole-module analysis to know if the semantic differences matter. Heck, you could probably mostly get away porting from a lexically scoped language to a dynamically scoped language without putting any effort into fixing up scoping issues. I hope our tools eventually get good enough to perform these sort of transformations, but at this point, it's far from being safe to trust automated tools for this sort of thing.

malux85 6y ago

Yeah I was thinking it might be better to use something like LLVM IR as the “data structure” and then train deep nets to go $(LANG) => IR and then IR => $(LANG), trained with a lot of idiomatic well written code for each Lang, then the nets might be able to glean the intent more than implementation details. Not sure though, I’m not an expert in LLVM IR, so I’m not sure how feasible that is...

loopz 6y ago
Good call on LLVM. Was thinking about that too, but had to look up IR. This one looked interesting: https://idea.popcount.org/2013-07-24-ir-is-better-than-assembly/ https://idea.popcount.org/2013-07-24-ir-is-better-than-assem... IR looks to be targetting machine code, so not sure it'll be the right format to embed high-level language concepts.
londons_explore 6y ago
$(LANG) => IR can be done by LLVM itself... And that conversion misses out important details like comments. Perhaps the ideal solution takes as input both the original code and the IR to generate output.
antoinealb 6y ago
The issue is that LANG -> IR is a lossy transform: some constructs are lost, some language specific optimization passes could be applied, etc. Going from LANG1 -> LANG2 starts with more information, so it could in theory work better.
- webmaven 6y ago
  It would be interesting to see what neural translation systems (in general, not just for code) use as an implicit internal IR.
- Certhas 6y ago
  It's lossy, the inverse is not well defined. But you could try to train a NN on reconstructing plausible (idiomatic) inputs leading to the current IR...

screye 6y ago

I have seen some ML methods that learn explicit structural mappings [1][2]. I think ML is great when used as such. Even RL can work, because it operates over discrete spaces. But casting any problem with a discrete correct answer into floating point calculations that mimic probability is a bad idea. [1] : Learning to Compose Neural Networks for Question Answering [2] : Inferring and Executing Programs for Visual Reasoning

tasogare 6y ago

How do you handle generics? Go doesn’t have them, right? C# has an open source compiler (Rosyln) with an API that exposes lot of information, so for advanced needs there is this road too.

loopz 6y ago
It was just a bunch of sed. It wouldn't handle generics in a general manner, so was just a script to fill a specific need in the span of a period. What was "amazing" is how much of C# didn't need any translation into Go code, for core business logic. It was up and running within say 20-30 minutes, though needed a tweak now and then (sort of like modern-day "AI" ;).

Veedrac 6y ago

An ML method is able to outperform hard-coded translators by large margins, supporting a wide mix of languages like Python → C++ and translation between varied native APIs, but it's not impressive impressive because you can do the same thing with `sed`? Maybe take a look at the translations in the paper and come back to give a second opinion?