5 ms·

Do you really want to convert the lines one-to-one? This seems pretty wasteful. Surely a better way is to just do step 3, drawing inspiration (and copying where

by orf 10y ago

Do you really want to convert the lines one-to-one? This seems pretty wasteful. Surely a better way is to just do step 3, drawing inspiration (and copying where appropriate) from the original code based on the features rather than the structure of the original code. For example gtk-doc generates documentation from comments in c code. I'm guessing there is some perl code to parse c code, which i would substitute for the great 'pycparser' library. With this method you would copy the existing code, fix it, then throw it away. The methods to generate HTML from the comments is likely a lot of code. Sure, you could copy whatever method the current perl code does... Or you could substitute a lot of it for Jinja2. Perhaps these examples are incorrect in the context of this specific project, but i don't see the point in copying the lines one for one. Copy the meaning of the code into idiomatic python from the get go, and test the output against the known good perl code. I doubt perl code copied one for one is ever going to be idiomatic python code, so why bother, especially if it takes so long? Edit: Yeah, tests are good. No tests are bad. Having a complete understanding of the code doesn't require translating it line-for-line. Re-writing a project in a different language is a breaking change, but in the context of this project your initial tests could be (for every bit of C code you can possibly find that has gtk-doc comments): > gtk-doc.py > a > gtk-doc.pl > b > diff a b After you can find no differences, great, release it as an alpha or beta. People can then feed back any internal, weird quirky behavior they depend on. Rinse and repeat. My point is spending 1000000000 hours hand-converting perl into python then again re-writing it into idiomatic python so you can catch any theoretical small perl-specific edge case in what is effectively a complete rewrite and a major version change is IMO a bit pointless, especially when converting Frankenstein python-perl into idiomatic Python may result in deleting large parts of the said slow-to-convert code.

douche 10y ago
Hopefully there is a comprehensive set of integration and acceptance tests to verify that the output of the new tool matches the output of the old tool, bugs and weirdness and all. Unfortunately, this is rarely the case.
- orf 10y ago
  Isn't that a null point? If they don't exist then this endeavour is dangerous no matter the method you go about it - translating perl to python doesn't ensure its a 1-1 mapping functionality wise, and if you're going to rewrite it in idomatic python anyway....
- mnm1 10y ago
  Run the old tool, run the new tool, compare outputs. If you can't do this, you shouldn't be rewriting anything.
  - JoshTriplett 10y ago
    If you don't have a comprehensive set of test inputs, then "compare the outputs" doesn't suffice. (If you do, then by all means, take that approach.)
  - majewsky 10y ago
    I do this a lot (end-to-end tests that just `diff` the results from the current code with the expectation that I previously committed into the repo), and it makes one appreciate how much work it is to make a program's output deterministic. When you look around some of my code, there's a lot of loops over maps/dicts/hashes where, instead of iterating over the map directly, I first gather the keys, sort them, and then iterate over these to ensure that the test is deterministic.
    - tofflos 10y ago
      Plus one for deterministic software. It's an incredibly useful property when you want to test something. Particularly when doing major version upgrades and when replacing entire systems. Unfortunately strict determinism and test coverage often go hand in hand. Either the developer gets it or (s)he doesn't. Fortunately you often do get a least a moderate level of determinism even from the most awful software.
ams6110 10y ago
One problem is that the "known good perl code" likely doesn't have comprehensive tests, it was probably written before TDD became prevalent. So you'll probably need an initial phase to develop tests for all the functionality so you can verify that your new code still passes all the tests. Also you probably still have to understand every line of the old code, because within will be buried the many undocumented special case workarounds and bug fixes that code accumulates over time. See the classic Spolsky "Things You Should Never Do" https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...
- jlg23 10y ago
  > One problem is that the "known good perl code" likely doesn't have comprehensive tests, it was probably written before TDD became prevalent. I'd argue that perl made TDD mainstream, though we did not call it that way 20 years ago. Well, 30, but I did not join in until the mid 90s. Perl's test protocol TAP was around since at least 1988[1] and back then when I was coding perl for a living, I rarely encountered "known good perl code" that did not have tests (notable exception: usemod wiki). [1] https://testanything.org/history.html https://testanything.org/history.html
  - BWStearns 10y ago
    Having used TAP I'm glad it existed for the things it arguably precipitated but I will never do it voluntarily again.
  - logicallee 10y ago
    >I'd argue that perl made TDD mainstream, because wtf else are you going to do when you develop using lines like next if (/^(#|$)/); (totally 100% typical totally normal line - I found it by clicking the source for the very first CPAN module for a very common word - I knew I would find a typical line like that. I swear there was zero bias to this methodology). I just use this example because it ends in what is line noise. I'm pretty sure it means "don't process this line (skip to the nexct one) if it's a line that either starts with a #, or is an empty line). But how do I really know what it does? By testing it, of course! When good code is line noise, how else do you know that it works? Of course you're going to test it. :D - EDIT: Downvoters aren't disagreeing with me. You can't downvote me just because you don't like the truth. Perl coders have a heavy culture of testing. It's the only way anyone writes Perl code - period. This is in contrast with C++ programmers, for example, who historically were more content to reason about their code. Historical fact - sorry.
jlg23 10y ago
This requires you to full understand upfront all of the code you are trying to convert. While this is feasible in small projects, for a large code base this can be very, very dangerous. The advantages of going the slow way: * you can reuse test-cases * you have something to check the program flow against (in worst case with print statements) * by the time you are done you really should know the code and step #3 is a breeze PS: Your example of "just use jinja2" - have you verified that a rewrite is non-trivial? Have you checked that there are test cases you can run against your code or are you risking to manually have to verify that your new version does the same?
smacktoward 10y ago
Sure, you can absolutely write a new program with the same feature list instead of more closely porting the old logic over to the new language. But now you have a new problem: a Chesterton's Fence (https://www.chesterton.org/taking-a-fence-down/ https://www.chesterton.org/taking-a-fence-down/) problem. The old logic is in the form it is (knotty and complicated, most probably) because previous developers made it that way in response to problems they were facing at the time -- problems you may not be aware of, because their old changes made those problems go away. Those knots represent places where those previous developers learned that the problem they were trying to solve wasn't as simple as it first appeared. You can definitely simplify the porting job if you just ignore all that and write new logic that does The Simplest Thing That Could Possibly Work, but by doing so you can set yourself up for future heartburn as all those forgotten edge cases come roaring back. You end up re-learning the hard way lessons your predecessors learned the first time around.
bsder 10y ago
It is Perl code to parse X using regular expressions. This is always the worst kind of Perl code imaginable to try to tinker with. It will be fiddly. It will have exceptional cases piled upon exceptional cases. It will have hideous numbers of undocumented corners. All because it isn't a real, actual, fscking parser. Because we don't really need a parser for this, right now, do we, really? Yeah, this is one of those cases where "Do the simplest thing" breaks down horribly and very rarely does someone have the wherewithal to apply 2x4 cluebats to the people who persist in writing parsers with regular expressions. Yes, I have done so (in Python actually). However, my code has a little "counter" in the comments along the lines of "You have debugged a regular expression in this code: <n> times. Use a real parser you idiot." It is depressing how large I have let that number get before I actually pull my head out of my ass.
- echlebek 10y ago
  > "Do the simplest thing" You have to do the simplest thing that could actually, possibly work!
- zzzcpan 10y ago
  Yeah, proper parsers are actually super easy once you are comfortable with them and take much much less time than a pile of regexps. Like higher-order parser combinators. You can implement them for most languages in no time and then just construct your parser from a few basic higher-order primitives: regexp tokens, alternations, sequences and repetitions. Getting there though takes a bit of practice, from recursive descent parsers to parser generators.
  - weaksauce 10y ago
    are you handwriting your parsers or using some kind of parser generator typically?
    - girvo 10y ago
      I usually just pull a Parsing Expression Grammar library in.
klibertp 10y ago
> Do you really want to convert the lines one-to-one? Yes, as much as possible! > This seems pretty wasteful. No, it's not! The more 1 to 1 the rewrite is, the more of it is possible to be done automatically. Assuming your language gives you enough expressive power, it's often faster to write a library of shims for semantics and translate syntax automatically... in the worst case with a bunch of regexes - even that works better than manually writing the code from scratch, even if you read and drew inspiration from another implementation. As an example, I started porting Python code, the PyParsing library, to Io[1]. I stripped docstrings for the moment, but if you open it side-by-side with the Python code, you'll see that it's almost line-to-line identical, except for the syntax. To do this I first wrote a library emulating some of Python features in Io[2], then passed the original code to a series of around 20 regexes, which resulted in not-working-but-close Io, which I manually corrected. Now, writing 1.5k (it's about a half of PyParsing for now, not finished) lines of a library (not to mention 3 times as much docstrings, which you can reuse if you keep your translation close to original), even if you had previously a quite good understanding of such a library, wouldn't take two days. Which is exactly how much time it took me using this approach. Of course, how beneficial this method may be to you in practice depends on the target language and how well it matches features of the original or how easy it is to extend it. Io and Racket are two examples of languages extremely well suited to this approach, but you can do the same with JavaScript, Python, Ruby, PERL and many other high-level, meta-programmable languages, just with a bit more effort. Anyway, my two cents, as I recently had such an experience :) [1] https://github.com/piotrklibert/ioparsing/blob/master/src/parsing.io https://github.com/piotrklibert/ioparsing/blob/master/src/pa... [2] https://github.com/piotrklibert/ioparsing/blob/master/src/syntax.io https://github.com/piotrklibert/ioparsing/blob/master/src/sy... and str.io
MaulingMonkey 10y ago
> Do you really want to convert the lines one-to-one? This seems pretty wasteful. I did this once - converting a few kloc of Java to C# - I'm sure it helped that these are very similar languages in many respects. I actually copied the Java source code into C# source files, and then tweaked them until they compiled. The initial commit even had e.g. a "polyfill" for Java's ArrayList<T> class. https://github.com/MaulingMonkey/poly2tri-cs https://github.com/MaulingMonkey/poly2tri-cs By virtue of being such a mechanical translation, it went very fast. I intentionally avoided translating into C# idioms - or even naming conventions - in the initial commit. I wanted to avoid refactoring the entire codebase simultaneously - my experience has been that multi kloc refactors are a great way to bite off more than I can chew, and spend a lot of time in refactoring or debugging hell, and to create an unreviewable mess, all of which is very slow. By saving all the refactoring into C# idioms etc. for followup changelists, the translation went so smoothly that I still haven't learned the math behind constrained Delaunay triangulation. A non-1:1 translation I'm sure would've forced me to learn it, if only for debugging purposes. ArrayList<T> is now gone - replaced with standard List<T> and IEnumerable<T> from .NET's standard libraries, foo.setX(value) replaced with foo.X = value;, etc. This isn't an approach I'd recommend for all such projects, but I'm surprised just how well it worked for that conversion. The initial translation took maybe a few hours (if that), and the subsequent cleanup a few more.
- candiodari 10y ago
  Given how powerful it is, when used correctly, it's tough to see why refactoring has such a bad reputation amongst software developers. For instance, using refactoring, when possible, provides far better guarantees of correctness than tests do.

by orf 10y ago

douche 10y ago

Hopefully there is a comprehensive set of integration and acceptance tests to verify that the output of the new tool matches the output of the old tool, bugs and weirdness and all. Unfortunately, this is rarely the case.

orf 10y ago
Isn't that a null point? If they don't exist then this endeavour is dangerous no matter the method you go about it - translating perl to python doesn't ensure its a 1-1 mapping functionality wise, and if you're going to rewrite it in idomatic python anyway....
mnm1 10y ago
Run the old tool, run the new tool, compare outputs. If you can't do this, you shouldn't be rewriting anything.
- JoshTriplett 10y ago
  If you don't have a comprehensive set of test inputs, then "compare the outputs" doesn't suffice. (If you do, then by all means, take that approach.)
- majewsky 10y ago
  I do this a lot (end-to-end tests that just `diff` the results from the current code with the expectation that I previously committed into the repo), and it makes one appreciate how much work it is to make a program's output deterministic. When you look around some of my code, there's a lot of loops over maps/dicts/hashes where, instead of iterating over the map directly, I first gather the keys, sort them, and then iterate over these to ensure that the test is deterministic.
  - tofflos 10y ago
    Plus one for deterministic software. It's an incredibly useful property when you want to test something. Particularly when doing major version upgrades and when replacing entire systems. Unfortunately strict determinism and test coverage often go hand in hand. Either the developer gets it or (s)he doesn't. Fortunately you often do get a least a moderate level of determinism even from the most awful software.

ams6110 10y ago

One problem is that the "known good perl code" likely doesn't have comprehensive tests, it was probably written before TDD became prevalent. So you'll probably need an initial phase to develop tests for all the functionality so you can verify that your new code still passes all the tests. Also you probably still have to understand every line of the old code, because within will be buried the many undocumented special case workarounds and bug fixes that code accumulates over time. See the classic Spolsky "Things You Should Never Do" https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ https://www.joelonsoftware.com/2000/04/06/things-you-should-...

jlg23 10y ago
> One problem is that the "known good perl code" likely doesn't have comprehensive tests, it was probably written before TDD became prevalent. I'd argue that perl made TDD mainstream, though we did not call it that way 20 years ago. Well, 30, but I did not join in until the mid 90s. Perl's test protocol TAP was around since at least 1988[1] and back then when I was coding perl for a living, I rarely encountered "known good perl code" that did not have tests (notable exception: usemod wiki). [1] https://testanything.org/history.html https://testanything.org/history.html
- BWStearns 10y ago
  Having used TAP I'm glad it existed for the things it arguably precipitated but I will never do it voluntarily again.
- logicallee 10y ago
  >I'd argue that perl made TDD mainstream, because wtf else are you going to do when you develop using lines like next if (/^(#|$)/); (totally 100% typical totally normal line - I found it by clicking the source for the very first CPAN module for a very common word - I knew I would find a typical line like that. I swear there was zero bias to this methodology). I just use this example because it ends in what is line noise. I'm pretty sure it means "don't process this line (skip to the nexct one) if it's a line that either starts with a #, or is an empty line). But how do I really know what it does? By testing it, of course! When good code is line noise, how else do you know that it works? Of course you're going to test it. :D - EDIT: Downvoters aren't disagreeing with me. You can't downvote me just because you don't like the truth. Perl coders have a heavy culture of testing. It's the only way anyone writes Perl code - period. This is in contrast with C++ programmers, for example, who historically were more content to reason about their code. Historical fact - sorry.

jlg23 10y ago

This requires you to full understand upfront all of the code you are trying to convert. While this is feasible in small projects, for a large code base this can be very, very dangerous. The advantages of going the slow way: * you can reuse test-cases * you have something to check the program flow against (in worst case with print statements) * by the time you are done you really should know the code and step #3 is a breeze PS: Your example of "just use jinja2" - have you verified that a rewrite is non-trivial? Have you checked that there are test cases you can run against your code or are you risking to manually have to verify that your new version does the same?

smacktoward 10y ago

Sure, you can absolutely write a new program with the same feature list instead of more closely porting the old logic over to the new language. But now you have a new problem: a Chesterton's Fence (https://www.chesterton.org/taking-a-fence-down/ https://www.chesterton.org/taking-a-fence-down/) problem. The old logic is in the form it is (knotty and complicated, most probably) because previous developers made it that way in response to problems they were facing at the time -- problems you may not be aware of, because their old changes made those problems go away. Those knots represent places where those previous developers learned that the problem they were trying to solve wasn't as simple as it first appeared. You can definitely simplify the porting job if you just ignore all that and write new logic that does The Simplest Thing That Could Possibly Work, but by doing so you can set yourself up for future heartburn as all those forgotten edge cases come roaring back. You end up re-learning the hard way lessons your predecessors learned the first time around.

bsder 10y ago

It is Perl code to parse X using regular expressions. This is always the worst kind of Perl code imaginable to try to tinker with. It will be fiddly. It will have exceptional cases piled upon exceptional cases. It will have hideous numbers of undocumented corners. All because it isn't a real, actual, fscking parser. Because we don't really need a parser for this, right now, do we, really? Yeah, this is one of those cases where "Do the simplest thing" breaks down horribly and very rarely does someone have the wherewithal to apply 2x4 cluebats to the people who persist in writing parsers with regular expressions. Yes, I have done so (in Python actually). However, my code has a little "counter" in the comments along the lines of "You have debugged a regular expression in this code: <n> times. Use a real parser you idiot." It is depressing how large I have let that number get before I actually pull my head out of my ass.

echlebek 10y ago
> "Do the simplest thing" You have to do the simplest thing that could actually, possibly work!
zzzcpan 10y ago
Yeah, proper parsers are actually super easy once you are comfortable with them and take much much less time than a pile of regexps. Like higher-order parser combinators. You can implement them for most languages in no time and then just construct your parser from a few basic higher-order primitives: regexp tokens, alternations, sequences and repetitions. Getting there though takes a bit of practice, from recursive descent parsers to parser generators.
- weaksauce 10y ago
  are you handwriting your parsers or using some kind of parser generator typically?
  - girvo 10y ago
    I usually just pull a Parsing Expression Grammar library in.

klibertp 10y ago

> Do you really want to convert the lines one-to-one? Yes, as much as possible! > This seems pretty wasteful. No, it's not! The more 1 to 1 the rewrite is, the more of it is possible to be done automatically. Assuming your language gives you enough expressive power, it's often faster to write a library of shims for semantics and translate syntax automatically... in the worst case with a bunch of regexes - even that works better than manually writing the code from scratch, even if you read and drew inspiration from another implementation. As an example, I started porting Python code, the PyParsing library, to Io[1]. I stripped docstrings for the moment, but if you open it side-by-side with the Python code, you'll see that it's almost line-to-line identical, except for the syntax. To do this I first wrote a library emulating some of Python features in Io[2], then passed the original code to a series of around 20 regexes, which resulted in not-working-but-close Io, which I manually corrected. Now, writing 1.5k (it's about a half of PyParsing for now, not finished) lines of a library (not to mention 3 times as much docstrings, which you can reuse if you keep your translation close to original), even if you had previously a quite good understanding of such a library, wouldn't take two days. Which is exactly how much time it took me using this approach. Of course, how beneficial this method may be to you in practice depends on the target language and how well it matches features of the original or how easy it is to extend it. Io and Racket are two examples of languages extremely well suited to this approach, but you can do the same with JavaScript, Python, Ruby, PERL and many other high-level, meta-programmable languages, just with a bit more effort. Anyway, my two cents, as I recently had such an experience :) [1] https://github.com/piotrklibert/ioparsing/blob/master/src/parsing.io https://github.com/piotrklibert/ioparsing/blob/master/src/pa... [2] https://github.com/piotrklibert/ioparsing/blob/master/src/syntax.io https://github.com/piotrklibert/ioparsing/blob/master/src/sy... and str.io

MaulingMonkey 10y ago

> Do you really want to convert the lines one-to-one? This seems pretty wasteful. I did this once - converting a few kloc of Java to C# - I'm sure it helped that these are very similar languages in many respects. I actually copied the Java source code into C# source files, and then tweaked them until they compiled. The initial commit even had e.g. a "polyfill" for Java's ArrayList<T> class. https://github.com/MaulingMonkey/poly2tri-cs https://github.com/MaulingMonkey/poly2tri-cs By virtue of being such a mechanical translation, it went very fast. I intentionally avoided translating into C# idioms - or even naming conventions - in the initial commit. I wanted to avoid refactoring the entire codebase simultaneously - my experience has been that multi kloc refactors are a great way to bite off more than I can chew, and spend a lot of time in refactoring or debugging hell, and to create an unreviewable mess, all of which is very slow. By saving all the refactoring into C# idioms etc. for followup changelists, the translation went so smoothly that I still haven't learned the math behind constrained Delaunay triangulation. A non-1:1 translation I'm sure would've forced me to learn it, if only for debugging purposes. ArrayList<T> is now gone - replaced with standard List<T> and IEnumerable<T> from .NET's standard libraries, foo.setX(value) replaced with foo.X = value;, etc. This isn't an approach I'd recommend for all such projects, but I'm surprised just how well it worked for that conversion. The initial translation took maybe a few hours (if that), and the subsequent cleanup a few more.

candiodari 10y ago
Given how powerful it is, when used correctly, it's tough to see why refactoring has such a bad reputation amongst software developers. For instance, using refactoring, when possible, provides far better guarantees of correctness than tests do.