4 ms·

Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x. Similarly, he's using 40x speedup as a rule of thumb. I usual

Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x. Similarly, he's using 40x speedup as a rule of thumb. I usually think of Python as 20x slower than C. Personally I'd be loathe to convert a working Python system to Go, but it sounds like he had good reasons. I do wonder a bit whether divide-and-conquer or a C extension might not have worked instead.

jerf 7y ago
"Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x." This has been my extensive experience as well. I wouldn't be able to use Go if it was that much more verbose than Python. It certainly isn't as succinct as Python by any means, but it's not the night-and-day nightmare a lot of HN posters seem to think it is... provided you actually learn the language. In fact, one of the questions that I've found coming up in my head is... if you take the huge, huge pile of features that Python or other languages bring to the table that Go doesn't, and all their corresponding disadvantages in terms of having to learn them all, and how they all interact, etc.... and that's all you get in real, production code... is it really worth it? Because let's not mince words... it's a long list of features, all of which superficially seem awesome. And I can craft one-liners that would be a dozen lines or more of Go... but usually those one-liners have a lot of single-letter variables in them to focus on all the awesome syntax and features. When I get into real code with real variable names, the advantage fades fast. I find this to be food for thought. I still haven't fully integrate it into my worldview. But I can definitely say I feel like the cost/benefit matrix I had in my head even three years ago has shifted a lot. Perhaps it would be fair to say I haven't necessarily lowered my estimate of the benefits of all the fancy features, but my estimates of their costs have gone significantly up. A lot of them are really cheap in the moment you're writing them down for the first time and using them, but carry hidden long-term costs that I feel like younger me was not accounting for properly, especially if you are not the only developer.
- bmurphy1976 7y ago
  I've been saying for a long time that every abstraction has a cost. Sometimes that cost is hard to quantify or externalized, but our inability to quantify it doesn't mean the cost doesn't exist. Abstractions have to at least pay for themselves many times over to be worth the extra cognitive burden and we need to get better at measuring these trade-offs. The success of languages like Go hint that there's more costs than we have traditionally been willing to acknowledge.
  - erik_seaberg 7y ago
    The one-time cost of learning a good abstraction is strictly less than the ongoing cost of understanding and then continually reimplementing it by hand. The purpose of a high-level language is to make programs more concise and clear; a language that doesn't do this may somehow become popular but that shouldn't be mistaken for successful.
    - marcus_holmes 7y ago
      > The one-time cost of learning a good abstraction is strictly less than the ongoing cost of understanding and then continually reimplementing it by hand. There is a danger of a "no true Scotsman" fallacy here - it's easy to define a "good" abstraction as one that is worth the one-time cost of learning it. So, given that there are both "good" and "bad" abstractions in every language, is the net gain from learning them all greater than the cognitive effort of having to learn them all? I'd argue that no, it's not. Absolutely reducing the total number of abstractions in the language reduces the cognitive load of working in that language, even if it requires us to be more verbose.
      - erik_seaberg 7y ago
        I look on it as falling out of the https://en.wikipedia.org/wiki/Rule_of_least_power https://en.wikipedia.org/wiki/Rule_of_least_power. When I see reuse of an abstraction I know not only what it does but what it doesn't do. If I skip that and start writing custom code, well, that code could do anything at all so everyone has to continually reread it carefully. I find poring over tedious boilerplate to be a waste of precious lifetime compared to learning better building blocks.
- AnimalMuppet 7y ago
  Well... fancy features/abstractions are pretty low cost, if you use just one. It's when they interact that they really get expensive unless they are almost flawlessly designed to work together (and even then, they just get somewhat more expensive). It's like the first donut has 100 calories, the second has 200, the third has 400, and the fourth has 800. If you eat the whole box, it's really going to hurt. Rather than eat them all, you need to pick the one or two that you really want, and leave the rest in the box.
- stubish 7y ago
  You wouldn't take the huge pile of features in an all or nothing. You get to pick and choose. I think the authors wish list is similar to most people who are experienced with more expressive languages. I don't see calls for Python f-strings or async syntax to end up in Go. But something like a list-comprehension syntax I think would be really popular (based on what I saw when it was introduced to Python, originally 'why' but now many people's favorite feature). Interesting how Python is now considered a huge pile of features, considering how long it took for things like a ternary operator to end up in the language. Maybe Go is in it's Python 1.5.2 lifecycle stage ;)
  - Cthulhu_ 7y ago
    > You get to pick and choose. In theory; in practice though, you need to have all developers (and yourself) strictly aligned on what features to use. In my limited experience with Scala and working with Scala developers, there's as many styles and preferences as there are developers on a codebase.
  - jerf 7y ago
    "Interesting how Python is now considered a huge pile of features" I've been tracking Python since 1.5.2 was common and 2.0 was just coming out. For me, it's not been a problem because the new features were added incrementally. But I've seen new programmers try to come at it in 2019 and 2020, swallowing what I got spread out in ~15 major releases over ~20 years all in one big chunk, and learning even the core language now is definitely "a pile of features", to say nothing of the various library ecosystems. I don't even mean this necessarily as a criticism per se, just a description. It's not an easy-to-learn language anymore, and I don't recommend that people model it as such. It's a power tool for developers, not an easy learning language. And, yes, the "you can pick and choose the features" is a non-starter in a team environment. At best, the team can pick and choose, and that takes a rather strong hand and alignment to achieve. It is a very common case that you'll just end up with what your teammates use.
    - downerending 7y ago
      That's a good point. One of the original appeals of Python is that it started as a pedagogical language, and it shows. Even Python2 today still has most of that feel. Python3 on the other hand, just isn't. You're pretty much required to constantly deal with Unicode and its related issues, and this is a burden for beginners. In Python2, you can kick that can down the road almost indefinitely.
- downerending 7y ago
  I have pretty good mastery of Python2, and in my hands, a few lines of Python would blow up into a page of Go. This really frustrated me and was one of the reasons I bounced off of it. Other reasons: no fork(), poor make integration, weird source dir structure, and I once caught one of the go tools trying to write system files in /usr/lib or something (which fortunately failed on permissions). Can't recall the specifics of that last, and perhaps it was somehow misconfigured. Python2 has a good feature set, and one of the downsides of Python3 is that it greatly enlarges what you need to know without much expanding what you can do. It's a real problem, and it would be a shame if it ended up where C++ is these days. ("Poor Joe--a C++ spec fell off a high shelf and killed him...")
peteretep 7y ago
> it sounds like he had good reasons Meh. > Subversion-to-git conversion of the Gnu Compiler Collection history, at over 280K commits (over 1.6M individual change actions), was the straw that broke the camel’s back. Using PyPy with every optimization on semi-custom hardware tuned for this job still yielded test conversion times of over 9 hours, which is death on the recipe-debugging cycle. Just how often does one need to convert the Gnu Compiler Collection from Subversion to Git in under 9 hours?
- downerending 7y ago
  I was wondering that, too. My guess is that he's using this as a (large) set of test data to test his "recipes" and has to do that a lot. Having to run many nine-hour trials serially is painful. Not sure I'd have bothered, but it's a defensible choice. And produced this interesting example of comparable implementations.
- pdonis 7y ago
  > Just how often does one need to convert the Gnu Compiler Collection from Subversion to Git in under 9 hours? For this particular repository, once the conversion is complete, it's complete. But there are plenty of other old projects out there with large repositories in old version control systems. Reposurgeon is a general tool. Also, one repository conversion does not mean one run of reposurgeon and you're done. Reposurgeon is meant to be run multiple times since each run is likely to uncover issues that have to be addressed, and the way to address them is to do another run with updated parameters. Reposurgeon also has an interactive mode where you can explore a repository and test the results of various possible transformations.
gameswithgo 7y ago
When you have a system that is at big scale, even a 2x speedup can relieve a huge amount of problems. I think that Go, which is very inexpressive for modern languages, only expanded the code size that much, should make people consider whether we should ever be using interpreted, dynamic typed languages. Perhaps various JITted/compiled and statically typed languages can offer the same productivity on medium to large sized projects while just being a lot faster on top of it. People who love Python may especially enjoy Nim, or F#
- melling 7y ago
  If you like F#, it’s based on OCaml.
- nimmer 7y ago
  Nim is the closest language to Python. Translating to Python is a breeze and you get speeds comparable with pure C.
  - downerending 7y ago
    You seem like a Nimmer. What would you describe as the biggest downsides of Nim wrt Python? (aside from the fact that it's currently a niche language.)
    - nimmer 7y ago
      The only two that I can think of are due to static typing: You cannot enforce types only where you want, like mypy. Yet, type inference and automated conversions help quite a bit. There's an unofficial REPL in the compiler but you cannot do dir() or tab-complete methods/procs. You have to rely on documentation, nimsuggests or IDEs instead.
- downerending 7y ago
  Having done a significant amount of programming in both Python-like and statically typed languages, I think the idea that they offer similar productivity (i.e., time to correct, maintainable implementation) is simply false. And in a number of cases I've turned an existing program in a language like C or Java into a faster program in Python. One of the benefits of a fluent language is that you can explore more of the possibility space and look for better solutions. That's difficult and costly in static languages.

jerf 7y ago

"Interesting. I'd have expected more than a 50% code expansion going to Go, maybe even 3x or 5x." This has been my extensive experience as well. I wouldn't be able to use Go if it was that much more verbose than Python. It certainly isn't as succinct as Python by any means, but it's not the night-and-day nightmare a lot of HN posters seem to think it is... provided you actually learn the language. In fact, one of the questions that I've found coming up in my head is... if you take the huge, huge pile of features that Python or other languages bring to the table that Go doesn't, and all their corresponding disadvantages in terms of having to learn them all, and how they all interact, etc.... and that's all you get in real, production code... is it really worth it? Because let's not mince words... it's a long list of features, all of which superficially seem awesome. And I can craft one-liners that would be a dozen lines or more of Go... but usually those one-liners have a lot of single-letter variables in them to focus on all the awesome syntax and features. When I get into real code with real variable names, the advantage fades fast. I find this to be food for thought. I still haven't fully integrate it into my worldview. But I can definitely say I feel like the cost/benefit matrix I had in my head even three years ago has shifted a lot. Perhaps it would be fair to say I haven't necessarily lowered my estimate of the benefits of all the fancy features, but my estimates of their costs have gone significantly up. A lot of them are really cheap in the moment you're writing them down for the first time and using them, but carry hidden long-term costs that I feel like younger me was not accounting for properly, especially if you are not the only developer.

bmurphy1976 7y ago
I've been saying for a long time that every abstraction has a cost. Sometimes that cost is hard to quantify or externalized, but our inability to quantify it doesn't mean the cost doesn't exist. Abstractions have to at least pay for themselves many times over to be worth the extra cognitive burden and we need to get better at measuring these trade-offs. The success of languages like Go hint that there's more costs than we have traditionally been willing to acknowledge.
- erik_seaberg 7y ago
  The one-time cost of learning a good abstraction is strictly less than the ongoing cost of understanding and then continually reimplementing it by hand. The purpose of a high-level language is to make programs more concise and clear; a language that doesn't do this may somehow become popular but that shouldn't be mistaken for successful.
  - marcus_holmes 7y ago
    > The one-time cost of learning a good abstraction is strictly less than the ongoing cost of understanding and then continually reimplementing it by hand. There is a danger of a "no true Scotsman" fallacy here - it's easy to define a "good" abstraction as one that is worth the one-time cost of learning it. So, given that there are both "good" and "bad" abstractions in every language, is the net gain from learning them all greater than the cognitive effort of having to learn them all? I'd argue that no, it's not. Absolutely reducing the total number of abstractions in the language reduces the cognitive load of working in that language, even if it requires us to be more verbose.
    - erik_seaberg 7y ago
      I look on it as falling out of the https://en.wikipedia.org/wiki/Rule_of_least_power https://en.wikipedia.org/wiki/Rule_of_least_power. When I see reuse of an abstraction I know not only what it does but what it doesn't do. If I skip that and start writing custom code, well, that code could do anything at all so everyone has to continually reread it carefully. I find poring over tedious boilerplate to be a waste of precious lifetime compared to learning better building blocks.
AnimalMuppet 7y ago
Well... fancy features/abstractions are pretty low cost, if you use just one. It's when they interact that they really get expensive unless they are almost flawlessly designed to work together (and even then, they just get somewhat more expensive). It's like the first donut has 100 calories, the second has 200, the third has 400, and the fourth has 800. If you eat the whole box, it's really going to hurt. Rather than eat them all, you need to pick the one or two that you really want, and leave the rest in the box.
stubish 7y ago
You wouldn't take the huge pile of features in an all or nothing. You get to pick and choose. I think the authors wish list is similar to most people who are experienced with more expressive languages. I don't see calls for Python f-strings or async syntax to end up in Go. But something like a list-comprehension syntax I think would be really popular (based on what I saw when it was introduced to Python, originally 'why' but now many people's favorite feature). Interesting how Python is now considered a huge pile of features, considering how long it took for things like a ternary operator to end up in the language. Maybe Go is in it's Python 1.5.2 lifecycle stage ;)
- Cthulhu_ 7y ago
  > You get to pick and choose. In theory; in practice though, you need to have all developers (and yourself) strictly aligned on what features to use. In my limited experience with Scala and working with Scala developers, there's as many styles and preferences as there are developers on a codebase.
- jerf 7y ago
  "Interesting how Python is now considered a huge pile of features" I've been tracking Python since 1.5.2 was common and 2.0 was just coming out. For me, it's not been a problem because the new features were added incrementally. But I've seen new programmers try to come at it in 2019 and 2020, swallowing what I got spread out in ~15 major releases over ~20 years all in one big chunk, and learning even the core language now is definitely "a pile of features", to say nothing of the various library ecosystems. I don't even mean this necessarily as a criticism per se, just a description. It's not an easy-to-learn language anymore, and I don't recommend that people model it as such. It's a power tool for developers, not an easy learning language. And, yes, the "you can pick and choose the features" is a non-starter in a team environment. At best, the team can pick and choose, and that takes a rather strong hand and alignment to achieve. It is a very common case that you'll just end up with what your teammates use.
  - downerending 7y ago
    That's a good point. One of the original appeals of Python is that it started as a pedagogical language, and it shows. Even Python2 today still has most of that feel. Python3 on the other hand, just isn't. You're pretty much required to constantly deal with Unicode and its related issues, and this is a burden for beginners. In Python2, you can kick that can down the road almost indefinitely.
downerending 7y ago
I have pretty good mastery of Python2, and in my hands, a few lines of Python would blow up into a page of Go. This really frustrated me and was one of the reasons I bounced off of it. Other reasons: no fork(), poor make integration, weird source dir structure, and I once caught one of the go tools trying to write system files in /usr/lib or something (which fortunately failed on permissions). Can't recall the specifics of that last, and perhaps it was somehow misconfigured. Python2 has a good feature set, and one of the downsides of Python3 is that it greatly enlarges what you need to know without much expanding what you can do. It's a real problem, and it would be a shame if it ended up where C++ is these days. ("Poor Joe--a C++ spec fell off a high shelf and killed him...")

peteretep 7y ago

> it sounds like he had good reasons Meh. > Subversion-to-git conversion of the Gnu Compiler Collection history, at over 280K commits (over 1.6M individual change actions), was the straw that broke the camel’s back. Using PyPy with every optimization on semi-custom hardware tuned for this job still yielded test conversion times of over 9 hours, which is death on the recipe-debugging cycle. Just how often does one need to convert the Gnu Compiler Collection from Subversion to Git in under 9 hours?

downerending 7y ago
I was wondering that, too. My guess is that he's using this as a (large) set of test data to test his "recipes" and has to do that a lot. Having to run many nine-hour trials serially is painful. Not sure I'd have bothered, but it's a defensible choice. And produced this interesting example of comparable implementations.
pdonis 7y ago
> Just how often does one need to convert the Gnu Compiler Collection from Subversion to Git in under 9 hours? For this particular repository, once the conversion is complete, it's complete. But there are plenty of other old projects out there with large repositories in old version control systems. Reposurgeon is a general tool. Also, one repository conversion does not mean one run of reposurgeon and you're done. Reposurgeon is meant to be run multiple times since each run is likely to uncover issues that have to be addressed, and the way to address them is to do another run with updated parameters. Reposurgeon also has an interactive mode where you can explore a repository and test the results of various possible transformations.

gameswithgo 7y ago

When you have a system that is at big scale, even a 2x speedup can relieve a huge amount of problems. I think that Go, which is very inexpressive for modern languages, only expanded the code size that much, should make people consider whether we should ever be using interpreted, dynamic typed languages. Perhaps various JITted/compiled and statically typed languages can offer the same productivity on medium to large sized projects while just being a lot faster on top of it. People who love Python may especially enjoy Nim, or F#

melling 7y ago
If you like F#, it’s based on OCaml.
nimmer 7y ago
Nim is the closest language to Python. Translating to Python is a breeze and you get speeds comparable with pure C.
- downerending 7y ago
  You seem like a Nimmer. What would you describe as the biggest downsides of Nim wrt Python? (aside from the fact that it's currently a niche language.)
  - nimmer 7y ago
    The only two that I can think of are due to static typing: You cannot enforce types only where you want, like mypy. Yet, type inference and automated conversions help quite a bit. There's an unofficial REPL in the compiler but you cannot do dir() or tab-complete methods/procs. You have to rely on documentation, nimsuggests or IDEs instead.
downerending 7y ago
Having done a significant amount of programming in both Python-like and statically typed languages, I think the idea that they offer similar productivity (i.e., time to correct, maintainable implementation) is simply false. And in a number of cases I've turned an existing program in a language like C or Java into a faster program in Python. One of the benefits of a fluent language is that you can explore more of the possibility space and look for better solutions. That's difficult and costly in static languages.