10 ms·

> The higher-order function style also tends to be much slower, so when writing a loop there's a decision burden every time between saving a line or two and wri

> The higher-order function style also tends to be much slower, so when writing a loop there's a decision burden every time between saving a line or two and writing fast code. (TIMTOWTDI is not really the Go ethos.) And programmers usually prefer concision, so the net result is often slow, HOF-heavy code. This is very real. We do it all the time in JavaScript: Object.entries({...spreadMerge, ...secondSpread}).map().filter().find(x) Not once do I think about how many things are initialized, iterated over, etc. Rarely would I do something like this in Go. Whether it matters is a good question, but the point is true: language facilities make you care more or less about performance depending on what it does.

jolux 5y ago
This is a problem of optimization, not language design. There’s no inherent reason a chain of functions like that has to be less performant than a for-loop. You need high-performance iterators in the language, and a good compiler.
- Cthulhu_ 5y ago
  One of Go's objectives is to have a simple and fast compiler; emphasizing simple and readable code will ensure that. It's one reason why type arguments were put off for as long as they have been.
  - dmitriid 5y ago
    `.map().filter().find(x)` is significantly more simple and readable than for-looping for every small thing in Go.
    - throwaway894345 5y ago
      I disagree. Fewer characters != simpler or even more readable. Map/filter/etc are certainly more clever, and I enjoy writing clever code, but I reserve it for my personal projects bprecisely because it isn’t so clear. Note that Python has these first class functions as well as various kinds of comprehensions and still it’s frowned upon to use anything more complex than the simplest comprehensions.
      - creata 5y ago
        Alright, if it's not too much trouble, can you show me what the equivalent of the following code would be in Go? I think this is extremely clear as-is, but I'd love to see how you can apparently make it even clearer by avoiding iterator combinators. https://play.rust-lang.org/?version=nightly&mode=release&edition=2018&gist=fbb03d43417b13234c9146c6a205f6c2 https://play.rust-lang.org/?version=nightly&mode=release&edi...
        throwaway894345 5y ago
        Personally I think this is clearer, but it's quite contrived either way: https://play.golang.org/p/FpUWwm5yZ8n https://play.golang.org/p/FpUWwm5yZ8n
        creata 5y ago
        Nice! Out of curiosity, if the iterators weren't of the same type (e.g. one was an iterator over the values of a hashmap and the other was over the items in a list) what would you have used instead of for _, m := range maps { ... }?
        coder543 5y ago
        If you could provide the hypothetical Rust code you're referring to, that would probably make your question more clear here. I know both Rust and Go rather well, and I'm confused about which direction you're wanting to take the code.
        throwaway894345 5y ago
        Just as with Rust, you'd have to convert both maps to the same type before you could iterate them over together (or chain them, in Rust).
      - jolux 5y ago
        Map and filter are extremely simple, far simpler than a for-loop. They can only do two things: transform items in a collection, and remove items from a collection. After mapping you know for sure that a collection will still have the same number of elements. After filtering you know that all the elements remaining still have the same type. For-loops can do anything. It’s a question of what you’re most familiar with and used to.
    - patates 5y ago
      I really like chaining array methods but you should also never forget it's incredibly easy to introduce crazy computational complexity with them.
      - coder543 5y ago
        Rust has lazy iterators, so a chain of iterator steps like this has the same complexity as a single for loop over the iterator. There's no reason Go couldn't do that same thing.
        stouset 5y ago
        In fact, Rust's version compiles down to pretty much the optimal looping form. It will even vectorize your iterations using SIMD.
        throwaway894345 5y ago
        Rust is seriously cool in this regard, but Go aspires to be different, namely more C-like and less C++-like. This means a dumber, faster compiler that is easier to intuit about, for better or worse. Rust is such an amazing piece of engineering (moreso an amazing process and culture for producing an amazing language), but I think Go’s approach is right for a whole lot of software—lots of things don’t need to be insanely performant (lots of things are chugging along in Python, which runs 2-3 orders of magnitude slower than Go) but we do need to be able to iterate on them quickly in a development context involving dozens or hundreds of other people.
      - jolux 5y ago
        Not if you have solid lazy iterator support in the language. If they’re strict and every link in the chain is adding O(n) to the complexity then yes, this is a problem.
    - twobitshifter 5y ago
      In the case we’d need a little context of what the map, filter, and find operations are, but if map is multiple by 2 and filter is less than 200 you can just replace the filter and map with a single if in the for loop that performs the multiplications and check and return immediately when you find X. That’s pretty obvious, always takes only one run, doesn’t reallocate, and doesn’t add concepts, which is of course adding complexity.
      - stouset 5y ago
        Yes and we could replace all of that with an implementation in assembly which both compiles and runs faster. But we don't do that because for the entire history of computing, we've come to realize that abstractions enable us to reason about programs at a higher level without having to be bogged down in irrelevant details and that getting those details right once lets us avoid the inevitable bugs that occur while reimplementing them thousands of times. By absolutely every objective measure we know in software engineering let triples = ints.map(|x| x * 3) is strictly superior to var triples = []int for value := range ints { triples = append(triples, value * 8) } The fact that this even has to be argued is just bonkers to me. The second one is trivial to introduce bugs into, reading it requires mentally filtering out 90% of the code as irrelevant minutiae, and it's even worse from a performance perspective unless you remember each and every time to allocate a result array of the appropriate length to avoid reallocation. None of the details of allocating a result array or iterating over indices/values are important from the purpose of expressing the problem to be solved, and yet every time someone has to read this solution those things are of greater visibility than the actual business logic. Worse, repetitive and unnecessary boilerplate like this hides bugs. Did you catch the fact that I'm multiplying by the wrong value in the golang version? Did you catch that I'm accidentally tripling the indices and not the values themselves? If you did, would you at least acknowledge that the first bug is much easier to identify in the first example than the second, and that the second bug is strictly impossible? As an added bonus, the first one (in Rust at least) is automatically vectorized using SIMD for optimal performance. This happens even as you chain additional operations to arbitrary complexity. If you still somehow think the second version is "better", then whatever your argument is can trivially be repurposed to say this version is better still: int* triples = malloc(sizeof(int) * array_len); for (int i = 0; i < array_len; i++) { triples[i] = array[i] * 3; } If those details of creating a result array and explicitly enumerating are important, why isn't maintaining your own counter and directly indexing? At least the C version somewhat encourages you to allocate a result array of the correct size.
  - kubb 5y ago
    This is the decade-long argument of "let's sacrifice all our programmers to the devil so that we don't have work as hard on the compiler". I don't buy it, and it has never been demonstrated that the cumbersome style of writing go programs is helping compilation times, yet people keep saying it as if it's a fact.
    - jolux 5y ago
      The cumbersome style seems to have become a strong part of the culture, almost a masochism about the limited features of the language. I find this ironic because I believe Go was intended to be highly pragmatic originally but it seems to have evolved an almost Scheme-like dogmatism about language complexity.
      - Serow225 5y ago
        Thank you for naming this. I've always struggled with how to convey this to people.
      - skywhopper 5y ago
        Do you have specific examples you’re thinking of? Calling it masochism is weirdly insulting. Adding complexity is a one-way street. Languages live for decades. There’s no reason to rush adding features in a half-baked way, and the tradeoffs in terms of compiler complexity are very real.
        stouset 5y ago
        Virtually every large go program I've read has asymptotically reached 50%+ of its code being if res, err := fn(...); err != nil { fmt.Errorf("Look ma, I'm a human exception handler: ", err) } This is held up as a positive thing, because handling errors is explicit and good. Somehow, the Rust equivalent of `?` is implicit and bad, despite desugaring to almost literally the exact same thing and being actually impossible to forget or overlook due to the return type of the function enforcing a Result. In the same breath, devs will assert that the language is "low boilerplate" (despite 50%+ LOC being just one type of boilerplate) because Error Handling is Important. Which seemingly ignores that the important part of error handling is actually ensuring that errors are handled and not the mechanistic act of self-flagellation by wrapping every line of code in nearly-identical error handling ceremony. Which is unfortunate because golang doesn't do so great a job with the first part. It's surprisingly easy to actually forget to do something with the error while carrying on with a return value whose semantics are undefined (unlike Rust's Result, where you can have an error or a value and not both).
- throwaway894345 5y ago
  This seems like the “sufficiently smart compiler” argument. A sufficiently complex compiler could certainly optimize this away, but building and maintaining such a compiler while also keeping compilation times low is an enormous challenge. Rust does an impressive job here, but Rust was built from the ground up to handle these problems and it also trades compile times to achieve those optimizations.
  - pjmlp 5y ago
    Not really on this case, a common optimization pass on functional programming languages since the 70's is to flatten such kind of calls. Rust long compile times are mainly caused by LLVM, and the amount of IR that is given to it.
    - bollu 5y ago
      Can you point me to a paper from the 70s that explains how to flatten such calls? The best that I know is "stream fusion, to completeness" which was published in something like 2019!
      - deleted 5y ago
        [deleted]
      - pjmlp 5y ago
        You can get the book "Lisp on Small Pieces" from 1994 https://www.amazon.com/exec/obidos/ASIN/0521562473/acmorg-20 https://www.amazon.com/exec/obidos/ASIN/0521562473/acmorg-20 It is not 70's, but 26 years should be enough to get hold of such knowledge. Or "Modern Compiler Implementation" from 1998, a little more young. https://www.cs.princeton.edu/~appel/modern/ https://www.cs.princeton.edu/~appel/modern/ Or rather "The Implementation of Functional Programming Languages", from 1986, now getting closer to 70's. https://www.microsoft.com/en-us/research/publication/the-implementation-of-functional-programming-languages/ https://www.microsoft.com/en-us/research/publication/the-imp... I can still have a look into my SIGPLAN paper collection to dust off some ML papers.
    - 5y ago
- masklinn 5y ago
  > This is a problem of optimization, not language design. It is also a problem of langage design though: JS’s HoFs are spec-defined as returning Arrays, which makes fusion way more complicated. Internal / external iteration also have different optimisation profiles e.g. Rust initially used internal iteration, switched to external because that was much more conducive to optimisations, but re-introduced an internal supplement because some combinations / patterns optimisé much better through internal iteration.
  - jolux 5y ago
    This is true, it does depend on how the implementation works.
  - creata 5y ago
    > switched to external because that was much more conducive to optimisations Source? I thought conventional wisdom was that internal iteration was much easier for compilers to digest.
    - masklinn 5y ago
      https://mail.mozilla.org/pipermail/rust-dev/2013-June/004364.html https://mail.mozilla.org/pipermail/rust-dev/2013-June/004364... Was probably the main starting point for that change. Though I misremembered performances being the main gist of the switch, ergonomics and flexibility were. With https://www.reddit.com/r/rust/comments/5ez38g/rusts_iterators_are_inefficient_and_heres_what_we/ https://www.reddit.com/r/rust/comments/5ez38g/rusts_iterator... as the… not counterpoint but reintroduction of internal iterators. Linking to the Reddit thread rather that the article directly because lots of comments from important rust folks.
jrimbault 5y ago
Rarely does a week go by without me wishing (for a few seconds) those js methods were all lazy.

jolux 5y ago

This is a problem of optimization, not language design. There’s no inherent reason a chain of functions like that has to be less performant than a for-loop. You need high-performance iterators in the language, and a good compiler.

Cthulhu_ 5y ago
One of Go's objectives is to have a simple and fast compiler; emphasizing simple and readable code will ensure that. It's one reason why type arguments were put off for as long as they have been.
- dmitriid 5y ago
  `.map().filter().find(x)` is significantly more simple and readable than for-looping for every small thing in Go.
  - throwaway894345 5y ago
    I disagree. Fewer characters != simpler or even more readable. Map/filter/etc are certainly more clever, and I enjoy writing clever code, but I reserve it for my personal projects bprecisely because it isn’t so clear. Note that Python has these first class functions as well as various kinds of comprehensions and still it’s frowned upon to use anything more complex than the simplest comprehensions.
    - creata 5y ago
      Alright, if it's not too much trouble, can you show me what the equivalent of the following code would be in Go? I think this is extremely clear as-is, but I'd love to see how you can apparently make it even clearer by avoiding iterator combinators. https://play.rust-lang.org/?version=nightly&mode=release&edition=2018&gist=fbb03d43417b13234c9146c6a205f6c2 https://play.rust-lang.org/?version=nightly&mode=release&edi...
      - throwaway894345 5y ago
        Personally I think this is clearer, but it's quite contrived either way: https://play.golang.org/p/FpUWwm5yZ8n https://play.golang.org/p/FpUWwm5yZ8n
        creata 5y ago
        Nice! Out of curiosity, if the iterators weren't of the same type (e.g. one was an iterator over the values of a hashmap and the other was over the items in a list) what would you have used instead of for _, m := range maps { ... }?
        coder543 5y ago
        If you could provide the hypothetical Rust code you're referring to, that would probably make your question more clear here. I know both Rust and Go rather well, and I'm confused about which direction you're wanting to take the code.
        throwaway894345 5y ago
        Just as with Rust, you'd have to convert both maps to the same type before you could iterate them over together (or chain them, in Rust).
    - jolux 5y ago
      Map and filter are extremely simple, far simpler than a for-loop. They can only do two things: transform items in a collection, and remove items from a collection. After mapping you know for sure that a collection will still have the same number of elements. After filtering you know that all the elements remaining still have the same type. For-loops can do anything. It’s a question of what you’re most familiar with and used to.
  - patates 5y ago
    I really like chaining array methods but you should also never forget it's incredibly easy to introduce crazy computational complexity with them.
    - coder543 5y ago
      Rust has lazy iterators, so a chain of iterator steps like this has the same complexity as a single for loop over the iterator. There's no reason Go couldn't do that same thing.
      - stouset 5y ago
        In fact, Rust's version compiles down to pretty much the optimal looping form. It will even vectorize your iterations using SIMD.
        throwaway894345 5y ago
        Rust is seriously cool in this regard, but Go aspires to be different, namely more C-like and less C++-like. This means a dumber, faster compiler that is easier to intuit about, for better or worse. Rust is such an amazing piece of engineering (moreso an amazing process and culture for producing an amazing language), but I think Go’s approach is right for a whole lot of software—lots of things don’t need to be insanely performant (lots of things are chugging along in Python, which runs 2-3 orders of magnitude slower than Go) but we do need to be able to iterate on them quickly in a development context involving dozens or hundreds of other people.
    - jolux 5y ago
      Not if you have solid lazy iterator support in the language. If they’re strict and every link in the chain is adding O(n) to the complexity then yes, this is a problem.
  - twobitshifter 5y ago
    In the case we’d need a little context of what the map, filter, and find operations are, but if map is multiple by 2 and filter is less than 200 you can just replace the filter and map with a single if in the for loop that performs the multiplications and check and return immediately when you find X. That’s pretty obvious, always takes only one run, doesn’t reallocate, and doesn’t add concepts, which is of course adding complexity.
    - stouset 5y ago
      Yes and we could replace all of that with an implementation in assembly which both compiles and runs faster. But we don't do that because for the entire history of computing, we've come to realize that abstractions enable us to reason about programs at a higher level without having to be bogged down in irrelevant details and that getting those details right once lets us avoid the inevitable bugs that occur while reimplementing them thousands of times. By absolutely every objective measure we know in software engineering let triples = ints.map(|x| x * 3) is strictly superior to var triples = []int for value := range ints { triples = append(triples, value * 8) } The fact that this even has to be argued is just bonkers to me. The second one is trivial to introduce bugs into, reading it requires mentally filtering out 90% of the code as irrelevant minutiae, and it's even worse from a performance perspective unless you remember each and every time to allocate a result array of the appropriate length to avoid reallocation. None of the details of allocating a result array or iterating over indices/values are important from the purpose of expressing the problem to be solved, and yet every time someone has to read this solution those things are of greater visibility than the actual business logic. Worse, repetitive and unnecessary boilerplate like this hides bugs. Did you catch the fact that I'm multiplying by the wrong value in the golang version? Did you catch that I'm accidentally tripling the indices and not the values themselves? If you did, would you at least acknowledge that the first bug is much easier to identify in the first example than the second, and that the second bug is strictly impossible? As an added bonus, the first one (in Rust at least) is automatically vectorized using SIMD for optimal performance. This happens even as you chain additional operations to arbitrary complexity. If you still somehow think the second version is "better", then whatever your argument is can trivially be repurposed to say this version is better still: int* triples = malloc(sizeof(int) * array_len); for (int i = 0; i < array_len; i++) { triples[i] = array[i] * 3; } If those details of creating a result array and explicitly enumerating are important, why isn't maintaining your own counter and directly indexing? At least the C version somewhat encourages you to allocate a result array of the correct size.
- kubb 5y ago
  This is the decade-long argument of "let's sacrifice all our programmers to the devil so that we don't have work as hard on the compiler". I don't buy it, and it has never been demonstrated that the cumbersome style of writing go programs is helping compilation times, yet people keep saying it as if it's a fact.
  - jolux 5y ago
    The cumbersome style seems to have become a strong part of the culture, almost a masochism about the limited features of the language. I find this ironic because I believe Go was intended to be highly pragmatic originally but it seems to have evolved an almost Scheme-like dogmatism about language complexity.
    - Serow225 5y ago
      Thank you for naming this. I've always struggled with how to convey this to people.
    - skywhopper 5y ago
      Do you have specific examples you’re thinking of? Calling it masochism is weirdly insulting. Adding complexity is a one-way street. Languages live for decades. There’s no reason to rush adding features in a half-baked way, and the tradeoffs in terms of compiler complexity are very real.
      - stouset 5y ago
        Virtually every large go program I've read has asymptotically reached 50%+ of its code being if res, err := fn(...); err != nil { fmt.Errorf("Look ma, I'm a human exception handler: ", err) } This is held up as a positive thing, because handling errors is explicit and good. Somehow, the Rust equivalent of `?` is implicit and bad, despite desugaring to almost literally the exact same thing and being actually impossible to forget or overlook due to the return type of the function enforcing a Result. In the same breath, devs will assert that the language is "low boilerplate" (despite 50%+ LOC being just one type of boilerplate) because Error Handling is Important. Which seemingly ignores that the important part of error handling is actually ensuring that errors are handled and not the mechanistic act of self-flagellation by wrapping every line of code in nearly-identical error handling ceremony. Which is unfortunate because golang doesn't do so great a job with the first part. It's surprisingly easy to actually forget to do something with the error while carrying on with a return value whose semantics are undefined (unlike Rust's Result, where you can have an error or a value and not both).
throwaway894345 5y ago
This seems like the “sufficiently smart compiler” argument. A sufficiently complex compiler could certainly optimize this away, but building and maintaining such a compiler while also keeping compilation times low is an enormous challenge. Rust does an impressive job here, but Rust was built from the ground up to handle these problems and it also trades compile times to achieve those optimizations.
- pjmlp 5y ago
  Not really on this case, a common optimization pass on functional programming languages since the 70's is to flatten such kind of calls. Rust long compile times are mainly caused by LLVM, and the amount of IR that is given to it.
  - bollu 5y ago
    Can you point me to a paper from the 70s that explains how to flatten such calls? The best that I know is "stream fusion, to completeness" which was published in something like 2019!
    - deleted 5y ago
      [deleted]
    - pjmlp 5y ago
      You can get the book "Lisp on Small Pieces" from 1994 https://www.amazon.com/exec/obidos/ASIN/0521562473/acmorg-20 https://www.amazon.com/exec/obidos/ASIN/0521562473/acmorg-20 It is not 70's, but 26 years should be enough to get hold of such knowledge. Or "Modern Compiler Implementation" from 1998, a little more young. https://www.cs.princeton.edu/~appel/modern/ https://www.cs.princeton.edu/~appel/modern/ Or rather "The Implementation of Functional Programming Languages", from 1986, now getting closer to 70's. https://www.microsoft.com/en-us/research/publication/the-implementation-of-functional-programming-languages/ https://www.microsoft.com/en-us/research/publication/the-imp... I can still have a look into my SIGPLAN paper collection to dust off some ML papers.
  - 5y ago
masklinn 5y ago
> This is a problem of optimization, not language design. It is also a problem of langage design though: JS’s HoFs are spec-defined as returning Arrays, which makes fusion way more complicated. Internal / external iteration also have different optimisation profiles e.g. Rust initially used internal iteration, switched to external because that was much more conducive to optimisations, but re-introduced an internal supplement because some combinations / patterns optimisé much better through internal iteration.
- jolux 5y ago
  This is true, it does depend on how the implementation works.
- creata 5y ago
  > switched to external because that was much more conducive to optimisations Source? I thought conventional wisdom was that internal iteration was much easier for compilers to digest.
  - masklinn 5y ago
    https://mail.mozilla.org/pipermail/rust-dev/2013-June/004364.html https://mail.mozilla.org/pipermail/rust-dev/2013-June/004364... Was probably the main starting point for that change. Though I misremembered performances being the main gist of the switch, ergonomics and flexibility were. With https://www.reddit.com/r/rust/comments/5ez38g/rusts_iterators_are_inefficient_and_heres_what_we/ https://www.reddit.com/r/rust/comments/5ez38g/rusts_iterator... as the… not counterpoint but reintroduction of internal iterators. Linking to the Reddit thread rather that the article directly because lots of comments from important rust folks.

jrimbault 5y ago

Rarely does a week go by without me wishing (for a few seconds) those js methods were all lazy.