12 ms·
Fun with Go Iterators
- pdimitar 2y agoMight be interesting to make a library that competes with https://github.com/samber/lo https://github.com/samber/lo?
- gtramont 2y agoBack when the Go team announced generics, I had a go (pun intended) at it: https://github.com/gtramontina/go-extlib https://github.com/gtramontina/go-extlib -- might resurrect it one day. `lo` is pretty comprehensive though.
- mihaitodor 2y agoHere's one: https://github.com/szmcdull/glinq https://github.com/szmcdull/glinq It doesn't do function chaining though.
- mseepgood 2y agoIt's not interesting. Boring people did this a million times, the result is always the same: just use for loops.
- _lvbh 2y ago> Tips for lazy developers > I cannot recommend it, but in case you are too lazy for repeating lo. everywhere, you can import the entire library into the namespace. import ( . "github.com/samber/lo" ) TIL. Crazy
- s6af7ygt 2y agoI just hate the fact that Go is super simple and clear but people try to make it complex with this kind of stuff. :( Makes me sad.
- deleted 2y ago[deleted]
- Savageman 2y agoI like how the author uses a test to run arbitrary code, this is exactly how I do it too!
- TurboHaskal 2y agoLife without REPLs.
- dlock17 2y agoI just use play.go.dev
- gtramont 2y agoUnfortunately the chain approach breaks down when if you need to `map` to a different type, for example. Go does not allow generic typed methods.
- daghamm 2y agoI never understood why that limitation exist. Can someone explain the reason for this?
- jerf 2y agoStraight from the designers: https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#no-parameterized-methods https://go.googlesource.com/proposal/+/refs/heads/master/des...
- thefounder 2y agohttps://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#No-parameterized-methods https://go.googlesource.com/proposal/+/refs/heads/master/des...
- mihaitodor 2y agoI recall reading some details around this on https://blog.merovius.de https://blog.merovius.de. Maybe it was this article: https://blog.merovius.de/posts/2018-06-03-why-doesnt-go-have-variance-in https://blog.merovius.de/posts/2018-06-03-why-doesnt-go-have.... Compilation speed plays a big factor when deciding which features are added to Golang and I think they'd have a hard time maintaining the current compilation speed if they remove this limitation.
- mseepgood 2y agohttps://go.dev/doc/faq#generic_methods https://go.dev/doc/faq#generic_methods
- deleted 2y ago[deleted]
- dilap 2y agoThe Reverse implementation seems off to me -- it runs through the iterator twice, once collecting into a slice, and then a second time filling the same slice in reverse. (So basically the first Collect call is only being used to find the length of the iterated sequence.) I'm not sure about Go conventions†, but I imagine it would be considered better form to only run through the iterator once, reversing the collected slice in-place via a series of swaps. († Are iterators even expected/required to be reusable? If they are reusable, are they expected to be stable?)
- jerf 2y agoThat eliminates one of the main reasons to use this approach. Function chaining as most people write is awful for performance because it involves creating a separate array for each step in the chain. Given that most programs are actually more memory-blocked than CPU blocked, this is a bad tradeoff. Composing Go iterators, and composing iterators in general, is preferable because it doesn't have to create all the intermediate arrays. A bad reverse wrecks that back up. Still, you need the option, and while reverse is one of the more common iterators, it's still usually avoidable if you need to. But if at all possible I'd suggest a "reverse" type-specialized to slices, and as necessary and possible, type-specialized to whatever other types you are using to actually crawl a value "backwards" rather than collecting a full iterator into a slice. (Then again, I'm not a fan of this approach in imperative languages in general, due to the generalized difficulty in refactoring code written in this style and the fact that observationally, people just don't refactor it once written and it affords a style rife with repetition. One of the most important considerations about code is how easily it can be refactored. In Haskell, this approach is excellent, precisely because it refactors very, very well in Haskell. In imperative languages, it tends not to, thus, serving as another example of why you can't just blindly bring over nice things from one language into another without verifying they haven't become sour in the process.)
- dilap 2y agoYeah, that all makes sense!, but I think it's not relevent to the code in question, which is this: func (i *Iterator[V]) Reverse() *Iterator[V] { collect := i.Collect() counter := len(collect) - 1 for e := range i.iter { collect[counter] = e counter-- } return From(collect) } So this code creates a slice from the iterator in the call to Collect(), and then fills the slice again in reverse by running the iterator again, which I think is wrong (or at least not-ideal). (Your broader point about wanting to avoid creating an intermediate array at all for iterators and using type information to intelligently reverse "at the source" definitely still stands, in a broader context, though.)
- skybrian 2y agoI think it would be more idiomatic to use statements, not expressions. That is, it’s ok to use local variables for intermediate values in a pipeline.
- simiones 2y agoYou often end up with a lot of extraneous variables with no useful names if you do that. A lot of the time the intermediate results in a pipeline are almost meaningless.
- skybrian 2y agoThe main thing is not to do things that would be fine in other languages when they result in complications in the one you’re using. Some people want to write an entire library rather than writing statements. Why? Also, local names can sometimes be useful documentation when the function names don’t really get the point across (perhaps because they’re at a different level of abstraction). Or alternatively, in Go it’s idiomatic to keep them short.
- simiones 2y ago> Some people want to write an entire library rather than writing statements. Why? Because you want to make code more readable, and getting rid of extraneous intermediate results is one way of achieving that.
- integrii 2y agoCall me crazy, but I don't like any of this. Make more named functions. Keep your logic flat and explicit. I believe go wants you to code this way as well. Imagine the horrors this kind of function chaining creates. Actually, you don't have to. It's JavaScript.
- lifthrasiir 2y agoEven JS doesn't really use functional styles that much. In fact, the whole reason for functional styles is the decomposability: any language with easy function chaining like `x |> f |> g |> h` will also have an easy extraction of any part of the chain (say, `x |> fg |> h where fg = f . g`). It is not a good idea to use chaining when the language supports no such feature, as it would be much harder to work with such chains then.
- arethuza 2y ago.Net seems to handle it pretty well with LINQ? https://medium.com/@sanjanasw99/an-in-depth-guide-to-linq-in-net-understanding-implementing-and-utilizing-22eddc92b13a https://medium.com/@sanjanasw99/an-in-depth-guide-to-linq-in...
- lifthrasiir 2y agoLINQ is a very dense syntactic sugar for some selected `IEnumerable` methods. It will surely look like chaining in typical uses and thus is useful, but different from an arbitrary chaining.
- bilinguliar 2y agoWhen Go eventually lets all this horrible syntax sugar into the standard library, we will meet again in the Zig community.
- tapirl 2y agoIt is a sad fact that Go is becoming more and more like JavaScript (and deviating from C).
- 2y ago
- openasocket 2y agoI work with Go a lot at my job, and I definitely prefer functional programming so I chafe at the language a bit. I was excited to incorporate iterators into our code. Unfortunately, I'm also working in an environment where we are memory constrained, so on our hot path we need to not allocate at all. I tried playing with the iterators a bit, and couldn't get it to produce something that didn't allocate. I got close, but as much as a tried I couldn't get below 1 allocation per loop (not per loop iteration, per loop). Which in any other setting would be fine, but not for our use case.
- _lvbh 2y agoThis is probably one of the areas where Zig shines. I'm mostly a Gopher but reading Zig is just as easy while ensuring that no allocations are hidden
- kunley 2y agoI know complaining about downvote is not the right thing, but the above is someone else's comment, not mine. Why was it downvoted (I see it grayed) ? It's an useful information without zealotry or "we-know-betterism".
- atomic128 2y agoI have not looked at Go's iterator range loop implementation yet. So somebody tell me if I'm wrong here. My guess is that Go is probably wrapping the body of the range loop into a closure, and passing that closure into the iterator function as the yield function. A break in the body of the loop becomes a "return false" in the closure. The allocation is probably the closure environment struct (giving access to variables prior to the range loop). This closure might escape through the iterator function so Go can't just put the environment struct onto the stack, it has to escape to the heap. The cost is small but it's not free. Usually, not having to think about the allocation is an advantage. In the rare case can't afford the iterator, do it differently. Go is great.
- saghm 2y agoI've seen issues in Go codebases a couple times where a _lot_ of effort has been spend trying to track down allocations and optimize memory usage. It sounds like the parent comment is describing writing new code striving to avoid allocations, which probably isn't something that Go is that much harder for than similar languages, but I feel like it's one of the more ambiguous languages in terms of the amount of context needed to be able to identify if a given variable is allocated on the heap or not. A pointer might be a pointer to the stack, or a pointer to the heap, or a pointer to an interface that might _also_ have a heap allocation on the concrete-typed value behind that. If you see a slice, it might be a heap allocation, or it might be a reference to a static array, or it might be a reference to another slice...which has the same possibility to be either a heap allocation, a reference to a static array, or just be another link in the chain to a different slice. This is a place where I feel like the type of simplicity that Go touts doesn't actually feel like it's optimizing for the right thing. Having a single type for all pointers certainly has a sort of abstract simplicity to it, but I feel like it doesn't actually make things simpler when using it in the long run. My perspective is that "conceptual" simplicity is a balancing act between not having too many concepts but also not having concepts being too confusing individually, and I'm surprised that Go is used for domains like needing to completely avoid allocations in a hot path when the language doesn't really feel like it's designed to make easy.
- Spivak 2y agoIt's funny the author throws a dig at Python for its syntax that actively discourages this kind of code. Like… my guy you're not taking the hint. Python makes things it doesn't want you to do ugly as sin. It's why lambda is so awkward and clunky.
- deleted 2y ago[deleted]
- rolux 2y agoYes. The following is a lot more concise: a = [1, 2, 3, 4] print([v*v for v in reversed(a) if v*v % 2 == 0])
- libria 2y agoI think the above is a good idea of what's wrong with python (and Go), because in your example the list comp is evaluated in what seems to be this order: FOURTH-> print([THIRD-> v*v for v in FIRST-> reversed(a) if SECOND-> v*v % 2 == 0]) Which is all over the place. I'd rather see: a = [1, 2, 3, 4] a = reversed(a) a = [v*v for v in a] a = [w for w in a if a % 2 == 0] print(a)
- alfons_foobar 2y agoThis. I often use generator expressions for the intermediate values (so I don't allocate a new list for each step), but I find this to be much more readable.
- xnacly 2y agoI get that, i still dont like to write f5(f4(f3(f2(f1())))) instead of writing f1().f2().f3().f4().f5()
- mbrumlow 2y ago> My issue with the go way of iterators is, you can’t chain them like you would in JavaScrip Because it’s not JavaScript, and that is a good thing.
- eweise 2y agoBut not because you can't easily chain functions. That's just a deficiency in the language design.
- mbrumlow 2y agoNo. It’s a feature. Chaining makes some of the worst unreadable code.
- Varriount 2y agoIt always bugs me when I see that pattern in JavaScript, because each `map`, etc. call is an array allocation. Yeah, yeah, I know that the VM's memory allocators are likely optimized for fast allocation, but that doesn't make the allocation completely free.
- binary132 2y agoI’m trying to understand whether this is intended to make Go seem bad or whether it’s just coming across that way to me.
- ugjka 2y agoI looked at the code and it was giving me a headache, been messing around with Go for a decade, i do not like this
- binary132 2y agoYeah, same, used Go fulltime professionally for 10 years, not into this newer stuff.
- arp242 2y agoPeople have been trying to do this sort of thing in Go for as long as I remember; it's nothing new and has not and most likely never will gain much uptake. The stages of learning a language are something along the lines of: 1. Force language patterns from previous language 2. Frustration 3. Write library to make it easier 4. Anger 5. Acceptance
- pragma_x 2y agoI absolutely love it when we can take advantage of Go's type system and add additional traits and behaviors to existing types like this. That said, I noticed something odd here. In order for a module like this to really shine, I think all these operations need to be functionally pure. Right now, some of these mutate the iterator's `iter` method mid-stream, which is about as side-effect-ful as you can get. ``` func (i Iterator[V]) Map(f func(V) V) Iterator[V] { cpy := i.iter i.iter = func(yield func(V) bool) { for v := range cpy { v = f(v) if !yield(v) { return } } } return i } ``` Unless I'm misreading that, `i.iter` has new behavior after this call. A better way would be to return a new _iterator_ with the custom iter behavior instead. ``` func (i Iterator[V]) Map(f func(V) V) Iterator[V] { // create a fresh iterator around a custom closure (NewIterator() is hypothetical in this case) return NewIterator(func(yield func(V) bool) { for v := range i.iter { v = f(v) if !yield(v) { return } } }) } ```
- tantivy 2y agoThis flagged for me right away too. I would be badly surprised if a Map-style chained method mutated the memory of the receiver.
- kunley 2y agoMight be written in a hurry and maybe the author was thinking that not allocating new Iterator, which is in fact wrapper around iter.Seq, will produce less heap garbage. But I guess it's the same amount of garbage, because the size of Iterator is the same as iter.Seq which is allocated anyway, it's just different type of the object being discarded
- relistan 2y agoIf they allocated, people would complain about that. If they don’t, people complain about mutation. :shrug: Personally, the article’s implementation seems fine to me. The iter is a private field of a throwaway struct created on the fly in order to support chaining. If anyone is then relying on the (private) contents of that struct, I think that’s user error. I can’t see personally why you’d do that.
- deleted 2y ago[deleted]
- kubb 2y agoGo people will do this and they'll be content: a := []int{1,2,3,4} it := slices.All(a) it = slices.Reverse(it) it = slices.Map(it) it = slices.Filter(it, func(i int) bool { return i % 2 == 0 }) slices.ForEach(it, func(i int) { fmt.Println(i) }) I don't judge the Go enjoyers, but I prefer writing TypeScript to Go which says it all. Type-inferred arrow lambda for function arguments would go such a long way in making this code nicer... And not make compilation slower at all. it = slices.Filter(it, i => i % 2 == 0) slices.ForEach(it, i => fmt.Println(i))
- jerf 2y agoNo, Go programmers would write a := []int{1, 2, 3, 4} out := []int{} for idx := range a { val := a[len(a)-idx-1] if mappedVal := Map(val); mappedVal % 2 == 0 { out = append(out, mappedVal) fmt.Println(mappedVal) } } In modern Go I might write a reverse iterator, that index is a bit hairy, which would cut out the 'val :=' line, but as there is not yet a standard library option for that I'll leave it out.
- kubb 2y agoMany Go programmers would claim this is cleaner and more understandable, unfortunately I'm not one of them.
- int_19h 2y agoThe bigger problem is that it's not efficiently composable. If your caller needs to additionally filter `out`, that's another loop with a copy.
- deleted 2y ago[deleted]
- thegeekpirate 2y agoThe proper term is deforestation https://en.wikipedia.org/wiki/Deforestation_(computer_science) https://en.wikipedia.org/wiki/Deforestation_(computer_scienc..., and I have seen Go libraries which do this
- vyskocilm 2y agoShameless plug. I had experimented with Go iterators a while ago and did a https://github.com/gomoni/it https://github.com/gomoni/it It was updated to 1.23, so it is as idiomatic as I can get. And yes it has a map method between two types. Just a single simple trick used.
- kunley 2y agoNice rewrite for 1.23. Btw, just sent you a PR for a typo in the readme.
- indulona 2y ago> My issue with the go way of iterators is, you can’t chain them like you would in JavaScript You are not supposed to chain them. This addiction to try and chain everything everywhere all the time is so freaking weird and has been for a very long time. Not only you are completely losing grasp on what is going on and write code prone to errors, but you are making it unreadable for other people that will be maintaining or just reading your code who will come long after you are gone from the company or abandon your library. This is where Go's simplicity approach and splitting each action into its own for loop or block of code is a godsend for maintainability.
- eweise 2y agoIts really not. For example, a map function tells you that there will be the exact number of outputs as inputs. A for loop doesn't have any guarantees. You have to read each line inside the loop to understand what its doing. In practice, having to be so explicit causes many more issues. I've never experienced as many mishandled errors on java projects as I have in Go.
- AndyKluger 2y ago1. That's a good looking Hugo theme! 2. Implicitly chain everything all the time! In Factor, you might do it as: reverse [ sq ] [ even? ] map-filter [ . ] each Or with a little less optimizing: reverse [ sq ] map [ even? ] filter [ . ] each The least obvious thing is that the period is the pretty-print function.
- qudat 2y agoWe just released a go pkg that uses the new iter pkg. We were so excited by the interface in large part because of how simple iterators are to use. https://github.com/picosh/pubsub/blob/main/pubsub.go#L18 https://github.com/picosh/pubsub/blob/main/pubsub.go#L18 We have seen in other languages like JS and python the power of iterators and we are happy to see it in Go
- tpoacher 2y agoSince the article is making a not-so-subtle jab at python being unable to do chain operations, I'm making my annual rounds to point out that implementing simple, straightforward chain functionality in python is as simple as a two-line function definition: def chain( Accumulant, *Functions_list ): for f in Functions_list: Accumulant = f( Accumulant ) return Accumulant https://sr.ht/~tpapastylianou/chain-ops-python/ https://sr.ht/~tpapastylianou/chain-ops-python/
- icar 2y agoThis reminds me of RxJS (https://rxjs.dev/ https://rxjs.dev/)