11 ms·
Iterators in Go: A proposed extension
- packetlost 3y agoNot sure how I feel about this particular approach to iterators. I'd love for there to be some generate way to `range` over a generic collection, but idk about this.
- liquidgecka 3y agoHonestly I would just prefer an interface that can be implemented (not this interface.. one like it that is better thought out). type Iterator interface{} { // Returns the next object and true, or nil and false if iteration has ended. func NextIteration() (interface{}, bool) } Even better (in my opinion) if it also adds a finalizer function to help clean up resources used during iteration and some kind of error handling.. but that might be asking too much given my track record of getting golang improvements approved. I still want else on for statements dang it! =)
- kevmo314 3y agoThis approach feels quite invasive compared to adding a standard interface that `range` accepts. Some discussions have happened before, eg https://github.com/golang/go/discussions/54245 https://github.com/golang/go/discussions/54245 A standard interface feels a cleaner than these nested callback functions.
- deleted 3y ago[deleted]
- christophilus 3y agoThe function-based approach is kind of nice because you can use defer for cleanup, and you don’t have to write an entire struct and track state via props, etc. I kind of like this proposal.
- earthboundkid 3y agoThe interface version is nicer for consumers in a world where you can’t range over a function, but the function version is way easier to write, so once you can range over functions, the advantages of the interface versions mostly go away.
- downvotetruth 3y agoBut what about gonerators? /s
- anonymoushn 3y agoThis proposal is about generators.
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- benatkin 3y agoYes please, goroutines and gonerators.
- Someone 3y agoIf go had monads…
- zer8k 3y ago`Gonerator` is the pattern for side-effect only deleting a list. Just implement gonerator and your lovely list will delete itself. for k := i.getNext() { yeet k } I'm willing to negotiate on `yeet` but its close enough to `yield` it feels natural.
- galkk 3y agoIterators are great, I really loved simplicity of `yield` in c# and python. Go's proposed syntax is god awful. Why they are doing that to devs and themselves? Should we expect it to improve in like 15-20 versions, like it happened with any and generic mix/max functions?
- cpuguy83 3y agoIt requires people to comment, on the issue where it is being discussed, in order to improve.
- joneholland 3y agoThis is a weird approach. They are essentially manually implementing the “yield return” functionality from c#. C# defines an interface IEnumerable<T> with a “next” method, and originally you’d need to implement this interface which was a bit of boiler plate. They added “yield return” which would create compiler generated syntax sugar to return a generator function like this go proposal, but your code looked identical to a non iterative version. The go proposal feels like the worst of both worlds and like the team misunderstood the purpose.
- anonymoushn 3y agoThe proposal is more powerful than the C# "yield return" functionality because it allows the yield function to be passed to subroutines.
- jcparkyn 3y agoHow is that better than the C# solution of just making the subroutine return an IEnumerable, then yielding its elements in the calling function? (aside from maybe marginally better perf characteristics)?
- anonymoushn 3y agoIt works with subroutines that don't know anything about iterators or yielding but do know about first-class functions. It's also less boilerplate, since there isn't a `yield from`, I guess.
- agent281 3y agoIt seems like a different kind of boilerplate: instead the functions need to take a yield so you are threading the function through you program. I don't know if it's better to expect a return type or to force a function argument. I lean towards the former, but that might be because of familiarity.
- imiric 3y agoI'm also not seeing why this is necessary. Range over int is pure syntax sugar, and while range over func could be useful, iterators can be easily implemented with current Go features. I love Go because of its simplicity, and because it strictly adheres to the "there's only one way to do it" mantra. Adding syntax sugar muddles this, and introducing another keyword and construct makes the code less readable. I hope that this proposal is rejected, and that it's not a sign of Go becoming yet another kitchen sink language.
- emmelaich 3y agoI'm a go newb, but would you do this with channels and gofuncs now? The readability this provides is considerable.
- d1str0 3y agoThats how I would do it. You can have a goroutine send on a channel (buffered or non buffered), and then another function can consume the chan as needed which lets the goroutine generate results as needed as well.
- slashdev 3y agoAnd make a simple iterator into a multithreaded program with channels and goroutines? Slower. More complex. Why?!? Just call your function in a loop.
- Groxx 3y agoWhich is to say: yes! Many do it with channels currently. It's awful, but they do it.
- konart 3y ago> Slower Why would that be slower?
- Mawr 3y agoThe design of channels is constrained by the need to support concurrency. If you do not need concurrency, using something designed to support it will be suboptimal. And because it's specifically concurrency, the overhead will be substantial. ... that's the theory behind it. For more grounded explanations, see [1] & [2]. [1] https://news.ycombinator.com/item?id=29510751 https://news.ycombinator.com/item?id=29510751 [2] https://ewencp.org/blog/golang-iterators/index.html https://ewencp.org/blog/golang-iterators/index.html
- Groxx 3y agoI'm guessing this is following the same / highly similar motivation as rsc's coroutines post about a month ago: https://research.swtch.com/coro https://research.swtch.com/coro I welcome people embracing function closures more, as I feel they're a drastically under-used language tool (particularly in Go where everyone crams everything into single-func interfaces which are almost exclusively far worse). More exposure to it helps people become comfortable with it, and now that we have generics it's finally not a pile of pain and agony at all times. And just using functions makes it very flexible / doesn't repeat the "X for me but not for thee" issues of past Go versions. I'm not entirely sure how I feel about this proposal though. It kinda feels like it's trying to put a round peg into a square hole. If you try hard enough, everything can go in the square hole! But that doesn't mean it's a good fit.
- anonymoushn 3y agoIt would be nice if the proposal defined the semantics of getting single item from the iterator (or detecting that it is out of items) and exposed an api for this, then defined the behavior of range as equivalent to some pattern using these primitives. As is the proposal says "there should be some coro.Pull type thing" which is a bit terse and says nothing about whether the implementation of coro.Pull will be fast or whether it will be built on range + channels
- znkr 3y agohttps://research.swtch.com/coro https://research.swtch.com/coro
- anonymoushn 3y agoThis does not contain anything about the proposed implementation of Pull, of course.
- Groxx 3y agoWouldn't this be the implementation? func Pull[V any](push func(yield func(V) bool)) (pull func() (V, bool), stop func()) { ... // snipped because it's big }
- ashton314 3y agoWait, so you return `false` to terminate the iterator? Stupid!! Now I can’t have an iterator that returns Booleans. UPDATE: I misread the article. This isn’t what happens. My bad. Honestly, I feel like Go focuses so much on superficial simplicity that you wind up with bad/leaky abstractions all over the place as well as bad primitive obsession. /rant
- marcopolo 3y agoSure you can. The bool is a signal telling the thing yielding values to stop.
- leijurv 3y agoNo that's not at all how this works. The iterator could "return" a boolean by passing a boolean to "yield". The return value of "yield" just indicates if the generator should continue, or stop because the actual loop has exited (due to break or return).
- ashton314 3y agoOh fie you’re right and I misread that. Thanks for correcting me. I’ve updated my comment.
- Jtsummers 3y ago> Now I can't have an iterator that returns Booleans. No, you've misunderstood it. The iterator function gets a yield function which can be called with a value (the value to be yielded). The yield function returns a boolean indicating whether the process should continue or terminate. func SomeIterator(yield func(bool) bool) bool { for whatever { if !yield(a_calculated_boolean_value) { return false } } return true } There, yielding boolean values without incident.
- flanked-evergl 3y agoI have used some iterator libraries for go and they do make for much more manageable code, especially if you start nesting iterators, like iterating over JSON which reads from an iterator over a blob from blob storage. And it also makes it simpler to apply filters at various points. Using callbacks also works but makes for nastier code.
- veber-alex 3y agoThis proposal seems pretty bad to me. I think Go lacks the features to make this ergonomic and flexible. Before adding iterators the language should add tuples so that you won't need to special case 0,1 and 2 artity functions. And shorter syntax for anonymous functions to make writing this less verbose, I believe this has been proposed before. Also the special case for "0..N" ranges seems very unnecessary.
- mutatio 3y agoI was thinking exactly this. Coming from Rust, but also as a fan of Go, I can't help that the building blocks just aren't there to construct a satisfying form of iterators, and like you said that would be language support for tuples and expanding the flexibility of Go's generics for methods to support the composition Rust allows, such as: iter.skip(10).take(10).map()...
- reichstein 3y agoThis design avoids the inefficiencies and most of the problems of having an?_iterator_, like the classical Java `hasNext`/`moveNext` or C# `MoveNext`/`Current` interfaces. Those iterators require you to keep the state between elements as values, effectively CPS transforming and defunctionalizing the control flow of actually iterating the elements. This iterator function here is really just a `forEach`/`Each` method where the callback can return a boolean. You can't easily use that to split iteration, say iterate the first five elements, then go do something else, and come back to iterate the rest, doing something completely different. But that also means it doesn't have to worry, as much, about the iterator going stale because the underlying structure changes (`ConcurrentModificationError`), or not knowing when to clean up, because somebody might call `hasNext`at any later time. So this is basically just a nicer `forEach` method.
- kubb 3y agoI much preferred the iterate by function approach described by one of the blogger guys a while ago (Muratori? Bendersky? Cassettieri? I can't quite recall the name atm).
- c0balt 3y agoHmm, it looks interesting but it seems to lack a precise need that's not covered atm imo. I also wonder how defer would be handled. After the last yield? After each yield? Should defer be executed when an iterator is not fully consumed?
- _0w8t 3y agoDefer will be executed normally when the function exits. yield here is not a statement but a normal function.
- _0w8t 3y agoI.e. C# and many other languages transform a function with the yield statement into a state machine implementing the iteration protocol. This proposal does the opposite. It transforms the loop body into a closure passed into the iterator.
- hknmtt 3y agoa) one item at a time means slow because when you are iterating maps/slice/array you have it all in memory so it is fast. with one item you have to constantly rewrite memory, move data and whatnot b) yo can always use for loop in a traditional for way with standard arguments(init, stop, next) c) yield is poor way of doing channels, i do not see why we should be degrading our code d) this entire premise is stupid :D
- jjallen 3y agoone of the reasons I love Go is because of its simplicity. It just works and it works well. You can even use with without GC if your code has to be fast. I don't see what the fuss is about with other new statically typed languages. So I oppose almost anything that will make Go less simple. Probably the only exception I would make is enums. I don't want my brain to have to work any harder reading code. Edit adding: the whole point of Go is simplicity and speed. That's why it's called Go in the first place. Let's keep it that way.
- hnlmorg 3y agoI want Go to remain simple but having written a hell of a lot of code in Go that needs to iterate over a stack whose size cannot be computed ahead of the iteration, I can confirm that writing such code in Go does get very complicated very fast. There are a lot of footguns to take into account: - how do you acknowledge the end of a stack? - how do you cancel the operation mid iteration? - how do you make it thread safe? - how do you make fast? The proposal here simplifies these concerns and thus makes it easy to write and use iterators. It makes it easier to read iterators too. And it does so without adding hacks to the standard library. This is one of those instances where the proposal might seem to add complexity but actually it simplifies the code a lot (unlike generics, for example, which complicates the syntax of your code in its effort to dedup lines of code. To be clear, I’m not being critical against generics here. Just using that as an example to show contest).
- ilyt 3y ago> But there are some disadvantages to this way of doing it. First, we have to wait until the whole slice has been generated before we can start to process the first element of it. Uh, range over channel ? The author seems to be very bad at go func Items() (items chan Item) will return channel that then can be just for a := range Items() {}
- progbits 3y agoChannels come with significant overhead [1]. While that is fine (or unavoidable) when communicating between goroutines, in single threaded code that iterates some custom container you don't want a channel. That is why stdlib map and vector have their own range magic, but as library author this isn't available to you. That is the point of the proposal. [1] https://syslog.ravelin.com/so-just-how-fast-are-channels-anyway-4c156a407e45 https://syslog.ravelin.com/so-just-how-fast-are-channels-any...
- masklinn 3y agoChannels are also not great when you want to break off / cancel iteration. The overhead also compounds when you compose / chain iterators. Plus the "lag" they induce due to context switching, although I think I saw an optimisation (proposal) where the runtime would directly switch from one routine to the other in some cases of channel interactions? Or am I thinking of an other language entirely?
- agent281 3y agorsc also talked about the performance of channels vs coroutines in his blog post. If memory serves, it was 10x reduction in looping overhead. https://research.swtch.com/coro https://research.swtch.com/coro
- Waterluvian 3y agoI have no opinion on this specific feature, but I have a general feeling that by far the hardest thing for language developers is to know when to stop. I feel this might be the case because language developers are themselves the worst reference for a typical user. They’re experts and no matter how big a language gets, they easily wield it. Even if they recognize that complexity is going up, they can’t truly grok the cost of that. I think Bob Ross talked about this often. That it can be tricky to know when you’re done. And it can be easy to clutter the canvas.
- binary132 3y agoI’ll cop to not having paid much attention to Go generics, but handwaving over my dubious syntax, what’s wrong with: ``` type Iter[T] interface { More() bool It() T } type mapOver[T] struct { Iter[T]; func (T) T f } func (m mapOver[T]) It() T { return m.f(m.Iter.It()) } func Map[T] (it Iter[T], f func(T) T) Iter[T] { return mapOver{it, f} } type Range { to, at int } func (r Range) More() bool { return r.at < r.to } func (r Range) It() (is int) { is = r.at; r.at += 1 } func DoEach[T] (it Iter[T], f func(T) T) { for it.More() { f(it.It()) } } // etc iter.DoEach(iter.Range(10), fx.Stdout(fns.MultBy(2))) // or whatever ``` Translating to English: If you can do it just fine with the syntax you have, why do you need to add it to the core language?
- earthboundkid 3y agoThis was already proposed and rejected. See the prior discussions on GitHub. https://github.com/golang/go/discussions/54245 https://github.com/golang/go/discussions/54245
- binary132 3y agoNobody has to accept it. You can do it yourself today. That’s the point of my comment.
- bborud 3y agoThis looks like a very complicated way of solving something for which there is already a solution: channels and contexts. Here's the pattern I often use: https://go.dev/play/p/mrQFLAc11QU https://go.dev/play/p/mrQFLAc11QU It should be doable to express this somewhat simpler using a (genericized) helper function to take out some of the work of writing the ListItems function. Since there is already a way to do this I don't think the language should be cluttered with more ways to essentially do the same thing. It is better to push people to develop a vocabulary of idioms that uses fewer features rather than introduce "more stuff".
- cpuguy83 3y agoThis does not solve the problem. With channels you are preparing things (even just one thing) ahead of time and you have to do it in a separate goroutine. One could possibly (haven't tried or given more than 2s of thought) implement a channel (or at least part of it) using iterators.
- binary132 3y agoUsing a channel for something is using a mutex for it. Nobody should suggest a mutex as a solution for implementing iterators.