9 ms·

I'm a go newb, but would you do this with channels and gofuncs now? The readability this provides is considerable.

d1str0 3y ago
Thats how I would do it. You can have a goroutine send on a channel (buffered or non buffered), and then another function can consume the chan as needed which lets the goroutine generate results as needed as well.
slashdev 3y ago
And make a simple iterator into a multithreaded program with channels and goroutines? Slower. More complex. Why?!? Just call your function in a loop.
- Groxx 3y ago
  Which is to say: yes! Many do it with channels currently. It's awful, but they do it.
- konart 3y ago
  > Slower Why would that be slower?
  - Mawr 3y ago
    The design of channels is constrained by the need to support concurrency. If you do not need concurrency, using something designed to support it will be suboptimal. And because it's specifically concurrency, the overhead will be substantial. ... that's the theory behind it. For more grounded explanations, see [1] & [2]. [1] https://news.ycombinator.com/item?id=29510751 https://news.ycombinator.com/item?id=29510751 [2] https://ewencp.org/blog/golang-iterators/index.html https://ewencp.org/blog/golang-iterators/index.html
    - konart 3y ago
      Yeah, I kind forgot we don't have anything concurrency related here. Thanks for the links!
  - slashdev 3y ago
    Channels and the goroutine scheduler need to be thread safe. They use locks and/or synchronized instructions that run much slower because they need to synchronize between the different cores of your CPU (or between CPUs, if you have more than one, which is slower still.) Running goroutines also saves and restores registers, trash part of the cache, or move to another core with a cold cache. Threads have more overhead, and processes more still, but that still is very slow.
jasonhansel 3y ago
As I recall, the performance of that approach is quite bad, even with Go's concurrency support.
liquidgecka 3y ago
That kind of suffers if you want to be able to cheaply cancel the iteration. Lets say the logic is: for item := range list { if condition(item) { break } } That is nice and readable, but if the thing your iterating is returning items in a batch then this gets far, far uglier. Its possible to end up with one loop that fetches x items at a time, and another that iterates over those items.. etc. But if you switch that to channels you maintain the readability and everything is nice! .. Except, you need a way to cancel the iteration, and it can't be closing the channel or the writer will panic. items, cancelFunc := apiCall() for item := range items { if condition(items) { cancelFunc() for _ := range items { } break } } close(items) Now, this is even worse if you need some kind of finalization of the goroutine that spawns to write to the given channel. Now you have a wait group, or a second channel.. etc. If its possible to error in the middle of iteration then things get complicated as well.
- jppittma 3y ago
  I may not understand what you're looking for, but what's wrong with this: topLoop: for _, batch := range list { for _, item := range batch { if condition(item) { break topLoop } } } The language also supports a limited goto. Combined with select and context, I've never seen a situation I felt I couldn't adequately describe.
Kamq 3y ago
I would probably just do the following: type Iterator[T any] interface { HasNext() bool GetNext() T } And then use it like: for i.HasNext() { current := i.GetNext() // do things here } It's an extra line, sure, but it's pretty obvious what's going on.
- pkolaczk 3y ago
  This is the way how Java iterators work. And having worked with them for many years I must say - please don't go that route. Having separate hasNext/next methods is terrible for implementers of the iterator. The problem arises when you need to actually compute the next item to learn if it exists. So then your hasNext is actually effectful and needs to cache the result internally. There are so many ways to get it wrong, I've even seen some terrible implementations which required the caller to call hasNext or otherwise they would not advance the iterator. Iterator design is another area where sum types would help simplify things a lot. Just one method returning Some/None would be so much better.
  - masklinn 3y ago
    > Iterator design is another area where sum types would help simplify things a lot. Just one method returning Some/None would be so much better. It would be better, but since go has MRV and zero values func Next() (hasNext bool, nextValue T) works fine, this is not Java or C# where you need to split the protocol into two separate operations.
  - Kamq 3y ago
    Since we're talking about implementing it in usercode, as opposed to adding it to the language, we fortunately have the option to rework it (or have a couple completing implementations to see what works in practice). If you're looking for those particular features, I'd recommend adopting go's `, ok` idiom that appears in several places in the language. A type like: type Iterator[T any] interface { Next() (T, bool) } Lets you write code like this, if that's more to your liking: for current, ok := i.Next(); ok; current, ok = i.Next() { // do things here } Which, really seems like a more go-like way to do this in the language as well (ranging over a function then becomes syntactic sugar for the above).

Thats how I would do it. You can have a goroutine send on a channel (buffered or non buffered), and then another function can consume the chan as needed which lets the goroutine generate results as needed as well.

slashdev 3y ago

And make a simple iterator into a multithreaded program with channels and goroutines? Slower. More complex. Why?!? Just call your function in a loop.

Groxx 3y ago
Which is to say: yes! Many do it with channels currently. It's awful, but they do it.
konart 3y ago
> Slower Why would that be slower?
- Mawr 3y ago
  The design of channels is constrained by the need to support concurrency. If you do not need concurrency, using something designed to support it will be suboptimal. And because it's specifically concurrency, the overhead will be substantial. ... that's the theory behind it. For more grounded explanations, see [1] & [2]. [1] https://news.ycombinator.com/item?id=29510751 https://news.ycombinator.com/item?id=29510751 [2] https://ewencp.org/blog/golang-iterators/index.html https://ewencp.org/blog/golang-iterators/index.html
  - konart 3y ago
    Yeah, I kind forgot we don't have anything concurrency related here. Thanks for the links!
- slashdev 3y ago
  Channels and the goroutine scheduler need to be thread safe. They use locks and/or synchronized instructions that run much slower because they need to synchronize between the different cores of your CPU (or between CPUs, if you have more than one, which is slower still.) Running goroutines also saves and restores registers, trash part of the cache, or move to another core with a cold cache. Threads have more overhead, and processes more still, but that still is very slow.

jasonhansel 3y ago

As I recall, the performance of that approach is quite bad, even with Go's concurrency support.

liquidgecka 3y ago

That kind of suffers if you want to be able to cheaply cancel the iteration. Lets say the logic is: for item := range list { if condition(item) { break } } That is nice and readable, but if the thing your iterating is returning items in a batch then this gets far, far uglier. Its possible to end up with one loop that fetches x items at a time, and another that iterates over those items.. etc. But if you switch that to channels you maintain the readability and everything is nice! .. Except, you need a way to cancel the iteration, and it can't be closing the channel or the writer will panic. items, cancelFunc := apiCall() for item := range items { if condition(items) { cancelFunc() for _ := range items { } break } } close(items) Now, this is even worse if you need some kind of finalization of the goroutine that spawns to write to the given channel. Now you have a wait group, or a second channel.. etc. If its possible to error in the middle of iteration then things get complicated as well.

jppittma 3y ago
I may not understand what you're looking for, but what's wrong with this: topLoop: for _, batch := range list { for _, item := range batch { if condition(item) { break topLoop } } } The language also supports a limited goto. Combined with select and context, I've never seen a situation I felt I couldn't adequately describe.

Kamq 3y ago

I would probably just do the following: type Iterator[T any] interface { HasNext() bool GetNext() T } And then use it like: for i.HasNext() { current := i.GetNext() // do things here } It's an extra line, sure, but it's pretty obvious what's going on.

pkolaczk 3y ago
This is the way how Java iterators work. And having worked with them for many years I must say - please don't go that route. Having separate hasNext/next methods is terrible for implementers of the iterator. The problem arises when you need to actually compute the next item to learn if it exists. So then your hasNext is actually effectful and needs to cache the result internally. There are so many ways to get it wrong, I've even seen some terrible implementations which required the caller to call hasNext or otherwise they would not advance the iterator. Iterator design is another area where sum types would help simplify things a lot. Just one method returning Some/None would be so much better.
- masklinn 3y ago
  > Iterator design is another area where sum types would help simplify things a lot. Just one method returning Some/None would be so much better. It would be better, but since go has MRV and zero values func Next() (hasNext bool, nextValue T) works fine, this is not Java or C# where you need to split the protocol into two separate operations.
- Kamq 3y ago
  Since we're talking about implementing it in usercode, as opposed to adding it to the language, we fortunately have the option to rework it (or have a couple completing implementations to see what works in practice). If you're looking for those particular features, I'd recommend adopting go's `, ok` idiom that appears in several places in the language. A type like: type Iterator[T any] interface { Next() (T, bool) } Lets you write code like this, if that's more to your liking: for current, ok := i.Next(); ok; current, ok = i.Next() { // do things here } Which, really seems like a more go-like way to do this in the language as well (ranging over a function then becomes syntactic sugar for the above).