5 ms·
Go range iterators demystified
- pjmlp 2y agoI am all for having them in the language, however the way they have been designed, or how the new magic fields for structure aligment (in Go 1.23) are being designed, this shows how attacking other languages as PhD level complexity and then coming out with such special case designs, is kind of ironic. I give it 10 more years of such special cased improved, for Go not to be any better than the languages the community regularly complains about, while Go is "perfect".
- jdougan 2y agoIf I wanted ranges I'd have probably done them in a similar way to the ones in D aka. Dlang. Several methods instead of 1, but very clear.
- rplnt 2y agoWouldn't that always require you to have a dedicated type? I don't know D but I'm imagining what python has. Nice thing about this go approach is you can have just functions as shown in examples. One practical use is functions building range functions on top of existing interfaces. Or as the filters examples shown, you can create filter generators on regular slices/maps.
- jdougan 2y agoAndrei Alexandrescu's design notes. He approached it from his prior experience with C++. https://www.informit.com/articles/printerfriendly/1407357 https://www.informit.com/articles/printerfriendly/1407357
- tsimionescu 2y agoCreating a small wrapper type over a slice/map is trivial in Go (`type X[T] []T`) , and then you can define the range functions as methods on that slice type. If they allowed generic instance methods it would be even simpler.
- jdougan 2y agoExactly. Similar in D. And using multiple methods leaves obvious places to expand the capability (bidirectional, random access, etc.)
- masklinn 2y ago> Creating a small wrapper type over a slice/map is trivial in Go And yet it's specifically one thing rsc did not want. Further issues described in the rangefunc proposals: - it would require the desugaring to run off of method-set analysis of userland types, something which does not currently exist - it severely complicates resource management around the iterator, as you need a 3-step iteration for resource-bearing iterators (acquire iterator, defer cleanup, perform iteration) And one not actually listed explicitly: for the limited amount of optimisations the Go compiler does, internal iteration is a lot easier to optimise as it pretty much inlines down to a `for` loop, the termination of which is much easier to analyse than bouncing through a bunch of pull calls. Not only that, but `for range` works off of underlying type, so this is already valid Go: import "fmt" type Foo []int func main() { f := Foo([]int{1, 2, 3}) for _, v := range f { fmt.Println(v) } }
- neonsunset 2y agoOne could approach it the other way. Once more projects adopt all kinds of wrapper functions and types, the deficiency of Go will become a more widespread knowledge as compiler will get progressively less able to cope with added abstractions. Hopefully it will put the common fallacy of "Go or Rust" to rest as the weight class and capabilities are on opposite ends of spectrum, with much closer comparison being "Rust or C# or Swift or Kotlin" if one looks for a Rust alternative that makes a tradeoff of not forcing many small decision to reduce decision fatigue by conceding to a reasonable extent certain areas Rust excels at. In any case, for its touted simplicity Go sure doesn't look like a simple and straightforward to follow language anymore. The 3-step approach to iteration is also well solved[0] in C#, and works even in rather complex cases like `File.ReadLines(...)` where the line iterator internally handles IO, file handle acquisition and disposal. Just `foreach (var msg in File.ReadLines("messages.jsonl"))` and you won't be able to make a footgun out of it. This also applies to the usage chained with filter/map/etc. var messages = File .ReadLines("messages.jsonl") .Select(line => JsonSerializer.Deserialize<Message>(line)) .ToArray(); [0]: https://sharplab.io/#v2:EYLgtghglgdgPgAQEwEYCwAoBAGABAlAOgEkB5Abkx3yIBlYBHSrPAwgFQFMAPAF0IBSAZwD2MZpgBmIgE6cIAYwAWuABQA3CDNxhOQoRADmnXLFwAxKABtOmXPdyEASvIAm9GHtUAiXfqN6hABWojBW3gCUdg6EAMqcNgq8qlawJgC8AHy4wmLxMlAQqQBenDKEACJ6ZYUlnAA8ALJ6BsaZKWlRGA49uAD8fbi8SjIiAO64nhMAotwKnAAOvFBiPuYiAK4wrricYEsAnripnoRnkRFdAN7R9gQAnKp+rZyEAIIbw7IRzAC+mJg5ApZDtmv5jKoCHgPl8ZAAaGh4Lh8H5AA= https://sharplab.io/#v2:EYLgtghglgdgPgAQEwEYCwAoBAGABAlAOgEk... note .GetEnumerator() and .Dispose() calls
- frou_dh 2y ago"Community" is kinda the key word there, because when someone is "along for the ride" over a period of many years then a ratcheting of complexity is easier to cope with than it is to arrive cold in a big/complex language.
- g15jv2dp 2y agoI guess people who'd like to get into go today can just go look elsewhere, then...
- heavenlyblue 2y agoThis is why we need to keep reinventing languages to grow another generation of juniors in seniors because they are not intelligent enough to correctly use abstraction
- deleted 2y ago[deleted]
- pjmlp 2y agoI agree with the idea, as I imagine someone landing today on C++23, C23, C# 12, Java 22, will have a much hard time that those of us that know them pretty much since their first baby steps as programming languages. As per my own experience since the mid 1980's. However, exactly because Go had such an history of language evolution to learn from since FORTRAN came to be in 1958, maybe some of the early decisions could have been done better, instead of Apple style, "we are not doing X, for years later, X is actually something we want".
- tapirl 2y agoAgree. Now, the old built-in iterators (containers) are incompatible with the new custom iterators. But it looks the Go core team never care about the consistency problem. The old built-in generics are also incompatible with the new custom generics.
- WhyNotHugo 2y agoAll this added complexity seems to be just sugar syntax too. The examples here clearly prove that we could write clean code that achieved the same functionality without this feature. We could also just use an Iterator pattern for such use cases too. Go used to be such a simple language. I wonder what's driving them to keep adding complexity to the language itself. It makes me respect Harelang's goal of "freeze the language once 1.0 is out" a lot more.
- bheadmaster 2y agoThe proposal [0] explains the motivation behind this "syntax sugar": There is no standard way to iterate over a sequence of values in Go. For lack of any convention, we have ended up with a wide variety of approaches. Each implementation has done what made the most sense in that context, but decisions made in isolation have resulted in confusion for users. In the standard library alone, we have archive/tar.Reader.Next, bufio.Reader.ReadByte, bufio.Scanner.Scan, container/ring.Ring.Do, database/sql.Rows, expvar.Do, flag.Visit, go/token.FileSet.Iterate, path/filepath.Walk, go/token.FileSet.Iterate, runtime.Frames.Next, and sync.Map.Range, hardly any of which agree on the exact details of iteration. Even the functions that agree on the signature don’t always agree about the semantics. For example, most iteration functions that return (T, bool) follow the usual Go convention of having the bool indicate whether the T is valid. In contrast, the bool returned from runtime.Frames.Next indicates whether the next call will return something valid. When you want to iterate over something, you first have to learn how the specific code you are calling handles iteration. This lack of uniformity hinders Go’s goal of making it easy to easy to move around in a large code base. People often mention as a strength that all Go code looks about the same. That’s simply not true for code with custom iteration. [0] https://github.com/golang/go/discussions/56413 https://github.com/golang/go/discussions/56413
- WhyNotHugo 2y agoAh, there was no standard so they defined a new one. Classic https://xkcd.com/927/ https://xkcd.com/927/
- meling 2y agoI was hoping the blog author would have revealed some plans for supporting a new iteration API in dolt. The range over func API is particularly useful if you need to iterate and compute over something that doesn’t all fit in memory (as is necessary for a slice and map).
- zachmu 2y agoSay more? What are you using Dolt for that you would find this useful?
- _cenw 2y agoThis feels like it's undermining channels, the feature go really wants you to use in other places too (but people tend to still use mutexes). Channels aren't quite as lazy (if you use an unbuffered one you supply one element in advance), but they're close.
- jakjak123 2y agoChannels complicate error handling quite a bit, so I tend to avoid them
- tapirl 2y agoCould you elaborate more?
- jakjak123 2y agoIf you have shipped some task to a channel, or is waiting for some work to complete on a channel, there is no native way to propagate the error that your task may have failed. Also if a error did happen during processing the task you put on the channel, the stacktrace suddenly is not the whole story anymore. Channels also has no way to make sure the context.Context is reasonably propagated
- tapirl 2y agoIs it possible to enhance channels to fit into the iterator need? I mean using them without always creating new goroutines.
- thwarted 2y agoHere's a rough implementation of iterators using channels that I was experimenting with. It needs some syntactic sugar to do proper cleanup of the go routine if the loop exits early; probably the iterator function needs to return a func that closes the channel called after the loop, and some error handling/closed channel checking. https://gist.github.com/thwarted/a6e5e7ca5ce552311a7d5ece13d298ac https://gist.github.com/thwarted/a6e5e7ca5ce552311a7d5ece13d...
- tapirl 2y agoCorrection: // Only care about the iteration count for range aContainer { ... } // Just the values for v := range myChannel { ... } // Indexes and values (or keys and values for a map) for i, v := range mySlice { ... }
- neonsunset 2y agoThis looks incredibly clunky and counter-intuitive. Compare that to yield return, .Select and .Where methods in C#, or Filter and Map in many popular languages - it's not a good look. Or when writing manually, compare it to static IEnumerable<T> Filter<T>( IEnumerable<T> source, Func<T, bool> predicate) { foreach (var item in source) { if (predicate(item)) yield return item; } } Could also compare to how easy it is to use for extremely common patterns in general purpose code: var numbers = Enumerable.Range(0, 10); var even = numbers.Where(n => n % 2 is 0); var strings = even.Select(n => n.ToString());
- doctor_eval 2y agoIsn’t the only real difference that the yield function is being passed into the iterator instead of being a reserved word? I don’t think it’s clunky, although it took a few minutes for me to get it.
- pjmlp 2y agoNo, before C# got generators, some of the machinery had to be manually implemented, with yeld, the compiler generates the necessary implementation for IEnumerable.
- pansa2 2y agoIn C#, `yield` is a kind of return, not a function call. The two approaches to iteration are quite different (external vs internal iteration).
- deergomoo 2y agoI’m glad this is being added for the ergonomic benefits. But the number of times the article points out cases where conventions will need to be formed by the community makes me fear this will add to the already longer-than-I’d-like list of anti-patterns or footguns that Go’s design and type system makes easy to fall into.
- Someone 2y agoFTA: “Maybe you just want to use the range keyword to iterate over every element of your collection. Easy enough. func (s Slice) All() func(yield func(i int) bool) { return func(yield func(i int) bool) { for i := range s { if !yield(s[i]) { return } } } } ” So, for an “easy enough” example correctly, you have to write func five times in order to, if I understand this correctly, wrote a function returning a function that takes a function as an argument? For comparison, C# does that this way (https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/statements/yield https://learn.microsoft.com/en-us/dotnet/csharp/language-ref...) IEnumerable<int> ProduceEvenNumbers(int upto) { for (int i = 0; i <= upto; i += 2) { yield return i; } } Yes, that introduces “magic” where the runtime figures out that ProduceEvenNumbers won’t continue, but why give such functions the flexibility not to listen to such requests (in the golang version, is forgetting the if and just yielding instead ever useful?)
- Seb-C 2y agoI don't know why most examples about the Go iterators shows unnecessarily convoluted examples. The authors probably don't know as much of the language as they think they do. Go is far from perfect and certainly has a lot of flaws, but IMO the iterators design fits well into the language and does not deserve the criticism. The range only needs a function, so what is the point of calling a function that returns a function? This is enough to make a range iterator, you don't have to call `All` manually: func (slice Slice) All(yield func(string) bool) { for _, s := range slice { if !yield(s) { break } } } https://go.dev/play/p/7AYOXzlHU4t?v=gotip https://go.dev/play/p/7AYOXzlHU4t?v=gotip In Go, the expression `object.method` already returns the method as a function, with `object` being already bound in the scope of the function. The syntax is so that if you have this definition: type T string func (t T) f(a int) int { return a } Then: - `T.f` returns `func(t T, a int) int` - `T("whatever").f` returns `func(a int) int` where `t` is already set to `"whatever"` - `T("whatever").f(42)` calls the function and returns `42`
- zachmu 2y agoThis is true, you can just return the method without invoking it. I don't tend to use that convention though, because most of the time when I use this pattern, I'm returning a function that uses a parameter I pass in. You can see this in the predicate example. I prefer keeping my call sites consistent and always using functions that return a function when invoked, rather than sometimes invoking them and sometimes just passing the func reference. YMMV.
- rochak 2y agoThe syntax just doesn’t sit right with me due to some reason. It gives me the same heebie jeebies as Python’s decorators. Not a fan of either. Maybe I need to get used to them.
- Hendrikto 2y agoI think Python iterators are designed very cleanly. They just take the function they decorate, and return another. It does not get simpler yet more flexible than this.
- foldr 2y agoThere isn't any new syntax here.
- tapirl 2y ago`range aFunction` is kinda of new syntax, though I agree it is more a new semantic thing.
- pansa2 2y agoDefining a Go iterator is similar to defining a Python decorator that takes an argument. Both involve defining a function that returns another function, with the inner function having another function as its parameter. This isn’t a particularly difficult concept, but manipulating functions in this way does feel unusual in such heavily-procedural languages. This has two downsides: * Programmers who only use those languages are unfamiliar with the concept, and * The languages’ syntaxes aren’t designed to make it particularly clear.
- rundev 2y ago`yield` being a function that is passed into the iterator seems like suboptimal design to me. Questions like "What happens if I store `yield` somewhere and call it long after the loop ended?" and "What happens if I call `yield` from another thread during the loop?" naturally arise. None of this can happen with a `yield` keyword like in JavaScript or C#. So why did the Go-lang people go for this design?
- TheDong 2y agoIt is an explicit goal of the go team to minimize the number of keywords in the language. Simple languages have fewer keywords so go must have few keywords. https://go.dev/ref/spec#Keywords https://go.dev/ref/spec#Keywords Look how simple that is. This is why things like ‘close(channel)’ are magic builtin functions, not keywords (more complicated) or a method like ‘channel.Close’ (works with interfaces and consistent with files and such, so not simple).
- pansa2 2y ago> It is an explicit goal of the go team to minimize the number of keywords in the language. It's understandable - because unfortunately people judge languages by very shallow metrics. Several times I've seen people use "number of keywords" as a proxy for language complexity. However, that's completely misguided. `static` in C++ (and, IMO, `for` in Go) demonstrate that overloading a keyword to mean multiple things is harder to understand than having a larger number of more meaningful keywords.
- weinzierl 2y agoI (still) find it confusing that in C++ the symbols to declare a pointer and a reference are the same as the ones for dereference and address-of. That pointers declared with * in Go are more like references (&) and that there are no true pointers (I think) does not really help.
- pjmlp 2y agoGo has true pointers. If you mean pointers arithmetic, that can be achieved with unsafe package.
- pansa2 2y agoWhat's the rationale behind Go choosing internal iteration (the iterator calls a function for each value, like Ruby) over external (the iterator returns each value, like Python and C#)? My understanding is that internal iteration makes it easier to write iterators (producers) but harder to write the consuming code. That's why Go needs to re-write the body of each `for` loop as a function body, including special handling for `break`, `return` etc. External iteration OTOH makes it harder to write producers but easier to write consumers. Python and C# therefore allow external iterators to be written via coroutines/generators. Wouldn't Go's goroutines make the coroutine approach to external iterators straightforward? Whereas the re-writing necessary for internal iterators seems convoluted?
- deleted 2y ago[deleted]