3 ms·
Returning large arrays is the kind of bad idiom you see in languages lacking abstract iterator constructs, generators, or lexical closures. Go isn't such a mini
by wahern 6y ago
Returning large arrays is the kind of bad idiom you see in languages lacking abstract iterator constructs, generators, or lexical closures. Go isn't such a minimalist language; in fact, it's the one area where it excels, a fact often overlooked by those obsessed with generics. It's indeed perplexing why they'd add an interface that accumulates everything into a giant array by default, especially in a context that you know will cause performance problems, and used by a community with many inexperienced programmers--including ones (e.g. devops) not accustomed to thinking of these details--who will invariably fall into this trap.
- dexen 6y ago>why they'd add an interface that accumulates everything into a giant array by default While I can't speak for Go's authors' mindset, here's my view: One array of records is about as simple interface as possible, and it's good enough for many use cases. Contrasting with that, reading the directory with repeated readdir()'s or equivalent is fraught with hidden pitfalls. You end up being responsible for sorting, for resuming syscall in case of a signal or other interruption, for handling changes to the list of files during read... and probably other concerns above and beyond that.
- sanderjd 6y agoThe suggestion isn't repeated readdir calls, it is some kind of iterator, which would specify and be implemented to solve the problems you mention (which already have to be dealt with inside the array-returning readdir call, by the way).
- hombre_fatal 6y agoWhat's annoying is that filepath.Walk(), something that you'd think was the performance-minded approach, also loads all the filenames into a slice to sort it for some reason. If an API is going to arbitrarily sort filenames, at least let us pass in a comparator function so we have some control. That would actually be a decent improvement to its API.
- jeffbee 6y agoWhat is this entire subthread even about? os.(File).Readdir does not, by default, even compile. The caller has to provide a parameter of how many entries to read at a time, and reading "all of them" is an intentional choice.
- wahern 6y agoAh. Perhaps the issue was ReadDirFS.ReadDir and GlobFS.Glob proposals, both of which are one-shot. I don't write much Go so missed that both os.ReadDir and ReadDirFile.ReadDir support streaming.