9 ms·
Writing a custom iterator in modern C++
- arcticbull 6y agoThere's six different kinds of iterator?! Off hand, is it just me or... - Isn't an iterator that outputs data to its recipient container not an iterator, but rather some type of filling API? - Isn't a random access iterator not an iterator but rather a slice view? - Isn't a contiguous iterator a leaky abstraction in that an iterator is just an interface that shouldn't expose how the data backing it is stored? This iterator kindedness feels very C++ in the worst way possible. IMO the only iterator type that makes any sense is an "Input Iterator" and everything else is some sort of hot mess of inconsistency and, uh, C++. Even with enough time I'm not sure I could design a worse API.
- deleted 6y ago[deleted]
- shakna 6y ago> Isn't a random access iterator not an iterator? Why would it not be? It still allows you to iterate over an entire collection, however the order is left undefined so that certain performance characteristics can be exposed to the underlying implementation. This is the kind of tool you'd reach for if you don't care about the order of a collection, but you do care about performance, and probably care about the entire collection. For example _pairs_ from Lua, used to iterate over key/values in tables, is a random-access iterator. Traversal of the collection is still happening.
- arcticbull 6y agoThe way the random access iterator was described in the article is not that it allows iteration over an un-ordered collection but rather that it allows random access within the collection. This would not necessarily map well to an un-ordered collection. This appears to be born out by the API which supports arbitrary addition and subtraction to the iterator. I could of course be mistaken. What you're describing based on my experience with other languages is simply an "input iterator" over an un-ordered collection. An iterator does not guarantee deterministic/repeatable ordering, just defined ordering (as in, guarantees it won't visit the same element twice), in my experience.
- millstone 6y agoRandom access iterators mean you can efficiently advance or retreat an iterator by an arbitrary amount. So a vector (via pointer arithmetic) but not a linked list.
- arcticbull 6y agoRight but why should this matter at the type level? All you need to implement is next() and annotate it with documentation of the performance characteristics.
- missblit 6y agoYou can think of it as implementing `seek`, not just `next`. `RandomAccessIterator` implements the `+=` and `-=` operations that `BidirectionalIterator` does not. Algorithms that accept `RandomAccessIterator` can then assume that calling += is possible.
- arcticbull 6y agoWhat's the advantage to using a random access iterator over a get() indexed? Like to me the point of an iterator is that it abstracts over access kind, and exposes the minimum viable view, which is sequential forward and possibly backward access. If you want indexed access why wouldn't you just, you know, use anything that implements operator[]?
- missblit 6y agoJust that iterators is the API that C++ settled on for doing this in a generic way (well concepts in C++20, but I haven't learned that yet). It's totally 100% kosher to write a function that calls `.at()` instead of `operator+=`. As long as it only needs to work on vectors, strings, or other stuff with a `.at()` method. But if the function works on iterators instead it can work on ANY data structure that also fulfills the requirements of RandomAccessIterator API. String Views, raw pointers, a 3rd party Cord library, etc. There's no inherent meaning to the iterator API requiring `operator+=` instead of something closer to `get()`, except that it means that the iterator itself doesn't need to know anything about container bounds (like I said above, raw pointers work as RandomAccessIterators).
- brighton36 6y agoC/C++ may be better understood as a template processing system around assembly.
- glouwbug 6y agoWell, I mean, iterating over a deque, map, vector, and list, in its own, already require the pointer to the next element to decide whether to increment by the type of its size, reference the `next` pointer, or the left/right pointer, or a combination of any of the above 3. It's not the language's fault, it's computer science
- robscallsign 6y agoKeep in mind that most of the details you mentioned are for the developer of the library, not the end user/consumer of the library. Ultimately most consumers of the library care if they can call std::sort(YourSpecialContainer.begin(), YourSpecialContainer.end()); in a consistent manner with some semblance that the use of templates and interfaces result in nearly zero overhead of the abstraction.
- vardump 6y ago"nearly zero overhead" I can't call anything "nearly zero overhead" that can generate large amounts of code. Not at least with a pure conscience. I've seen enough many cases of better performance with more "overhead" but drastically smaller code size.
- vardump 6y agoSorry for answering my own reply. Zero overhead cult is pretty strong. People should profile instead. Sigh.
- benibela 6y agoUsually people say std::sort is faster than qsort
- flqn 6y agoIt is, since the compiler can inline the comparator.
- vardump 6y agoIt's true whenever the function call overhead for the comparison operation dominates. Inlining a trivial operation (often just one instruction) is one extreme. At the other extreme L1C cache (typically just 32 kB) gets constantly trashed by numerous type specialized functions. Individually fast when microbenchmarked, but collectively slow. That's why profiling is a must when high performance is required. Microbenchmarks can be very misleading.
- creato 6y agoIt's funny how two of the most common bandwagon posts you can see on HN and other programming venues is: * C++ is a terrible bloated language that no one should use. * The power of modern computers is wasted on lazy software engineers writing software with too many layers of abstraction.
- david2ndaccount 6y agoTo many, these statements are saying the same thing. Your average C++ codebase is a bloated mess with too many layers of abstraction.
- mzsmyk 6y agoYou don't need to use a "terribly bloated language" to write efficient software.
- mehrdadn 6y agoI mean, C++ has lots of compile-time abstractions, but unlike with other languages, it can collapse a lot of them during compilation via its inlining and other optimizations. You don't get that with a language like Python...
- adwn 6y agoI don't see a contradiction between these two statements. There are more languages than C++ and Python. You could use Rust, or some of the languages targeting the JVM or the CLI.
- gpderetta 6y ago> Isn't an iterator that outputs data to its recipient container not an iterator, but rather some type of filling API? Sure, but it makes sense for this api to have the same interface as an iterator so that existing iterators can be used without an adapter. > Isn't a random access iterator not an iterator but rather a slice view? It is an iterator that supports additional functions. You can use a random access oterators with functions that take output iterators, forward iterators, multipass iterators etc. Of you had untelated concepts for each use case the system would be more complex. > Isn't a contiguous iterator a leaky abstraction in that an iterator is just an interface that shouldn't expose how the data backing it is stored? Seeing thought abstractions and specializing algorithms is what makes STL style generic code efficient. And it is not inconsistent at all. Stepanov has written in grat details about the theoretical fundations of his design.
- quietbritishjim 6y agoYou seem to have mixed up an objection to the API with an objection to the user of the word "iterator" for something a bit more general than the normal/original meaning. That conflation isn't terribly helpful. Words can have more than one meaning. So long as everyone's on the same page, that's OK.
- millstone 6y agoIterators are inconsistent, but this plays to the strengths of C++'s duck-typed generics. The same code can either copy or insert, depending on whether you use std::begin or std::back_inserter; this isn't principled but it's powerful. The "six types of iterators" overstates it. You don't need to sweat those details unless you are writing boost or something. Just add what you need. I miss C++ iterators in other languages. For example, in Rust it's hard to talk about "a position in a String" which may be advanced or retreated. Usually you give up, use an integer index, and this is much worse.
- Tyr42 6y agoBacking up a utf8 stream is nontrivial though.
- millstone 6y agoThis is an interesting point. Walking backwards through a UTF8 byte stream is easy: just look for the next non-continuation byte, and Rust really can do this, though the API is buried under `str.chars().rev()`. C++ is structured around iterators that point into; this enables e.g. writing into the guts of a std::map but not the guts of a std::string if you care about Unicode, since the write might change the string's byte length.
- emmanueloga_ 6y agoIt would be nice to write a DSL to generate some of the C++ boilerplate (say, containers, iterators, etc) required as a baseline before starting to solve the actual problem. A DSL could codify rules like the ones described in this article and generate at least a rough first pass to be hand-tuned later.
- pumpkinpal 6y agoBoost STL Interfaces provides this functionality as a library. Not just for iterators but both views and containers of sequences too. https://www.boost.org/doc/libs/1_75_0/doc/html/stl_interfaces.html https://www.boost.org/doc/libs/1_75_0/doc/html/stl_interface...
- mhdhn 6y agoWow, no, really? wouldn't hat would spoil the "fun" of all the endless arcana?
- quietbritishjim 6y agoThe classic iterator categories (input, output, ...) discussed in this article have been deprecated in C++20, and replaced with new concepts. I would expect a post from 2020 claiming to be about iterators "in modern C++" to be specifically about that change, but actually it's just about the legacy categories, so this is a pretty worthless article.
- cma 6y agoIt is mentioned in the article early on: "The first thing to do is to assign the iterator some properties. Until C++17 this is done by tagging it with the tag dispatch mechanism, while C++20 uses concepts: in this article I will follow the traditional approach."
- DoofusOfDeath 6y agoHonest question: what does it mean for something to be "deprecated" in (specifically) the c++ language spec?
- millstone 6y agoThe spec itself defines it as: Normative for the current edition of the Standard, but having been identified as a candidate for removal from future revisions This is a very blunt tool. In practice, a feature may be deprecated if it is: 1. Legacy inherited from C that serves no modern purpose. The register keyword. 2. Features hardly anybody implemented. Exported templates is the only example I am aware of. Like falling off the edge of the map. 3. Features that proved to be a bad idea, like incrementing a bool. 4. Features that proved to be wholly inadequate, like codecvt for dealing with Unicode. 5. Features that are reasonably new in C++ but obviated by new stuff. You don't need std::bind if you have lambdas. This is my outsider's perspective; please chime in if you have standards committee insight.
- DoofusOfDeath 6y agoThanks for the explanation. Given my impression that the standards committee places very strong emphasis on backwards compatibility, I didn't expect there to be a policy to allow for breaking changes.
- ribs 6y agoFirst sentence: “ An iterator is an object that points to an element inside a container” This is not the primary purpose of an iterator. Does anyone want to hazard a guess as to what an iterator’s primary purpose is? You? You? Bueller?
- millstone 6y agoI think you are arguing that iterators are "primarily" for iteration. But in C++ they really are like an abstraction over C pointers. For example, you can hold a std::list::iterator, and use it to efficiently delete from a linked list, without ever iterating. Sometimes C pointers are used for iteration, but I would not say that is their primary purpose.
- jhayward 6y ago> I think you are arguing that iterators are "primarily" for iteration I really wonder why people put up with something named this way then. "That fool thinks iterators are for iteration!" Clarity of abstraction and good naming are important, unless the practitioners are masochists.
- gpderetta 6y agoCursor would be a better name probably, but I wouldn't say the name is misleading, as iteration is an important use case.
- missblit 6y agoThe primary purpose of an iterator is to allow algorithms to be written in a generic manner without needing to know details about the type it's operating on. Since the algorithm only operates on the iterator type, any number of collections or other code that exposes that iterator type can be plugged into the same algorithm. Take `std::copy_n` for example. You _could_ use this to copy from one vector to another: std::vector<int> a = {1,2,3}, b = {0,0,0}; std::copy_n(a.begin(), a.size(), b.begin()); But iterators also allow the input and output collection types to be different: std::vector<int> a = {1,2,3}; std::array<int,3> b; std::copy_n(a.begin(), a.size(), b.begin()); Or even for the output to not be associated with a container at all. std::vector<int> a = {1,2,3}; std::copy_n(a.begin(), a.size(), std::ostream_iterator<int>(std::cout, " "));
- pumpkinpal 6y agoSomewhat related but C++20 ranges do not replace iterators and instead build a higher level of abstraction upon iterators. Ranges are implemented in terms of iterator pairs.
- mehrdadn 6y agoYeah. That actually sucks; often iterators doesn't fit the iteration model of some sequences, but ranges would fit right in if they didn't require separability into two iterators.
- tlb 6y agoCan you give an example of such a sequence?
- mehrdadn 6y agoAny kind of generator is basically like this. Imagine one for listing the contents of a directory... maybe it looks like this: class DirectoryChildren { void *dir; void *buffer; size_t buffer_size; size_t buffer_offset; ... public: bool next(FileInfo *info); }; Bear in mind the buffer only represents a chunk of intermediate results fetched from the OS; when we reach its end, we have to request more data from the OS. If you had to make an iterator for this, what would you do? You'd basically need to either (a) create iterators that at least duplicate the entire state of DirectoryChildren on the stack, or (b) make iterators allocate on the heap. Both of these suck, because (1) iterators are passed around and copied all over the place on the assumption that these are fundamentally cheap operations, which is an assumption that easily breaks (like here), and (2) the iterators fundamentally do not have any meaning or utility as separate entities from the range itself.
- pumpkinpal 6y agoThis functionality already exists in the C++ standard library as std::filesystem::(recursive_)directory_iterator with the default constructor creating a sentinel value. This is solution (1) and this is due to the restriction that prior to C++20 iterators were required to be the same type. C++20 relaxes this restriction and allows the sentinel to be any type as long as it is equality comparable with the other iterator. Since on *nix systems readdir is used under the hood, the sentinel can simply be an empty type that is only equal when the actual iterator's last cached dirent pointer is equal to NULL. The sentinel itself can be an empty type.
- fjfaase 6y agoIterators are a very powerful concept, however, it is very cumbersome and often impossible, to implement some interesting iterator based on the standard iterators. This stems from the idea that any iterator can based on a single pointer and use pointer comparison for indicating the start and the end of the contents. The most common case, are forward output iterators, for which I usually define a class that defines the methods more and next, that can be used as: for (MyIterator it(data); it.more(); it.next())
- Olipro 6y agoAn iterator is a class. It can contain any private data you desire and mutate in any wonderful and/or stupid ways you desire. How it decides to compare equal to the .end() instance is also your call. Your example is not a compliant iterator (as in, wouldn't work in a range-based for loop) - yet the same behaviour would be achievable through your own operator++() and operator bool(),
- fjfaase 6y agoI agree, but I feel it is a little sad that such a powerful programming concept as an iterator is hard to implement in C++ if you want to make full use of it. Other program languages have made better choices with respect to this and allow you to write cleaner, shorter code.
- saurik 6y agoWhy not just use those two operators instead of more and next?
- Thorrez 6y ago>difference_type A forward iterator doesn't have a difference_type. A random access iterator does.