4 ms·

Yeah, and I genuinely want to like Rust and I pick it up a few times every year (and have done for the last decade), but each time I get burned by trying to do

Yeah, and I genuinely want to like Rust and I pick it up a few times every year (and have done for the last decade), but each time I get burned by trying to do something that would be trivial (and safe!) in Go or C. Most recently, I'm trying to build a non-allocating, minimally copying `Lines` iterator which reads from an internal `Reader` into an internal buffer and then yields slices of the internal buffer on each call to `next` (each slice represents a single line, and it's an error if a line exceeds the capacity of the internal buffer); however, as far as I can tell, this is unworkable without some unsafety because the signature is `fn next(self: &mut Lines<'a>) -> Result<&'a [u8]>` which doesn't work because Rust thinks the mutable reference to self must outlive 'a, and explicitly setting the mutable self reference to 'a violates the trait. If I forego the trait and just make a thing with a `next()` method, then I can't use it in a loop without triggering some multiple mutable borrows error (each loop iteration constitutes a mutable borrow and for some reason these borrows are considered to be concurrent). The only thing I can think to do is have a `scan()` method that finds the next newline and notes its location inside the `Lines` struct and a separate `line()` method that actually fetches the resulting slice from the buffer. I don't run into this in C or Go, and for all of the difficulty of battling the borrow checker, I'm not getting any extra safety (in this case).

masklinn 3y ago
What you're talking about is the canonical example for generic associated types, which were stabilised late 2022: https://blog.rust-lang.org/2022/11/03/Rust-1.65.0.html#generic-associated-types-gats https://blog.rust-lang.org/2022/11/03/Rust-1.65.0.html#gener...
- throwaway894345 3y ago
  Ah, good to know. Thanks!
burntsushi 3y ago
Alternatively, use a closure: https://docs.rs/bstr/latest/bstr/io/trait.BufReadExt.html#method.for_byte_line https://docs.rs/bstr/latest/bstr/io/trait.BufReadExt.html#me... There are downsides to this approach because it uses internal iteration while most things in Rust use external iteration. But shit happens. Go doesn't even have a first class concept of iterators as an abstraction (yet), and its standard library contains patterns for both internal (sync.Map) and external (bufio.Scanner) iteration. > The only thing I can think to do is have a `scan()` method that finds the next newline and notes its location inside the `Lines` struct and a separate `line()` method that actually fetches the resulting slice from the buffer. Yup that works too. That's basically the design used by the `streaming-iterator` crate: https://docs.rs/streaming-iterator/latest/streaming_iterator/trait.StreamingIterator.html https://docs.rs/streaming-iterator/latest/streaming_iterator... > I don't run into this in C or Go Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them. And Go lets you iterate over a fixed set of built-in types. (Currently. Maybe it's changing: https://research.swtch.com/coro https://research.swtch.com/coro) Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without.
- masklinn 3y ago
  TBF you could do it in most languages which do have iteration abstraction. However assuming trying to keep to 0-alloc odds are good you'd probably just be telling the user to git gud as the slices would become nonsensical on every refill of the buffer.
  - burntsushi 3y ago
    Yes, 0-alloc and minimal copying is a key constraint here. :-) If you take that away, then there's no real problem here.
    - masklinn 3y ago
      I could easily imagine trying to do it because I can rather than because I actually need to. Also because of the missing middle: in Go (or Java, or C#, or even python) you could hand out slices to a large buffer, but discard the buffer rather than refill it in place, relying on the GC to clean things up (and possibly give you the same buffer on the next allocation). So you get an efficiency middle ground where you allocate more than strictly necessary but not for every line, and maintain correctness. In Rust that’d require some sort of Rc projection which I’m not sure even exists?
- throwaway894345 3y ago
  > Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them. My goal isn’t “implement an Iterator trait”, I just want to iterate over lines in a file, so I don’t especially care that Go and C don’t have an iterator type. > Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without. I’m not allocating, so the GC doesn’t matter. The Go and C versions look essentially the same—it’s only the Rust version that I had a hard time with because of the borrow checker.
  - burntsushi 3y ago
    Great, then the closure approach should work perfectly! > I just want to iterate over lines in a file No... you don't. You specifically said you wanted to do this without allocating and minimal copying. That's a different problem than "just iterate over lines in a file." > I’m not allocating, so the GC doesn’t matter. Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up.

masklinn 3y ago

What you're talking about is the canonical example for generic associated types, which were stabilised late 2022: https://blog.rust-lang.org/2022/11/03/Rust-1.65.0.html#generic-associated-types-gats https://blog.rust-lang.org/2022/11/03/Rust-1.65.0.html#gener...

throwaway894345 3y ago
Ah, good to know. Thanks!

burntsushi 3y ago

Alternatively, use a closure: https://docs.rs/bstr/latest/bstr/io/trait.BufReadExt.html#method.for_byte_line https://docs.rs/bstr/latest/bstr/io/trait.BufReadExt.html#me... There are downsides to this approach because it uses internal iteration while most things in Rust use external iteration. But shit happens. Go doesn't even have a first class concept of iterators as an abstraction (yet), and its standard library contains patterns for both internal (sync.Map) and external (bufio.Scanner) iteration. > The only thing I can think to do is have a `scan()` method that finds the next newline and notes its location inside the `Lines` struct and a separate `line()` method that actually fetches the resulting slice from the buffer. Yup that works too. That's basically the design used by the `streaming-iterator` crate: https://docs.rs/streaming-iterator/latest/streaming_iterator/trait.StreamingIterator.html https://docs.rs/streaming-iterator/latest/streaming_iterator... > I don't run into this in C or Go Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them. And Go lets you iterate over a fixed set of built-in types. (Currently. Maybe it's changing: https://research.swtch.com/coro https://research.swtch.com/coro) Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without.

masklinn 3y ago
TBF you could do it in most languages which do have iteration abstraction. However assuming trying to keep to 0-alloc odds are good you'd probably just be telling the user to git gud as the slices would become nonsensical on every refill of the buffer.
- burntsushi 3y ago
  Yes, 0-alloc and minimal copying is a key constraint here. :-) If you take that away, then there's no real problem here.
  - masklinn 3y ago
    I could easily imagine trying to do it because I can rather than because I actually need to. Also because of the missing middle: in Go (or Java, or C#, or even python) you could hand out slices to a large buffer, but discard the buffer rather than refill it in place, relying on the GC to clean things up (and possibly give you the same buffer on the next allocation). So you get an efficiency middle ground where you allocate more than strictly necessary but not for every line, and maintain correctness. In Rust that’d require some sort of Rc projection which I’m not sure even exists?
throwaway894345 3y ago
> Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them. My goal isn’t “implement an Iterator trait”, I just want to iterate over lines in a file, so I don’t especially care that Go and C don’t have an iterator type. > Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without. I’m not allocating, so the GC doesn’t matter. The Go and C versions look essentially the same—it’s only the Rust version that I had a hard time with because of the borrow checker.
- burntsushi 3y ago
  Great, then the closure approach should work perfectly! > I just want to iterate over lines in a file No... you don't. You specifically said you wanted to do this without allocating and minimal copying. That's a different problem than "just iterate over lines in a file." > I’m not allocating, so the GC doesn’t matter. Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up.