4 ms·

Alternatively, use a closure: https://docs.rs/bstr/latest/bstr/io/trait.BufReadExt.html#method.for_byte_line https://docs.rs/bstr/latest/bstr/io/trait.BufReadEx

Alternatively, use a closure: https://docs.rs/bstr/latest/bstr/io/trait.BufReadExt.html#method.for_byte_line https://docs.rs/bstr/latest/bstr/io/trait.BufReadExt.html#me... There are downsides to this approach because it uses internal iteration while most things in Rust use external iteration. But shit happens. Go doesn't even have a first class concept of iterators as an abstraction (yet), and its standard library contains patterns for both internal (sync.Map) and external (bufio.Scanner) iteration. > The only thing I can think to do is have a `scan()` method that finds the next newline and notes its location inside the `Lines` struct and a separate `line()` method that actually fetches the resulting slice from the buffer. Yup that works too. That's basically the design used by the `streaming-iterator` crate: https://docs.rs/streaming-iterator/latest/streaming_iterator/trait.StreamingIterator.html https://docs.rs/streaming-iterator/latest/streaming_iterator... > I don't run into this in C or Go Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them. And Go lets you iterate over a fixed set of built-in types. (Currently. Maybe it's changing: https://research.swtch.com/coro https://research.swtch.com/coro) Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without.

masklinn 3y ago
TBF you could do it in most languages which do have iteration abstraction. However assuming trying to keep to 0-alloc odds are good you'd probably just be telling the user to git gud as the slices would become nonsensical on every refill of the buffer.
- burntsushi 3y ago
  Yes, 0-alloc and minimal copying is a key constraint here. :-) If you take that away, then there's no real problem here.
  - masklinn 3y ago
    I could easily imagine trying to do it because I can rather than because I actually need to. Also because of the missing middle: in Go (or Java, or C#, or even python) you could hand out slices to a large buffer, but discard the buffer rather than refill it in place, relying on the GC to clean things up (and possibly give you the same buffer on the next allocation). So you get an efficiency middle ground where you allocate more than strictly necessary but not for every line, and maintain correctness. In Rust that’d require some sort of Rc projection which I’m not sure even exists?
throwaway894345 3y ago
> Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them. My goal isn’t “implement an Iterator trait”, I just want to iterate over lines in a file, so I don’t especially care that Go and C don’t have an iterator type. > Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without. I’m not allocating, so the GC doesn’t matter. The Go and C versions look essentially the same—it’s only the Rust version that I had a hard time with because of the borrow checker.
- burntsushi 3y ago
  Great, then the closure approach should work perfectly! > I just want to iterate over lines in a file No... you don't. You specifically said you wanted to do this without allocating and minimal copying. That's a different problem than "just iterate over lines in a file." > I’m not allocating, so the GC doesn’t matter. Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up.
  - throwaway894345 3y ago
    > No... you don't. You specifically said you wanted to do this without allocating and minimal copying. That's a different problem than "just iterate over lines in a file." Yes, of course. The point is iterating over the lines of a file given the aforementioned constraints and not “implementing Rust’s Iterator abstraction”. Whether or not Go or C have iterator abstractions is immaterial. > Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up. GC only manages allocations. There are no allocations here to manage.
    - burntsushi 3y ago
      > GC only manages allocations. There are no allocations here to manage. You're missing the forest for the trees. The GC is integrally tied to lifetimes. If you have a `*Foo` in Go, that may or may not be on the stack. It might be on the stack, thus no allocation, if the Go compiler can prove something about its lifetime and usage. Same deal with `[]byte`. Maybe that's tied to an array on a stack somewhere. Maybe not. AFAIK, in order to get guarantees about this in Go you need to drop down into `unsafe`. > Yes, of course. The point is iterating over the lines of a file given the aforementioned constraints and not “implementing Rust’s Iterator abstraction”. Whether or not Go or C have iterator abstractions is immaterial. That's why the very first thing I said to you was to point out the closure approach. I even linked you to real code (that I've written) that does it. Yet 'round and 'round we go.

masklinn 3y ago

TBF you could do it in most languages which do have iteration abstraction. However assuming trying to keep to 0-alloc odds are good you'd probably just be telling the user to git gud as the slices would become nonsensical on every refill of the buffer.

burntsushi 3y ago
Yes, 0-alloc and minimal copying is a key constraint here. :-) If you take that away, then there's no real problem here.
- masklinn 3y ago
  I could easily imagine trying to do it because I can rather than because I actually need to. Also because of the missing middle: in Go (or Java, or C#, or even python) you could hand out slices to a large buffer, but discard the buffer rather than refill it in place, relying on the GC to clean things up (and possibly give you the same buffer on the next allocation). So you get an efficiency middle ground where you allocate more than strictly necessary but not for every line, and maintain correctness. In Rust that’d require some sort of Rc projection which I’m not sure even exists?

throwaway894345 3y ago

> Of course you don't. Neither C or Go even have abstractions called "iteration" at all. They have patterns for them. My goal isn’t “implement an Iterator trait”, I just want to iterate over lines in a file, so I don’t especially care that Go and C don’t have an iterator type. > Besides, Go has a garbage collector. You should expect all sorts of patterns involving memory/copying to change when you move from a language with a GC to one without. I’m not allocating, so the GC doesn’t matter. The Go and C versions look essentially the same—it’s only the Rust version that I had a hard time with because of the borrow checker.

burntsushi 3y ago
Great, then the closure approach should work perfectly! > I just want to iterate over lines in a file No... you don't. You specifically said you wanted to do this without allocating and minimal copying. That's a different problem than "just iterate over lines in a file." > I’m not allocating, so the GC doesn’t matter. Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up.
- throwaway894345 3y ago
  > No... you don't. You specifically said you wanted to do this without allocating and minimal copying. That's a different problem than "just iterate over lines in a file." Yes, of course. The point is iterating over the lines of a file given the aforementioned constraints and not “implementing Rust’s Iterator abstraction”. Whether or not Go or C have iterator abstractions is immaterial. > Of course it does. The GC manages the lifetimes for you. In your C code, you manage the lifetimes yourself and likely rely on the caller to not fuck things up. GC only manages allocations. There are no allocations here to manage.
  - burntsushi 3y ago
    > GC only manages allocations. There are no allocations here to manage. You're missing the forest for the trees. The GC is integrally tied to lifetimes. If you have a `*Foo` in Go, that may or may not be on the stack. It might be on the stack, thus no allocation, if the Go compiler can prove something about its lifetime and usage. Same deal with `[]byte`. Maybe that's tied to an array on a stack somewhere. Maybe not. AFAIK, in order to get guarantees about this in Go you need to drop down into `unsafe`. > Yes, of course. The point is iterating over the lines of a file given the aforementioned constraints and not “implementing Rust’s Iterator abstraction”. Whether or not Go or C have iterator abstractions is immaterial. That's why the very first thing I said to you was to point out the closure approach. I even linked you to real code (that I've written) that does it. Yet 'round and 'round we go.