3 ms·

Here's a wart. Why is reading lines from a file so hard? It's one of the first things people are going to need to do in a programming language. On top of handl

by rspeer 10y ago

Here's a wart. Why is reading lines from a file so hard? It's one of the first things people are going to need to do in a programming language. On top of handling errors (which I understand is necessary, and which the ? operator makes easier), it requires importing BufReader and BufRead, and wrapping a reference to a file handle in BufReader. Nobody is going to know how to do this unless they come across it in Rust By Example or on Stack Overflow. Shouldn't a standard library be able to abstract over fiddly details like this? Why is it my job to tell it how to buffer? I say this as someone who is enthusiastic about Rust, but still learning.

douche 10y ago
That's a good point. It's slightly reminiscent of Haskell, where you have to start trying to understand the IO monad and do notation to do the most trivial real-world examples. In other languages, you might do File.ReadAllLines("foo.txt"), and bam, you're done.
- dripton 10y ago
  Unless foo.txt is too big, in which case, bam, out of memory.
  - rspeer 10y ago
    Iterators exist.
  - int_19h 10y ago
    In C# (from which the original example is), these days - and for like 6 years now - what you do is File.ReadLines, and you get a lazy enumerator.
  - sametmax 10y ago
    No, Python, ruby and such have a common interface for all lazy iterations. In Python you can just loop on anything declaring the __iter__ method, and it will lazily yield results little by little. IO related objects implement it plus an additional layer of interface so that you can do: with open('file', [mode, encoding]) as f: To get an auto closing file handle and then choose: - for `line in f` to lazy read it line by line. This calls __iter__. - `f.read([byte_count])` to read it all or some bunch of bytes. `f.seek(index)` to move around, etc. - `f.readlines()` to get a list of all lines in memory. - `f.close()` if you wish to close the file manually instead of letting the `with` keyword doing it work you. This interface works for most IO, including files, sockets, in memory buffers, etc. So you can choose an automatic lazy loading, a manual loading, load everything in memory, etc. And still have a lot of control. I think rust should get some traits to expose such a common high level interface on top of the current way it deals with files, to ease simple operations. Just make sure the documentation states what you can do to go lower level. There is a similar trend with many basic topics trying to do that as 3rd party libs too. click makes creating cmd UI very easy on top of argparse which is lower level. pendulum is higher level than datetime. requests higher than urllib. etc. Those all make doing those very common operation super easy. Now rust is not meant to be Python/Ruby/whatever, being very low level, and checking safety at compile time implies very different requirements. But those communities have some good concepts on API ergonomics and it would be a shame to not steal ideas from them.
    - kazagistar 10y ago
      The Rust team seems to not want to write trait impls where it would be inefficient to use those traits. If a method doesn't exist, its often because there is a slightly better way. They also insist that memory usage, especially any heap allocations, be very clear and explicit: allocating and attaching a buffer is probably not OK within the internal logic of the library designs. The interface for reading is split into multiple parts. Read is the lower level interface for any stream of bytes, and is what lets you read a specific number of bytes into a preallocated buffer. Seek is implemented for files, but not all streams, because in most cases it makes no sense. There are also a number of file-specific methods directly attached to File for handling permissions and sync and such. BufRead is the trait that lets you read all the lines, but it is explicitly not implemented for File, since using it for a File would be slow, and generally the wrong way to do things. It does have an implementation for Stdin, because that is already buffered, but if you want to associate a buffer with a file, you have to do it yourself by wrapping it in a ReadBuffer. If I were to find a problem with the File documentation, it is that it does not mention BufReader.
      - sametmax 10y ago
        Make senses. I tried to apply a solution to the wrong problem.
steveklabnik 10y ago
> Why is reading lines from a file so hard? It's not so much that it's hard, as is there's lots of options. Can you afford to read it all into memory as a String? Do you need to only read it bit by bit? > Why is it my job to tell it how to buffer? Systems langauges need to expose these kinds of details and levels of control. > Nobody is going to know how to do this I googled "rust open a file and read by lines" and got these results: https://users.rust-lang.org/t/read-a-file-line-by-line/1585 https://users.rust-lang.org/t/read-a-file-line-by-line/1585 https://doc.rust-lang.org/std/io/ https://doc.rust-lang.org/std/io/ https://doc.rust-lang.org/std/fs/struct.File.html https://doc.rust-lang.org/std/fs/struct.File.html The first two show you directly how, though the first one is dealing with the error message. The last one shows reading it all in. I have some more work to do :)
- rspeer 10y ago
  It would be silly and wasteful to read the whole file into memory just so I can iterate its lines. I'm not sure why that's an option you'd need to accomodate. However, many languages make a reasonable assumption that you can afford to fit each line in memory, and provide an obvious way to do this.
  - steveklabnik 10y ago
    > I'm not sure why that's an option you'd need to accomodate. Again, it's about control. Maybe you're only loading a small configuration file, and so fetching it all in one go is better than dealing with a buffer. > an obvious way to do this. If you search for 'line' or 'lines' in rustdoc, the correct thing is right near the top, which will show you how to use it with BufRead.
    - rspeer 10y ago
      > Maybe you're only loading a small configuration file, and so fetching it all in one go is better than dealing with a buffer. If it's small, it can't be that much worse to buffer. What's wrong with reasonable defaults? Can you tell me what it means to search for 'line' or 'lines' in rustdoc? Is this a command-line tool? I'm aware of the 'rustdoc' that generates documentation, but not of anything by that name that searches documentation, and Googling isn't turning up much of anything except the Rust book here. It would be great to have an offline way to search Rust documentation and examples, as things like this make it very hard to write Rust on a plane, for example. Googling often gives answers that are wrong or pre-1.0. You seem to have better luck at searching for the right things, perhaps because you are an expert in the language and know exactly what to search for. (Amusing related anecdote: I googled for "how to return self in rust" and got a suicide prevention website. Jeez, Google, it's not that bad.)

by rspeer 10y ago

douche 10y ago

That's a good point. It's slightly reminiscent of Haskell, where you have to start trying to understand the IO monad and do notation to do the most trivial real-world examples. In other languages, you might do File.ReadAllLines("foo.txt"), and bam, you're done.

dripton 10y ago
Unless foo.txt is too big, in which case, bam, out of memory.
- rspeer 10y ago
  Iterators exist.
- int_19h 10y ago
  In C# (from which the original example is), these days - and for like 6 years now - what you do is File.ReadLines, and you get a lazy enumerator.
- sametmax 10y ago
  No, Python, ruby and such have a common interface for all lazy iterations. In Python you can just loop on anything declaring the __iter__ method, and it will lazily yield results little by little. IO related objects implement it plus an additional layer of interface so that you can do: with open('file', [mode, encoding]) as f: To get an auto closing file handle and then choose: - for `line in f` to lazy read it line by line. This calls __iter__. - `f.read([byte_count])` to read it all or some bunch of bytes. `f.seek(index)` to move around, etc. - `f.readlines()` to get a list of all lines in memory. - `f.close()` if you wish to close the file manually instead of letting the `with` keyword doing it work you. This interface works for most IO, including files, sockets, in memory buffers, etc. So you can choose an automatic lazy loading, a manual loading, load everything in memory, etc. And still have a lot of control. I think rust should get some traits to expose such a common high level interface on top of the current way it deals with files, to ease simple operations. Just make sure the documentation states what you can do to go lower level. There is a similar trend with many basic topics trying to do that as 3rd party libs too. click makes creating cmd UI very easy on top of argparse which is lower level. pendulum is higher level than datetime. requests higher than urllib. etc. Those all make doing those very common operation super easy. Now rust is not meant to be Python/Ruby/whatever, being very low level, and checking safety at compile time implies very different requirements. But those communities have some good concepts on API ergonomics and it would be a shame to not steal ideas from them.
  - kazagistar 10y ago
    The Rust team seems to not want to write trait impls where it would be inefficient to use those traits. If a method doesn't exist, its often because there is a slightly better way. They also insist that memory usage, especially any heap allocations, be very clear and explicit: allocating and attaching a buffer is probably not OK within the internal logic of the library designs. The interface for reading is split into multiple parts. Read is the lower level interface for any stream of bytes, and is what lets you read a specific number of bytes into a preallocated buffer. Seek is implemented for files, but not all streams, because in most cases it makes no sense. There are also a number of file-specific methods directly attached to File for handling permissions and sync and such. BufRead is the trait that lets you read all the lines, but it is explicitly not implemented for File, since using it for a File would be slow, and generally the wrong way to do things. It does have an implementation for Stdin, because that is already buffered, but if you want to associate a buffer with a file, you have to do it yourself by wrapping it in a ReadBuffer. If I were to find a problem with the File documentation, it is that it does not mention BufReader.
    - sametmax 10y ago
      Make senses. I tried to apply a solution to the wrong problem.

steveklabnik 10y ago

> Why is reading lines from a file so hard? It's not so much that it's hard, as is there's lots of options. Can you afford to read it all into memory as a String? Do you need to only read it bit by bit? > Why is it my job to tell it how to buffer? Systems langauges need to expose these kinds of details and levels of control. > Nobody is going to know how to do this I googled "rust open a file and read by lines" and got these results: https://users.rust-lang.org/t/read-a-file-line-by-line/1585 https://users.rust-lang.org/t/read-a-file-line-by-line/1585 https://doc.rust-lang.org/std/io/ https://doc.rust-lang.org/std/io/ https://doc.rust-lang.org/std/fs/struct.File.html https://doc.rust-lang.org/std/fs/struct.File.html The first two show you directly how, though the first one is dealing with the error message. The last one shows reading it all in. I have some more work to do :)

rspeer 10y ago
It would be silly and wasteful to read the whole file into memory just so I can iterate its lines. I'm not sure why that's an option you'd need to accomodate. However, many languages make a reasonable assumption that you can afford to fit each line in memory, and provide an obvious way to do this.
- steveklabnik 10y ago
  > I'm not sure why that's an option you'd need to accomodate. Again, it's about control. Maybe you're only loading a small configuration file, and so fetching it all in one go is better than dealing with a buffer. > an obvious way to do this. If you search for 'line' or 'lines' in rustdoc, the correct thing is right near the top, which will show you how to use it with BufRead.
  - rspeer 10y ago
    > Maybe you're only loading a small configuration file, and so fetching it all in one go is better than dealing with a buffer. If it's small, it can't be that much worse to buffer. What's wrong with reasonable defaults? Can you tell me what it means to search for 'line' or 'lines' in rustdoc? Is this a command-line tool? I'm aware of the 'rustdoc' that generates documentation, but not of anything by that name that searches documentation, and Googling isn't turning up much of anything except the Rust book here. It would be great to have an offline way to search Rust documentation and examples, as things like this make it very hard to write Rust on a plane, for example. Googling often gives answers that are wrong or pre-1.0. You seem to have better luck at searching for the right things, perhaps because you are an expert in the language and know exactly what to search for. (Amusing related anecdote: I googled for "how to return self in rust" and got a suicide prevention website. Jeez, Google, it's not that bad.)