4 ms·

Programs should really use unbuffered i/o for large files read only once (yes i know Linus doesn’t like unbuffered i/o but he’s wrong) > This comes into play w

Programs should really use unbuffered i/o for large files read only once (yes i know Linus doesn’t like unbuffered i/o but he’s wrong) > This comes into play when you copy or access huge files that are going to be read exactly once

caf 7y ago
Readahead is still useful for large files read sequentially once, and that needs to be buffered. Such programs should use posix_fadvise().
- paulsutter 7y ago
  you can readahead as far as you like with unbuffered io
  - caf 7y ago
    If you are reading unbuffered (ie. O_DIRECT) then you are reading directly into the memory block the user supplied, so you cannot read ahead - there's nowhere to put the extra data.
    - paulsutter 7y ago
      Of course you can read ahead using multiple buffers, you can issue as many reads as you want concurrently
      - caf 7y ago
        I think it is pretty clear I was referring to kernel-mediated readahead. Sure, you can achieve the same thing in userspace using async IO or threads.
        paulsutter 7y ago
        I think it’s pretty clear that I was referring to reading large files using direct io, and your comment was totally irrelevant
        caf 7y ago
        Reading large files using direct IO defeats kernel read-ahead, which means you have to take on the complexity of reimplementing it in userspace, or the performance hit of not having it. This is a good reason for programs to use buffered IO even when they are reading a large file once, so yes my comment was entirely on point.
        paulsutter 7y ago
        Your comments were misleading, even if you want to argue they were on point. I know that you're knowledgable about the kernel, I ready your other comments. The following comments are just not true, except in bizarro world where all code must be in the kernel: - "Readahead... needs to be buffered" - "reading unbuffered (ie. O_DIRECT)... you cannot read ahead" Good application code needs to work around kernel limitations all the time. Your suggestion of fadvise is reasonable, except in practice fadvise rarely makes a measurable difference. The only way to get what you really want is to code it the way you want it, and fortunately, there are multiple ways to do that.

caf 7y ago

Readahead is still useful for large files read sequentially once, and that needs to be buffered. Such programs should use posix_fadvise().

paulsutter 7y ago
you can readahead as far as you like with unbuffered io
- caf 7y ago
  If you are reading unbuffered (ie. O_DIRECT) then you are reading directly into the memory block the user supplied, so you cannot read ahead - there's nowhere to put the extra data.
  - paulsutter 7y ago
    Of course you can read ahead using multiple buffers, you can issue as many reads as you want concurrently
    - caf 7y ago
      I think it is pretty clear I was referring to kernel-mediated readahead. Sure, you can achieve the same thing in userspace using async IO or threads.
      - paulsutter 7y ago
        I think it’s pretty clear that I was referring to reading large files using direct io, and your comment was totally irrelevant
        caf 7y ago
        Reading large files using direct IO defeats kernel read-ahead, which means you have to take on the complexity of reimplementing it in userspace, or the performance hit of not having it. This is a good reason for programs to use buffered IO even when they are reading a large file once, so yes my comment was entirely on point.
        paulsutter 7y ago
        Your comments were misleading, even if you want to argue they were on point. I know that you're knowledgable about the kernel, I ready your other comments. The following comments are just not true, except in bizarro world where all code must be in the kernel: - "Readahead... needs to be buffered" - "reading unbuffered (ie. O_DIRECT)... you cannot read ahead" Good application code needs to work around kernel limitations all the time. Your suggestion of fadvise is reasonable, except in practice fadvise rarely makes a measurable difference. The only way to get what you really want is to code it the way you want it, and fortunately, there are multiple ways to do that.