5 ms·

io_uring still has its wrinkles. We are scrambling right now to fix a problem due to change in behavior exposed to user-space from the io_uring kernel module i

io_uring still has its wrinkles. We are scrambling right now to fix a problem due to change in behavior exposed to user-space from the io_uring kernel module in later kernels. Turns out that in earlier kernels (Ubuntu 19.04 5.3.0-51-generic #44-Ubuntu SMP) io_uring will not return short reads/writes (that's where you ask for e.g. 8k, but there's only 4k in the buffer cache, so the call doesn't signal as complete and blocks until all 8k has been transferred). In later kernels (not sure when the behavior changed, but the one shipped with Fedora 32 has the new behavior) io_uring returns partial (short) reads to user space. e.g. You ask for 8k but there's only 4k in the buffer cache, so the call signals complete with a return of only 4k read, not the 8k you asked for. Userspace code now has to cope with this where it didn't before. You could argue (and kernel developers did :-) that this was always possible, so user code needs to be aware of this. But it didn't used to do that :-). Change for user space is bad, mkay :-).

magicalhippo 6y ago
I know nothing about io_uring but looking at the man page[1] of readv I see it returns number of bytes read. For me as a developer that's an unmistakable flag that partial reads is possible. Was readv changed? The man page also states that partial reads is possible, but I guess that might have been added later? If it always returned bytes read, it would hardly be the first case where the current behavior is mistaken for the specification. My fondest memory of that is all the OpenGL 1.x programs that broke when OpenGL 2.x was released. [1]: http://man7.org/linux/man-pages/man2/readv.2.html http://man7.org/linux/man-pages/man2/readv.2.html
- jra_samba_org 6y ago
  Well pread/pwrite have the same return values, and historically for disk reads they block or return a device error. pread only returns a short value on EOF.
  - magicalhippo 6y ago
    Well, the man page does say that "The readv() system call works just like read(2) except that multiple buffers are filled". If we go to read(2) we find "It is not an error if [the return value] is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now [...], or because read() was interrupted by a signal." As an outsider, I'd never rely on this returning the requested number of bytes. If I required N bytes, I'd write use a read loop. But I do agree that the RWF_NOWAIT flag mentioned in your other comment doesn't help, as it suggests the default is to block.
  - loeg 6y ago
    Well, or EINTR if your signal handlers are not SA_RESTART.
    - jra_samba 6y ago
      For EINTR it never returns a short read, as the only way to see EINTR is a return of -1 with errno==EINTR. We handle signals fine.
      - loeg 6y ago
        Sure, I was imprecise. A signal can cause a read to return a short result.
- jra_samba 6y ago
  Also, note the preadv2 man page which has a flags field with one flag defined as: ------------------------------- RWF_NOWAIT (since Linux 4.14) Do not wait for data which is not immediately available. If this flag is specified, the preadv2() system call will return instantly if it would have to read data from the backing storage or wait for a lock. If some data was successfully read, it will return the number of bytes read. ------------------------------- This implies that "standard" pread/preadv/preadv2 without that flag (which is only available for preadv2) will block waiting for all bytes (or short return on EOF) and you need to set a flag to get the non-blocking behavior you're describing here. Otherwise the flag would be the inverse - RWF_WAIT, implying the standard behavior is the non-blocking one, not the blocking one. The blocking behavior is what we were expecting (and previously got) out of io_uring, so it was an unpleasant surprise to see the behavior change visible to user-space in later kernels.
  - altfredd 6y ago
    > If this flag is specified, the preadv2() system call will return instantly if it would... wait for a lock. Doesn't this sound a bit different from ordinary short reads? Receiving EAGAIN usually happens under fairly specific conditions (signal interruption), but I'd imagine, that filesystem code has a great deal of locks. For example, FUSE filesystems can support signal interruptions via EAGAIN, but they are not guaranteed to. You can end up in situation, when FUSE filesystem hangs, and you can not interrupt the thread, which reads from it. I suspect, that RWF_WAIT is a "fix" for similar situations and not the opposite of default behavior.
jra_samba 6y ago
It was really interesting how this was found. A user started describing file corruption when copying to/from Windows with the io_uring VFS module loaded. Tests using the Linux kernel cifsfs client and the Samba libsmbclient libraries/smbclient user-space transfer utility couldn't reproduce the problem, neither could running Windows against Samba on Ubuntu 19.04. What turned out to be happening was a combination of things. Firstly, the kernel changed so an SMB2_READ request against Samba with io_uring loaded was sometimes hitting a short read, where some of file data was already in the buffer cache, so io_uring now returned a short read to smbd. We returned this to the client, as in the SMB2 protocol it isn't an error to return a short read, the client is supposed to check read returns and then re-issue another read request for any missing bytes. The Linux kernel cifsfs client and Samba libsmbclient/smbclient did this correctly. But it turned out that Windows10 clients and MacOSX Catalina (maybe earlier versions of clients too, I don't have access to those) clients have a horrible bug, where they're not checking read returns when doing pipeline reads. When trying to read a 10GB file for example, they'll issue a series of 1MB reads at 1MB boundaries, up to their SMB2 credit limit, without waiting for replies. This is an excellent way to improve network file copy performance as you fill the read pipe without waiting for reply latency - indeed both Linux cifsfs and smbclient do exactly the same. But if one of those reads returns a short value, Windows10 and MacOSX Catalina DON'T GO BACK AND RE-READ THE MISSING BYTES FROM THE SHORT READ REPLY !!!! This is catastrophic, and will corrupt any file read from the server (the local client buffer cache fills the file contents I'm assuming with zeros - I haven't checked, but the files are corrupt as checked by SHA256 hashing anyway). That's how we discovered the behavior and ended up leading back to the io_uring behavior change. And that's why I hate it when kernel interfaces expose changes to user-space :-).
- andoma 6y ago
  So it's a Windows and MacOS bug then? Ie, no shadow should fall on io_uring really? That said, nice dig figuring this out. These type of bugs can be really frustrating to round up.
  - jra_samba 6y ago
    It is a Windows and MacOS bug, but only visible with the io_uring change in behavior. Problem is we have to fix the server, as even when buggy, one billion clients can never be wrong.

magicalhippo 6y ago

I know nothing about io_uring but looking at the man page[1] of readv I see it returns number of bytes read. For me as a developer that's an unmistakable flag that partial reads is possible. Was readv changed? The man page also states that partial reads is possible, but I guess that might have been added later? If it always returned bytes read, it would hardly be the first case where the current behavior is mistaken for the specification. My fondest memory of that is all the OpenGL 1.x programs that broke when OpenGL 2.x was released. [1]: http://man7.org/linux/man-pages/man2/readv.2.html http://man7.org/linux/man-pages/man2/readv.2.html

jra_samba_org 6y ago
Well pread/pwrite have the same return values, and historically for disk reads they block or return a device error. pread only returns a short value on EOF.
- magicalhippo 6y ago
  Well, the man page does say that "The readv() system call works just like read(2) except that multiple buffers are filled". If we go to read(2) we find "It is not an error if [the return value] is smaller than the number of bytes requested; this may happen for example because fewer bytes are actually available right now [...], or because read() was interrupted by a signal." As an outsider, I'd never rely on this returning the requested number of bytes. If I required N bytes, I'd write use a read loop. But I do agree that the RWF_NOWAIT flag mentioned in your other comment doesn't help, as it suggests the default is to block.
- loeg 6y ago
  Well, or EINTR if your signal handlers are not SA_RESTART.
  - jra_samba 6y ago
    For EINTR it never returns a short read, as the only way to see EINTR is a return of -1 with errno==EINTR. We handle signals fine.
    - loeg 6y ago
      Sure, I was imprecise. A signal can cause a read to return a short result.
jra_samba 6y ago
Also, note the preadv2 man page which has a flags field with one flag defined as: ------------------------------- RWF_NOWAIT (since Linux 4.14) Do not wait for data which is not immediately available. If this flag is specified, the preadv2() system call will return instantly if it would have to read data from the backing storage or wait for a lock. If some data was successfully read, it will return the number of bytes read. ------------------------------- This implies that "standard" pread/preadv/preadv2 without that flag (which is only available for preadv2) will block waiting for all bytes (or short return on EOF) and you need to set a flag to get the non-blocking behavior you're describing here. Otherwise the flag would be the inverse - RWF_WAIT, implying the standard behavior is the non-blocking one, not the blocking one. The blocking behavior is what we were expecting (and previously got) out of io_uring, so it was an unpleasant surprise to see the behavior change visible to user-space in later kernels.
- altfredd 6y ago
  > If this flag is specified, the preadv2() system call will return instantly if it would... wait for a lock. Doesn't this sound a bit different from ordinary short reads? Receiving EAGAIN usually happens under fairly specific conditions (signal interruption), but I'd imagine, that filesystem code has a great deal of locks. For example, FUSE filesystems can support signal interruptions via EAGAIN, but they are not guaranteed to. You can end up in situation, when FUSE filesystem hangs, and you can not interrupt the thread, which reads from it. I suspect, that RWF_WAIT is a "fix" for similar situations and not the opposite of default behavior.

jra_samba 6y ago

It was really interesting how this was found. A user started describing file corruption when copying to/from Windows with the io_uring VFS module loaded. Tests using the Linux kernel cifsfs client and the Samba libsmbclient libraries/smbclient user-space transfer utility couldn't reproduce the problem, neither could running Windows against Samba on Ubuntu 19.04. What turned out to be happening was a combination of things. Firstly, the kernel changed so an SMB2_READ request against Samba with io_uring loaded was sometimes hitting a short read, where some of file data was already in the buffer cache, so io_uring now returned a short read to smbd. We returned this to the client, as in the SMB2 protocol it isn't an error to return a short read, the client is supposed to check read returns and then re-issue another read request for any missing bytes. The Linux kernel cifsfs client and Samba libsmbclient/smbclient did this correctly. But it turned out that Windows10 clients and MacOSX Catalina (maybe earlier versions of clients too, I don't have access to those) clients have a horrible bug, where they're not checking read returns when doing pipeline reads. When trying to read a 10GB file for example, they'll issue a series of 1MB reads at 1MB boundaries, up to their SMB2 credit limit, without waiting for replies. This is an excellent way to improve network file copy performance as you fill the read pipe without waiting for reply latency - indeed both Linux cifsfs and smbclient do exactly the same. But if one of those reads returns a short value, Windows10 and MacOSX Catalina DON'T GO BACK AND RE-READ THE MISSING BYTES FROM THE SHORT READ REPLY !!!! This is catastrophic, and will corrupt any file read from the server (the local client buffer cache fills the file contents I'm assuming with zeros - I haven't checked, but the files are corrupt as checked by SHA256 hashing anyway). That's how we discovered the behavior and ended up leading back to the io_uring behavior change. And that's why I hate it when kernel interfaces expose changes to user-space :-).

andoma 6y ago
So it's a Windows and MacOS bug then? Ie, no shadow should fall on io_uring really? That said, nice dig figuring this out. These type of bugs can be really frustrating to round up.
- jra_samba 6y ago
  It is a Windows and MacOS bug, but only visible with the io_uring change in behavior. Problem is we have to fix the server, as even when buggy, one billion clients can never be wrong.