4 ms·

Yes, I'd like both! Asynchronous io_uring is good for things that take a lot of time, e.g. io. Its really just a queue between the user-space and kernel threa

Yes, I'd like both! Asynchronous io_uring is good for things that take a lot of time, e.g. io. Its really just a queue between the user-space and kernel threads. But it doesn't replace just making a synchronous syscall for small things e.g. stating files etc. A lot of those syscalls are called in quick succession, often with no logic between them. The Soares paper I linked too showed how buffering up a list of syscalls, doing a single context switch to the kernel, executing them and then switching back is a major gain to a wide variety of applications. Sychronous syscall batching would be much more straightforward to adopt than aysnc io; imagine if cstdlib file stuff all used buffered syscalls - everyone would benefit! Whereas I would imagine it adopting true async io would be very hard because it is a exposing a blocking api to its user.

the8472 7y ago
You can use io_uring for synchronous batching too. All you have to do is submit a batch and set min_complete to the number of submitted items when calling io_uring_enter. That's a single syscall, assuming you already have some thread-local ring lying around just for batching.
- yxhuvud 7y ago
  You probably also need to set the flag that enables the events to be linked, so that they happen in the correct order.
  - the8472 7y ago
    Only when order matters. For the given example of stating a lot of files it wouldn't.
codetrotter 7y ago
> A lot of those syscalls are called in quick succession, often with no logic between them. Speaking of which, I noticed on macOS that the “du” command-line utility seems much slower than a third-party GUI utility called DaisyDisk [1] at collecting the sizes of everything on a disk. This has me wondering what DaisyDisk is doing, if anything, that “du” isn’t, to achieve this. One theory of mine is that perhaps APFS caches metadata about file sizes and that this can be accessed faster than walking all directories and stat’ing all the things. An alternative theory I have is that perhaps GUI apps just run with a higher priority. If anyone knows, I’d be delighted to learn. A third possibly is that all of this was just due to “noise”. For example, disks happening to be under less I/O pressure when I ran DaisyDisk than when I’ve run “du”. My perception of their relative speed is based on very little scientific rigor. [1]: https://daisydiskapp.com/ https://daisydiskapp.com/
- the8472 7y ago
  It might just be using more threads and thus filling command queues better.
  - winrid 7y ago
    Right. It's likely multithreaded whereas du is not. You can find some multithreaded versions of du though.
- IAmEveryone 7y ago
  One of the features of APFS is supposedly a cache for total size of directory trees. As far as I remember, it is something the OS decides to turn on or off for different folders. It’s wasteful for /tmp, for example, because it makes file creation and deletion slightly slower. I’m not entirely sure if du would want to tap into that. A lot has changed in MacOS, and it’s not entirely clear if we still have a universal definition of “file size”. You can let iCloud store large files now, for example, truncating the local file when it is rarely accessed. Don’t remember the exact name, sorry. There is a MacOS-native way to explore recursive size, by the way: about this Mac -> storage -> manage -> documents -> file browser. This is limited to your user folder by default, but you can just create a soft link to take you to /
  - derefr 7y ago
    > You can let iCloud store large files now, for example, truncating the local file when it is rarely accessed. Do you know how that translates to the BSD layer? Does such a file appear to exist and have a certain size, but is e.g. exclusively read-locked, unless you manually download it? Or does fopen(2) block, trigger an iCloud download, and then unblock once it's complete?
namibj 7y ago
Stat and open are supported by io_uring in 5.6

the8472 7y ago

You can use io_uring for synchronous batching too. All you have to do is submit a batch and set min_complete to the number of submitted items when calling io_uring_enter. That's a single syscall, assuming you already have some thread-local ring lying around just for batching.

yxhuvud 7y ago
You probably also need to set the flag that enables the events to be linked, so that they happen in the correct order.
- the8472 7y ago
  Only when order matters. For the given example of stating a lot of files it wouldn't.

codetrotter 7y ago

> A lot of those syscalls are called in quick succession, often with no logic between them. Speaking of which, I noticed on macOS that the “du” command-line utility seems much slower than a third-party GUI utility called DaisyDisk [1] at collecting the sizes of everything on a disk. This has me wondering what DaisyDisk is doing, if anything, that “du” isn’t, to achieve this. One theory of mine is that perhaps APFS caches metadata about file sizes and that this can be accessed faster than walking all directories and stat’ing all the things. An alternative theory I have is that perhaps GUI apps just run with a higher priority. If anyone knows, I’d be delighted to learn. A third possibly is that all of this was just due to “noise”. For example, disks happening to be under less I/O pressure when I ran DaisyDisk than when I’ve run “du”. My perception of their relative speed is based on very little scientific rigor. [1]: https://daisydiskapp.com/ https://daisydiskapp.com/

the8472 7y ago
It might just be using more threads and thus filling command queues better.
- winrid 7y ago
  Right. It's likely multithreaded whereas du is not. You can find some multithreaded versions of du though.
IAmEveryone 7y ago
One of the features of APFS is supposedly a cache for total size of directory trees. As far as I remember, it is something the OS decides to turn on or off for different folders. It’s wasteful for /tmp, for example, because it makes file creation and deletion slightly slower. I’m not entirely sure if du would want to tap into that. A lot has changed in MacOS, and it’s not entirely clear if we still have a universal definition of “file size”. You can let iCloud store large files now, for example, truncating the local file when it is rarely accessed. Don’t remember the exact name, sorry. There is a MacOS-native way to explore recursive size, by the way: about this Mac -> storage -> manage -> documents -> file browser. This is limited to your user folder by default, but you can just create a soft link to take you to /
- derefr 7y ago
  > You can let iCloud store large files now, for example, truncating the local file when it is rarely accessed. Do you know how that translates to the BSD layer? Does such a file appear to exist and have a certain size, but is e.g. exclusively read-locked, unless you manually download it? Or does fopen(2) block, trigger an iCloud download, and then unblock once it's complete?

namibj 7y ago

Stat and open are supported by io_uring in 5.6