4 ms·

First of all, thank you for presenting a succinct take on this viewpoint from the other side of the fence from where I am at. So how can I learn from this? (As

First of all, thank you for presenting a succinct take on this viewpoint from the other side of the fence from where I am at. So how can I learn from this? (Asking very aggressively, especially for Internet writing, to make the contrast unmistakable. And contrast helps with perceiving differences and mistakes.) (You also don’t owe me any of your time or mental bandwidth, whatsoever.) So here goes: Question 1: How come "speed", "performance", race conditions and st_ino keep getting brought up? Speed (latency), physically writing things out to storage (sequentially, atomically (ACID), all of HDD NVME SSD ODD FDD tape, "haskell monad", event horizons, finite speed of light and information, whatever) as well as race conditions all seem to boil down to the same thing. For reliable systems like accounting the path seems to be ACID or the highway. And "unreliable" systems forget fast enough that computers don’t seem to really make a difference there. Question 2: Does throughput really matter more than latency in everyday application? Question 3 (explanation first, this time): The focus on inode numbers is at least understandable with regards to the history of C and unix-like operating systems and GNU coreutils. What about this basic example? Just make a USB thumb drive "work" for storing files (ignoring nand flash decay and USB). Without getting tripped up in libc IO buffering, fflush, kernel buffering (Hurd if you prefer it over Linux or FreeBSD), more than one application running on a multi-core and/or time-sliced system (to really weed out single-core CPUs running only a single user-land binary with blocking IO).

dijit 5mo ago
> Does throughput really matter more than latency in everyday application? In my experience latency and throughput are intrinsically linked unless you have the buffer-space to handle the throughput you want. Which you can't guarantee on all the systems where GNU Coreutils run.
- dapperdrake 5mo ago
  Higher throughput increases the risk of high latency. Low latency increases the risk of "wasted cycles”, i.e. lowers (machine) throughput. Helps with human discovery throughput, though. The sled.rs people had a well readable take on this in their performance guide.
awesome_dude 5mo ago
> Question 2: > Does throughput really matter more than latency in everyday application? IME as a user, hell yes Getting a video I don't mind if it buffers a moment, but once it starts I need all of that data moving to my player as quickly as possible OTOH if there's no wait, but the data is restricted (the amount coming to my player is less than the player needs to fully render the images), the video is "unwatchable"
- WJW 5mo ago
  I don't mean to nitpick, but absolute values for both of these matter much less than how much it is compared to "enough". As long as the throughput is enough to prevent the video from stuttering, it doesn't matter if the data is moved to your video player program at 1 GB/s or 1 TB/s. Conversely, you say you don't mind if a video buffers for a moment but I'm willing to bet there's some value of "a moment" where it becomes "too long". Nobody is willing to wait an hour buffering before their video starts. The perception of speed in using a computer is almost entirely latency driven these days. Compare using `rg` or `git` vs loading up your banking website.
- jorvi 5mo ago
  Hell no. Linux desktop (and the kernel) felt awful for such a long time because everyone was optimizing for server and workstation workloads. Its the reason CachyOS (and before that Linux Zen and.. Licorix?) are a thing. For good UX, you heavily prioritize latency over throughput. No one cares if copying a file stalls for a moment or takes 2 seconds longer if that ensures no hitches in alt tabbing, scrolling or mouse movement.
  - deleted 5mo ago
    [deleted]
  - pjmlp 5mo ago
    How many talks have you seen at USENIX that care about UNIX as desktop OS? Exactly.
  - ragall 5mo ago
    When Kon Colivas introduced a scheduler optimized for desktop latency, about 15 years ago, the amount of abuse he got from Linux developers was astonishing, and he ended up quitting for good. I remember compiling it on my laptop and noticing how it made a huge improvement in the useability of X and desktop environment.
duped 5mo ago
Just want to point out that race conditions are a correctness problem, not a performance problem.
- dapperdrake 5mo ago
  Accurate a.k.a. "correct" implementation of ACID needs a single (central) source of truth and temporal serializability (or something close to that). In practice this always "impacts" performance. If I understand it correctly, then in physics this is called an event horizon.
  - duped 5mo ago
    Not necessarily. Most race conditions violate the `A` in ACID, but the finicky thing about atomicity is that N > 1 sequential actions that in and of themselves are atomic violates atomicity. So any atomic store is possible to misuse if you can compose multiple atomic operations on it. In addition ACID isn't always provided by the floor beneath your programs but by designing the programs on top to uphold it and/or not require it, allowing you to relax the constraints from your lower level interfaces for performance reasons.
    - dapperdrake 5mo ago
      Firstly, atomicity and/or thread-safety not composing is where the Consistency and Isolation come in. The "application layer" always has to enforce its own consistency guarantees. If the lower layers are total garbage, then the system is garbage. And the "speed" of the lower layers can be infinitely fast and it doesn’t matter, if the application has a latency floor. So optimize it all you want.
ericbarrett 5mo ago
Coreutils are not only used in interactive contexts. They are the primitives that make up the countless shell scripts which glue systems together. Any edge case will be encountered and the resulting poor performance will impact somebody, somewhere. Here's a related example of what happens when you change a shell primitive's behavior - even interactively. Back in the 2000s, Linux distributions started adding color output to the ls command via a default "alias ls=/bin/ls --color=auto". You know: make directories blue, symlinks cyan, executables purple; that kind of thing. Somebody thought it would be a nice user experience upgrade. I was working at a NAS (NFS remote box) vendor in tech support. We frequently got calls from folks who had just switched to Linux from Solaris, or had just moved their home directories from local disk to NFS. They would complain that listing a directory with a lot of files would hang. If it came back at all, it would be in minutes or hours! The fix? "unalias ls". Because calling "/bin/ls" would execute a single READDIR (the NFS RPC), which was 1 round-trip to the server and only a few network packets; but calling "/bin/ls --color=auto" would add a STAT call for every single file in the directory to figure out what color it should be - sequentially, one-by-one, confirming the success of each before the next iteration. If you had 30,000 files with a round-trip time of 1ms that's 30 seconds. If you had millions...well, either you waited for hours or you power-cycled the box. (This was eventually fixed with NFSv3's READDIRPLUS.) Now I'm sure whomever changed that alias did not intend it, but they caused thousands of people thousands of hours of lost productivity. I was just one guy in one org's tech support group, and I saw at least a dozen such cases, not all of which were lucky enough to land in the queue of somebody who'd already seen the problem. So I really appreciate GNU coreutils' commitment to sane behavior even at the edges. If you do systems work long enough, you will ride those edges, and a tool which stays steady in your hand - or script - is invaluable.
- dapperdrake 5mo ago
  In short, NFS has a terrible data model and only pretends to be a file system.
  - pjmlp 5mo ago
    Hence why even on UNIX people moved on from NFS, but on Linux it keeps being the remote filesystem many reach for.
dapperdrake 5mo ago
Additional point: The point of data storage is to be a singleton. (Backups are desireable, anyhow.)

dijit 5mo ago

> Does throughput really matter more than latency in everyday application? In my experience latency and throughput are intrinsically linked unless you have the buffer-space to handle the throughput you want. Which you can't guarantee on all the systems where GNU Coreutils run.

dapperdrake 5mo ago
Higher throughput increases the risk of high latency. Low latency increases the risk of "wasted cycles”, i.e. lowers (machine) throughput. Helps with human discovery throughput, though. The sled.rs people had a well readable take on this in their performance guide.

awesome_dude 5mo ago

> Question 2: > Does throughput really matter more than latency in everyday application? IME as a user, hell yes Getting a video I don't mind if it buffers a moment, but once it starts I need all of that data moving to my player as quickly as possible OTOH if there's no wait, but the data is restricted (the amount coming to my player is less than the player needs to fully render the images), the video is "unwatchable"

WJW 5mo ago
I don't mean to nitpick, but absolute values for both of these matter much less than how much it is compared to "enough". As long as the throughput is enough to prevent the video from stuttering, it doesn't matter if the data is moved to your video player program at 1 GB/s or 1 TB/s. Conversely, you say you don't mind if a video buffers for a moment but I'm willing to bet there's some value of "a moment" where it becomes "too long". Nobody is willing to wait an hour buffering before their video starts. The perception of speed in using a computer is almost entirely latency driven these days. Compare using `rg` or `git` vs loading up your banking website.
jorvi 5mo ago
Hell no. Linux desktop (and the kernel) felt awful for such a long time because everyone was optimizing for server and workstation workloads. Its the reason CachyOS (and before that Linux Zen and.. Licorix?) are a thing. For good UX, you heavily prioritize latency over throughput. No one cares if copying a file stalls for a moment or takes 2 seconds longer if that ensures no hitches in alt tabbing, scrolling or mouse movement.
- deleted 5mo ago
  [deleted]
- pjmlp 5mo ago
  How many talks have you seen at USENIX that care about UNIX as desktop OS? Exactly.
- ragall 5mo ago
  When Kon Colivas introduced a scheduler optimized for desktop latency, about 15 years ago, the amount of abuse he got from Linux developers was astonishing, and he ended up quitting for good. I remember compiling it on my laptop and noticing how it made a huge improvement in the useability of X and desktop environment.

duped 5mo ago

Just want to point out that race conditions are a correctness problem, not a performance problem.

dapperdrake 5mo ago
Accurate a.k.a. "correct" implementation of ACID needs a single (central) source of truth and temporal serializability (or something close to that). In practice this always "impacts" performance. If I understand it correctly, then in physics this is called an event horizon.
- duped 5mo ago
  Not necessarily. Most race conditions violate the `A` in ACID, but the finicky thing about atomicity is that N > 1 sequential actions that in and of themselves are atomic violates atomicity. So any atomic store is possible to misuse if you can compose multiple atomic operations on it. In addition ACID isn't always provided by the floor beneath your programs but by designing the programs on top to uphold it and/or not require it, allowing you to relax the constraints from your lower level interfaces for performance reasons.
  - dapperdrake 5mo ago
    Firstly, atomicity and/or thread-safety not composing is where the Consistency and Isolation come in. The "application layer" always has to enforce its own consistency guarantees. If the lower layers are total garbage, then the system is garbage. And the "speed" of the lower layers can be infinitely fast and it doesn’t matter, if the application has a latency floor. So optimize it all you want.

ericbarrett 5mo ago

Coreutils are not only used in interactive contexts. They are the primitives that make up the countless shell scripts which glue systems together. Any edge case will be encountered and the resulting poor performance will impact somebody, somewhere. Here's a related example of what happens when you change a shell primitive's behavior - even interactively. Back in the 2000s, Linux distributions started adding color output to the ls command via a default "alias ls=/bin/ls --color=auto". You know: make directories blue, symlinks cyan, executables purple; that kind of thing. Somebody thought it would be a nice user experience upgrade. I was working at a NAS (NFS remote box) vendor in tech support. We frequently got calls from folks who had just switched to Linux from Solaris, or had just moved their home directories from local disk to NFS. They would complain that listing a directory with a lot of files would hang. If it came back at all, it would be in minutes or hours! The fix? "unalias ls". Because calling "/bin/ls" would execute a single READDIR (the NFS RPC), which was 1 round-trip to the server and only a few network packets; but calling "/bin/ls --color=auto" would add a STAT call for every single file in the directory to figure out what color it should be - sequentially, one-by-one, confirming the success of each before the next iteration. If you had 30,000 files with a round-trip time of 1ms that's 30 seconds. If you had millions...well, either you waited for hours or you power-cycled the box. (This was eventually fixed with NFSv3's READDIRPLUS.) Now I'm sure whomever changed that alias did not intend it, but they caused thousands of people thousands of hours of lost productivity. I was just one guy in one org's tech support group, and I saw at least a dozen such cases, not all of which were lucky enough to land in the queue of somebody who'd already seen the problem. So I really appreciate GNU coreutils' commitment to sane behavior even at the edges. If you do systems work long enough, you will ride those edges, and a tool which stays steady in your hand - or script - is invaluable.

dapperdrake 5mo ago
In short, NFS has a terrible data model and only pretends to be a file system.
- pjmlp 5mo ago
  Hence why even on UNIX people moved on from NFS, but on Linux it keeps being the remote filesystem many reach for.

dapperdrake 5mo ago

Additional point: The point of data storage is to be a singleton. (Backups are desireable, anyhow.)