5 ms·

That is getting considerably easier with the addition of pidfds, though: https://lwn.net/SubscriberLink/794707/93ffb35438fd3710/ https://lwn.net/SubscriberLink/

by AnssiH 7y ago

That is getting considerably easier with the addition of pidfds, though: https://lwn.net/SubscriberLink/794707/93ffb35438fd3710/ https://lwn.net/SubscriberLink/794707/93ffb35438fd3710/

mehrdadn 7y ago
Posted 1 week ago! Do you know what took them so long to realize this is the correct way to do it? Windows has had it for some two decades...
- caspper69 7y ago
  Unfortunately, as Linux has matured, many of its developers refuse to adapt technologies for the sole reason that it "stinks" of Microsoft. Cutler and his team were pretty forward thinking given that NT is over 30 years old. See also: the async i/o headaches and the inability to WaitForSingleObject()/WaitForMultipleObjects() (or an analogue) in Linux. It's a shame really.
  - pas 7y ago
    re: WaitForMultipleObjects, finally something like that is likely coming to Linux: https://www.reddit.com/r/linux/comments/ck77gm/valve_proposes_kernel_changes_to_help_gaming https://www.reddit.com/r/linux/comments/ck77gm/valve_propose... > It's a shame really. Well, yes, and of course, yet at the same time it's FOSS, so ... if someone really needed it, they should have proposed a patch. ¯\_(ツ)_/¯
    - blattimwind 7y ago
      > Well, yes, and of course, yet at the same time it's FOSS, so ... if someone really needed it, they should have proposed a patch. ¯\_(ツ)_/¯ It's a bit like doing I/O that doesn't throw your data away... just use O_DIRECT, bring your own I/O manager and page cache and scheduling and simply implement all the low-level compatibility stuff yourself if you need such a weird thing ¯\_(ツ)_/¯
      - pas 7y ago
        I mean MySQL did that with libaio (which needs O_DIRECT anyway), Oracle on Linux did that and some with ASM (their own filesystem basically), also Ceph with BlueStore (LevelDB as a FS). Because general purpose filesystems and I/O susbsystems are just that, general purpose. PostgreSQL regularly woes about the problems of the Linux I/O and fs APIs, but ... to my knowledge they did not do much else. (Other than working around the problems in userspace, and taking the performance hit.) If you want I/O that doesn't throw you data away, use the perfectly fine buffered I/O on let's say XFS. Fast, safe, sane. If you need more juice, add NVMe, and/or MD-RAID / LVM. And eventually you need to scale out to multiple nodes anyway, and then the performance of the cluster consistency subsystem will be the weak point.
    - caspper69 7y ago
      RHEL had an entire async i/o subsystem. It was rejected upstream.
      - pas 7y ago
        Could you provide some details about this?
    - hombre_fatal 7y ago
      Things don't get accepted into mainline just because you did the work. This is a weird misconception people have about open source. You also see it when people ask "why did you build this instead of just contributing it to $largeProject?"
      - pas 7y ago
        Huh. I never said it'll be magically accepted. But you also can't say that it's somehow impossible to get anything other than what Linus et al. think is the very epitome of perfect. Upstreaming stuff is an effort. Contributing to such a complex and fragile pile of code/crap/C must be done through the kernel devs. But if you want to add a new FS, I/O scheduler, bytecode VM, or whatever, it's possible. Look at Ceph, btrfs, bcachefs, all the SSD/NAND oriented FS-es, look at eBPF, and the XDP [eXpress Data Path] IO Visor network thingie, or WireGuard [which is still not merged as far as I know, because it tried to bring its own crypto library instead of using the already existing stuff]. All the stuff getting upstreamed from Android piece by piece. A lot of kernel devs are happy about this or that, but they are pragmatists. And Linus doesn't veto big ideas just because, but does not want to pull sub-par implementations. (See KDBUS for example.)
  - kasabali 7y ago
    > many of its developers refuse to adapt technologies for the sole reason that it "stinks" of Microsoft. I don't where did you get that idea but it is not true at all. Linux developers have a history of ignoring (and reinventing) technologies and interfaces developed in other Unix systems, too. You can listen any Bryan Cantrill rant for more details. The claim that they refuse to adapt technologies that "stink" Microsoft is a mischaracterization.
twic 7y ago
> Beyond the ability to unambiguously specify which process should be waited for, this change will eventually enable another interesting feature: it will make it possible to wait for a process that is not a child — something that waitid() cannot do now. Since a pidfd is a file descriptor, it can be passed to another process via an SCM_RIGHTS datagram in the usual manner. The recipient of a pidfd will, once this functionality is completed, be able to use it in most of the ways that the parent can to operate on (or wait for) the associated process. So, to wait for a process that is not your child, do you have to get the relevant pidfd from its parent? In which case, this doesn't help all that much. Or is there some other way to get pidfds for arbitrary processes?
- mehrdadn 7y ago
  Perhaps pidfd_open(pid, ...)? https://lwn.net/Articles/789023/ https://lwn.net/Articles/789023/ I find it bizarre they called it "pidfd_" rather than just "process_" or "proc_"... almost seems like they deliberately avoided the obvious?
  - coldtea 7y ago
    What obvious? The command creates and returns a pidfd for the pid passed, not a process or a proc (which would be conflated with other concepts in Linux).
    - mehrdadn 7y ago
      But you're opening the object (which already exists), not a file descriptor (which doesn't exist yet). And when you create or open a kernel object, you always get back a (new) file descriptor representing that object. See perf_event_open, mkfifo, etc. I'm not really sure how it'd have conflated with anything to call it proc_open or something like that, but I'll take your word on it.
  - cyphar 7y ago
    It's because the object you get is a file descriptor. In fact it's exactly equivalent to getting a file descriptor for /proc/$pid -- Christian (the person who developed the patchsets) quite cleverly solidified a trick that some folks knew about for several years (that you could use /proc/$pid as a race-free way of checking if a process has died if you grabbed a handle while it was still alive). Before pidfd_send_signal(2) there wasn't a way to use that "interface" nicely. But now it's a first-class citizen (and Christian had to fight a lot of battles to get this in over several releases). It's really cool work and I have high hopes it will be used far and wide because it solves so many individual problems in one fell swoop.

by AnssiH 7y ago

That is getting considerably easier with the addition of pidfds, though: https://lwn.net/SubscriberLink/794707/93ffb35438fd3710/ https://lwn.net/SubscriberLink/794707/93ffb35438fd3710/

mehrdadn 7y ago

Posted 1 week ago! Do you know what took them so long to realize this is the correct way to do it? Windows has had it for some two decades...

caspper69 7y ago
Unfortunately, as Linux has matured, many of its developers refuse to adapt technologies for the sole reason that it "stinks" of Microsoft. Cutler and his team were pretty forward thinking given that NT is over 30 years old. See also: the async i/o headaches and the inability to WaitForSingleObject()/WaitForMultipleObjects() (or an analogue) in Linux. It's a shame really.
- pas 7y ago
  re: WaitForMultipleObjects, finally something like that is likely coming to Linux: https://www.reddit.com/r/linux/comments/ck77gm/valve_proposes_kernel_changes_to_help_gaming https://www.reddit.com/r/linux/comments/ck77gm/valve_propose... > It's a shame really. Well, yes, and of course, yet at the same time it's FOSS, so ... if someone really needed it, they should have proposed a patch. ¯\_(ツ)_/¯
  - blattimwind 7y ago
    > Well, yes, and of course, yet at the same time it's FOSS, so ... if someone really needed it, they should have proposed a patch. ¯\_(ツ)_/¯ It's a bit like doing I/O that doesn't throw your data away... just use O_DIRECT, bring your own I/O manager and page cache and scheduling and simply implement all the low-level compatibility stuff yourself if you need such a weird thing ¯\_(ツ)_/¯
    - pas 7y ago
      I mean MySQL did that with libaio (which needs O_DIRECT anyway), Oracle on Linux did that and some with ASM (their own filesystem basically), also Ceph with BlueStore (LevelDB as a FS). Because general purpose filesystems and I/O susbsystems are just that, general purpose. PostgreSQL regularly woes about the problems of the Linux I/O and fs APIs, but ... to my knowledge they did not do much else. (Other than working around the problems in userspace, and taking the performance hit.) If you want I/O that doesn't throw you data away, use the perfectly fine buffered I/O on let's say XFS. Fast, safe, sane. If you need more juice, add NVMe, and/or MD-RAID / LVM. And eventually you need to scale out to multiple nodes anyway, and then the performance of the cluster consistency subsystem will be the weak point.
  - caspper69 7y ago
    RHEL had an entire async i/o subsystem. It was rejected upstream.
    - pas 7y ago
      Could you provide some details about this?
  - hombre_fatal 7y ago
    Things don't get accepted into mainline just because you did the work. This is a weird misconception people have about open source. You also see it when people ask "why did you build this instead of just contributing it to $largeProject?"
    - pas 7y ago
      Huh. I never said it'll be magically accepted. But you also can't say that it's somehow impossible to get anything other than what Linus et al. think is the very epitome of perfect. Upstreaming stuff is an effort. Contributing to such a complex and fragile pile of code/crap/C must be done through the kernel devs. But if you want to add a new FS, I/O scheduler, bytecode VM, or whatever, it's possible. Look at Ceph, btrfs, bcachefs, all the SSD/NAND oriented FS-es, look at eBPF, and the XDP [eXpress Data Path] IO Visor network thingie, or WireGuard [which is still not merged as far as I know, because it tried to bring its own crypto library instead of using the already existing stuff]. All the stuff getting upstreamed from Android piece by piece. A lot of kernel devs are happy about this or that, but they are pragmatists. And Linus doesn't veto big ideas just because, but does not want to pull sub-par implementations. (See KDBUS for example.)
- kasabali 7y ago
  > many of its developers refuse to adapt technologies for the sole reason that it "stinks" of Microsoft. I don't where did you get that idea but it is not true at all. Linux developers have a history of ignoring (and reinventing) technologies and interfaces developed in other Unix systems, too. You can listen any Bryan Cantrill rant for more details. The claim that they refuse to adapt technologies that "stink" Microsoft is a mischaracterization.

twic 7y ago

> Beyond the ability to unambiguously specify which process should be waited for, this change will eventually enable another interesting feature: it will make it possible to wait for a process that is not a child — something that waitid() cannot do now. Since a pidfd is a file descriptor, it can be passed to another process via an SCM_RIGHTS datagram in the usual manner. The recipient of a pidfd will, once this functionality is completed, be able to use it in most of the ways that the parent can to operate on (or wait for) the associated process. So, to wait for a process that is not your child, do you have to get the relevant pidfd from its parent? In which case, this doesn't help all that much. Or is there some other way to get pidfds for arbitrary processes?

mehrdadn 7y ago
Perhaps pidfd_open(pid, ...)? https://lwn.net/Articles/789023/ https://lwn.net/Articles/789023/ I find it bizarre they called it "pidfd_" rather than just "process_" or "proc_"... almost seems like they deliberately avoided the obvious?
- coldtea 7y ago
  What obvious? The command creates and returns a pidfd for the pid passed, not a process or a proc (which would be conflated with other concepts in Linux).
  - mehrdadn 7y ago
    But you're opening the object (which already exists), not a file descriptor (which doesn't exist yet). And when you create or open a kernel object, you always get back a (new) file descriptor representing that object. See perf_event_open, mkfifo, etc. I'm not really sure how it'd have conflated with anything to call it proc_open or something like that, but I'll take your word on it.
- cyphar 7y ago
  It's because the object you get is a file descriptor. In fact it's exactly equivalent to getting a file descriptor for /proc/$pid -- Christian (the person who developed the patchsets) quite cleverly solidified a trick that some folks knew about for several years (that you could use /proc/$pid as a race-free way of checking if a process has died if you grabbed a handle while it was still alive). Before pidfd_send_signal(2) there wasn't a way to use that "interface" nicely. But now it's a first-class citizen (and Christian had to fight a lot of battles to get this in over several releases). It's really cool work and I have high hopes it will be used far and wide because it solves so many individual problems in one fell swoop.