5 ms·

In my opinion, the fact that procfs is the only API for so many things is one of the biggest problems with Linux. BSDs have sysctl(), macOS has mach_* functions

In my opinion, the fact that procfs is the only API for so many things is one of the biggest problems with Linux. BSDs have sysctl(), macOS has mach_* functions and, of course, Windows has a real API too. Plain text interfaces lead to complicated, potentially insecure code (especially in C!), they're prone to race conditions and slow. I wish it was possible to retrieve that information using real syscalls. I think it's a better approach than, for example, inventing a faster way to read procfs: https://lwn.net/Articles/813827/ https://lwn.net/Articles/813827/

hamburglar 4y ago
Totally agreed. Any time I’ve found myself parsing proc, I’ve felt like I was doing something foolish and unsafe in lieu of a “real” api.
CamJN 4y ago
MacOS’ KERN_PROCARGS2 sysctl is an exception to this, it is very unintuitive to parse and every single piece of code that tries to parse the results that I’ve found on the internet has been wrong, including those from Apple, Google, and Microsoft. I wound up making a library to do it (https://getargv.narzt.cam/ https://getargv.narzt.cam/) because apparently people need help.
- convolvatron 4y ago
  I just ran into this and its not documented and there are very examples. I will definitely be looking at your library, thank you. it sounds fishy, but just because sysctl is a mess doesn't necessarily imply that structured kernel interfaces are a bad idea
xxpor 4y ago
Even if they insist on a file based interface (it is a UNIX, so fair enough), in modern times it would be nice if they used a "real" data format. Yeah, it's not like JSON parsers have never had bugs, but on average they'll be MUCH better than everyone and their mother hand rolling a C based bespoke parser. Obviously you'd need a new name to not break backwards compatibility.
- kbrazil 4y ago
  jc[0] supports proc files. Converts them to JSON or YAML. (I am the author) [0] https://kellyjonbrazil.github.io/jc/docs/parsers/proc https://kellyjonbrazil.github.io/jc/docs/parsers/proc
  - henrydark 4y ago
    I just had a quick read of the pid/stat parser, and the regex pattern starts with ^, but there's no $. Doesn't this mean that this parser suffers exactly the bug of the original post?
    - prattmic 4y ago
      See https://news.ycombinator.com/item?id=34097179 https://news.ycombinator.com/item?id=34097179. A comm is limited to 64 bytes, so I don't think it is possible to fit a long enough comm to match the full regexp.
      - henrydark 4y ago
        Right, it's not a security problem on its own, but it can make the regex not match at all causing jc to return an error. So jc suffers from the parsing bug mentioned in the post. [edit:] In order to get jc to return an error one has to actually read the regex. Here is a file name that gets it to return an error: bad) S 1 2 3 4 5
        kbrazil 4y ago
        It doesn't look like jc suffers from this bug since the regex match is greedy: % echo '2001 (my (file) with) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc {"pid":2001,"comm":"my (file) with","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"} Edit: looks like I can tighten up the signature matching regex for the "magic" syntax per the issue found above. The greedy regex matching for the parser does seem to work fine, though.
marcodiego 4y ago
I love the fact that I can play with the leds of my device without specialized tools. Yes, it is sysfs, not procfs, but it is the same idea.
- nijave 4y ago
  Many of the APIs on Windows are pretty trivial to interact with using PowerShell commandlets. Similarly, many SaaS based tools have CLIs to interact with their arbitrarily complex APIs. You can still have easy abstractions while providing a way around them for times they don't work well (acquiring structured data)
woodruffw 4y ago
Linux does have libproc, which is meant (IIUC) to mirror the BSD-style libproc. It wouldn't surprise me if it's just parsing the same files under the hood, however, and correspondingly has the same bugs. But then again, bugs in one place is potentially a better state of affairs than bugs in many places (?).
the8472 4y ago
Having C functions isn't all that much better. You have replaced a crufty text format with crufty data structures full of paddings, unions, bitfields, VLAs, unaligned nested structs and other crazy stuff. Look at ioctls or cmsg. With C structs + 3rd-party kernel drivers you can even get UB because the driver returns data that is invalid under the struct definition (e.g. incorrect alignment, invalid bools).
- touisteur 4y ago
  I think a proper formal grammar would do the trick, with maybe a canonical implementation...
touisteur 4y ago
Netlink is the place to look for some of these info, https://twitter.com/dvyukov/status/1605539242506997765 https://twitter.com/dvyukov/status/1605539242506997765 . Loads and loads of stuff in netlink.
mdaverde 4y ago
This is changing! (or technically, has changed!) eBPF recently added the ability to look through internal data structures through iterators [0] so instead of parsing text we can run a program that traverses through all the task_structs and pushes the exact information we want to userspace in the form the developer wants. So, alongside other tradeoffs, it's more flexible than syscalls. [0] https://developers.facebook.com/blog/post/2022/03/31/bpf-iterator-retrieving-kernel-data-with-flexibility-and-efficiency/ https://developers.facebook.com/blog/post/2022/03/31/bpf-ite...

hamburglar 4y ago

Totally agreed. Any time I’ve found myself parsing proc, I’ve felt like I was doing something foolish and unsafe in lieu of a “real” api.

CamJN 4y ago

MacOS’ KERN_PROCARGS2 sysctl is an exception to this, it is very unintuitive to parse and every single piece of code that tries to parse the results that I’ve found on the internet has been wrong, including those from Apple, Google, and Microsoft. I wound up making a library to do it (https://getargv.narzt.cam/ https://getargv.narzt.cam/) because apparently people need help.

convolvatron 4y ago
I just ran into this and its not documented and there are very examples. I will definitely be looking at your library, thank you. it sounds fishy, but just because sysctl is a mess doesn't necessarily imply that structured kernel interfaces are a bad idea

xxpor 4y ago

Even if they insist on a file based interface (it is a UNIX, so fair enough), in modern times it would be nice if they used a "real" data format. Yeah, it's not like JSON parsers have never had bugs, but on average they'll be MUCH better than everyone and their mother hand rolling a C based bespoke parser. Obviously you'd need a new name to not break backwards compatibility.

kbrazil 4y ago
jc[0] supports proc files. Converts them to JSON or YAML. (I am the author) [0] https://kellyjonbrazil.github.io/jc/docs/parsers/proc https://kellyjonbrazil.github.io/jc/docs/parsers/proc
- henrydark 4y ago
  I just had a quick read of the pid/stat parser, and the regex pattern starts with ^, but there's no $. Doesn't this mean that this parser suffers exactly the bug of the original post?
  - prattmic 4y ago
    See https://news.ycombinator.com/item?id=34097179 https://news.ycombinator.com/item?id=34097179. A comm is limited to 64 bytes, so I don't think it is possible to fit a long enough comm to match the full regexp.
    - henrydark 4y ago
      Right, it's not a security problem on its own, but it can make the regex not match at all causing jc to return an error. So jc suffers from the parsing bug mentioned in the post. [edit:] In order to get jc to return an error one has to actually read the regex. Here is a file name that gets it to return an error: bad) S 1 2 3 4 5
      - kbrazil 4y ago
        It doesn't look like jc suffers from this bug since the regex match is greedy: % echo '2001 (my (file) with) S 1888 2001 1888 34816 2001 4202496 428 0 0 0 0 0 0 0 20 0 1 0 75513 115900416 297 18446744073709551615 4194304 5100612 140737020052256 140737020050904 140096699233308 0 65536 4 65538 18446744072034584486 0 0 17 0 0 0 0 0 0 7200240 7236240 35389440 140737020057179 140737020057223 140737020057223 140737020059606 0' | jc --proc {"pid":2001,"comm":"my (file) with","state":"S","ppid":1888,"pgrp":2001,"session":1888,"tty_nr":34816,"tpg_id":2001,"flags":4202496,"minflt":428,"cminflt":0,"majflt":0,"cmajflt":0,"utime":0,"stime":0,"cutime":0,"cstime":0,"priority":20,"nice":0,"num_threads":1,"itrealvalue":0,"starttime":75513,"vsize":115900416,"rss":297,"rsslim":18446744073709551615,"startcode":4194304,"endcode":5100612,"startstack":140737020052256,"kstkeep":140737020050904,"kstkeip":140096699233308,"signal":0,"blocked":65536,"sigignore":4,"sigcatch":65538,"wchan":18446744072034584486,"nswap":0,"cnswap":0,"exit_signal":17,"processor":0,"rt_priority":0,"policy":0,"delayacct_blkio_ticks":0,"guest_time":0,"cguest_time":0,"start_data":7200240,"end_data":7236240,"start_brk":35389440,"arg_start":140737020057179,"arg_end":140737020057223,"env_start":140737020057223,"env_end":140737020059606,"exit_code":0,"state_pretty":"Sleeping in an interruptible wait"} Edit: looks like I can tighten up the signature matching regex for the "magic" syntax per the issue found above. The greedy regex matching for the parser does seem to work fine, though.

marcodiego 4y ago

I love the fact that I can play with the leds of my device without specialized tools. Yes, it is sysfs, not procfs, but it is the same idea.

nijave 4y ago
Many of the APIs on Windows are pretty trivial to interact with using PowerShell commandlets. Similarly, many SaaS based tools have CLIs to interact with their arbitrarily complex APIs. You can still have easy abstractions while providing a way around them for times they don't work well (acquiring structured data)

woodruffw 4y ago

Linux does have libproc, which is meant (IIUC) to mirror the BSD-style libproc. It wouldn't surprise me if it's just parsing the same files under the hood, however, and correspondingly has the same bugs. But then again, bugs in one place is potentially a better state of affairs than bugs in many places (?).

the8472 4y ago

Having C functions isn't all that much better. You have replaced a crufty text format with crufty data structures full of paddings, unions, bitfields, VLAs, unaligned nested structs and other crazy stuff. Look at ioctls or cmsg. With C structs + 3rd-party kernel drivers you can even get UB because the driver returns data that is invalid under the struct definition (e.g. incorrect alignment, invalid bools).

touisteur 4y ago
I think a proper formal grammar would do the trick, with maybe a canonical implementation...

touisteur 4y ago

Netlink is the place to look for some of these info, https://twitter.com/dvyukov/status/1605539242506997765 https://twitter.com/dvyukov/status/1605539242506997765 . Loads and loads of stuff in netlink.

mdaverde 4y ago

This is changing! (or technically, has changed!) eBPF recently added the ability to look through internal data structures through iterators [0] so instead of parsing text we can run a program that traverses through all the task_structs and pushes the exact information we want to userspace in the form the developer wants. So, alongside other tradeoffs, it's more flexible than syscalls. [0] https://developers.facebook.com/blog/post/2022/03/31/bpf-iterator-retrieving-kernel-data-with-flexibility-and-efficiency/ https://developers.facebook.com/blog/post/2022/03/31/bpf-ite...