5 ms·

I wish /proc|/sys would just agree on serialization format and just serialize the data into some defined format instead of having a bunch of files that all need

by ilyt 4y ago

I wish /proc|/sys would just agree on serialization format and just serialize the data into some defined format instead of having a bunch of files that all need their own parser

stefan_ 4y ago
Well it's too late now. But I thought the plan was for all of that stuff to move to Netlink? Not that that isn't a terrible very horrible API either.
bradfitz 4y ago
Advent of Proc
jcelerier 4y ago
we could even name the tool to query such serialized data, procctl, provided by the systemd-proc package
st_goliath 4y ago
While procfs has a lot of historical baggage, sysfs is rather specific about the layout and providing only a single value per file, as plain ASCII, rather than using anything complex that has to be parsed. Structure is implemented via the filesystem. In return, the kernel side API for sysfs is also a lot cleaner and allows to more-or-less expose individual variables as tuning knobs for a driver. Of course there are edge cases, and there are e.g. some binary interfaces as well (e.g. for providing direct register access, or implementing a firmware upload interface for a device). ABI compat issues aside, I think that implementing "a standardized [structured] record format" as suggested in the comments here is a rather bad idea, going into exactly the wrong direction by adding complexity rather than reducing it, which would definitely cause even more parsing related issues in the long run.
- ilyt 4y ago
  >While procfs has a lot of historical baggage, sysfs is rather specific about the layout and providing only a single value per file, as plain ASCII, rather than using anything complex that has to be parsed. Structure is implemented via the filesystem. I'd rather have structured file than to have open 30k files (for say conntrack) Hell, just example from the article, /proc/<PID>/stat has 52 parameters. That would be 52 opens and reads with single value per file. > ABI compat issues aside, I think that implementing "a standardized [structured] record format" as suggested in the comments here is a rather bad idea, going into exactly the wrong direction by adding complexity rather than reducing it, which would definitely cause even more parsing related issues in the long run. It's literally the opposite. You have to implement it once on kernel side and once in userspace vs every special format that currently needs
  - throwaway09223 4y ago
    Linux has already half-solved this, eg with netlink. If you're on a system with huge numbers of connections, reading from /proc/net/tcp get extremely slow. Modern tools query connection state using netlink instead (ss vs netstat). This was done by necessity: /proc/net/tcp actually doesn't work at scale. I agree with you, serializing and deserializing files with records is a terrible idea - it cannot be performant. JSON fixes parsing ambiguity but at a cost of being even slower. We already know it won't work. We have already solved this for specific parts of /proc and it works great. All we have to do is finish the work and provide the rest of proc via netlink as well (or whatever else similar non-text based system for querying structured records)
    - dale_glass 4y ago
      JSON can be parsed very quickly: https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson CBOR could be another option: https://en.wikipedia.org/wiki/CBOR https://en.wikipedia.org/wiki/CBOR
      - the8472 4y ago
        JSON cannot be efficient for binary data or even utf8 strings because it needs escaping which means you can't have a zero-copy implementation. CBOR is better in that regard, yeah.
        ilyt 4y ago
        All it needs to beat is stringifying and formatting that /proc|/sys already does
      - throwaway09223 4y ago
        It can't (simdjson is fast compared to other text parsing - it's still extremely slow compared to native types), but that fact actually doesn't matter because the problem isn't the parsing. Even the construction of these text files in proc is too slow. You can't read /proc/net/tcp within a reasonable amount of time on a system with hundreds of thousands of connections. Even allocating/churning memory to store a textual representation becomes a problematic overhead.
ajross 4y ago
FWIW: sysfs tried to do this already. In general each node corresponds to one "thing", with a reasonably standard set of stringification schemes, and with a path that acts as a self-describing schema. Obviously in practice it ends up that every driver or subsystem ends up doing funny nonsense (e.g. uevent nodes have their own sub-schema with shell-style variables, etc...). You can't really prevent that. People do funny nonsense in other self-describing data formats like JSON and XML all the time too. There's only so much you can do with a framework. But /proc is... extremely old, and very heavily used by userspace. In practice it's never going to change.
- ilyt 4y ago
  > You can't really prevent that. People do funny nonsense in other self-describing data formats like JSON and XML all the time too. There's only so much you can do with a framework. Sure but you will get more of that if the convention is too simplistic. "one file per value" breaks really fast, just cat /proc/net/nf_conntrack or even just proc/<pid>/stats and see just how many values single entry (file/connection) has. Doesn't need to be some ASN.1 monstrosity, could be simple conventions like "this is how key/value proc/sys file should look, this is how tabular file should look etc." Make all escaping use same syntax, make every table separator be \t etc. > But /proc is... extremely old, and very heavily used by userspace. In practice it's never going to change. eh, just mount it in /proc2
eminence32 4y ago
I've been working on a library[1] that aims to have fairly complete support for the procfs filesystem, so that you can hide away these annoying parsing quirks. But for some casual usage of /proc/ where you only need one tiny bit of information, it's often better to just roll your own parser instead of bringing in a 3rd party library. It's these small one-off cases that would really benefit from a standardized serialization format like you propose. [1] https://github.com/eminence/procfs https://github.com/eminence/procfs
- idealmedtech 4y ago
  It would be great if the kernel itself provided a header only definition of such a format, so you could focus on the data and not the parsing. Would also be able to integrate into their extensive testing infrastructure.
  - ilyt 4y ago
    Realistically it could stay mostly as-is, some text values with separators, just define exactly how to quote stuff and what the separator. Then you could just have "load a single value" function that does the unquoting, "load K/V" function for stuff like /proc/meminfo, and "load table" for stuff like /proc/<pid>/stat. Maybe "load records" for stuff like /proc/net/nf_conntrack which is essentially list of KV pairs.
saalweachter 4y ago
While we're wishing in one hand, how is it that our programs still take an input of an array of strings, that get escaped and unescaped and split randomly by our shell scripts?
- ilyt 4y ago
  That is entirely due to sh/bash and friends being a terrible programming language. All sensible ones allow you to just pass an array of parameters to command execution and not worry about spaces in them
  - Spivak 4y ago
    The irony is that I’ve had to add shell word splitting into in my real programming language code because of how many people prefer run(f”command {arg} -v -p{opt} {target}”) to run([“command”, arg, “-v”, “-p”, opt, target])
  - saalweachter 4y ago
    That's only, like, half the problem. It's an array of string parameters. No semantics! Just an variety of customs about what it means when a parameters begins with a - or a -- or if you have -- by itself preceding some characters, how to break lists of arguments with separators, what happens when you pass the same argument twice, etc etc etc. To choose the worst possible solution better than that, we could instead be passing in a single string with a JSON dictionary that says things like '{ "recursive": True, "force": True, "files": [ "file1", "file2", "file3" ]}'
  - NathanOsullivan 4y ago
    FWIW bash supports arrays, and running commands with an array providing the arguments
emmelaich 4y ago
osquery covers those fwiw, and can produce json.

by ilyt 4y ago

I wish /proc|/sys would just agree on serialization format and just serialize the data into some defined format instead of having a bunch of files that all need their own parser

stefan_ 4y ago

Well it's too late now. But I thought the plan was for all of that stuff to move to Netlink? Not that that isn't a terrible very horrible API either.

bradfitz 4y ago

Advent of Proc

jcelerier 4y ago

we could even name the tool to query such serialized data, procctl, provided by the systemd-proc package

st_goliath 4y ago

While procfs has a lot of historical baggage, sysfs is rather specific about the layout and providing only a single value per file, as plain ASCII, rather than using anything complex that has to be parsed. Structure is implemented via the filesystem. In return, the kernel side API for sysfs is also a lot cleaner and allows to more-or-less expose individual variables as tuning knobs for a driver. Of course there are edge cases, and there are e.g. some binary interfaces as well (e.g. for providing direct register access, or implementing a firmware upload interface for a device). ABI compat issues aside, I think that implementing "a standardized [structured] record format" as suggested in the comments here is a rather bad idea, going into exactly the wrong direction by adding complexity rather than reducing it, which would definitely cause even more parsing related issues in the long run.

ilyt 4y ago
>While procfs has a lot of historical baggage, sysfs is rather specific about the layout and providing only a single value per file, as plain ASCII, rather than using anything complex that has to be parsed. Structure is implemented via the filesystem. I'd rather have structured file than to have open 30k files (for say conntrack) Hell, just example from the article, /proc/<PID>/stat has 52 parameters. That would be 52 opens and reads with single value per file. > ABI compat issues aside, I think that implementing "a standardized [structured] record format" as suggested in the comments here is a rather bad idea, going into exactly the wrong direction by adding complexity rather than reducing it, which would definitely cause even more parsing related issues in the long run. It's literally the opposite. You have to implement it once on kernel side and once in userspace vs every special format that currently needs
- throwaway09223 4y ago
  Linux has already half-solved this, eg with netlink. If you're on a system with huge numbers of connections, reading from /proc/net/tcp get extremely slow. Modern tools query connection state using netlink instead (ss vs netstat). This was done by necessity: /proc/net/tcp actually doesn't work at scale. I agree with you, serializing and deserializing files with records is a terrible idea - it cannot be performant. JSON fixes parsing ambiguity but at a cost of being even slower. We already know it won't work. We have already solved this for specific parts of /proc and it works great. All we have to do is finish the work and provide the rest of proc via netlink as well (or whatever else similar non-text based system for querying structured records)
  - dale_glass 4y ago
    JSON can be parsed very quickly: https://github.com/simdjson/simdjson https://github.com/simdjson/simdjson CBOR could be another option: https://en.wikipedia.org/wiki/CBOR https://en.wikipedia.org/wiki/CBOR
    - the8472 4y ago
      JSON cannot be efficient for binary data or even utf8 strings because it needs escaping which means you can't have a zero-copy implementation. CBOR is better in that regard, yeah.
      - ilyt 4y ago
        All it needs to beat is stringifying and formatting that /proc|/sys already does
    - throwaway09223 4y ago
      It can't (simdjson is fast compared to other text parsing - it's still extremely slow compared to native types), but that fact actually doesn't matter because the problem isn't the parsing. Even the construction of these text files in proc is too slow. You can't read /proc/net/tcp within a reasonable amount of time on a system with hundreds of thousands of connections. Even allocating/churning memory to store a textual representation becomes a problematic overhead.

ajross 4y ago

FWIW: sysfs tried to do this already. In general each node corresponds to one "thing", with a reasonably standard set of stringification schemes, and with a path that acts as a self-describing schema. Obviously in practice it ends up that every driver or subsystem ends up doing funny nonsense (e.g. uevent nodes have their own sub-schema with shell-style variables, etc...). You can't really prevent that. People do funny nonsense in other self-describing data formats like JSON and XML all the time too. There's only so much you can do with a framework. But /proc is... extremely old, and very heavily used by userspace. In practice it's never going to change.

ilyt 4y ago
> You can't really prevent that. People do funny nonsense in other self-describing data formats like JSON and XML all the time too. There's only so much you can do with a framework. Sure but you will get more of that if the convention is too simplistic. "one file per value" breaks really fast, just cat /proc/net/nf_conntrack or even just proc/<pid>/stats and see just how many values single entry (file/connection) has. Doesn't need to be some ASN.1 monstrosity, could be simple conventions like "this is how key/value proc/sys file should look, this is how tabular file should look etc." Make all escaping use same syntax, make every table separator be \t etc. > But /proc is... extremely old, and very heavily used by userspace. In practice it's never going to change. eh, just mount it in /proc2

eminence32 4y ago

I've been working on a library[1] that aims to have fairly complete support for the procfs filesystem, so that you can hide away these annoying parsing quirks. But for some casual usage of /proc/ where you only need one tiny bit of information, it's often better to just roll your own parser instead of bringing in a 3rd party library. It's these small one-off cases that would really benefit from a standardized serialization format like you propose. [1] https://github.com/eminence/procfs https://github.com/eminence/procfs

idealmedtech 4y ago
It would be great if the kernel itself provided a header only definition of such a format, so you could focus on the data and not the parsing. Would also be able to integrate into their extensive testing infrastructure.
- ilyt 4y ago
  Realistically it could stay mostly as-is, some text values with separators, just define exactly how to quote stuff and what the separator. Then you could just have "load a single value" function that does the unquoting, "load K/V" function for stuff like /proc/meminfo, and "load table" for stuff like /proc/<pid>/stat. Maybe "load records" for stuff like /proc/net/nf_conntrack which is essentially list of KV pairs.

saalweachter 4y ago

While we're wishing in one hand, how is it that our programs still take an input of an array of strings, that get escaped and unescaped and split randomly by our shell scripts?

ilyt 4y ago
That is entirely due to sh/bash and friends being a terrible programming language. All sensible ones allow you to just pass an array of parameters to command execution and not worry about spaces in them
- Spivak 4y ago
  The irony is that I’ve had to add shell word splitting into in my real programming language code because of how many people prefer run(f”command {arg} -v -p{opt} {target}”) to run([“command”, arg, “-v”, “-p”, opt, target])
- saalweachter 4y ago
  That's only, like, half the problem. It's an array of string parameters. No semantics! Just an variety of customs about what it means when a parameters begins with a - or a -- or if you have -- by itself preceding some characters, how to break lists of arguments with separators, what happens when you pass the same argument twice, etc etc etc. To choose the worst possible solution better than that, we could instead be passing in a single string with a JSON dictionary that says things like '{ "recursive": True, "force": True, "files": [ "file1", "file2", "file3" ]}'
- NathanOsullivan 4y ago
  FWIW bash supports arrays, and running commands with an array providing the arguments

emmelaich 4y ago

osquery covers those fwiw, and can produce json.