4 ms·

This seems like a good place to ask. I saw this comment[1] few weeks back > This is one of the superior aspects of FreeBSD: no parsing of human-readable strin

by zokier 6y ago

This seems like a good place to ask. I saw this comment[1] few weeks back > This is one of the superior aspects of FreeBSD: no parsing of human-readable strings to get back machine-readable information about the process table. It's available directly in machine-readable form via sysctl(). And started wondering how deep does that go in (Free)BSD? Is all info available in neat strucured format programmatically, or are there other places where you'd need to do Linux-like parsing of special files? Is there some good reading material of this aspect of the system? I hope this isn't too inflammatory Linux vs BSD question, I did not intend value judgement on either approach. [1] https://news.ycombinator.com/item?id=23425867 https://news.ycombinator.com/item?id=23425867

toast0 6y ago
As a long time user, this feels like something that's true but not. The userland tools to dump kernel information (ps, ifconfig, etc) get back binary data from the kernel, not human readable strings, so it's true, but other programs that might need that information would often be expected to call the same tools and then parse the human readable forms until the recent inclusion of libxo can provide machine readable outputs. libxo was added in FreeBSD 11.0, and I'm not sure how far it's been implemented; when I was still managing a large fleet of FreeBSD machines, we had a bunch of 10.x, so it didn't makes sense to update our scripts to use libxo.
- JdeBP 6y ago
  viraptor was talking about C programs having to parse human-readble strings from /proc in Linux to get machine-readable data. That the output of the `ps` command is only human-readable is a complete red herring. That's not what a C program would use, and not the C API that viraptor was talking about.
trasz 6y ago
I think this comes from the different development model. Passing machine-readable stuff instead of parsing strings is convenient, but requires cooperation between parties, usually the kernel and the userspace. In FreeBSD it's trivial: I can change the kernel, userspace, and documentation all in a single commit. In Linux it's much more involved - you never know which userspace version is going to interface with whatever you've tinkered with; people who put together distributions are separate from actual developers of various parts etc.
loeg 6y ago
> And started wondering how deep does that go in (Free)BSD? Is all info available in neat strucured format programmatically, In short: not exactly, but it's somewhat better than Linux. Sysctls are a commonly used, structured tree of mostly integers. The thing Linux does where procfs/sysfs are a bunch of unstructured text files? FreeBSD mostly doesn't do that. Here are some sort of off-hand exceptions or oddball things that might be less desirable: 1. To get the disk/partition topology, there is a sysctl that spits out an XML graph: kern.geom.confxml. So that's sort of text parsing, but it's got a well-defined grammar that many libraries exist to parse already. 2. Some sysctls just dump opaque kernel ABI binary structures. These are machine parseable by C programs as long as they were compiled with the same headers as the kernel; parsing in non-C or across ABIs is less good. If these interfaces need to be portable, they are usually exported in a more portable way by libc or another base system C library. > or are there other places where you'd need to do Linux-like parsing of special files? The only special filesystem you need in FreeBSD is devfs (/dev), and it is populated by devices (or directories containing devices, symlinks to other devices, etc). So if a device exposes an interface that is arbitrary text, sure, you could have something like the Linux situation. But that is atypical. You absolutely do not need /proc or /sys; I don't have either on my desktop FreeBSD workstation. The common convention is to use ioctls instead. Ioctls are a pretty lax type system, but the FreeBSD ones do embed the size of the structure in the identity of the ioctl, so you have at least a slight guarantee of noticing ABI changes and being able to provide backwards compatibility. > Is there some good reading material of this aspect of the system? The canonical book here is FreeBSD: Design and Implementation, 2nd Edition. It's not super recent, but many of the underlying design principles are accurate. Many of the subsystems described function in basically the same way today. I would also plug Joe Kong's FreeBSD Device Drivers if you're interested in writing device drivers specifically (full disclosure: he is a friend).
yjftsjthsd-h 6y ago
Sorta-related: there's an effort to use libxo to make this kind of thing nicer: https://wiki.freebsd.org/LibXo https://wiki.freebsd.org/LibXo
- JdeBP 6y ago
  Again, though, viraptor was talking about the APIs used in C programming, not what one has in shell script programming. When it comes to shell script programming, the obvious mechanism has already existed in FreeBSD for quite a while now. It's vis (vis(1), unvis(1), vis(3)). I have an alternative to ps that vis-encodes the fields in its output, and from experience most and possibly all of the well-known and widely-documented (by Greg Wooledge and others) pitfalls that are warned about do indeed go away with simple application of vis encoding. It does the same on Linux-based operating systems, too, although one has to go and get an unvis tool from somewhere. Yes, the code to read the process table on Linux in the first place does indeed involve decoding human-readable strings from stat files into machine form. (-: * https://unix.stackexchange.com/a/593198/5132 https://unix.stackexchange.com/a/593198/5132
JdeBP 6y ago
There's an enormous amount of stuff available via sysctl(). It is essentially a huge, hierarchically extensible, system call API that passes (usually, but not always) machine-readable data to programs. The sysctl(3) manual page has a lot of stuff, but it doesn't cover anywhere near everything. All of the transmit/receive statistics for the second Ethernet interface on one of my machines are various numeric values under dev.bge.1.stats , for example. The machine-readable process table record structure is given as a C language structure, struct kinfo_proc, in the <sys/user.h> header.
tal8d 6y ago
I was porting a diagnostic command the other day, which displayed SMT status for the IBM power processor - written by a programmer at IBM. It was essentially a directory walk strncmp loop over /proc. The overwhelming sadness compelled me to spend a day of trying to figure out how far back I could trace direct evidence proving the existence of _SC_NPROCESSORS_CONF. Unfortunately I couldn't find the source code for SVR4.0MP, only references in interface manuals.

by zokier 6y ago

toast0 6y ago

As a long time user, this feels like something that's true but not. The userland tools to dump kernel information (ps, ifconfig, etc) get back binary data from the kernel, not human readable strings, so it's true, but other programs that might need that information would often be expected to call the same tools and then parse the human readable forms until the recent inclusion of libxo can provide machine readable outputs. libxo was added in FreeBSD 11.0, and I'm not sure how far it's been implemented; when I was still managing a large fleet of FreeBSD machines, we had a bunch of 10.x, so it didn't makes sense to update our scripts to use libxo.

JdeBP 6y ago
viraptor was talking about C programs having to parse human-readble strings from /proc in Linux to get machine-readable data. That the output of the `ps` command is only human-readable is a complete red herring. That's not what a C program would use, and not the C API that viraptor was talking about.

trasz 6y ago

I think this comes from the different development model. Passing machine-readable stuff instead of parsing strings is convenient, but requires cooperation between parties, usually the kernel and the userspace. In FreeBSD it's trivial: I can change the kernel, userspace, and documentation all in a single commit. In Linux it's much more involved - you never know which userspace version is going to interface with whatever you've tinkered with; people who put together distributions are separate from actual developers of various parts etc.

loeg 6y ago

> And started wondering how deep does that go in (Free)BSD? Is all info available in neat strucured format programmatically, In short: not exactly, but it's somewhat better than Linux. Sysctls are a commonly used, structured tree of mostly integers. The thing Linux does where procfs/sysfs are a bunch of unstructured text files? FreeBSD mostly doesn't do that. Here are some sort of off-hand exceptions or oddball things that might be less desirable: 1. To get the disk/partition topology, there is a sysctl that spits out an XML graph: kern.geom.confxml. So that's sort of text parsing, but it's got a well-defined grammar that many libraries exist to parse already. 2. Some sysctls just dump opaque kernel ABI binary structures. These are machine parseable by C programs as long as they were compiled with the same headers as the kernel; parsing in non-C or across ABIs is less good. If these interfaces need to be portable, they are usually exported in a more portable way by libc or another base system C library. > or are there other places where you'd need to do Linux-like parsing of special files? The only special filesystem you need in FreeBSD is devfs (/dev), and it is populated by devices (or directories containing devices, symlinks to other devices, etc). So if a device exposes an interface that is arbitrary text, sure, you could have something like the Linux situation. But that is atypical. You absolutely do not need /proc or /sys; I don't have either on my desktop FreeBSD workstation. The common convention is to use ioctls instead. Ioctls are a pretty lax type system, but the FreeBSD ones do embed the size of the structure in the identity of the ioctl, so you have at least a slight guarantee of noticing ABI changes and being able to provide backwards compatibility. > Is there some good reading material of this aspect of the system? The canonical book here is FreeBSD: Design and Implementation, 2nd Edition. It's not super recent, but many of the underlying design principles are accurate. Many of the subsystems described function in basically the same way today. I would also plug Joe Kong's FreeBSD Device Drivers if you're interested in writing device drivers specifically (full disclosure: he is a friend).

yjftsjthsd-h 6y ago

Sorta-related: there's an effort to use libxo to make this kind of thing nicer: https://wiki.freebsd.org/LibXo https://wiki.freebsd.org/LibXo

JdeBP 6y ago
Again, though, viraptor was talking about the APIs used in C programming, not what one has in shell script programming. When it comes to shell script programming, the obvious mechanism has already existed in FreeBSD for quite a while now. It's vis (vis(1), unvis(1), vis(3)). I have an alternative to ps that vis-encodes the fields in its output, and from experience most and possibly all of the well-known and widely-documented (by Greg Wooledge and others) pitfalls that are warned about do indeed go away with simple application of vis encoding. It does the same on Linux-based operating systems, too, although one has to go and get an unvis tool from somewhere. Yes, the code to read the process table on Linux in the first place does indeed involve decoding human-readable strings from stat files into machine form. (-: * https://unix.stackexchange.com/a/593198/5132 https://unix.stackexchange.com/a/593198/5132

JdeBP 6y ago

There's an enormous amount of stuff available via sysctl(). It is essentially a huge, hierarchically extensible, system call API that passes (usually, but not always) machine-readable data to programs. The sysctl(3) manual page has a lot of stuff, but it doesn't cover anywhere near everything. All of the transmit/receive statistics for the second Ethernet interface on one of my machines are various numeric values under dev.bge.1.stats , for example. The machine-readable process table record structure is given as a C language structure, struct kinfo_proc, in the <sys/user.h> header.

tal8d 6y ago

I was porting a diagnostic command the other day, which displayed SMT status for the IBM power processor - written by a programmer at IBM. It was essentially a directory walk strncmp loop over /proc. The overwhelming sadness compelled me to spend a day of trying to figure out how far back I could trace direct evidence proving the existence of _SC_NPROCESSORS_CONF. Unfortunately I couldn't find the source code for SVR4.0MP, only references in interface manuals.