3 ms·

> The idea is not to have one unconfigurable binary reciever, It _is_ configurable, why do you think it is not? This idea that somehow one can fit any interac

> The idea is not to have one unconfigurable binary reciever, It _is_ configurable, why do you think it is not? This idea that somehow one can fit any interaction with hardware into a filesystem ends when hardware becomes complex enough to simply NOT fit (like modern GPU). How do you guarantee atomicity when saving configuration with tar(1)? Linux recently deprecated textual conntracking data interface in favour of shipping binary data over netlink socket. Reason -- read() speed in real life high load scenarios. What could Plan9 do here?

strictfp 15y ago
I actually tried to point out that configurability _is_ very possible with such an API. Your complexity argument does not hold. If I can control something with a C API, why not with a file API? Take atomicity as an example. C has no inherent concept of atomicity. Yet you can acheive atomicity by using C. So why not with a REST or file-based interface? You also mention speed. Interesting since complexity and performance worries are the two roots to premature optimization if you ask me. Since this solution is still on a concept level I am of the opinion that one should avoid such discussions at this stage. But I see no _inherent_ speed limitations in a file-based API. Send binary data if you want. Text is only for readability.
- adobriyan 15y ago
  > Take atomicity as an example. C has no inherent concept of atomicity. Yet you can acheive atomicity by using C. So why not with a REST or file-based interface? You achieve atomicity because inside kernel handler of relevant imaginary ioctl(2) which dumps all hardware configuration there is synchronization primitive which provides necessary guarantees, so this imaginary ioctl(HW_GET_STATE) writes coherent state to userspace. If you merely expose individual hw config into flat files, POSIX directory semantics does not guarantee you atomicity. So, you have to invent something instead, like additional file which kinda sorta provides locking. But if you invented it, you have implicitly mandated that ALL users must be nice and go through locking API/file, whatever. At this point, nothing prevents rogue (and more importantly) buggy userspace from messing with well-behaving users. At this point, you've failed to provide something another developer is asking you to provide, namely, a guarantee that hw state will be coherent.
  - deleted 15y ago
    [deleted]
- adobriyan 15y ago
  Regarding speed and knuth argument. At this stage real world examples show that once scale of the state becomes sufficiently large (large number of conntracks on big firewalls), text interface always lose. So, play with toy flat text file interface if you want, but you might as well do it right from the very beginning. Right now, /proc/stat slowness is being discussed in linux-kernel. Part of the problem: heuristics determining how many pages needs to be allocated for buffer containing text info sucks because integers written in decimal are variable sized. Kernel first allocate 1 page, only to fill it and see that 1 page is not enough for big enough machines (NR_CPUS * NR_INTERRUPTS) 2 patches are proposed: first is to allocate 2 pages from the beginning, second is (no kidding) to print decimals faster (printf "%u" can be made faster since it's known that "unsigned int" is going to be dumped and C has pathetic support for compile-time evaluation). But the correct from every angle (except existing /proc/stat users) answer is obvious: dump BINARY info already into userspace buffer which will be appropriately sized because it's easy to multiply 3 integers). Simply ship information to who is asking, without print bullshit. Real programming languages and environments should easily eat result (Python's struct.unpack springs to mind). Excuses for programming languages will not. Can you transform /proc/bus/pci/00/00.0 content into one-dimensional array of bytes with you favourite PL? And if you can't, whose problem is it? And yet another example, Linux USB bus sniffer kernel module (don't remember exact name/config option) gained binary interface deprecating text one. As for _inherent_ speed limitations, read(2) does memcpy(), so you have to provide mmap(2) for your file (most virtual Linux filesystems doesn't do it for majority of files).
angersock 15y ago
This idea that somehow one can fit any interaction with hardware into a filesystem ends when hardware becomes complex enough to simply NOT fit (like modern GPU). The idea that one can fit any interaction with hardware into the mere toggling of registers and stuffing of byte buffers... I don't know what FS you're using, but any good modern filesystem (say, ZFS or something similar) should prove more than adequate for handling that many/that size nodes. Everything else can probably be fixed in the kernel level--for example, duplicating the hardware directory prior to tar'ing it, and have the kernel/FS note what actions to do to guarantee atomicity. I mean, we do the same thing with sharing /dev/dsp anyways (I believe daemons can be set up to handles this), so what's the deal?
- adobriyan 15y ago
  We're discussing anything that can do open(), read, write(), provides files and directories. Do you understand that virtual filesystems do not have problems with inode counts? Dupping a directory means a) hw state is saved on open(), in which case you're holding it unnecessary because read() simply may not happen b) hw state is saved on first read(). OK, what to do with parallel dumpers? How to determine when transaction starts and ends? What to do if dumper process is sigkilled? ioctl(HW_GET_STATE) doesn't have these problems, kernel knows how to kill process inside itself. Try to save /sys/class/net/lo/* coherently and write down all assumptions which you did.

strictfp 15y ago

I actually tried to point out that configurability _is_ very possible with such an API. Your complexity argument does not hold. If I can control something with a C API, why not with a file API? Take atomicity as an example. C has no inherent concept of atomicity. Yet you can acheive atomicity by using C. So why not with a REST or file-based interface? You also mention speed. Interesting since complexity and performance worries are the two roots to premature optimization if you ask me. Since this solution is still on a concept level I am of the opinion that one should avoid such discussions at this stage. But I see no _inherent_ speed limitations in a file-based API. Send binary data if you want. Text is only for readability.

adobriyan 15y ago
> Take atomicity as an example. C has no inherent concept of atomicity. Yet you can acheive atomicity by using C. So why not with a REST or file-based interface? You achieve atomicity because inside kernel handler of relevant imaginary ioctl(2) which dumps all hardware configuration there is synchronization primitive which provides necessary guarantees, so this imaginary ioctl(HW_GET_STATE) writes coherent state to userspace. If you merely expose individual hw config into flat files, POSIX directory semantics does not guarantee you atomicity. So, you have to invent something instead, like additional file which kinda sorta provides locking. But if you invented it, you have implicitly mandated that ALL users must be nice and go through locking API/file, whatever. At this point, nothing prevents rogue (and more importantly) buggy userspace from messing with well-behaving users. At this point, you've failed to provide something another developer is asking you to provide, namely, a guarantee that hw state will be coherent.
- deleted 15y ago
  [deleted]
adobriyan 15y ago
Regarding speed and knuth argument. At this stage real world examples show that once scale of the state becomes sufficiently large (large number of conntracks on big firewalls), text interface always lose. So, play with toy flat text file interface if you want, but you might as well do it right from the very beginning. Right now, /proc/stat slowness is being discussed in linux-kernel. Part of the problem: heuristics determining how many pages needs to be allocated for buffer containing text info sucks because integers written in decimal are variable sized. Kernel first allocate 1 page, only to fill it and see that 1 page is not enough for big enough machines (NR_CPUS * NR_INTERRUPTS) 2 patches are proposed: first is to allocate 2 pages from the beginning, second is (no kidding) to print decimals faster (printf "%u" can be made faster since it's known that "unsigned int" is going to be dumped and C has pathetic support for compile-time evaluation). But the correct from every angle (except existing /proc/stat users) answer is obvious: dump BINARY info already into userspace buffer which will be appropriately sized because it's easy to multiply 3 integers). Simply ship information to who is asking, without print bullshit. Real programming languages and environments should easily eat result (Python's struct.unpack springs to mind). Excuses for programming languages will not. Can you transform /proc/bus/pci/00/00.0 content into one-dimensional array of bytes with you favourite PL? And if you can't, whose problem is it? And yet another example, Linux USB bus sniffer kernel module (don't remember exact name/config option) gained binary interface deprecating text one. As for _inherent_ speed limitations, read(2) does memcpy(), so you have to provide mmap(2) for your file (most virtual Linux filesystems doesn't do it for majority of files).

angersock 15y ago

This idea that somehow one can fit any interaction with hardware into a filesystem ends when hardware becomes complex enough to simply NOT fit (like modern GPU). The idea that one can fit any interaction with hardware into the mere toggling of registers and stuffing of byte buffers... I don't know what FS you're using, but any good modern filesystem (say, ZFS or something similar) should prove more than adequate for handling that many/that size nodes. Everything else can probably be fixed in the kernel level--for example, duplicating the hardware directory prior to tar'ing it, and have the kernel/FS note what actions to do to guarantee atomicity. I mean, we do the same thing with sharing /dev/dsp anyways (I believe daemons can be set up to handles this), so what's the deal?

adobriyan 15y ago
We're discussing anything that can do open(), read, write(), provides files and directories. Do you understand that virtual filesystems do not have problems with inode counts? Dupping a directory means a) hw state is saved on open(), in which case you're holding it unnecessary because read() simply may not happen b) hw state is saved on first read(). OK, what to do with parallel dumpers? How to determine when transaction starts and ends? What to do if dumper process is sigkilled? ioctl(HW_GET_STATE) doesn't have these problems, kernel knows how to kill process inside itself. Try to save /sys/class/net/lo/* coherently and write down all assumptions which you did.