4 ms·

> invoking them as a separate process from the main application does not count Why not? What is your worry? That starting a process is too slow? That data t

> invoking them as a separate process from the main application does not count Why not? What is your worry? That starting a process is too slow? That data transfer is too slow? Something else? I agree that process startup may be meaningful overhead in the rare cases when the actual computation is very fast. But sharing uncompressed data between programs is essentially zero-cost. Separate processes look particularly elegant and modular to me. They are much easier to debug and profile than bolted-in dependencies, since the running time and input/output of each process are trivial to isolate. Of course, having both a library and a cli interface for the same computational brick is always better, but the unix-style tool is an essential thing to have.

hresvelgr 3y ago
Performance is not what concerns me, it's design. To what end is it useful to have distinctions between libraries and applications? If I have a JSON parsing library, it might be nice to run one-off scripts that grab a certain value and pipe it somewhere else. It's also useful to deserialise in complex manners inside an app of my design. My belief is that programs whether they are intended to or not usually end up being composed with other programs so on that basis, having the distinction of bin/lib doesn't serve a useful end to me at least.
gonzus 3y ago
I agree 100% with using processes as the natural unit of isolation. I also believe we could solve many of the integration issues between multiple tools, by having them all optionally spit out a universal, easy / easier to parse format -- even JSON would be good here. So you could do ls --foo --bar --json | jq ... and be able to process the output to your heart's content, without having to do any kind of white space parsing; as a free bonus, you get automatic support for file names with embedded white space... The best we have for now is jc (https://kellyjonbrazil.github.io/jc/ https://kellyjonbrazil.github.io/jc/), which knows how to do this for a closed (but growing?) set of commands: ls -l *.db | jc --ls | jq .
dale_glass 3y ago
As someone who writes software that does a lot of this: Process management gets very annoying, very quickly. When you run a helper program soon you have to consider process management. What if it crashes? What if it gets stuck? What if you crash and it remains running? What if the user uses the same program elsewhere? None of this gets work done. A text stream is an awful API. Many times the called program doesn't intend to provide an API, or doesn't commit to stability. Your code breaks because version 5.0.2 fixed a typo, and you relied on the typo for your parsing. A text stream is an awful API. Instead of a nice protocol you get to parse lots of text, deal with escaping and quoting. You better hope the called program does it competently. It may well not, then you have a problem. A text stream is an awful API. No normal program will dump or read a JPEG over stdin/stdout, so if you need to communicate some sort of binary data now you need another communication channel. That may involve commandline arguments, killing, restarting the program, and reestablishing the state. More process management fun. A text stream is an awful API. You'll find yourself doing things like assembling multiple lines of output into a single coherent concept, and trying to detect where something ends when the program doesn't necessarily provide a clear indication. A text stream is an awful API. Sometimes programs print stuff before it took effect, and may not ever give you a clear indication of "now it's been applied". You may need to somehow test for it, retry operations, insert wait states. 99% of the effort invested in this doesn't get work done. You're spending it on management that wouldn't exist if you were using a sane API like DBus or similar, where you don't deal with process management, where things are broken down into nice fields, and where the API is intended as an API.
troupo 3y ago
> Separate processes look particularly elegant and modular to me. They are much easier to debug and profile than bolted-in dependencies Debugging async rocesses is definitely not an easy task. As is managing them in systems that have no management and supervision capabilities (aka nearly all of them). What happens when your process errors out? Gets kille dby OS? Gets stuck? Fork bombs? They are also quite expensive to start in OS terms.

hresvelgr 3y ago

Performance is not what concerns me, it's design. To what end is it useful to have distinctions between libraries and applications? If I have a JSON parsing library, it might be nice to run one-off scripts that grab a certain value and pipe it somewhere else. It's also useful to deserialise in complex manners inside an app of my design. My belief is that programs whether they are intended to or not usually end up being composed with other programs so on that basis, having the distinction of bin/lib doesn't serve a useful end to me at least.

gonzus 3y ago

I agree 100% with using processes as the natural unit of isolation. I also believe we could solve many of the integration issues between multiple tools, by having them all optionally spit out a universal, easy / easier to parse format -- even JSON would be good here. So you could do ls --foo --bar --json | jq ... and be able to process the output to your heart's content, without having to do any kind of white space parsing; as a free bonus, you get automatic support for file names with embedded white space... The best we have for now is jc (https://kellyjonbrazil.github.io/jc/ https://kellyjonbrazil.github.io/jc/), which knows how to do this for a closed (but growing?) set of commands: ls -l *.db | jc --ls | jq .

dale_glass 3y ago

As someone who writes software that does a lot of this: Process management gets very annoying, very quickly. When you run a helper program soon you have to consider process management. What if it crashes? What if it gets stuck? What if you crash and it remains running? What if the user uses the same program elsewhere? None of this gets work done. A text stream is an awful API. Many times the called program doesn't intend to provide an API, or doesn't commit to stability. Your code breaks because version 5.0.2 fixed a typo, and you relied on the typo for your parsing. A text stream is an awful API. Instead of a nice protocol you get to parse lots of text, deal with escaping and quoting. You better hope the called program does it competently. It may well not, then you have a problem. A text stream is an awful API. No normal program will dump or read a JPEG over stdin/stdout, so if you need to communicate some sort of binary data now you need another communication channel. That may involve commandline arguments, killing, restarting the program, and reestablishing the state. More process management fun. A text stream is an awful API. You'll find yourself doing things like assembling multiple lines of output into a single coherent concept, and trying to detect where something ends when the program doesn't necessarily provide a clear indication. A text stream is an awful API. Sometimes programs print stuff before it took effect, and may not ever give you a clear indication of "now it's been applied". You may need to somehow test for it, retry operations, insert wait states. 99% of the effort invested in this doesn't get work done. You're spending it on management that wouldn't exist if you were using a sane API like DBus or similar, where you don't deal with process management, where things are broken down into nice fields, and where the API is intended as an API.

troupo 3y ago

> Separate processes look particularly elegant and modular to me. They are much easier to debug and profile than bolted-in dependencies Debugging async rocesses is definitely not an easy task. As is managing them in systems that have no management and supervision capabilities (aka nearly all of them). What happens when your process errors out? Gets kille dby OS? Gets stuck? Fork bombs? They are also quite expensive to start in OS terms.