4 ms·

These tools are not composable once you venture outside of a shell, and if I have to write anything beyond 5 lines, I'm not using one. There needs to be a way t

These tools are not composable once you venture outside of a shell, and if I have to write anything beyond 5 lines, I'm not using one. There needs to be a way to make these applications usable as software libraries, and no, invoking them as a separate process from the main application does not count. This is the reason the functionality of these tools have been reinvented many times over and why they get extra cruft added is because they're actually not composable everywhere that matters. edit: typo

vlovich123 3y ago
Did you read the article? The entire last half is how to apply it to UIs and what that looks like (Obsidian, Canva, etc).
- hresvelgr 3y ago
  I did. I'm talking about CLI apps, though that wasn't clear I will admit.
eternityforest 3y ago
We need more standard special purpose interfaces. Not unstructured RPC protocols, but very specific "Every plugin of this type has this function that does exactly this" kinds of things. Linux people seem to tolerate bash partly because they DO have a special purpose modular system. They algorithmically manipulate text in batch rather than realtime mode to produce an output on a regular basis(I'm not sure for what, besides compiling and building, but they all say they have use cases). Apparently it works great if you process lots of text files. Perhaps a GitHub awesome list. I've been meaning to start one for very common standards in general(18650 batteries, tripod thread, etc). We have LADSPA and LV2 for audio, shell commands for text, etc, What else do we have, and what should we have?
enriquto 3y ago
> invoking them as a separate process from the main application does not count Why not? What is your worry? That starting a process is too slow? That data transfer is too slow? Something else? I agree that process startup may be meaningful overhead in the rare cases when the actual computation is very fast. But sharing uncompressed data between programs is essentially zero-cost. Separate processes look particularly elegant and modular to me. They are much easier to debug and profile than bolted-in dependencies, since the running time and input/output of each process are trivial to isolate. Of course, having both a library and a cli interface for the same computational brick is always better, but the unix-style tool is an essential thing to have.
- hresvelgr 3y ago
  Performance is not what concerns me, it's design. To what end is it useful to have distinctions between libraries and applications? If I have a JSON parsing library, it might be nice to run one-off scripts that grab a certain value and pipe it somewhere else. It's also useful to deserialise in complex manners inside an app of my design. My belief is that programs whether they are intended to or not usually end up being composed with other programs so on that basis, having the distinction of bin/lib doesn't serve a useful end to me at least.
- gonzus 3y ago
  I agree 100% with using processes as the natural unit of isolation. I also believe we could solve many of the integration issues between multiple tools, by having them all optionally spit out a universal, easy / easier to parse format -- even JSON would be good here. So you could do ls --foo --bar --json | jq ... and be able to process the output to your heart's content, without having to do any kind of white space parsing; as a free bonus, you get automatic support for file names with embedded white space... The best we have for now is jc (https://kellyjonbrazil.github.io/jc/ https://kellyjonbrazil.github.io/jc/), which knows how to do this for a closed (but growing?) set of commands: ls -l *.db | jc --ls | jq .
- dale_glass 3y ago
  As someone who writes software that does a lot of this: Process management gets very annoying, very quickly. When you run a helper program soon you have to consider process management. What if it crashes? What if it gets stuck? What if you crash and it remains running? What if the user uses the same program elsewhere? None of this gets work done. A text stream is an awful API. Many times the called program doesn't intend to provide an API, or doesn't commit to stability. Your code breaks because version 5.0.2 fixed a typo, and you relied on the typo for your parsing. A text stream is an awful API. Instead of a nice protocol you get to parse lots of text, deal with escaping and quoting. You better hope the called program does it competently. It may well not, then you have a problem. A text stream is an awful API. No normal program will dump or read a JPEG over stdin/stdout, so if you need to communicate some sort of binary data now you need another communication channel. That may involve commandline arguments, killing, restarting the program, and reestablishing the state. More process management fun. A text stream is an awful API. You'll find yourself doing things like assembling multiple lines of output into a single coherent concept, and trying to detect where something ends when the program doesn't necessarily provide a clear indication. A text stream is an awful API. Sometimes programs print stuff before it took effect, and may not ever give you a clear indication of "now it's been applied". You may need to somehow test for it, retry operations, insert wait states. 99% of the effort invested in this doesn't get work done. You're spending it on management that wouldn't exist if you were using a sane API like DBus or similar, where you don't deal with process management, where things are broken down into nice fields, and where the API is intended as an API.
bluetomcat 3y ago
On a modern Unix-like OS, these tools aren’t as simple as they used to be in the 1970s. The man page for ls alone lists tens of options, corner cases and historical cruft. If those tools were really composable, they wouldn’t need so many options hidden in their man pages.
- csydas 3y ago
  I don't agree with this conclusion on the number of flags for the GNU core utils relating to it's composability; the flags are natural advanced extensions of what the tool is meant to do, and even without the flags the tools are straight forward enough that the default execution likely will meet the requirements of even new users. From my own experience teaching newbies about bash, the only hard part about all the flags was getting newbies comfortable with ignoring them at first and just trusting that `ls` lists files in the current directory, `sed` let's you manipulate text in a smart way, `grep` finds things, and so on. Once they got that in their head and understood that the flags just allow very specific operations that you previously needed to write a lot more code for, most people got it pretty fast and could punch out quick shell scripts to help with their work. But the base commands without a slew of flags was still quite useful for these newbies. Again though, that's just my experience and it was fairly narrow in scope for accomplishing specific troubleshooting tasks, so that may factor into why I saw success here; I don't know if the people I taught further evolved their scripting skills past basic troubleshooting, though I am fairly confident these persons understand the "kitchen tools" nature of the GNU core utils as many ended up writing scripts on their own for situations we never discussed, and the scripts were pretty okay. Edit: Typo, changed Exceptions to Extensions in the first paragraph
IshKebab 3y ago
I agree. The reason small tools are simple and have few bugs is that they don't do the whole job! They say "here are your modules! now you do the last bit of programming to integrate them and do all the integration testing please. good luck!"

vlovich123 3y ago

Did you read the article? The entire last half is how to apply it to UIs and what that looks like (Obsidian, Canva, etc).

hresvelgr 3y ago
I did. I'm talking about CLI apps, though that wasn't clear I will admit.

eternityforest 3y ago

We need more standard special purpose interfaces. Not unstructured RPC protocols, but very specific "Every plugin of this type has this function that does exactly this" kinds of things. Linux people seem to tolerate bash partly because they DO have a special purpose modular system. They algorithmically manipulate text in batch rather than realtime mode to produce an output on a regular basis(I'm not sure for what, besides compiling and building, but they all say they have use cases). Apparently it works great if you process lots of text files. Perhaps a GitHub awesome list. I've been meaning to start one for very common standards in general(18650 batteries, tripod thread, etc). We have LADSPA and LV2 for audio, shell commands for text, etc, What else do we have, and what should we have?

enriquto 3y ago

> invoking them as a separate process from the main application does not count Why not? What is your worry? That starting a process is too slow? That data transfer is too slow? Something else? I agree that process startup may be meaningful overhead in the rare cases when the actual computation is very fast. But sharing uncompressed data between programs is essentially zero-cost. Separate processes look particularly elegant and modular to me. They are much easier to debug and profile than bolted-in dependencies, since the running time and input/output of each process are trivial to isolate. Of course, having both a library and a cli interface for the same computational brick is always better, but the unix-style tool is an essential thing to have.

hresvelgr 3y ago
Performance is not what concerns me, it's design. To what end is it useful to have distinctions between libraries and applications? If I have a JSON parsing library, it might be nice to run one-off scripts that grab a certain value and pipe it somewhere else. It's also useful to deserialise in complex manners inside an app of my design. My belief is that programs whether they are intended to or not usually end up being composed with other programs so on that basis, having the distinction of bin/lib doesn't serve a useful end to me at least.
gonzus 3y ago
I agree 100% with using processes as the natural unit of isolation. I also believe we could solve many of the integration issues between multiple tools, by having them all optionally spit out a universal, easy / easier to parse format -- even JSON would be good here. So you could do ls --foo --bar --json | jq ... and be able to process the output to your heart's content, without having to do any kind of white space parsing; as a free bonus, you get automatic support for file names with embedded white space... The best we have for now is jc (https://kellyjonbrazil.github.io/jc/ https://kellyjonbrazil.github.io/jc/), which knows how to do this for a closed (but growing?) set of commands: ls -l *.db | jc --ls | jq .
dale_glass 3y ago
As someone who writes software that does a lot of this: Process management gets very annoying, very quickly. When you run a helper program soon you have to consider process management. What if it crashes? What if it gets stuck? What if you crash and it remains running? What if the user uses the same program elsewhere? None of this gets work done. A text stream is an awful API. Many times the called program doesn't intend to provide an API, or doesn't commit to stability. Your code breaks because version 5.0.2 fixed a typo, and you relied on the typo for your parsing. A text stream is an awful API. Instead of a nice protocol you get to parse lots of text, deal with escaping and quoting. You better hope the called program does it competently. It may well not, then you have a problem. A text stream is an awful API. No normal program will dump or read a JPEG over stdin/stdout, so if you need to communicate some sort of binary data now you need another communication channel. That may involve commandline arguments, killing, restarting the program, and reestablishing the state. More process management fun. A text stream is an awful API. You'll find yourself doing things like assembling multiple lines of output into a single coherent concept, and trying to detect where something ends when the program doesn't necessarily provide a clear indication. A text stream is an awful API. Sometimes programs print stuff before it took effect, and may not ever give you a clear indication of "now it's been applied". You may need to somehow test for it, retry operations, insert wait states. 99% of the effort invested in this doesn't get work done. You're spending it on management that wouldn't exist if you were using a sane API like DBus or similar, where you don't deal with process management, where things are broken down into nice fields, and where the API is intended as an API.

bluetomcat 3y ago

On a modern Unix-like OS, these tools aren’t as simple as they used to be in the 1970s. The man page for ls alone lists tens of options, corner cases and historical cruft. If those tools were really composable, they wouldn’t need so many options hidden in their man pages.

csydas 3y ago
I don't agree with this conclusion on the number of flags for the GNU core utils relating to it's composability; the flags are natural advanced extensions of what the tool is meant to do, and even without the flags the tools are straight forward enough that the default execution likely will meet the requirements of even new users. From my own experience teaching newbies about bash, the only hard part about all the flags was getting newbies comfortable with ignoring them at first and just trusting that `ls` lists files in the current directory, `sed` let's you manipulate text in a smart way, `grep` finds things, and so on. Once they got that in their head and understood that the flags just allow very specific operations that you previously needed to write a lot more code for, most people got it pretty fast and could punch out quick shell scripts to help with their work. But the base commands without a slew of flags was still quite useful for these newbies. Again though, that's just my experience and it was fairly narrow in scope for accomplishing specific troubleshooting tasks, so that may factor into why I saw success here; I don't know if the people I taught further evolved their scripting skills past basic troubleshooting, though I am fairly confident these persons understand the "kitchen tools" nature of the GNU core utils as many ended up writing scripts on their own for situations we never discussed, and the scripts were pretty okay. Edit: Typo, changed Exceptions to Extensions in the first paragraph

IshKebab 3y ago

I agree. The reason small tools are simple and have few bugs is that they don't do the whole job! They say "here are your modules! now you do the last bit of programming to integrate them and do all the integration testing please. good luck!"