13 ms·

I keep wondering about what seems to be the most important component of Unix philosophy: write many small programs that do one thing well and interface using te

I keep wondering about what seems to be the most important component of Unix philosophy: write many small programs that do one thing well and interface using text streams! Yes, modularity is important. However, in some cases, this philosophy has resulted in the "tangled mess held together by duct tape" kind of systems architecture that no one dares to touch for fear of breaking things. I think Unix philosophy is struggling with a fundamental dilemma: On one hand, creating systems from programs written by different people requires stronger formal guarantees in order to make interfaces more reliable, stronger guarantees than interfaces within one large program written by one person or a small team would require. On the other hand, creating systems from programms written by different people requires more flexibile interfaces that can deal with versioning, backward and forward compatibility, etc, something that is extremely difficult to do across programming languages without heaping on massive complexity (CORBA, WS-deathstar, ...) I think HTTP has shown that it can be done. But HTTP is also quite heavy weight. It doesn't exactly favor very small programs. Handling HTTP error codes is not something you'd want to do on every other function call. In any event, I think Unix philosophy is a good place to start but needs a refresh in light of a couple of decades worth of experience.

lmm 11y ago
The big problem I've found is serialization of sum types. thrift (or protobuf or any number of similar systems) is very good at serializing most of the data one tends to work with, with declarations that are suitably strict but easy to write. But it doesn't have a good way to represent "this field is an A or a B".
- nly 11y ago
  Thrift has unions, so does Apache Avro. Afaik Protocol Buffers is the only one where you have to string together one yourself with a required tag and a bunch of optionals.
  - cfallin 11y ago
    Protobufs has unions now too (they're called "oneofs"), as of version 2.6.0 released last year: https://github.com/google/protobuf/releases/tag/v2.6.0 https://github.com/google/protobuf/releases/tag/v2.6.0
- coderjames 11y ago
  Protobuf has somewhat recently added the 'oneof' feature which tries to deal with these union types. The one downside I found when trying to use it was that the 3rd-party C# Protobuf libraries didn't yet support it. See: https://developers.google.com/protocol-buffers/docs/proto#oneof https://developers.google.com/protocol-buffers/docs/proto#on...
justwannasing 11y ago
Are you saying you disagree with the use of subroutines, functions and interfaces? After all, they are essentially small programs that do one thing and do it well.
- cousin_it 11y ago
  Note the part about text streams in fauigerzigerk's comment. If all functions were limited to accepting a text stream and returning a text stream, the overall program would quickly become a ball of mud.
- fauigerzigerk 11y ago
  No, I don't disagree with the use of functions or interfaces at all, but functions in particular are a programming language construct and as such not easily extended across programming languages, processes and networks, mostly due to different type systems and error states. I think it's not an accident or historical vestige that Unix philosophy uses the word "program". It has technical as well as social implications that have barely changed in the past 50 years. Functions and programs are certainly not an either-or kind of thing but making them one and the same would require a very different kind of operating system than we have ever seen. Say, emacs OS :)
  - contingencies 11y ago
    Functions in particular are a programming language construct and as such not easily extended across programming languages, processes and networks, mostly due to different type systems and error states [...] it's not an accident or historical vestige that Unix philosophy uses the word "program". It has technical as well as social implications that have barely changed in the past 50 years. Insightful.
  - justwannasing 11y ago
    Then I think you are confused or inexperienced because Unix does not treat everything as a text stream and you say functions in a programming language are different so they are better is not a wise statement. It appears you don't understand the fundamental concept and think Unix hasn't improved over the years.
jalfresi 11y ago
Recently I've been exploring the idea that small programs should consist of two parts; a "backend" that you can communicate with via a strict message protocol like protobuf. It can handle multiple requests because it is a mini server, the message format is locked to a "type" and you communicate with it via RPC. The second part is a "frontend", a command-line client that does all the unixy stuff with text streams etc; but is just another RPC client to the backend. This provides flexibility; for quick duct-tape situations you use the front end and pipe anf ilter to your hearts content. Then when stuff needs to get serious, you can bypass the command-line front end and connect directly over RPC using a strict message format which has the ability to handle a couple (doesn't have to many) of concurrent requests. Of course this is early doors yet, but I think it might have legs in the flexibility in the text streams/hard message format debate, for very little additional work.
- forgottenpass 11y ago
  You're sort of describing the design idea common in unix-y areas to build libfoo that does all the work, then a foo frontend that makes tasks useable from the command line. I don't know how widely used this model is, but it's certainly out there. The difference is that in what I've described, the ABI is the common interface and an RPC server would be just another consumer of the library.
  - jalfresi 11y ago
    Yup, even better!
- fmoralesc 11y ago
  Incidentally, this is what neovim does.[1] [1]: http://neovim.io/doc/user/msgpack_rpc.html#msgpack-rpc http://neovim.io/doc/user/msgpack_rpc.html#msgpack-rpc
  - jalfresi 11y ago
    Thanks for the solid example. I especially like the use of the NVIM_LISTEN_ADDRESS environment variable for service discovery of the backend!
ajross 11y ago
> in some cases, this philosophy has resulted in the "tangled mess held together by duct tape" kind of systems architecture that no one dares to touch for fear of breaking things. Pretty much, just as Booch's philosophy has given us tangled messes of class hierarchies with cross references pointed every which way. And just as The Gang of Four gave us AbstractedTangleMessFactorySingleton. The distinction with the Unix-style pipe mess is that it exists and works an order of magnitude faster than other examples of bad code.
- jbooth 11y ago
  Additionally, it's not a religion. Every programmer should be exposed to Unix style programming just like every programmer should be exposed to functional programming. 100% purity in either case is just creating unnecessary problems.
  - jayvanguard 11y ago
    Amen. Extremism in programming, as in religion, is evil.
    - Retra 11y ago
      I would like to point out that most extremism is from the perspective of the listener, not the speaker. If I just wrote a book or article called "Functional programming is great," there'd be plenty of people replying with "Oh, but what about MY favorite style? It's not that great! You're just being extreme!" When the reality is that you can't take an opinion as the whole thought. But people do. Almost always. Regardless of the subject, people will think you're a fanatic for your position, no matter how temporary or experimental it is, unless you pay your dues in apologies and waffle terms.
      - saidajigumi 11y ago
        > I would like to point out that most extremism is from the perspective of the listener, not the speaker. Believe it or not, there is a significant contingent of crazy, death-threat-making jerks fixated on "The Unix Philosophy". As in, an experimental Unixy project[1] that made the rounds (including on HN) some years back and got a fair bit of attention. The project had a vision with scale similar to that of Atom, which is soaring on the front page today. But unlike Atom, this project was quietly dropped from public view because the author received a full blast of nutball hate from misguided Defenders of the Faith/Purity/Whatever. I've been using, studying, and working with *nix systems basically forever (longer than almost everyone reading this). But that experience made me rather allergic to ever using or hearing "The Unix Philosophy". I realized that it's heavily overused as an argument-stopper: "that's not the unix way (so <EOF> off)." I've since witnessed similar language used in many projects, taken in whole context, as a dogmatic excuse rather than an actual source of architectural wisdom. [1] Apologies for being deliberately vague. It's not my place to risk riling up the hate monsters vs. the creator of that project again. I'm still absolutely disgusted that Gamergate-level human insanity was leveled at someone putting forth what was, IMO, one of the most interesting Unix tooling experiments in the past decade.
marcosdumay 11y ago
For a while, Linux mostly had the philosophy that your software should come with a text interface, optionally a GUI, and a C library. That library part is for solving the messes created by tangled text interfaces. But nowadays lots of things only come for the GUI.
- mjevans 11y ago
  A lot of those are actually just task specific business logic style applications in scripting languages (often python, sometimes perl, etc). In this case the 'library' is often the base programming language it's self, as the shipped parts are entirely glue and configuration.
eli_gottlieb 11y ago
>On the other hand, creating systems from programms written by different people requires more flexibile interfaces that can deal with versioning, backward and forward compatibility, etc, something that is extremely difficult to do across programming languages without heaping on massive complexity (CORBA, WS-deathstar, ...) Personally, I think the LLVM project shows what the Right Thing is here: package most of the real functionality into libraries while putting the user interfaces into runnable programs. Everyone who wants to communicate with the functionality can just link the library. And if we want to do it statefully, well, handling global state across a large system is basically the unsolved problem of programming.
JustSomeNobody 11y ago
Anyone can make a mess of anything if they are not sufficiently careful. This is not unique to nix. Personally, I enjoy the nix philosophy of using small programs that do one thing well. Also, I prefer text to proprietary binary formats that _I_ can't do anything with. As a *nix and Windows developer, I've never understood the mentality of wanting to build huge monolithic applications. But, I do think the tide is turning. I think more and more developers are allowing themselves to be influenced by ideas outside of what they are used to and that's a great thing.
0xdeadbeefbabe 11y ago
It's your viewpoint that makes you think it becomes a mess of duct tape and not some failing of the philosophy. To improve the philosophy you look outside the box not inside it i.e. not at http. Something like, "everything is a file" is an improved philosophy (but I'm cheating of course because I know this is a plan9 philosophy to improve upon unix).
webkike 11y ago
I think maybe the Unix philosophy applies to programs that users (read: programmers) use in their own time, and to make other programs.
contingencies 11y ago
Unix philosophy is a good place to start but needs a refresh in light of a couple of decades worth of experience. It's always going to be a good place to start because it embodies a realistic, documented, proven and actionable compromise between engineering purity (the "formal guarantees" you describe) and just shipping something. For a collection of related thoughts, not limited to traditional unix philosophy but still unix philosophy-centric, check out my ansified fortune clone @ https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup

lmm 11y ago

The big problem I've found is serialization of sum types. thrift (or protobuf or any number of similar systems) is very good at serializing most of the data one tends to work with, with declarations that are suitably strict but easy to write. But it doesn't have a good way to represent "this field is an A or a B".

nly 11y ago
Thrift has unions, so does Apache Avro. Afaik Protocol Buffers is the only one where you have to string together one yourself with a required tag and a bunch of optionals.
- cfallin 11y ago
  Protobufs has unions now too (they're called "oneofs"), as of version 2.6.0 released last year: https://github.com/google/protobuf/releases/tag/v2.6.0 https://github.com/google/protobuf/releases/tag/v2.6.0
coderjames 11y ago
Protobuf has somewhat recently added the 'oneof' feature which tries to deal with these union types. The one downside I found when trying to use it was that the 3rd-party C# Protobuf libraries didn't yet support it. See: https://developers.google.com/protocol-buffers/docs/proto#oneof https://developers.google.com/protocol-buffers/docs/proto#on...

justwannasing 11y ago

Are you saying you disagree with the use of subroutines, functions and interfaces? After all, they are essentially small programs that do one thing and do it well.

cousin_it 11y ago
Note the part about text streams in fauigerzigerk's comment. If all functions were limited to accepting a text stream and returning a text stream, the overall program would quickly become a ball of mud.
fauigerzigerk 11y ago
No, I don't disagree with the use of functions or interfaces at all, but functions in particular are a programming language construct and as such not easily extended across programming languages, processes and networks, mostly due to different type systems and error states. I think it's not an accident or historical vestige that Unix philosophy uses the word "program". It has technical as well as social implications that have barely changed in the past 50 years. Functions and programs are certainly not an either-or kind of thing but making them one and the same would require a very different kind of operating system than we have ever seen. Say, emacs OS :)
- contingencies 11y ago
  Functions in particular are a programming language construct and as such not easily extended across programming languages, processes and networks, mostly due to different type systems and error states [...] it's not an accident or historical vestige that Unix philosophy uses the word "program". It has technical as well as social implications that have barely changed in the past 50 years. Insightful.
- justwannasing 11y ago
  Then I think you are confused or inexperienced because Unix does not treat everything as a text stream and you say functions in a programming language are different so they are better is not a wise statement. It appears you don't understand the fundamental concept and think Unix hasn't improved over the years.

jalfresi 11y ago

Recently I've been exploring the idea that small programs should consist of two parts; a "backend" that you can communicate with via a strict message protocol like protobuf. It can handle multiple requests because it is a mini server, the message format is locked to a "type" and you communicate with it via RPC. The second part is a "frontend", a command-line client that does all the unixy stuff with text streams etc; but is just another RPC client to the backend. This provides flexibility; for quick duct-tape situations you use the front end and pipe anf ilter to your hearts content. Then when stuff needs to get serious, you can bypass the command-line front end and connect directly over RPC using a strict message format which has the ability to handle a couple (doesn't have to many) of concurrent requests. Of course this is early doors yet, but I think it might have legs in the flexibility in the text streams/hard message format debate, for very little additional work.

forgottenpass 11y ago
You're sort of describing the design idea common in unix-y areas to build libfoo that does all the work, then a foo frontend that makes tasks useable from the command line. I don't know how widely used this model is, but it's certainly out there. The difference is that in what I've described, the ABI is the common interface and an RPC server would be just another consumer of the library.
- jalfresi 11y ago
  Yup, even better!
fmoralesc 11y ago
Incidentally, this is what neovim does.[1] [1]: http://neovim.io/doc/user/msgpack_rpc.html#msgpack-rpc http://neovim.io/doc/user/msgpack_rpc.html#msgpack-rpc
- jalfresi 11y ago
  Thanks for the solid example. I especially like the use of the NVIM_LISTEN_ADDRESS environment variable for service discovery of the backend!

ajross 11y ago

> in some cases, this philosophy has resulted in the "tangled mess held together by duct tape" kind of systems architecture that no one dares to touch for fear of breaking things. Pretty much, just as Booch's philosophy has given us tangled messes of class hierarchies with cross references pointed every which way. And just as The Gang of Four gave us AbstractedTangleMessFactorySingleton. The distinction with the Unix-style pipe mess is that it exists and works an order of magnitude faster than other examples of bad code.

jbooth 11y ago
Additionally, it's not a religion. Every programmer should be exposed to Unix style programming just like every programmer should be exposed to functional programming. 100% purity in either case is just creating unnecessary problems.
- jayvanguard 11y ago
  Amen. Extremism in programming, as in religion, is evil.
  - Retra 11y ago
    I would like to point out that most extremism is from the perspective of the listener, not the speaker. If I just wrote a book or article called "Functional programming is great," there'd be plenty of people replying with "Oh, but what about MY favorite style? It's not that great! You're just being extreme!" When the reality is that you can't take an opinion as the whole thought. But people do. Almost always. Regardless of the subject, people will think you're a fanatic for your position, no matter how temporary or experimental it is, unless you pay your dues in apologies and waffle terms.
    - saidajigumi 11y ago
      > I would like to point out that most extremism is from the perspective of the listener, not the speaker. Believe it or not, there is a significant contingent of crazy, death-threat-making jerks fixated on "The Unix Philosophy". As in, an experimental Unixy project[1] that made the rounds (including on HN) some years back and got a fair bit of attention. The project had a vision with scale similar to that of Atom, which is soaring on the front page today. But unlike Atom, this project was quietly dropped from public view because the author received a full blast of nutball hate from misguided Defenders of the Faith/Purity/Whatever. I've been using, studying, and working with *nix systems basically forever (longer than almost everyone reading this). But that experience made me rather allergic to ever using or hearing "The Unix Philosophy". I realized that it's heavily overused as an argument-stopper: "that's not the unix way (so <EOF> off)." I've since witnessed similar language used in many projects, taken in whole context, as a dogmatic excuse rather than an actual source of architectural wisdom. [1] Apologies for being deliberately vague. It's not my place to risk riling up the hate monsters vs. the creator of that project again. I'm still absolutely disgusted that Gamergate-level human insanity was leveled at someone putting forth what was, IMO, one of the most interesting Unix tooling experiments in the past decade.

marcosdumay 11y ago

For a while, Linux mostly had the philosophy that your software should come with a text interface, optionally a GUI, and a C library. That library part is for solving the messes created by tangled text interfaces. But nowadays lots of things only come for the GUI.

mjevans 11y ago
A lot of those are actually just task specific business logic style applications in scripting languages (often python, sometimes perl, etc). In this case the 'library' is often the base programming language it's self, as the shipped parts are entirely glue and configuration.

eli_gottlieb 11y ago

>On the other hand, creating systems from programms written by different people requires more flexibile interfaces that can deal with versioning, backward and forward compatibility, etc, something that is extremely difficult to do across programming languages without heaping on massive complexity (CORBA, WS-deathstar, ...) Personally, I think the LLVM project shows what the Right Thing is here: package most of the real functionality into libraries while putting the user interfaces into runnable programs. Everyone who wants to communicate with the functionality can just link the library. And if we want to do it statefully, well, handling global state across a large system is basically the unsolved problem of programming.

JustSomeNobody 11y ago

Anyone can make a mess of anything if they are not sufficiently careful. This is not unique to nix. Personally, I enjoy the nix philosophy of using small programs that do one thing well. Also, I prefer text to proprietary binary formats that _I_ can't do anything with. As a *nix and Windows developer, I've never understood the mentality of wanting to build huge monolithic applications. But, I do think the tide is turning. I think more and more developers are allowing themselves to be influenced by ideas outside of what they are used to and that's a great thing.

0xdeadbeefbabe 11y ago

It's your viewpoint that makes you think it becomes a mess of duct tape and not some failing of the philosophy. To improve the philosophy you look outside the box not inside it i.e. not at http. Something like, "everything is a file" is an improved philosophy (but I'm cheating of course because I know this is a plan9 philosophy to improve upon unix).

webkike 11y ago

I think maybe the Unix philosophy applies to programs that users (read: programmers) use in their own time, and to make other programs.

contingencies 11y ago

Unix philosophy is a good place to start but needs a refresh in light of a couple of decades worth of experience. It's always going to be a good place to start because it embodies a realistic, documented, proven and actionable compromise between engineering purity (the "formal guarantees" you describe) and just shipping something. For a collection of related thoughts, not limited to traditional unix philosophy but still unix philosophy-centric, check out my ansified fortune clone @ https://github.com/globalcitizen/taoup https://github.com/globalcitizen/taoup