5 ms·

Rich Hickey addresses this in his famous 2011 talk, titled Simple Made Easy. > Are we all not glad we don’t use the Unix method of communicating on the web? Ri

by epr 6y ago

Rich Hickey addresses this in his famous 2011 talk, titled Simple Made Easy. > Are we all not glad we don’t use the Unix method of communicating on the web? Right? Any arbitrary command string can be the argument list for your program, and any arbitrary set of characters can come out the other end. Let’s all write parsers. The way that I think about this is that the Unix philosophy, which this behavior is undoubtedly representative of, is at one end of a spectrum, with something like strict typing at the other end. Rich, being a big proponent of what is described in the article as "Lego-like" development clearly does not prefer either end of the spectrum, but something in-between. In my opinion as well, the future of software development is somewhere in the middle of this spectrum, although exactly where the line should be drawn is a matter of trade-offs, not absolute best and worst. My estimation is that seasoned developers who have worked in many languages and in a variety of circumstances have all internalized this.

_bxg1 6y ago
> Let’s all write parsers And yet, at least for the Unix tools I've used, nobody did write an elaborate parser. Instead, they all ended up using newlines to represent a sequence. There were never really nested structures at all. That least-common-denominator format they were forced into ended up making input and output really easy to quickly understand. Maybe the problem with Rest APIs is that JSON does too good of a job at making it easy to represent complex structures? Maybe we'd be better off using CSV as our data format for everything.
- epr 6y ago
  What trade-offs make the most sense are highly dependent on the problem being addressed. If there were ever an instance where text being the universal input and output type made sense, it would be in a Unix environment. That being said, what percentage of shell scripts in the wild would you estimate contain at least one use of grep or a regular expression in general?
- thezilch 6y ago
  Agreed. Thinking in arrays is very powerful. Simple. Say Unix tools returned JSON. What good is that without a schema? Instead of a parser, let's all write schema transforms?
- nanomonkey 6y ago
  > And yet, at least for the Unix tools I've used, nobody did write an elaborate parser. sed and awk are the elaborate parsers that are used in complex unix command line scripting.
  - jb3689 6y ago
    I think what the OP is getting at is that the people writing the other command line tools didn't have to write that logic (and thankfully we can instead centralize it in a limited set of tools like sed and awk)
    - nanomonkey 6y ago
      I may be mistaken, but I assumed that they meant that no parsing needed to be done between components, and that they just stream together, which is not really the case. You often have to use sed and awk (eleborate parsers) between different components to get the necessary portions of data from one component into the other...they don't just fit together like legos.
      - fiddlerwoaroof 6y ago
        Yeah but sed and awk are amazing “sharp tools” for this sort of task: they have a learning curve, but 100 characters of awk can do some amazing stuff.
  - _bxg1 6y ago
    There's some truth to that, although there's still something to be said for a lack of recursive structure. Regular expressions can only match a flat pattern. Working with a JSON expression, even if you already have a JSON parser on hand, is often going to add more complexity than parsing a flat line of raw text using a regular expression.
- notriddle 6y ago
  > Instead, they all ended up using newlines to represent a sequence. The biggest mistake in the design of Linux is that they allow newlines in filenames. Seriously, what the heck?!
- fiddlerwoaroof 6y ago
  Also, tools like jq, xsv and xmlstarlet really solve this so-called “problem”: you just need an widely-supported serialization Formats and good tools to focus on the subset of structured data and pull it out into easily-manipulated text.
_jal 6y ago
There is also an audience mismatch. Most of the criticism of pipelines is from professional programmers. A lot of the value of scripting, pipes and associated concepts is low barrier to entry. I agree with the value of strong typing. But I also remember how infuriating it was to learn to work with a type system when I was learning to write code. When I need a quick answer of some sort, iteratively piping crap at a series of "| grep | awk" is exactly what the doctor ordered. Sure, I could bullet proof it nicely and make it reusable by investing time the time to write it in something saner, but there's zero reason to - I'm not likely to ever want to perform the same action again.
jb3689 6y ago
> In my opinion as well, the future of software development is somewhere in the middle of this spectrum Unfortunately this is complexity in and of itself. I don't disagree that different cases require different tools, however the split should be strongly weighted in one direction. Mostly IPC over pipes with a few exceptions, mostly REST and JSON with a few exceptions, mostly language X with a few exceptions. Everyone will have their own preferences, but I think it's important to pick a side (or at least mostly pick a side) or else you accept chaos
- chubot 6y ago
  I think of it as a matter of scale. - At the scale of 1,000 to 1M lines of code, authors/programmers have some control, and you're dealing with a time scale of months or years. - At the scale of 10M to 100M lines of code, no organization is an author with complete control, and you're dealing with a time scale of decades. In the latter case, you can't make global changes to the system, so you end up with glue in the form of pipes and textual data dumps. You're going to have some chaos because you can't go and change every system to be consistent. So I would say the future of software development is at BOTH ends of the spectrum, not somewhere in the middle. ----- Good material on the subject: https://www.dreamsongs.com/Files/DesignBeyondHumanAbilitiesSimp.pdf https://www.dreamsongs.com/Files/DesignBeyondHumanAbilitiesS... If you look at the code in a web browser or operating system you will see this "lack of control". Once a codebase reaches a certain size, there's only so much you can do with it, and you end up with loosely coupled glue around it. I think a reasonably minimal Debian system has to have somewhere between 100M and 1B lines of code in it, and that whole thing becomes a blob that you dump on a virtually free computer like a Raspberry Pi, etc. I think if you look at the software that banks and airlines run on you will see the same thing. There's some code from the 60's or 70's written in some weird language at the heart of it. You don't really get to architect the system; it's more accurate to say that the system places its constraints on you.
dragonwriter 6y ago
> Are we all not glad we don’t use the Unix method of communicating on the web? Right? Any arbitrary command string can be the argument list for your program, and any arbitrary set of characters can come out the other end. Let’s all write parsers. Um, but that's exactly what we do use on the web. Oh, sure, there's some really popular formats for input and output with widely available parsers (HTML, XML, JSON), and HTTP itself (as well as other protocols) specifies headers and other formats that need to be parsed before you get to those body formats, including telling you what those headers tell you about the other formats so you can no which parsers you need to write, or use if you can find one written for you.
Izkata 6y ago
> Are we all not glad we don’t use the Unix method of communicating on the web? Uzbl is a collection of "web interface tools" that adhere to the Unix philosophy, that come together to create a browser. https://www.uzbl.org/ https://www.uzbl.org/
taneq 6y ago
> Are we all not glad we don’t use the Unix method of communicating on the web? Right? Any arbitrary command string can be the argument list for your program, and any arbitrary set of characters can come out the other end. He clearly uses a different web to the one I use. >.>

by epr 6y ago

_bxg1 6y ago

> Let’s all write parsers And yet, at least for the Unix tools I've used, nobody did write an elaborate parser. Instead, they all ended up using newlines to represent a sequence. There were never really nested structures at all. That least-common-denominator format they were forced into ended up making input and output really easy to quickly understand. Maybe the problem with Rest APIs is that JSON does too good of a job at making it easy to represent complex structures? Maybe we'd be better off using CSV as our data format for everything.

epr 6y ago
What trade-offs make the most sense are highly dependent on the problem being addressed. If there were ever an instance where text being the universal input and output type made sense, it would be in a Unix environment. That being said, what percentage of shell scripts in the wild would you estimate contain at least one use of grep or a regular expression in general?
thezilch 6y ago
Agreed. Thinking in arrays is very powerful. Simple. Say Unix tools returned JSON. What good is that without a schema? Instead of a parser, let's all write schema transforms?
nanomonkey 6y ago
> And yet, at least for the Unix tools I've used, nobody did write an elaborate parser. sed and awk are the elaborate parsers that are used in complex unix command line scripting.
- jb3689 6y ago
  I think what the OP is getting at is that the people writing the other command line tools didn't have to write that logic (and thankfully we can instead centralize it in a limited set of tools like sed and awk)
  - nanomonkey 6y ago
    I may be mistaken, but I assumed that they meant that no parsing needed to be done between components, and that they just stream together, which is not really the case. You often have to use sed and awk (eleborate parsers) between different components to get the necessary portions of data from one component into the other...they don't just fit together like legos.
    - fiddlerwoaroof 6y ago
      Yeah but sed and awk are amazing “sharp tools” for this sort of task: they have a learning curve, but 100 characters of awk can do some amazing stuff.
- _bxg1 6y ago
  There's some truth to that, although there's still something to be said for a lack of recursive structure. Regular expressions can only match a flat pattern. Working with a JSON expression, even if you already have a JSON parser on hand, is often going to add more complexity than parsing a flat line of raw text using a regular expression.
notriddle 6y ago
> Instead, they all ended up using newlines to represent a sequence. The biggest mistake in the design of Linux is that they allow newlines in filenames. Seriously, what the heck?!
fiddlerwoaroof 6y ago
Also, tools like jq, xsv and xmlstarlet really solve this so-called “problem”: you just need an widely-supported serialization Formats and good tools to focus on the subset of structured data and pull it out into easily-manipulated text.

_jal 6y ago

There is also an audience mismatch. Most of the criticism of pipelines is from professional programmers. A lot of the value of scripting, pipes and associated concepts is low barrier to entry. I agree with the value of strong typing. But I also remember how infuriating it was to learn to work with a type system when I was learning to write code. When I need a quick answer of some sort, iteratively piping crap at a series of "| grep | awk" is exactly what the doctor ordered. Sure, I could bullet proof it nicely and make it reusable by investing time the time to write it in something saner, but there's zero reason to - I'm not likely to ever want to perform the same action again.

jb3689 6y ago

> In my opinion as well, the future of software development is somewhere in the middle of this spectrum Unfortunately this is complexity in and of itself. I don't disagree that different cases require different tools, however the split should be strongly weighted in one direction. Mostly IPC over pipes with a few exceptions, mostly REST and JSON with a few exceptions, mostly language X with a few exceptions. Everyone will have their own preferences, but I think it's important to pick a side (or at least mostly pick a side) or else you accept chaos

chubot 6y ago
I think of it as a matter of scale. - At the scale of 1,000 to 1M lines of code, authors/programmers have some control, and you're dealing with a time scale of months or years. - At the scale of 10M to 100M lines of code, no organization is an author with complete control, and you're dealing with a time scale of decades. In the latter case, you can't make global changes to the system, so you end up with glue in the form of pipes and textual data dumps. You're going to have some chaos because you can't go and change every system to be consistent. So I would say the future of software development is at BOTH ends of the spectrum, not somewhere in the middle. ----- Good material on the subject: https://www.dreamsongs.com/Files/DesignBeyondHumanAbilitiesSimp.pdf https://www.dreamsongs.com/Files/DesignBeyondHumanAbilitiesS... If you look at the code in a web browser or operating system you will see this "lack of control". Once a codebase reaches a certain size, there's only so much you can do with it, and you end up with loosely coupled glue around it. I think a reasonably minimal Debian system has to have somewhere between 100M and 1B lines of code in it, and that whole thing becomes a blob that you dump on a virtually free computer like a Raspberry Pi, etc. I think if you look at the software that banks and airlines run on you will see the same thing. There's some code from the 60's or 70's written in some weird language at the heart of it. You don't really get to architect the system; it's more accurate to say that the system places its constraints on you.

dragonwriter 6y ago

> Are we all not glad we don’t use the Unix method of communicating on the web? Right? Any arbitrary command string can be the argument list for your program, and any arbitrary set of characters can come out the other end. Let’s all write parsers. Um, but that's exactly what we do use on the web. Oh, sure, there's some really popular formats for input and output with widely available parsers (HTML, XML, JSON), and HTTP itself (as well as other protocols) specifies headers and other formats that need to be parsed before you get to those body formats, including telling you what those headers tell you about the other formats so you can no which parsers you need to write, or use if you can find one written for you.

Izkata 6y ago

> Are we all not glad we don’t use the Unix method of communicating on the web? Uzbl is a collection of "web interface tools" that adhere to the Unix philosophy, that come together to create a browser. https://www.uzbl.org/ https://www.uzbl.org/

taneq 6y ago