14 ms·

This makes me realize: the one example I can think of where software has been "Lego-like" is Unix piping. I'm not a Unix purist or anything, but they really hit

• nwienert 6y ago
  In practice it’s totally inscrutable. I never remember or even feel comfortable guessing at anything more than the most basic. Meanwhile, any typed library in language X usually works immediately with no docs given a decent IDE.
  • _bxg1 6y ago
    I would argue that it thrives in those most basic cases, and isn't really suited to building truly complex systems. But I also don't think there's anything wrong with that. There's a use-case for simple pieces that are easy to snap together, and I think that use-case has been greatly under-served because lots of things that aim for it end up as complicated, multi-faceted APIs. You could almost say that micro-services are trying to follow in the Unix tradition. But the problem is that a) they don't really get used in that ideal, small-scale use-case because they're almost always written and consumed internally, not exposed to the public, and b) they do get used in those huge, complex cases where their lego-ness stops being a virtue and starts being a liability.
    • TAForObvReasons 6y ago
      > starts being a liability SaaS is all the rage now, and the business of SaaS is disincentivized from making "lego-like" software. The easier it is to interoperate, arguably the easier it is to replace or reimplement your service. Because a lot of the value of unix tools is derived from their interoperability, they play nice with each other.
      • _bxg1 6y ago
        Yeah, that is part of the problem too. Any for-profit corp would love to lock people in, and software megacorps no longer need to interoperate for their own survival. So we see increased lock-in across most of the (commercial) software world. An interesting inverse-example is Slack, which does lots of interop with other services, because it isn't a trillion-dollar company that can attempt to give you everything you need under a single roof.
    • galaxyLogic 6y ago
      > I would argue that it thrives in those most basic cases, and isn't really suited to building truly complex systems Right, just like LEGOS thrive in simple cases, are not suited for building complex systems. That is why the idea of "Software LEGOs" is not reality. Or you can say that Unix Pipes are "software LEGOs", with similar limitations.
  • jvanderbot 6y ago
    In your practice you have favored an alternative. (not "in practice", implying absolute) In my practice I have used both with great success. For logging, parsing, and displaying playback data from field systems, UNIX and UNIX-like tools have been incredible. VNLog in particular is a wonderful way to interact with data if you need just a bit of structure on top of unix outputs. And anyway, getting from not-so-great data to something that a typed library in language X can parse is a great job for plain old Unix tools.
    • nwienert 6y ago
      I’m saying in the sense of “time I put in to value I get out”, I’ve put quite a lot of time into bash, but even though I’ve done certain actions many times, I remember few. I recently picked up Swift and it was a dream. Never used it in nearly the same degree, but I was highly productive quickly. The thing is: I don’t think it’s a bad paradigm (everything a stream of text), it’s just the “IDE” sucks completely. If there was inline autocomplete, peek-ahead results, a “verbose mode” where the options are all more like typed full-English objects, instant fuzzy search of actions, multi line with an easy way to edit, etc, it would probably be fantastic and I’d be a 100x shell developer. It would also basically feel like an IDE at that point for a normal programming language!
  • amiga_500 6y ago
    In practice it's been massively successful for decades.
• ohazi 6y ago
  I think the best way to describe pipes to the uninitiated is in terms of copy & paste. Copy & paste is the basic, ubiquitous, doesn't-try-to-do-too-much IPC mechanism that allows normal users to shovel data from one program into another. As simple as it is, it's indispensable, and it's difficult to imagine trying to use a computer without this feature. The same applies to pipes, even though they work a little bit differently and are useful in slightly different situations. They're the "I just need to do this one thing" IPC mechanism for slightly more technical users.
  • boring_twenties 6y ago
    > difficult to imagine trying to use a computer without [copy & paste] Remember the first iPhone? I wasn't into it, but a bunch of my (senior developer) colleagues were. I asked them how they lived without copy & paste and they all told me it was just no big deal.
    • servercobra 6y ago
      A phone and a computer are used in very different ways. Even now, I probably use copy/paste once a week on my iPhone, and at least once an hour on my laptop, often more.
      • jimmaswell 6y ago
        I do it multiple times a day on my Android. I've even taken to using ctrl-c/x/v in Hacker's Keyboard because it's faster.
      • waheoo 6y ago
        I use copy and paste on my phone constantly. I would guess the reason you dont do it much on iphone is the lack of multitasking support. Android alt tabbing has always been a basic part of life and now split screening is a thing too. You lot getting widgets soon, maybe you'll get alt tab next year?
        • vxNsr 6y ago
• epr 6y ago
  Rich Hickey addresses this in his famous 2011 talk, titled Simple Made Easy. > Are we all not glad we don’t use the Unix method of communicating on the web? Right? Any arbitrary command string can be the argument list for your program, and any arbitrary set of characters can come out the other end. Let’s all write parsers. The way that I think about this is that the Unix philosophy, which this behavior is undoubtedly representative of, is at one end of a spectrum, with something like strict typing at the other end. Rich, being a big proponent of what is described in the article as "Lego-like" development clearly does not prefer either end of the spectrum, but something in-between. In my opinion as well, the future of software development is somewhere in the middle of this spectrum, although exactly where the line should be drawn is a matter of trade-offs, not absolute best and worst. My estimation is that seasoned developers who have worked in many languages and in a variety of circumstances have all internalized this.
  • _bxg1 6y ago
    > Let’s all write parsers And yet, at least for the Unix tools I've used, nobody did write an elaborate parser. Instead, they all ended up using newlines to represent a sequence. There were never really nested structures at all. That least-common-denominator format they were forced into ended up making input and output really easy to quickly understand. Maybe the problem with Rest APIs is that JSON does too good of a job at making it easy to represent complex structures? Maybe we'd be better off using CSV as our data format for everything.
    • epr 6y ago
      What trade-offs make the most sense are highly dependent on the problem being addressed. If there were ever an instance where text being the universal input and output type made sense, it would be in a Unix environment. That being said, what percentage of shell scripts in the wild would you estimate contain at least one use of grep or a regular expression in general?
    • thezilch 6y ago
      Agreed. Thinking in arrays is very powerful. Simple. Say Unix tools returned JSON. What good is that without a schema? Instead of a parser, let's all write schema transforms?
• shadowgovt 6y ago
  I think piping works well because it's an opinionated framework for IPC. The strong opinions it holds are: 1) data is a stream of bytes 2) data is only a stream of bytes That's it. And it turns out that's a pretty powerful abstraction... Except it requires the developer to write a lot of code to massage the data entering and/or leaving the pipe if either end of it thinks "stream of bytes" means something different. In the broad-and-flat space where it's most useful---text manipulation---it works great because every tool agrees what text is (kind of... Pipe some Unicode into something that only understands only ASCII and you're going to have a lousy day). When we get outside that space? So while, on the one hand, it allows a series of processes to go from a text file to your audio hardware (neat!), on the other hand, it allows you to accidentally pipe /dev/random directly into your audio hardware, which, here's hoping you don't have your headphones turned all the way up. This example also kind of handwaves something, in that you touched on it directly but called it a feature, not a bug: pipes are almost always the wrong tool if you do want structure. They're too flexible. It's way the wrong API for anything where you cannot afford to have any mistakes, because unless you include something in the pipe chain to sanity-check your data, who knows what comes out the other end?
  • XMPPwocky 6y ago
    A bit of a tangent, but... audio hardware can't really be treated as a stream of bytes either. You used to be able to cat things to /dev/dsp, but- that used something like 8khz, 8-bit, mono audio. That's horrendous. Because, with just a stream of bytes, you have to settle for the least common denominator- /dev/dsp had IOCTLs to set sample rate, number of channels, and bit depth, but... with just a stream of bytes, you can't do that. Similarly, video data via /dev/fb0 - AFAIK you don't even have defaults there to rely on, to display anything useful you need to do IOCTLs to find out about its format. When do you not want structure? Seriously- Plain, human-readable ASCII text is maybe a candidate - but even then there's implicit structure (things like handling CR/LF, tabs...) Unicode text? You know that's structure. (Ever had a read() call return in the middle of a UTF-8 multi-byte sequence? CSV? That's structure. Tab-separated columns? That's structure. Fixed-width columns? Also structure. You don't get to not have structure. Structure is always there. The question is whether you get to have a spec for your structure, or whether it's just "well, from the output it looks like column 3 here is always the hostname of the server I want, so I'll use 'cut' or 'awk' to extract it". That approach can work in practice, but...
• lmilcin 6y ago
  You put this very nicely. This is why functional programming and lisps are such fantastic development environment, because you can use components (functions) that are not very opinionated about what they are acting on.
• deleted 6y ago
  [deleted]
• deleted 6y ago
  [deleted]
• ajuc 6y ago
  It's a good analogy because you can make anything out of lego - even a car - but it won't be anything good for real use, just toys. BTW if I was making UNIX command line today it would use LinkedHashMaps for everything instead of text streams.
• jb3689 6y ago
  Another fascinating thing is how we as a community reacted to this simplicity. One thing in particular that I find interesting is the conventions that have been built up. Many tools don't just accept text streams but react the same way to a common set of options and assume line-separation among other things. None of these are defined in the interface but were good ideas that were adopted between projects
• crimsonalucard5 6y ago
  Unix piping is basically functional programming. If you ever wondered why some people are obsessed with functional programming this is the reason why: Functional programming forces every primitive in your program to be a Lego Block. A lot of functional programmers don't see the big picture. They see a sort of elegance with the functional style, they like the immutability but they can't explain the practical significance to the uninitiated. Functional Programming is the answer to the question that has plagued me as a programmer for years. How do I organize my program in such a way that it becomes endlessly re-useable from a practical standpoint? Functional programming transforms organ transplantation into lego building blocks. The "lego" is the "function" and "connecting two lego blocks" is "function composition". In short, another name for "Point free style" programming is "Lego building block style" programming.
  • EGreg 6y ago
    Want to build web apps from reusable blocks? Reason at a higher level about chatrooms, roles and permissions, credits? That was the thinking behind our open source project: https://qbix.com/platform https://qbix.com/platform Reusability on the web. Here is where we are going: https://qbix.com/QBUX/whitepaper.html#Distributed-Operating-System https://qbix.com/QBUX/whitepaper.html#Distributed-Operating-...
    • adamsea 6y ago
      Is there a sample project somewhere on Github? Would be helpful to see a concrete example of use to get a better idea of it.
  • xvedejas 6y ago
    You can't just compose two functions because they're both written in a functional programming language. The programmer has to have the foresight to make their types compatible. I think the novelty of the Unix pipe for interoperability is that they (typically) work on one agreed-upon kind of data: human readable, white-space separated. So a lot of tools "just work" with each other. There's no reason you can't do this with functional programming, but obviously you can do it with non-functional programming too, and you could certainly fail to do this with functional programming.
• jmchuster 6y ago
  Is is not the same description for REST API? You pass in a text body and get back a text body? Everyone uses JSON instead of a complicated data protocol?
  • ehnto 6y ago
    A UNIX pipeline is more like a set of operations on the same object passing through the pipeline, and typically that object is a text file. REST APIs represent an ability to interact with a repository of information with a really specific protocol, yet the response is not in that same format as the input. You couldn't pass the response of a REST API into another REST API without managing it externally. Consider cat myData | toolStripsWhitespace > myData Versus var myData = RestApi->read(myDataRecordID) var mutatedData = someMutation(myData) RestApi->update(myDataRecordID, mutatedData) You could of course write your ORM in a pipeline-like manner, many do. But that's got nothing to do with REST itself. var myData = new SomeObject(RestApi->read(myDataRecordID)) ->someMutation('params') ->someOtherMutation('params') RestApi->update(myDataRecordID, myData->toJSON())
    • jmchuster 6y ago
      Ah, so you're looking for something more like var myData = RestApi->get('/records', id=myDataRecordID) myData = RestApi->post('/tools/strips-whitespace', myData) So, you can certainly write such HTTP endpoints to work exactly as you've described, but it doesn't really make sense to make these network calls when you can just run such a simple function in memory. So, we usually say make a call to an api, do whatever manipulating we want in code, make a call to another api, manipulate, and so on and so on. If we removed the ability to manipulate in code and forced you to make calls to '/tools' endpoints for all of that, then we'd be in the same place as UNIX pipelines, no?
      • ehnto 6y ago
        On any old API maybe, but not a REST API. It's ignoring the core ideologies of REST, which I'll get to below the code, but despite that it's also just not the same in real world usage either: cat myData | stripWhitespace | allToUppercase | removeTheLetterC > myData Versus var myData RestAPI->get'/records', id=myDataRecordID) myData = RestAPI->post('/tools/strips-whitespace', myData) myData = RestAPI->post('/tools/allToUppercase', myData) myData = RestAPI->post('/tools/removeTheLetterC', myData) It's outside the scope of a REST API to have transformation endpoints. A REST API is supposed to allow simple access to a systems state, most often records in a database, in order to retrieve, store and update that state. REST standing for Representational State Transfer, and transformations don't fit into that at all. It's not a transfer of state from one system to another if you are giving it your data and having it manipulate it then give it back, it's asking one system to do work for you and give you the result back. Of course in the real world, you can put whatever endpoints you want into your API, and there are dozens of wildly different APIs being called RESTful, but regardless of any of that, rarely are they conducive to being used in pipes, because the point the GP was getting at is that tools are made in the UNIX ecosystem with the pipeline in mind, and that is not the case with REST endpoints. Thanks for the discussion by the way!
• taneq 6y ago
  > I'm not a Unix purist or anything, but they really hit on something special when it came to "code talking to other code". I agree, and it just hit me while reading your comment that the special thing is not just that you can plug any program into any other program. It's that if one program doesn't work cleanly with another, this enforced simplicity means that you can easily modify the output of one program to work with another program. Unix command-line programs aren't always directly composable but they're adaptable in a way that other interfaces aren't. It's not great for infrastructure, don't get me wrong. This isn't nuts and bolts. It's putty. But often putty is all you need to funnel the flow of information this one time.