7 ms·

Forgetting the history of Unix is coding us into a corner

93 points by lproven 3y ago

lproven 3y ago
The lessons of yesteryear's OS are getting lost in translation. This is by me on El Reg, and it's part 2 of this year's FOSDEM talk. It will (I hope!) make more sense in context with the rest.
- deleted 3y ago
  [deleted]
- gumby 3y ago
  It’s a ramble that ends up with a complaint that Wayland doesn’t fit into the stream file model that’s supposedly critical to “unix”. Block files have been in Unix forever and they don’t either. Maybe read “Worse is Better” before writing part 3. And “Unix” is orthogonal with FLOSS. Sorry to be such a downer — been reading el reg since it started.
  - lproven 3y ago
    > Maybe read “Worse is Better” before writing part 3. Way ahead o' ya, kid. https://www.theregister.com/2023/12/25/the_war_of_the_workstations/ https://www.theregister.com/2023/12/25/the_war_of_the_workst... Note that this is loosely based on part of a FOSDEM talk I gave 4 years ago.
- cjs_ac 3y ago
  Part 3 is going to be about Plan 9 and Redox OS, right?
  - lproven 3y ago
    It is indeed going to be about Plan 9. :-) The talk was a couple of weeks ago now and is already online: https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-way-forward-finding-a-path-to-what-comes-after-unix/ https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-... I linked to it in the first paragraph. You can read the script if you want. Redox, OTOH, no: it does not even enter into it. Although I do not address it, if pressed, I might concede that if there is a Rust involvement, it's R9, the successor to Harvey OS. https://github.com/r9os/r9 https://github.com/r9os/r9
- torstenvl 3y ago
  I read the article, and I didn't see this point made. Mostly it was a very long description of the history of Unix, with a tangent about microkernels. Your headings are What is 'Unix' anyway? Generations and the gaps between Turn it up to 11 I believe your ultimate point is about Wayland and perhaps that it should follow everything-is-a-file. But that comes so late in the article that you barely give it a passing mention, and it isn't consistent with the premise of the article.
  - lproven 3y ago
    > I believe your ultimate point is about Wayland Nah. :-) I am not terribly interested in Wayland, myself, and I suspect that in the fullness of time it will turn out to be a not-very-interesting blind alley.
    - torstenvl 3y ago
      Okay. So what is the thesis of this article and where is it discussed?
      - lproven 3y ago
        As it says, and links, in the first paragraph of the story: « This article is the second based on The Reg FOSS desk's talk at FOSDEM 2024 (part 1 was Drowning in code). » That para contains hyperlinks to both. The talk was a couple of weeks ago now and is already online: https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one- https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-... You can read the script if you want.
        torstenvl 3y ago
        The existence of more than one article based on a talk does not by itself suggest that they are dependencies of one another; nor does it excuse rambling. The title of the OP is "Forgetting the history of Unix is coding us into a corner." At no point do you explain who is forgetting what lessons, or why such forgetting is coding us into a corner. In any case, the history is interesting and clearly well-researched. I offer as constructive criticism that your structure and clarity could benefit from continued editing.
- Diris 3y ago
  I saw the talk at FOSDEM and greatly enjoyed it. Someone sitting next to me was busy coding on 9front haha
  - lproven 3y ago
    :-D Thanks! I had some great feedback from some 9Front folks and ended up meeting up with a bunch of them after the event. Given that I was half-expecting a lynch mob, this was a very pleasant surprise indeed... ;-)
didntcheck 3y ago
On the contrary, the widespread obsession with "Unix philosophy" often provides quasi-religious objections to suggestions we ever try new approaches to things. In particular, the desire to cram increasingly diverse and rich interfaces into 70s file IO is a strange sacrament
- seydar 3y ago
  What are some alternatives that you'd recommend people look at? I'm keen for any potential successor, but a lot things come back to effectively text, and anything we could build on top of it. Binary object formats seem like an alternative for faster parsing of structured data, so while that ability has always been it, it's a matter of people actually sticking to it. Maybe some coordination between the program and the shell?
  - LeonidasXIV 3y ago
    Some alternatives is a structured data interface, a bit like PowerShell where not everything needs to be serialized and deserialized and parsed in every program in a pipeline. Another approach is that the OS is basically a complete environment where everything is code. You can see the idea in the Smalltalk environments where you can theoretically interact with every object by sending messages. Lisp machines come to mind as well and one could even consider early personal computers that booted into BASIC as an idea of this (though in BASIC instead ov everything is a file everything is memory)
    - traverseda 3y ago
      Generally I think people are happy piping structured data around, the problem is that ties you into a particular data structure. JSON seems to be winning there, and I think people would be very happy if more commands had a JSON environment variable, or even if it was possible to output data on a /dev/stdjson pipe. Anyone know how to add a new stdout-like interface to unix-like OSes?
      - dale_glass 3y ago
bjiuyg7544 3y ago
[dead]
stcroixx 3y ago
It's not just unix, but programming languages themselves and general patterns for how software is built and how projects are run. It's what I'm most frustrated by in this industry. So much waste. I once thought surely this industry would mature, but now I don't see that happening in my lifetime. Reminds me of human civilizations before people developed language and wrote things down and almost all knowledge was lost with each new generation, keeping progress limited to a snails pace.
- gregjor 3y ago
  You might enjoy the insights in Richard Gabriel's book Patterns of Software.
bradley13 3y ago
The advantage (or disadvantage) of getting older: Seeing the younger generations come up with brilliant "new" ideas. New to them, whereas I think "yep, that was done 40 years ago, and 20 years ago, and now it's new all over again." Part of the problem is the sheer breadth of IT and software development today. 40 years ago, I felt like I understood nearly everything: I designed processors at the transistor level, programmed from machine code through assembly to Lisp, Prolog, Pascal and other languages. Worked on projects from tiny to huge. That's just not possible today. There are too many areas of application, too many frameworks and languages. Too many tools and platforms. Any individual developer is going to be in a niche. Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants. Today, it's all you can do to stay abreast of developments in your own field. There is little time to look for lessons from the past, or across to ideas from other, parallel fields.
- cdelsolar 3y ago
  As a former electrical engineer now web app/api/full stack developer I either know or can quickly learn all those things.
  - voakbasda 3y ago
    Not at sufficient depth to go head-to-head with an expert. If you are a Jack of All Trades, you will be Master of None.
    - andrewmcwatters 3y ago
      I disagree with the Jack of all trades, master of none cliché. You can effectively be a master of multiple fields because the true applicable depth of so many of them are completely shallow. There are entire niches in software development that you can dismiss wholesale. Virtually everything in the web that isn't the web standards themselves or from the most popular libraries isn't worth learning because it's just regurgitation. If you know CSS as an applicable professional level, any CSS framework you see is irrelevant. You can pick it up. Etc.
cladopa 3y ago
I believe the main corner was Unix itself. "Everything is a file" is not a lesson, it is an imposition that with some some things that are very different to files it is a very wrong thing. The lesson is: Minimise as much as possible the different interfaces necessary and reuse them. Interlisp or Lisp machines used Expressions so everything could be read and written by everything. Unix is a mess of binary formats, a mess of different console emulator formats, a mess of different text formats. And the article is just about one guy who does not like Wayland not using files for everything, which is fine for me.
- gpderetta 3y ago
  > Minimise as much as possible the different interfaces necessary and reuse them. Very very strongly agree. As a corollary a small number of too simple interfaces is better than a larger number of more sophisticate interfaces. "It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures." - Alan Perlis' Epigrams on Programming (1982).
RcouF1uZ4gsC 3y ago
Actually, I think forgetting the history of mainframes (which were before Unix) is worse. We are now reinventing a lot of concepts with regards to security, isolation, distributed computing. Unix was probably a step backward, yet people keep on testing it is the pinnacle of how to organize and abstract computing.
- lproven 3y ago
  Strongly agree, and I would point you at my feature column that was published on Xmas day. It's linked twice in these comments already, so I won't add yet another.
HackerThemAll 3y ago
“Those that fail to learn from history are doomed to repeat it.” Winston Churchill, 1948. As can be seen, it equally well applies to history of computing. We keep reinventing the same things only because young coders don't learn from computing history and outright dismiss anything that's more than 1 year old.
- gregjor 3y ago
  Churchill was paraphrasing George Santayana, and leaving off most of the original. The author of the article we're commenting on included the original quote. The context seems relevant here: Progress, far from consisting in change, depends on retentiveness. When change is absolute there remains no being to improve and no direction is set for possible improvement: and when experience is not retained, as among savages, infancy is perpetual. Those who cannot remember the past are condemned to repeat it. -- George Santayana
  - jiveturkey 3y ago
    I dunno. It seems to be captured quite concisely in the Churchill version.
bowsamic 3y ago
I'm really not a fan of UNIX in the slightest, other than as a basic kernel OS to build something nicer on, like what macOS did. I think it's much better to make it easier to write programs and have a rich system library than to glue together text streams. And in fact that's basically what has happened instead: the "UNIX philosophy" has largely been replaced by Python and the like. People now reach for Python for many tasks that were previously shell one-liners, and imo that's a positive development. When I was a kid learning how to code reading online, everyone was religiously defending the "UNIX philosophy" showing off their fancy and indecipherable one-liners (that don't work on my system bc I have GNU awk instead of whatever awk they have or whatever). This has basically died out year by year as so many people switch over to scripting languages with great standard libraries and super-powered easy to install libraries. Now, I'm not really a fan of the super overkill object-oriented approach that systems like mac and smalltalk take, but I think our modern approach is a good compromise
- jll29 3y ago
  On the topic of "UNIX shell one-liners versus Python", I enjoy letting my students solve the "make a list of the words in a file with their counts" exercise in Python, then I show them a sh one-liner. The Python scripts takes them from 10 minutes to two hours, they're buggy, and they cannot process big files - everything is in RAM (with a monster dict) - without a crash/system freeze. The power of UNIX, pipes and re-use.
  - d0mine 3y ago
    Space-separated case: import sys from collections import Counter L = Counter( word for line in sys.stdin for word in line.casefold().split() ) L is the words' multiset (the desired list of words with counts). How many words are there in English? 1M? Huge text [Unicode] files. Fancier tokenizer: for word in re.findall(r"\w+", line.casefold())
- tech_ken 3y ago
  The one thing holding Python back from completely eating my workflow of utils, glue, and hacks is environment isolation. Nothing sucks more than when activating some venv breaks a background tool I wrote 3 years ago and forgot about. I recognize that this is a surmountable problem with at most moderate effort, but getting to a point of complete frictionlessness would really move the whole ecosystem forward a lot, IMO.
jpgvm 3y ago
The main lesson from unix that is getting lost is composability of systems. So many silos and proprietary "cloud" systems, so much glue that needs to be constantly rewritten because there isn't any stability in an ecosystem composed of competing rather than cooperating actors. There are great things that have come along, I would say k8s has been very successful at introducing a highly interoperable, composable API for abstracting over a bunch of machines. But the trend towards hosted services that are incentivized to create lock-in and data-gravity isn't great.
- rjbwork 3y ago
  I don't think this lesson is getting lost accidentally, but intentionally. There was a brief moment in the late 00's/early 10's where companies were creating these rich web API's that would let you (and other services acting on your behalf) interact with their system in arbitrary and novel ways. But that turned out to not be very profitable (or at the very least, wouldn't let you create a walled garden that would get 100x exit for your VC's) so we've steadily moved to more locked down, less interactive, less open, less useful, but ultimately more profitable services. Emblematic of this journey is Twitter, I think.
amelius 3y ago
Curious, in what ways would an OS be different from Unix when invented today?
- kekebo 3y ago
  It may be written in Javascript.
  - amelius 3y ago
    Haha, yes. I was thinking more about things like containers, and the concepts around cgroups etc. that I think is currently a mess (from the user's point of view).
- fsflover 3y ago
  https://qubes-os.org https://qubes-os.org
  - lproven 3y ago
    This was indeed one of my inspirations.
- traverseda 3y ago
  It would probably look a lot like plan9
  - lproven 3y ago
    It would, yes. Here's that part: https://www.theregister.com/2024/02/21/successor_to_unix_plan_9/ https://www.theregister.com/2024/02/21/successor_to_unix_pla... But what I am proposing is a way to bring mainstream Unix apps to it, without bloating it with emulation.
javier_e06 3y ago
I started working on the embedded industry in the early 90s when other OS'es with the likes of Greenhills and VxWorks. My first introduction to a Unix system was Caldera Linux in 3 or 4 floppy disks. What call me attention the most compared with other embedded OS was the transparency of the system. We all had our door-stop manuals 10 lbs weight manuals for our embedded OS documentation where a collection of tools like Tornado (In VxWorks) gave us a way to interact with the OS. But Unix, in my understanding, needed no added tool. A kernel, a filesystem a shell... go. Many years a go a developer tried to introduce a patch in Linux to give graphics a leg up because Linux PC Desktop experience lagged behind Windows and Macs. The gate-keepers said no and the guy forked the kernel and went his own merry way. Going back to embedded real-time systems and Unix/Linux efforts like MonteVista and HardHat created smaller leaner footprints that started competing in the embedded field. The OS evolves wells. My lasts embedded projects with Yocto make the point. Unix/Linux, ideas that evolve.
- lproven 3y ago
  > My first introduction to a Unix system was Caldera Linux in 3 or 4 floppy disks. You know what, although I experimented with Lasermoon Linux/FT on CD in ~1996, and tried and failed to install Slackware 3.0 around then, the first Linux I ran as my daily desktop OS for a while was Caldera OpenLinux 1. Your memory deceives you: it came on a CD. It needed 3 or 4 boot diskettes to get to the point where it could find your CD drive and start the installation process. It had the first graphical installation in the Linux world, and while it wasn't a live environment and had no desktop, it did multitask: you could play Tetris while it installed.
  - javier_e06 3y ago
    You are right. The floppy disk was to load the drivers for CD. Cutting edge tech.
assimpleaspossi 3y ago
>Like it or not, this means that Linux isn't "a Unix-like OS" any more. It is a Unix. This needs to be corrected. Two Chinese centOS distros passing the Unix test does not make all of Linux and its distros "Unix".
- lproven 3y ago
  I disagree. If one distro can pass the test, then in principle, any distro can. If two distros passed, then it is case closed. The entire point of the argument is not "can Linux pass?" but "will anyone pay to put Linux through the test?" And the answer is yes, repeatedly, although not any more, and who can blame them. Unix has meant "passes the compatibility test" for ⅓ of a century. Linux passed. Once a Unix, always a Unix. It's Unix. FWIW the time to protest was a year ago when I wrote the "Unix is dead" story. Here was the brief discussion: https://news.ycombinator.com/item?id=34412022 https://news.ycombinator.com/item?id=34412022
  - assimpleaspossi 3y ago
    Linux is only the kernel. To pass the Unix certification: https://www.opengroup.org/membership/forums/platform/unix https://www.opengroup.org/membership/forums/platform/unix >The specification encompasses the base operating system environment, networking services, windowing system services, and internationalization aspects and programming languages.
    - lproven 3y ago
      I still disagree. I am not saying "all distros have passed." They haven't. I am saying "all distros _could_ pass, if anyone cared enough to pay." Do you think you can falsify that?
      - assimpleaspossi 3y ago
        All distros MIGHT pass if they meet the qualifications but just because these two distros passed does NOT mean ALL Linux distros can pass and it doesn't mean any other distros can pass.
moomin 3y ago
Yet another article fetishizing Plan 9 accompanied with a history lesson that doesn't support the conclusion. No, not everything is a file. Having a somewhat standardised way of accessing certain features in a stream-like way is cool, but it's not the fundamental feature of any unix, it's just a cute piece of user-space. And there's better ways to evaluate a design than aesthetic appreciation.
- lproven 3y ago
  Hang on. You haven't got to the bit about Plan 9 yet. It was only published today. https://www.theregister.com/2024/02/21/successor_to_unix_plan_9/ https://www.theregister.com/2024/02/21/successor_to_unix_pla... HN post (not mine, I was too slow): https://news.ycombinator.com/item?id=39451861 https://news.ycombinator.com/item?id=39451861
loeg 3y ago
I mean, the history of Unix is fun, whatever, but the next step (showing it is "coding us into a corner") is totally missing.
- lproven 3y ago
  Yes it is. As the saying once went: tune in next time for next week's exciting episode!
  - loeg 3y ago
    Seems clickbaity to title part 1 based on unpublished future parts.
    - lproven 3y ago
      Given that a 1000w article is quite long and the first cut of the talk was 6000w, how else would you handle it?
daniel_grady 3y ago
> For instance, macOS is a Unix™. It passes the tests, and Apple pays for the certification. But it hides most of the real Unix directory tree, its /etc is relatively empty, it doesn't have an X server – it's an optional extra. And most of all, it's not case sensitive. What does it mean for the operating system to be “case sensitive”? Certainly APFS is case sensitive, so this must refer to something else?
- wtallis 3y ago
  OS X / macOS has always defaulted to using filesystems that are case-preserving but not case-sensitive. You can cd to ~/library and see the contents of ~/Library and trying to mkdir ~/library will fail because ~/Library exists.
  - daniel_grady 3y ago
    Amazing, I can check off “learn something” early today!
deleted 3y ago
[deleted]
bigbillheck 3y ago
> So any Wayland evangelists out there, tell us: where in the file system can I find the files describing a window on the screen under the Wayland protocol? What file holds the coordinates of the window, its place in the Z-order, its colour depth, its contents? I haven't used X11 in several years, and it's been well over a decade since I've had to put any serious thought into it, but I honestly don't remember being able to do this there.
zokier 3y ago
The random Wayland bashing seems both completely unrelated to whatever the topic of the article is, and uncalled for. Wayland literally is a protocol that is communicated over a file (WAYLAND_DISPLAY). Sure, the file is a (UDS) socket, but it is still a file! Also Wayland is purely an userland thing, so it is very much constrained on what primitives kernel has to offer. The way the Wayland section is included there, it seems like author is heavily implying that it is prime example of "spiralling software complexity and bloat", which to me sounds mostly like they do not know what they are talking about. Core Wayland is distinctly smaller and simpler than X11 ever was. Sure, modern graphics (etc) are intrinsically complex and Wayland exposes some that, but it does it mostly by keeping Wayland itself as simple and transparent as possible, instead of making Wayland (or rather its implementations) hugely complex and bloated. But besides this Wayland thing, I'm really confused what the article is actually trying to say and how does the title relate in any way to the content? The intro implies that there are some lessons to be learned from UNIX history, but the article sure as hell doesn't cast any light on what those lessons might be. There is also this implication that we are somehow coding ourselves into a corner, but that is never actually expanded on what they are meaning with that? In terms of lessons learned, the key observation maybe is that UNIX peaked somewhere around SysIII before BSD etc hippies set their grubby hands on it and messed everything up.
taeric 3y ago
If we are going to go with witty quotes talking about older lessons that people reject, I am rather fond of https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule. I'm a little worried this encourages ignoring things that have been learned recently. But it is very hard to not be annoyed when you realize how much ground was covered early on.
euroderf 3y ago
When I think of the Unix approach that is based on text stream and the pipes that carry them, I think of telephony. (What a coinky-dink.) The Unix way is "in-band", where metadata (for telephony: signaling) is in the same stream (channel) as the data. It makes reinvention (upgrading) of the shell super difficult. Nobody can even agree precisely how to (for example) include column headers in a CSV stream. The other way is "out-of-band", where metadata/signaling is done in a parallel channel. Newer shells can try to approximate this by using JSON when the pipeline is appropriate. I kinda wish the latter (OOB) had taken hold back in the day.

93 points by lproven 3y ago

The lessons of yesteryear's OS are getting lost in translation. This is by me on El Reg, and it's part 2 of this year's FOSDEM talk. It will (I hope!) make more sense in context with the rest.

deleted 3y ago
[deleted]
gumby 3y ago
It’s a ramble that ends up with a complaint that Wayland doesn’t fit into the stream file model that’s supposedly critical to “unix”. Block files have been in Unix forever and they don’t either. Maybe read “Worse is Better” before writing part 3. And “Unix” is orthogonal with FLOSS. Sorry to be such a downer — been reading el reg since it started.
- lproven 3y ago
  > Maybe read “Worse is Better” before writing part 3. Way ahead o' ya, kid. https://www.theregister.com/2023/12/25/the_war_of_the_workstations/ https://www.theregister.com/2023/12/25/the_war_of_the_workst... Note that this is loosely based on part of a FOSDEM talk I gave 4 years ago.
cjs_ac 3y ago
Part 3 is going to be about Plan 9 and Redox OS, right?
- lproven 3y ago
  It is indeed going to be about Plan 9. :-) The talk was a couple of weeks ago now and is already online: https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-way-forward-finding-a-path-to-what-comes-after-unix/ https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-... I linked to it in the first paragraph. You can read the script if you want. Redox, OTOH, no: it does not even enter into it. Although I do not address it, if pressed, I might concede that if there is a Rust involvement, it's R9, the successor to Harvey OS. https://github.com/r9os/r9 https://github.com/r9os/r9
torstenvl 3y ago
I read the article, and I didn't see this point made. Mostly it was a very long description of the history of Unix, with a tangent about microkernels. Your headings are What is 'Unix' anyway? Generations and the gaps between Turn it up to 11 I believe your ultimate point is about Wayland and perhaps that it should follow everything-is-a-file. But that comes so late in the article that you barely give it a passing mention, and it isn't consistent with the premise of the article.
- lproven 3y ago
  > I believe your ultimate point is about Wayland Nah. :-) I am not terribly interested in Wayland, myself, and I suspect that in the fullness of time it will turn out to be a not-very-interesting blind alley.
  - torstenvl 3y ago
    Okay. So what is the thesis of this article and where is it discussed?
    - lproven 3y ago
      As it says, and links, in the first paragraph of the story: « This article is the second based on The Reg FOSS desk's talk at FOSDEM 2024 (part 1 was Drowning in code). » That para contains hyperlinks to both. The talk was a couple of weeks ago now and is already online: https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one- https://fosdem.org/2024/schedule/event/fosdem-2024-3095-one-... You can read the script if you want.
      - torstenvl 3y ago
        The existence of more than one article based on a talk does not by itself suggest that they are dependencies of one another; nor does it excuse rambling. The title of the OP is "Forgetting the history of Unix is coding us into a corner." At no point do you explain who is forgetting what lessons, or why such forgetting is coding us into a corner. In any case, the history is interesting and clearly well-researched. I offer as constructive criticism that your structure and clarity could benefit from continued editing.
Diris 3y ago
I saw the talk at FOSDEM and greatly enjoyed it. Someone sitting next to me was busy coding on 9front haha
- lproven 3y ago
  :-D Thanks! I had some great feedback from some 9Front folks and ended up meeting up with a bunch of them after the event. Given that I was half-expecting a lynch mob, this was a very pleasant surprise indeed... ;-)

didntcheck 3y ago

On the contrary, the widespread obsession with "Unix philosophy" often provides quasi-religious objections to suggestions we ever try new approaches to things. In particular, the desire to cram increasingly diverse and rich interfaces into 70s file IO is a strange sacrament

seydar 3y ago
What are some alternatives that you'd recommend people look at? I'm keen for any potential successor, but a lot things come back to effectively text, and anything we could build on top of it. Binary object formats seem like an alternative for faster parsing of structured data, so while that ability has always been it, it's a matter of people actually sticking to it. Maybe some coordination between the program and the shell?
- LeonidasXIV 3y ago
  Some alternatives is a structured data interface, a bit like PowerShell where not everything needs to be serialized and deserialized and parsed in every program in a pipeline. Another approach is that the OS is basically a complete environment where everything is code. You can see the idea in the Smalltalk environments where you can theoretically interact with every object by sending messages. Lisp machines come to mind as well and one could even consider early personal computers that booted into BASIC as an idea of this (though in BASIC instead ov everything is a file everything is memory)
  - traverseda 3y ago
    Generally I think people are happy piping structured data around, the problem is that ties you into a particular data structure. JSON seems to be winning there, and I think people would be very happy if more commands had a JSON environment variable, or even if it was possible to output data on a /dev/stdjson pipe. Anyone know how to add a new stdout-like interface to unix-like OSes?
    - dale_glass 3y ago

bjiuyg7544 3y ago

[dead]

stcroixx 3y ago

It's not just unix, but programming languages themselves and general patterns for how software is built and how projects are run. It's what I'm most frustrated by in this industry. So much waste. I once thought surely this industry would mature, but now I don't see that happening in my lifetime. Reminds me of human civilizations before people developed language and wrote things down and almost all knowledge was lost with each new generation, keeping progress limited to a snails pace.

gregjor 3y ago
You might enjoy the insights in Richard Gabriel's book Patterns of Software.

bradley13 3y ago

The advantage (or disadvantage) of getting older: Seeing the younger generations come up with brilliant "new" ideas. New to them, whereas I think "yep, that was done 40 years ago, and 20 years ago, and now it's new all over again." Part of the problem is the sheer breadth of IT and software development today. 40 years ago, I felt like I understood nearly everything: I designed processors at the transistor level, programmed from machine code through assembly to Lisp, Prolog, Pascal and other languages. Worked on projects from tiny to huge. That's just not possible today. There are too many areas of application, too many frameworks and languages. Too many tools and platforms. Any individual developer is going to be in a niche. Your web developer knows nothing of how to write ABAP, your SAP developer has no clue how to do embedded programming for a microcontroller, your real-time embedded programmer knows nothing of databases, your database admin thinks React is something to do with nuclear power plants. Today, it's all you can do to stay abreast of developments in your own field. There is little time to look for lessons from the past, or across to ideas from other, parallel fields.

cdelsolar 3y ago
As a former electrical engineer now web app/api/full stack developer I either know or can quickly learn all those things.
- voakbasda 3y ago
  Not at sufficient depth to go head-to-head with an expert. If you are a Jack of All Trades, you will be Master of None.
  - andrewmcwatters 3y ago
    I disagree with the Jack of all trades, master of none cliché. You can effectively be a master of multiple fields because the true applicable depth of so many of them are completely shallow. There are entire niches in software development that you can dismiss wholesale. Virtually everything in the web that isn't the web standards themselves or from the most popular libraries isn't worth learning because it's just regurgitation. If you know CSS as an applicable professional level, any CSS framework you see is irrelevant. You can pick it up. Etc.

cladopa 3y ago

I believe the main corner was Unix itself. "Everything is a file" is not a lesson, it is an imposition that with some some things that are very different to files it is a very wrong thing. The lesson is: Minimise as much as possible the different interfaces necessary and reuse them. Interlisp or Lisp machines used Expressions so everything could be read and written by everything. Unix is a mess of binary formats, a mess of different console emulator formats, a mess of different text formats. And the article is just about one guy who does not like Wayland not using files for everything, which is fine for me.

gpderetta 3y ago
> Minimise as much as possible the different interfaces necessary and reuse them. Very very strongly agree. As a corollary a small number of too simple interfaces is better than a larger number of more sophisticate interfaces. "It is better to have 100 functions operate on one data structure than to have 10 functions operate on 10 data structures." - Alan Perlis' Epigrams on Programming (1982).

RcouF1uZ4gsC 3y ago

Actually, I think forgetting the history of mainframes (which were before Unix) is worse. We are now reinventing a lot of concepts with regards to security, isolation, distributed computing. Unix was probably a step backward, yet people keep on testing it is the pinnacle of how to organize and abstract computing.

lproven 3y ago
Strongly agree, and I would point you at my feature column that was published on Xmas day. It's linked twice in these comments already, so I won't add yet another.

HackerThemAll 3y ago

“Those that fail to learn from history are doomed to repeat it.” Winston Churchill, 1948. As can be seen, it equally well applies to history of computing. We keep reinventing the same things only because young coders don't learn from computing history and outright dismiss anything that's more than 1 year old.

gregjor 3y ago
Churchill was paraphrasing George Santayana, and leaving off most of the original. The author of the article we're commenting on included the original quote. The context seems relevant here: Progress, far from consisting in change, depends on retentiveness. When change is absolute there remains no being to improve and no direction is set for possible improvement: and when experience is not retained, as among savages, infancy is perpetual. Those who cannot remember the past are condemned to repeat it. -- George Santayana
- jiveturkey 3y ago
  I dunno. It seems to be captured quite concisely in the Churchill version.

bowsamic 3y ago

I'm really not a fan of UNIX in the slightest, other than as a basic kernel OS to build something nicer on, like what macOS did. I think it's much better to make it easier to write programs and have a rich system library than to glue together text streams. And in fact that's basically what has happened instead: the "UNIX philosophy" has largely been replaced by Python and the like. People now reach for Python for many tasks that were previously shell one-liners, and imo that's a positive development. When I was a kid learning how to code reading online, everyone was religiously defending the "UNIX philosophy" showing off their fancy and indecipherable one-liners (that don't work on my system bc I have GNU awk instead of whatever awk they have or whatever). This has basically died out year by year as so many people switch over to scripting languages with great standard libraries and super-powered easy to install libraries. Now, I'm not really a fan of the super overkill object-oriented approach that systems like mac and smalltalk take, but I think our modern approach is a good compromise

jll29 3y ago
On the topic of "UNIX shell one-liners versus Python", I enjoy letting my students solve the "make a list of the words in a file with their counts" exercise in Python, then I show them a sh one-liner. The Python scripts takes them from 10 minutes to two hours, they're buggy, and they cannot process big files - everything is in RAM (with a monster dict) - without a crash/system freeze. The power of UNIX, pipes and re-use.
- d0mine 3y ago
  Space-separated case: import sys from collections import Counter L = Counter( word for line in sys.stdin for word in line.casefold().split() ) L is the words' multiset (the desired list of words with counts). How many words are there in English? 1M? Huge text [Unicode] files. Fancier tokenizer: for word in re.findall(r"\w+", line.casefold())
tech_ken 3y ago
The one thing holding Python back from completely eating my workflow of utils, glue, and hacks is environment isolation. Nothing sucks more than when activating some venv breaks a background tool I wrote 3 years ago and forgot about. I recognize that this is a surmountable problem with at most moderate effort, but getting to a point of complete frictionlessness would really move the whole ecosystem forward a lot, IMO.

jpgvm 3y ago

The main lesson from unix that is getting lost is composability of systems. So many silos and proprietary "cloud" systems, so much glue that needs to be constantly rewritten because there isn't any stability in an ecosystem composed of competing rather than cooperating actors. There are great things that have come along, I would say k8s has been very successful at introducing a highly interoperable, composable API for abstracting over a bunch of machines. But the trend towards hosted services that are incentivized to create lock-in and data-gravity isn't great.

rjbwork 3y ago
I don't think this lesson is getting lost accidentally, but intentionally. There was a brief moment in the late 00's/early 10's where companies were creating these rich web API's that would let you (and other services acting on your behalf) interact with their system in arbitrary and novel ways. But that turned out to not be very profitable (or at the very least, wouldn't let you create a walled garden that would get 100x exit for your VC's) so we've steadily moved to more locked down, less interactive, less open, less useful, but ultimately more profitable services. Emblematic of this journey is Twitter, I think.

amelius 3y ago

Curious, in what ways would an OS be different from Unix when invented today?

kekebo 3y ago
It may be written in Javascript.
- amelius 3y ago
  Haha, yes. I was thinking more about things like containers, and the concepts around cgroups etc. that I think is currently a mess (from the user's point of view).
fsflover 3y ago
https://qubes-os.org https://qubes-os.org
- lproven 3y ago
  This was indeed one of my inspirations.
traverseda 3y ago
It would probably look a lot like plan9
- lproven 3y ago
  It would, yes. Here's that part: https://www.theregister.com/2024/02/21/successor_to_unix_plan_9/ https://www.theregister.com/2024/02/21/successor_to_unix_pla... But what I am proposing is a way to bring mainstream Unix apps to it, without bloating it with emulation.

javier_e06 3y ago

I started working on the embedded industry in the early 90s when other OS'es with the likes of Greenhills and VxWorks. My first introduction to a Unix system was Caldera Linux in 3 or 4 floppy disks. What call me attention the most compared with other embedded OS was the transparency of the system. We all had our door-stop manuals 10 lbs weight manuals for our embedded OS documentation where a collection of tools like Tornado (In VxWorks) gave us a way to interact with the OS. But Unix, in my understanding, needed no added tool. A kernel, a filesystem a shell... go. Many years a go a developer tried to introduce a patch in Linux to give graphics a leg up because Linux PC Desktop experience lagged behind Windows and Macs. The gate-keepers said no and the guy forked the kernel and went his own merry way. Going back to embedded real-time systems and Unix/Linux efforts like MonteVista and HardHat created smaller leaner footprints that started competing in the embedded field. The OS evolves wells. My lasts embedded projects with Yocto make the point. Unix/Linux, ideas that evolve.

lproven 3y ago
> My first introduction to a Unix system was Caldera Linux in 3 or 4 floppy disks. You know what, although I experimented with Lasermoon Linux/FT on CD in ~1996, and tried and failed to install Slackware 3.0 around then, the first Linux I ran as my daily desktop OS for a while was Caldera OpenLinux 1. Your memory deceives you: it came on a CD. It needed 3 or 4 boot diskettes to get to the point where it could find your CD drive and start the installation process. It had the first graphical installation in the Linux world, and while it wasn't a live environment and had no desktop, it did multitask: you could play Tetris while it installed.
- javier_e06 3y ago
  You are right. The floppy disk was to load the drivers for CD. Cutting edge tech.

assimpleaspossi 3y ago

>Like it or not, this means that Linux isn't "a Unix-like OS" any more. It is a Unix. This needs to be corrected. Two Chinese centOS distros passing the Unix test does not make all of Linux and its distros "Unix".

lproven 3y ago
I disagree. If one distro can pass the test, then in principle, any distro can. If two distros passed, then it is case closed. The entire point of the argument is not "can Linux pass?" but "will anyone pay to put Linux through the test?" And the answer is yes, repeatedly, although not any more, and who can blame them. Unix has meant "passes the compatibility test" for ⅓ of a century. Linux passed. Once a Unix, always a Unix. It's Unix. FWIW the time to protest was a year ago when I wrote the "Unix is dead" story. Here was the brief discussion: https://news.ycombinator.com/item?id=34412022 https://news.ycombinator.com/item?id=34412022
- assimpleaspossi 3y ago
  Linux is only the kernel. To pass the Unix certification: https://www.opengroup.org/membership/forums/platform/unix https://www.opengroup.org/membership/forums/platform/unix >The specification encompasses the base operating system environment, networking services, windowing system services, and internationalization aspects and programming languages.
  - lproven 3y ago
    I still disagree. I am not saying "all distros have passed." They haven't. I am saying "all distros _could_ pass, if anyone cared enough to pay." Do you think you can falsify that?
    - assimpleaspossi 3y ago
      All distros MIGHT pass if they meet the qualifications but just because these two distros passed does NOT mean ALL Linux distros can pass and it doesn't mean any other distros can pass.

moomin 3y ago

Yet another article fetishizing Plan 9 accompanied with a history lesson that doesn't support the conclusion. No, not everything is a file. Having a somewhat standardised way of accessing certain features in a stream-like way is cool, but it's not the fundamental feature of any unix, it's just a cute piece of user-space. And there's better ways to evaluate a design than aesthetic appreciation.

lproven 3y ago
Hang on. You haven't got to the bit about Plan 9 yet. It was only published today. https://www.theregister.com/2024/02/21/successor_to_unix_plan_9/ https://www.theregister.com/2024/02/21/successor_to_unix_pla... HN post (not mine, I was too slow): https://news.ycombinator.com/item?id=39451861 https://news.ycombinator.com/item?id=39451861

loeg 3y ago

I mean, the history of Unix is fun, whatever, but the next step (showing it is "coding us into a corner") is totally missing.

lproven 3y ago
Yes it is. As the saying once went: tune in next time for next week's exciting episode!
- loeg 3y ago
  Seems clickbaity to title part 1 based on unpublished future parts.
  - lproven 3y ago
    Given that a 1000w article is quite long and the first cut of the talk was 6000w, how else would you handle it?

daniel_grady 3y ago

> For instance, macOS is a Unix™. It passes the tests, and Apple pays for the certification. But it hides most of the real Unix directory tree, its /etc is relatively empty, it doesn't have an X server – it's an optional extra. And most of all, it's not case sensitive. What does it mean for the operating system to be “case sensitive”? Certainly APFS is case sensitive, so this must refer to something else?

wtallis 3y ago
OS X / macOS has always defaulted to using filesystems that are case-preserving but not case-sensitive. You can cd to ~/library and see the contents of ~/Library and trying to mkdir ~/library will fail because ~/Library exists.
- daniel_grady 3y ago
  Amazing, I can check off “learn something” early today!

deleted 3y ago

[deleted]

bigbillheck 3y ago

> So any Wayland evangelists out there, tell us: where in the file system can I find the files describing a window on the screen under the Wayland protocol? What file holds the coordinates of the window, its place in the Z-order, its colour depth, its contents? I haven't used X11 in several years, and it's been well over a decade since I've had to put any serious thought into it, but I honestly don't remember being able to do this there.

zokier 3y ago

The random Wayland bashing seems both completely unrelated to whatever the topic of the article is, and uncalled for. Wayland literally is a protocol that is communicated over a file (WAYLAND_DISPLAY). Sure, the file is a (UDS) socket, but it is still a file! Also Wayland is purely an userland thing, so it is very much constrained on what primitives kernel has to offer. The way the Wayland section is included there, it seems like author is heavily implying that it is prime example of "spiralling software complexity and bloat", which to me sounds mostly like they do not know what they are talking about. Core Wayland is distinctly smaller and simpler than X11 ever was. Sure, modern graphics (etc) are intrinsically complex and Wayland exposes some that, but it does it mostly by keeping Wayland itself as simple and transparent as possible, instead of making Wayland (or rather its implementations) hugely complex and bloated. But besides this Wayland thing, I'm really confused what the article is actually trying to say and how does the title relate in any way to the content? The intro implies that there are some lessons to be learned from UNIX history, but the article sure as hell doesn't cast any light on what those lessons might be. There is also this implication that we are somehow coding ourselves into a corner, but that is never actually expanded on what they are meaning with that? In terms of lessons learned, the key observation maybe is that UNIX peaked somewhere around SysIII before BSD etc hippies set their grubby hands on it and messed everything up.

taeric 3y ago

If we are going to go with witty quotes talking about older lessons that people reject, I am rather fond of https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule https://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule. I'm a little worried this encourages ignoring things that have been learned recently. But it is very hard to not be annoyed when you realize how much ground was covered early on.

euroderf 3y ago

When I think of the Unix approach that is based on text stream and the pipes that carry them, I think of telephony. (What a coinky-dink.) The Unix way is "in-band", where metadata (for telephony: signaling) is in the same stream (channel) as the data. It makes reinvention (upgrading) of the shell super difficult. Nobody can even agree precisely how to (for example) include column headers in a CSV stream. The other way is "out-of-band", where metadata/signaling is done in a parallel channel. Newer shells can try to approximate this by using JSON when the pipeline is appropriate. I kinda wish the latter (OOB) had taken hold back in the day.