13 ms·

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of th

Pipes are wonderful! In my opinion you can’t extol them by themselves. One has to bask in a fuller set of features that are so much greater than the sum of their parts, to feel the warmth of Unix: (1) everything is text (2) everything (ish) is a file (3) including pipes and fds (4) every piece of software is accessible as a file, invoked at the command line (5) ...with local arguments (6) ...and persistent globals in the environment A lot of understanding comes once you know what execve does, though such knowledge is of course not necessary. It just helps. Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material.

ianai 6y ago
How’s it uncool? Just not “ mobile”?
- gorgoiler 6y ago
  Exactly. “Linux, ew!” A big part of teaching computer science to children is breaking this obsession with approaching a computer from the top down — the old ICT ways, and the love of apps — and learning that it is a machine under your own control the understanding of which is entirely tractable from the bottom up. Unlike the natural sciences, computer science (like math) is entirely man made, to its advantage. No microscopes, test tubes, rock hammers, or magnets required to investigate its phenomena. Just a keyboard.
  - carlmr 6y ago
    >No microscopes, test tubes, rock hammers, or magnets required to investigate its phenomena. They're full of magnets, and with electron microscope you can analyze what's going on in the hardware.
    - yjftsjthsd-h 6y ago
      Yeah, but it's not required for a lot of investigation.
  - jtbayly 6y ago
    Until stuff goes wrong with the hardware...
  - shrimp_emoji 6y ago
    My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin? multiple users on one computer?? user groups??? entering sudo password all the time????) - The design blunder of assuming that people will read insanely dense man pages (the amount of StackOverflow questions[0][1][2] that exist for anything that can be located in documentation are a testament to this) - And of course no bideo gambes and other things (although hopefully webapps will liberate us from OS dependence soon[3][4]) 0: https://stackoverflow.com/questions/3689838/whats-the-difference-between-head-working-tree-and-index-in-git https://stackoverflow.com/questions/3689838/whats-the-differ... 1: https://stackoverflow.com/questions/3528245/whats-the-difference-between-git-reset-mixed-soft-and-hard https://stackoverflow.com/questions/3528245/whats-the-differ... 2: https://stackoverflow.com/questions/3639342/whats-the-difference-between-git-reset-and-git-checkout https://stackoverflow.com/questions/3639342/whats-the-differ... 3: https://discord.com/ https://discord.com/ 4: http://wasm.continuation-labs.com/d3demo/ http://wasm.continuation-labs.com/d3demo/
    - anthk 6y ago
      - Python as a shell would be the worst crapware ever. Whitespace syntax, no proper file autocompletion, no good pipes, no nothing. Even Tclsh with readline would be better than a Python shell. - By mixing C and C++ you look like a clueless youngster. - /usr/local is to handle non-base/non-packages stuff so you don't trash your system. Under OpenBSD, /usr/local is for packages, everything else should be under /opt or maybe ~/src. - OpenBSD's man pages slap hard any outdated StackOverflow crap. Have fun with a 3 yo unusable answer. Anything NON-docummented on -release- gets wrong fast. Have a look on Linux HOWTO's. That's the future of the StackOverflow usefulness over the years. Near none. Because half of the answers will not apply.
      - blacksqr 6y ago
        > Even Tclsh with readline would be better than a Python shell In fact, I think Tcl would make an excellent sh replacement.
- remexre 6y ago
  My complaints with unix (as someone running linux on every device, starting to dip my toe into freebsd on a vps); apologies for lack of editing: > everything is text I'd often like to send something structured between processes without needing both sides to have to roll their own de/serialization of the domain types; in practice I end up using sockets + some thrown-together HTTP+JSON or TCP+JSON thing instead of pipes for any data that's not CSV-friendly > everything (ish) is a file > including pipes and fds To my mind, this is much less elegant when most of these things don't support seeking, and fnctls have to exist. It'd be nicer if there were something to declare interfaces like Varlink [0, 1], and a shell that allowed composing pipelines out of them nicely. > every piece of software is accessible as a file, invoked at the command line > ...with local arguments > ...and persistent global in the environment Sure, mostly fine; serialization still a wart for arguments + env vars, but one I run into much less > and common signalling facility As in Unix signals? tbh those seem ugly too; sigwinch, sighup, etc ought to be connected to stdin in some way; it'd be nice if there were a more general way to send arbitrary data to processes as an event > also nice if some files have magic properties like /dev/random or /proc or /dev/null userspace programs can't really extend these though, unless they expose FUSE filesystems, which sounds terrible and nobody does also, this results in things like [2]... > every program starts with 3 streams, stdin/stdout for work and stderr for out of band errors iow, things get trickier once I need more than one input one one output. :) I also prefer something like syslog over stderr, but again this is an argument for more structured things. [0]: https://varlink.org/ https://varlink.org/ [1]: ideally with sum type support though, and maybe full dependent types like https://dhall-lang.org/ https://dhall-lang.org/ [2]: https://www.phoronix.com/scan.php?page=news_item&px=UEFI-rm-root-directory https://www.phoronix.com/scan.php?page=news_item&px=UEFI-rm-...
  - blibble 6y ago
    > iow, things get trickier once I need more than one input one one output. :) you can pass as many FDs to a child as you want, 0/1/2 is just a convention
    - remexre 6y ago
      At that point you're writing something for /usr/libexec, not /usr/bin though. (i.e., at that point it becomes sufficiently inconvenient to use from the shell that no user-facing program would do so.)
- ma2rten 6y ago
  I was also skeptical of that statement. I did a google trends search and indeed there seems to be a slow decline in command-line related searches: https://trends.google.com/trends/explore?date=all&q=bash,awk,grep,wget https://trends.google.com/trends/explore?date=all&q=bash,awk...
laumars 6y ago
> everything is text Everything is a byte stream. Usually that means text but sometimes it doesn't. Which means you can do fun stuff like: - copy file systems over a network: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm... - stream a file into gzip - backup or restore an SD card using `cat`
- jrumbut 6y ago
  Also dd, which may be the disk destroyer but is also a great tool for binary file miracles. See one here: https://unix.stackexchange.com/questions/6852/best-way-to-remove-bytes-from-the-start-of-a-file https://unix.stackexchange.com/questions/6852/best-way-to-re...
  - dredmorbius 6y ago
    "data definition", from MVS JCL. https://news.ycombinator.com/user?id=dredmorbius https://news.ycombinator.com/user?id=dredmorbius
    - mjevans 6y ago
      I recall hearing conjecture that cc was already a routine, and that "copy and convert" became dd as a result in a really old system that may have been pre-unix. Researching, it appears that there are multiple attributions, because it's just that obvious and necessary a routine. https://www2.cs.duke.edu/csl/docs/unix_course/intro-98.html https://www2.cs.duke.edu/csl/docs/unix_course/intro-98.html https://unix.stackexchange.com/questions/6804/what-does-dd-stand-for https://unix.stackexchange.com/questions/6804/what-does-dd-s... https://en.wikipedia.org/wiki/Dd_(Unix) https://en.wikipedia.org/wiki/Dd_(Unix)
    - jrumbut 6y ago
      Yeah disk destroyer or data deleter or whatever is the old joke, not it's actual name. For those unaware, dd has often been used for stuff like overwriting a file with random noise as a homespun secure delete function, and it can do a lot of damage with small typos in the command. Could have used a \s or something to clarify.
jcranmer 6y ago
> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them.
- chooseaname 6y ago
  All of which are so much easier to deal with than some binary format you have no spec on. XML? SMH.
- 6gvONxR4sf7o 6y ago
  God help you if your paths have spaces in them.
  - gorgoiler 6y ago
    By “God” you may mean “libc”... ...if it was the 1970s. Nowadays we use Python etc for tools where it’s less acceptable for them to fall apart, though everything falls apart at some point, spaces or otherwise.
    - anthk 6y ago
      That's Perl role, not Python. You are like 20 years late, kid. Have fun with that shitty whitespace syntax. At least Perl can be refactorized.
      - bananabreakfast 6y ago
        Don't you mean Raku? lol
        naniwaduni 6y ago
        Perl 5 was more suitable for these particular use cases than perl 6 ever tried to be.
imglorp 6y ago
More Lego rules: (7) and common signalling facility (8) also nice if some files have magic properties like /dev/random or /proc or /dev/null (9) every program starts with 3 streams, stdin/stdout for work and stderr for out of band errors
- enedil 6y ago
  (9) every program starts with 3 streams, stdin/stdout for work and stderr for out of band errors This is false - if a process closes stdin/out/err, its children won't have these files open.
  - imglorp 6y ago
    That's correct, which is why I said _starts_ with the streams. The child can do what it wants with them after that.
    - DSMan195276 6y ago
      His point was that if you close stdin/stdout/stderr and then fork a child, the new child will not have all of those streams when it starts - the three streams are a convention, not something enforced by the kernel when creating a new process. That said, it's still a bit of a pedantic point, you can expect to always have those three fds in the same way you can expect `argv[0]` to represent the name of your executable (Which also isn't enforced by the kernel).
      - JdeBP 6y ago
        Actually, it's not a pedantic point. It's a security flaw waiting to happen (as it actually has) if one assumes that one always has three open file descriptors. The privileged program opens a sensitive data file for writing, it gets assigned (say) file descriptor 2, because someone in a parent process took some advice to "close your standard I/O because you are a dæmon" to heart, other completely unrelated library code somewhere else blithely logs network-supplied input to standard error without sanitizing it, and (lo!) there's a remotely accessible file overwrite exploit of the sensitive data file. Ironically, the correct thing to do as a dæmon is to not close standard I/O, but to leave setting it up to the service management subsystem, and use it as-is. That way, when the service management subsystem connects standard output and error to a pipe and arranges to log everything coming out the other end, daemontools-style, the idea actually works. (-:
bityard 6y ago
> (1) everything is text Not at all, you can pipe around all the binary you want. Until GNU tar added the 'z' option, the way to extract all files in a tarball was: `gunzip -c < foo.tar.gz | tar x` However, "text files" in Unix do have a very specific definition, if you want them to work with standard Unix text manipulation utilities like awk, sed, and diff: All lines of text end are terminated by a line feed, even the last one. I can't tell you how many "text files" I run across these days that don't have a trailing newline. It's as bad as mixing tabs and spaces, maybe worse.
- deleted 6y ago
  [deleted]
ssivark 6y ago
Serialized bytestreams do compose better than graphical applications. But that is setting a very low bar. For example, allowing passing around dicts/maps/json (and possibly other data structures) would already be a massive improvement. You know what might be even better — passing around objects you could interact with by passing messages (gasp! cue, Alan Kay). While posix and streams are nice (if you squint, files look like a poor man’s version of objects imposed on top of bytestreams), it’s about time we moved on to better things, and not lose sight of the bigger picture.
- fao_ 6y ago
  Yeah, but that makes every single file an object. Then you need plugins to deal with plain text objects, plugins to deal with every single type of object and their versions. Objects aren't human readable and if they get corrupted the object's format is very difficult to recover, you need an intimate knowledge of both the specific format used (of which there might be thousands of variations). Plaintext, however, if that gets corrupted it's human-readable. The data might not be more compact (Although TSV files tend to come out equal), but it's more robust against those kinds of changes. Have you ever examined a protobuf file without knowing what the protobuf structure is? I have, as part of various reverse-engineering I did. It's a nightmare, and without documentation (That's frequently out of date or nonexistent) it's almost impossible to figure out what the data actually is, and you never know if you've got it right. Even having part of the recovered data, I can't figure out what the rest of it is.
  - ssivark 6y ago
    But that seems like just ignoring the problem, like an ostrich shoving one’s head into the sand! > Have you ever examined a protobuf file without knowing what the protobuf structure is? Are you referring to an ASCII serialization or a binary serialization? In a typical scenario, your application needed structured data (hence protobuf), and then you serialized your protobuf into an ASCII bytestream... so how are you ever worse off with a protobuf compared to a text file, provided you make a good choice of serialization scheme? Clearly text files cannot carry enough structure, so we have to build more layers of scaffolding on top? Ultimately files are serialized on to disk as bits, and “ASCII” and “posix” are just “plugins” to be used when reading bitstreams (serialized file objects). Plaintext is not readable if the bits get corrupted or byte boundaries/endianness gets shifted! To have the same amount of of robustness with higher-order structures I imagine all one needs is a well-defined de/serialization protocol which would allow you to load new objects into memory? Am I missing something?
matheusmoreira 6y ago
1. All data is bytes 2. Everything is a file descriptor 3. File descriptors are things that support standard I/O system calls such as read and write
- Koshkin 6y ago
  You are describing Plan 9.
quickthrower2 6y ago
Can you give me any clue as to what execve does? I looked at the man page but none the wiser. Sounds like magic from what I read there. I'm from a Windows backgroud and not used to pipes.
- kccqzy 6y ago
  It replaces the executable in the current process with a different executable. It's kind of like spawning a new process with an executable, except it's the same PID, and file descriptors without CLOEXEC remain open in their original state.
  - uryga 6y ago
    would `become` be a good alternate name for it?
    - yjftsjthsd-h 6y ago
      That's loosely correct.
  - asdff 6y ago
    whats a simple example of using this?
    - gorgoiler 6y ago
      It’s how all new processes in Unix are created. There is no “spawn new process running this code” operation. Instead: (1) you duplicate yourself with fork() (2) the parent process continues about its business (3) the created process calls execve() to become something else This is the mechanism by which all new processes are created, from init (process 1) to anything you run from your shell. Real life examples here, in this example of a Unix shell: https://github.com/mirror/busybox/blob/master/shell/ash.c https://github.com/mirror/busybox/blob/master/shell/ash.c
    - Icathian 6y ago
      Bash is probably the best example of this style of working. You have a shell process that forks and spawns child processes from itself that are turned into whatever program you've called from your terminal.
deleted 6y ago
[deleted]
ryandrake 6y ago
> (2) everything (ish) is a file I'm not all that knowledgable about Unix history, but one thing that has always puzzled me was that for whatever reason network connections (generally) aren't files. While I can do: cat < /dev/ttyS0 to read from a serial device, I've always wondered why I can't do something like: bind /tmp/mysocket 12.34.56.78 80 cat < /tmp/mysocket It is weird that so many things in Unix are somehow twisted into files (/dev/fb0??), but network connections, to my knowledge, never managed to go that way. I know we have netcat but it's not the same.
- moonchild 6y ago
  In the c api, you can read/write to sockets. Wouldn't be difficult to implement something like your example with a named pipe. Additionally, there is /dev/tcp in bash.
- spijdar 6y ago
  The original authors of "UNIX" considered their creation to be fatally flawed, and created a successor OS called "Plan 9", partially to address this specific deficit, among others like the perceived tacked-on nature of GUIs on *nix. That's a whole rabbit hole to go down, but Plan 9 networking is much more like your hypothetical example [0]. Additionally, things like the framebuffer and devices are also exposed as pure files, whereas on Linux and most unix-like systems such devices are just endpoints for making ioctl() calls. [0] http://doc.cat-v.org/plan_9/4th_edition/papers/net/ http://doc.cat-v.org/plan_9/4th_edition/papers/net/
- DSMan195276 6y ago
  I think you're just seeing the abstraction fall apart a bit - for weirder devices, I would say they are "accessible" via a file, but you likely can't interact with it without special tools/syscalls/ioctls. For example, cat'ing your serial connection won't always work, occasionally you'll need to configure the serial device or tty settings first and things can get pretty messy. That's why programs like `minicom` exist. For networking, I don't think there's a particular reason it doesn't exist, but it's worth noting sockets are a little special and require a bit extra care (Which is why they have special syscalls like `shutdown()` and `send()` and `recv()`). If you did pass a TCP socket to `cat` or other programs (Which you can do! just a bit of fork-exec magic), you'd discover it doesn't always work quite right. And while I don't know the history on why such a feature doesn't exist, with the fact that tools like socat or curl do the job pretty well I don't think it's seen as that necessary.
devchix 6y ago
>(1) everything is text >(2) everything (ish) is a file I'm having a moment. I have to get logs out of AWS, the thing is frustrating to no end. There's a stream, it goes somewhere, it can be written to S3, topics and queues, but it's not like I can jack it to a huge file or many little files and run text processing filters on it. Am I stupid, behind the times, tied to a dead model, just don't get it? There's no "landing" anywhere that I can examine things directly. I miss the accessibility of basic units which I can examine and do things to with simple tools.
- macintux 6y ago
  I worked around that by writing a Python script to let me pick the groups of interest, download all of the data for a certain time period, and collapse them into a CSV file ordered by time stamp. Yes, it’s very annoying I have to do all that, but I commend Bezos for his foresight in demanding everything be driven by APIs.
dvirsky 6y ago
> Unix is seriously uncool with young people at the moment. Those damn kids with their loud rockn'roll music and their Windows machines. Back in my day we had he vocal stylings of Dean Martin and the verbal stylings of Linus Torvalds let me tell ya. Seriously though, I'm actually seeing younger engineers really taking the time to learn how to do shell magic, using vim, etc. It's like the generation of programmers who started up until the late 90s used those by default, people who like me started 15-20 years ago grew up on IDEs and GUIs, but I've seen a lot of 20-30 something devs who are really into the Unix way of doing things. The really cool kids are in fact doing it.
- abnry 6y ago
  I guess I am one of the young kids who think the unix command line is wicked cool. It makes the user experience on my laptop feel so much more powerful.
  - logicprog 6y ago
    Me too. Especially since I can regularly hit 100 WPM when typing, but I'm a terrible shot with the mouse - not to mention the fact that you can get really nice and satisfying keyboards (I use an IBM Model M at home and a CODE Cherry MX Clear at college) but mice are all kind of the same, and you have to move your hand a lot to get there. On that last point, my mouse hand gets wrist pain consistently way more than my other hand (which I only use for the keyboard) does. Add to all that the fact that the command line allows for a lot of really easy composition and automation, and it's significantly better for me. I can hardly function on a Windows computer!
    - divbzero 6y ago
      Ditto. For projects where I must work in Windows my secret weapon is WSL + tmux + fish + vim for everything without a strict Visual Studio dependency.
      - dvirsky 6y ago
jiggawatts 6y ago
> Pipes are wonderful! It's wonderful only if compared to worse things, pretending that PowerShell is not a thing, and that Python doesn't exist. UNIX pipes are a stringly-typed legacy that we've inherited from the 1970s. The technical constraints of its past have been internalised by its proponents, and lauded as benefits. To put it most succinctly, the "byte stream" nature of UNIX pipes means that any command that does anything high-level such as processing "structured" data must have a parsing step on its input, and then a serialization step after its internal processing. The myth of UNIX pipes is that it works like this: process1 | process2 | process3 | ... The reality is that physically, it actually works like this: (process1,serialize) | (parse,process2,serialize) | ... Where each one of those "parse" and "serialise" steps is unique and special, inflexible, and poorly documented. This is forced by the use of byte streams to connect each step. It cannot be circumvented! It's an inherent limitation of UNIX style pipes. This is not a limitation of PowerShell, which is object oriented, and passes strongly typed, structured objects between processing steps. It makes it much more elegant, flexible, and in effect "more UNIX than UNIX". If you want to see this in action, check out my little "challenge" that I posed in a similar thread on YC recently: https://news.ycombinator.com/item?id=23257776 https://news.ycombinator.com/item?id=23257776 The solution provided by "JoshuaDavid" really impressed me, because I was under the impression that that simple task is actually borderline impossible with UNIX pipes and GNU tools: https://news.ycombinator.com/item?id=23267901 https://news.ycombinator.com/item?id=23267901 Compare that to the much simpler equivalent in PowerShell: https://news.ycombinator.com/item?id=23270291 https://news.ycombinator.com/item?id=23270291 Especially take note half of that script is sample data! > Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material. UNIX is entirely too cool with young people. They have conflated the legacy UNIX design of the 1970s with their preferred open source software, languages, and platforms. There are better things out there, and UNIX isn't the be all and end all of system management.
- matheusmoreira 6y ago
  > Where each one of those "parse" and "serialise" steps is unique and special, inflexible, and poorly documented. This can also be a security concern. According to research, a great number of defects with security implications occur at the input handling layers: http://langsec.org http://langsec.org
Shared404 6y ago
>Unix is seriously uncool with young people... Not all of us. I much prefer the Unix way of doing things, it just makes more sense then just trying to become another Windows.
pjmlp 6y ago
Everything is a file until one needs to do UNIX IPC, networking or high performance graphics rendering.
lsofzz 6y ago
> Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material. I offer you my well wishes on that.
hhas01 6y ago
“(1) everything is text” LOL. What fantasy land are you living in? Pipes are great. Untyped, untagged pipes are not. They’re a frigging disaster in every way imaginable. “Unix is seriously uncool with young people at the moment.” Unix is half a century old. It’s old enough to be your Dad. Hell, it’s old enough to be your Grandad! It’s had 50 years to better itself and it hasn’t done squat. Linux? Please. That’s just a crazy cat lady living in Unix’s trash house. Come back when you’ve a modern successor to Plan 9 that is easy, secure, and doesn’t blow huge UI/UX chunks everywhere.
xkucf03 6y ago
Unix is seriously cool and I consider myself still young :-) But it could be better. See https://relational-pipes.globalcode.info/ https://relational-pipes.globalcode.info/

ianai 6y ago

How’s it uncool? Just not “ mobile”?

gorgoiler 6y ago
Exactly. “Linux, ew!” A big part of teaching computer science to children is breaking this obsession with approaching a computer from the top down — the old ICT ways, and the love of apps — and learning that it is a machine under your own control the understanding of which is entirely tractable from the bottom up. Unlike the natural sciences, computer science (like math) is entirely man made, to its advantage. No microscopes, test tubes, rock hammers, or magnets required to investigate its phenomena. Just a keyboard.
- carlmr 6y ago
  >No microscopes, test tubes, rock hammers, or magnets required to investigate its phenomena. They're full of magnets, and with electron microscope you can analyze what's going on in the hardware.
  - yjftsjthsd-h 6y ago
    Yeah, but it's not required for a lot of investigation.
- jtbayly 6y ago
  Until stuff goes wrong with the hardware...
- shrimp_emoji 6y ago
  My sense is that people do think Linux is "cool" (it's certainly distinguished in being free, logical, and powerful), but its age definitely shows. My biggest pain points are: - Bash is awful (should be replaced with Python, and there are so many Python-based shells now that reify that opinion) - C/C++ are awful - Various crusty bits of Linux that haven't aged well besides the above items (/usr/bin AND /usr/local/bin? multiple users on one computer?? user groups??? entering sudo password all the time????) - The design blunder of assuming that people will read insanely dense man pages (the amount of StackOverflow questions[0][1][2] that exist for anything that can be located in documentation are a testament to this) - And of course no bideo gambes and other things (although hopefully webapps will liberate us from OS dependence soon[3][4]) 0: https://stackoverflow.com/questions/3689838/whats-the-difference-between-head-working-tree-and-index-in-git https://stackoverflow.com/questions/3689838/whats-the-differ... 1: https://stackoverflow.com/questions/3528245/whats-the-difference-between-git-reset-mixed-soft-and-hard https://stackoverflow.com/questions/3528245/whats-the-differ... 2: https://stackoverflow.com/questions/3639342/whats-the-difference-between-git-reset-and-git-checkout https://stackoverflow.com/questions/3639342/whats-the-differ... 3: https://discord.com/ https://discord.com/ 4: http://wasm.continuation-labs.com/d3demo/ http://wasm.continuation-labs.com/d3demo/
  - anthk 6y ago
    - Python as a shell would be the worst crapware ever. Whitespace syntax, no proper file autocompletion, no good pipes, no nothing. Even Tclsh with readline would be better than a Python shell. - By mixing C and C++ you look like a clueless youngster. - /usr/local is to handle non-base/non-packages stuff so you don't trash your system. Under OpenBSD, /usr/local is for packages, everything else should be under /opt or maybe ~/src. - OpenBSD's man pages slap hard any outdated StackOverflow crap. Have fun with a 3 yo unusable answer. Anything NON-docummented on -release- gets wrong fast. Have a look on Linux HOWTO's. That's the future of the StackOverflow usefulness over the years. Near none. Because half of the answers will not apply.
    - blacksqr 6y ago
      > Even Tclsh with readline would be better than a Python shell In fact, I think Tcl would make an excellent sh replacement.
remexre 6y ago
My complaints with unix (as someone running linux on every device, starting to dip my toe into freebsd on a vps); apologies for lack of editing: > everything is text I'd often like to send something structured between processes without needing both sides to have to roll their own de/serialization of the domain types; in practice I end up using sockets + some thrown-together HTTP+JSON or TCP+JSON thing instead of pipes for any data that's not CSV-friendly > everything (ish) is a file > including pipes and fds To my mind, this is much less elegant when most of these things don't support seeking, and fnctls have to exist. It'd be nicer if there were something to declare interfaces like Varlink [0, 1], and a shell that allowed composing pipelines out of them nicely. > every piece of software is accessible as a file, invoked at the command line > ...with local arguments > ...and persistent global in the environment Sure, mostly fine; serialization still a wart for arguments + env vars, but one I run into much less > and common signalling facility As in Unix signals? tbh those seem ugly too; sigwinch, sighup, etc ought to be connected to stdin in some way; it'd be nice if there were a more general way to send arbitrary data to processes as an event > also nice if some files have magic properties like /dev/random or /proc or /dev/null userspace programs can't really extend these though, unless they expose FUSE filesystems, which sounds terrible and nobody does also, this results in things like [2]... > every program starts with 3 streams, stdin/stdout for work and stderr for out of band errors iow, things get trickier once I need more than one input one one output. :) I also prefer something like syslog over stderr, but again this is an argument for more structured things. [0]: https://varlink.org/ https://varlink.org/ [1]: ideally with sum type support though, and maybe full dependent types like https://dhall-lang.org/ https://dhall-lang.org/ [2]: https://www.phoronix.com/scan.php?page=news_item&px=UEFI-rm-root-directory https://www.phoronix.com/scan.php?page=news_item&px=UEFI-rm-...
- blibble 6y ago
  > iow, things get trickier once I need more than one input one one output. :) you can pass as many FDs to a child as you want, 0/1/2 is just a convention
  - remexre 6y ago
    At that point you're writing something for /usr/libexec, not /usr/bin though. (i.e., at that point it becomes sufficiently inconvenient to use from the shell that no user-facing program would do so.)
ma2rten 6y ago
I was also skeptical of that statement. I did a google trends search and indeed there seems to be a slow decline in command-line related searches: https://trends.google.com/trends/explore?date=all&q=bash,awk,grep,wget https://trends.google.com/trends/explore?date=all&q=bash,awk...

laumars 6y ago

> everything is text Everything is a byte stream. Usually that means text but sometimes it doesn't. Which means you can do fun stuff like: - copy file systems over a network: https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.htm... - stream a file into gzip - backup or restore an SD card using `cat`

jrumbut 6y ago
Also dd, which may be the disk destroyer but is also a great tool for binary file miracles. See one here: https://unix.stackexchange.com/questions/6852/best-way-to-remove-bytes-from-the-start-of-a-file https://unix.stackexchange.com/questions/6852/best-way-to-re...
- dredmorbius 6y ago
  "data definition", from MVS JCL. https://news.ycombinator.com/user?id=dredmorbius https://news.ycombinator.com/user?id=dredmorbius
  - mjevans 6y ago
    I recall hearing conjecture that cc was already a routine, and that "copy and convert" became dd as a result in a really old system that may have been pre-unix. Researching, it appears that there are multiple attributions, because it's just that obvious and necessary a routine. https://www2.cs.duke.edu/csl/docs/unix_course/intro-98.html https://www2.cs.duke.edu/csl/docs/unix_course/intro-98.html https://unix.stackexchange.com/questions/6804/what-does-dd-stand-for https://unix.stackexchange.com/questions/6804/what-does-dd-s... https://en.wikipedia.org/wiki/Dd_(Unix) https://en.wikipedia.org/wiki/Dd_(Unix)
  - jrumbut 6y ago
    Yeah disk destroyer or data deleter or whatever is the old joke, not it's actual name. For those unaware, dd has often been used for stuff like overwriting a file with random noise as a homespun secure delete function, and it can do a lot of damage with small typos in the command. Could have used a \s or something to clarify.

jcranmer 6y ago

> (1) everything is text And lists are space-separated. Unless you want them to be newline-separated, or NUL-separated, which is controlled by an option that may or may not be present for the command you're invoking, and is spelled completely differently for each program. Or maybe you just quote spaces somehow, and good luck figuring out who is responsible for inserting quotes and who is responsible for removing them.

chooseaname 6y ago
All of which are so much easier to deal with than some binary format you have no spec on. XML? SMH.
6gvONxR4sf7o 6y ago
God help you if your paths have spaces in them.
- gorgoiler 6y ago
  By “God” you may mean “libc”... ...if it was the 1970s. Nowadays we use Python etc for tools where it’s less acceptable for them to fall apart, though everything falls apart at some point, spaces or otherwise.
  - anthk 6y ago
    That's Perl role, not Python. You are like 20 years late, kid. Have fun with that shitty whitespace syntax. At least Perl can be refactorized.
    - bananabreakfast 6y ago
      Don't you mean Raku? lol
      - naniwaduni 6y ago
        Perl 5 was more suitable for these particular use cases than perl 6 ever tried to be.

imglorp 6y ago

More Lego rules: (7) and common signalling facility (8) also nice if some files have magic properties like /dev/random or /proc or /dev/null (9) every program starts with 3 streams, stdin/stdout for work and stderr for out of band errors

enedil 6y ago
(9) every program starts with 3 streams, stdin/stdout for work and stderr for out of band errors This is false - if a process closes stdin/out/err, its children won't have these files open.
- imglorp 6y ago
  That's correct, which is why I said _starts_ with the streams. The child can do what it wants with them after that.
  - DSMan195276 6y ago
    His point was that if you close stdin/stdout/stderr and then fork a child, the new child will not have all of those streams when it starts - the three streams are a convention, not something enforced by the kernel when creating a new process. That said, it's still a bit of a pedantic point, you can expect to always have those three fds in the same way you can expect `argv[0]` to represent the name of your executable (Which also isn't enforced by the kernel).
    - JdeBP 6y ago
      Actually, it's not a pedantic point. It's a security flaw waiting to happen (as it actually has) if one assumes that one always has three open file descriptors. The privileged program opens a sensitive data file for writing, it gets assigned (say) file descriptor 2, because someone in a parent process took some advice to "close your standard I/O because you are a dæmon" to heart, other completely unrelated library code somewhere else blithely logs network-supplied input to standard error without sanitizing it, and (lo!) there's a remotely accessible file overwrite exploit of the sensitive data file. Ironically, the correct thing to do as a dæmon is to not close standard I/O, but to leave setting it up to the service management subsystem, and use it as-is. That way, when the service management subsystem connects standard output and error to a pipe and arranges to log everything coming out the other end, daemontools-style, the idea actually works. (-:

bityard 6y ago

> (1) everything is text Not at all, you can pipe around all the binary you want. Until GNU tar added the 'z' option, the way to extract all files in a tarball was: `gunzip -c < foo.tar.gz | tar x` However, "text files" in Unix do have a very specific definition, if you want them to work with standard Unix text manipulation utilities like awk, sed, and diff: All lines of text end are terminated by a line feed, even the last one. I can't tell you how many "text files" I run across these days that don't have a trailing newline. It's as bad as mixing tabs and spaces, maybe worse.

deleted 6y ago
[deleted]

ssivark 6y ago

Serialized bytestreams do compose better than graphical applications. But that is setting a very low bar. For example, allowing passing around dicts/maps/json (and possibly other data structures) would already be a massive improvement. You know what might be even better — passing around objects you could interact with by passing messages (gasp! cue, Alan Kay). While posix and streams are nice (if you squint, files look like a poor man’s version of objects imposed on top of bytestreams), it’s about time we moved on to better things, and not lose sight of the bigger picture.

fao_ 6y ago
Yeah, but that makes every single file an object. Then you need plugins to deal with plain text objects, plugins to deal with every single type of object and their versions. Objects aren't human readable and if they get corrupted the object's format is very difficult to recover, you need an intimate knowledge of both the specific format used (of which there might be thousands of variations). Plaintext, however, if that gets corrupted it's human-readable. The data might not be more compact (Although TSV files tend to come out equal), but it's more robust against those kinds of changes. Have you ever examined a protobuf file without knowing what the protobuf structure is? I have, as part of various reverse-engineering I did. It's a nightmare, and without documentation (That's frequently out of date or nonexistent) it's almost impossible to figure out what the data actually is, and you never know if you've got it right. Even having part of the recovered data, I can't figure out what the rest of it is.
- ssivark 6y ago
  But that seems like just ignoring the problem, like an ostrich shoving one’s head into the sand! > Have you ever examined a protobuf file without knowing what the protobuf structure is? Are you referring to an ASCII serialization or a binary serialization? In a typical scenario, your application needed structured data (hence protobuf), and then you serialized your protobuf into an ASCII bytestream... so how are you ever worse off with a protobuf compared to a text file, provided you make a good choice of serialization scheme? Clearly text files cannot carry enough structure, so we have to build more layers of scaffolding on top? Ultimately files are serialized on to disk as bits, and “ASCII” and “posix” are just “plugins” to be used when reading bitstreams (serialized file objects). Plaintext is not readable if the bits get corrupted or byte boundaries/endianness gets shifted! To have the same amount of of robustness with higher-order structures I imagine all one needs is a well-defined de/serialization protocol which would allow you to load new objects into memory? Am I missing something?

matheusmoreira 6y ago

1. All data is bytes 2. Everything is a file descriptor 3. File descriptors are things that support standard I/O system calls such as read and write

Koshkin 6y ago
You are describing Plan 9.

quickthrower2 6y ago

Can you give me any clue as to what execve does? I looked at the man page but none the wiser. Sounds like magic from what I read there. I'm from a Windows backgroud and not used to pipes.

kccqzy 6y ago
It replaces the executable in the current process with a different executable. It's kind of like spawning a new process with an executable, except it's the same PID, and file descriptors without CLOEXEC remain open in their original state.
- uryga 6y ago
  would `become` be a good alternate name for it?
  - yjftsjthsd-h 6y ago
    That's loosely correct.
- asdff 6y ago
  whats a simple example of using this?
  - gorgoiler 6y ago
    It’s how all new processes in Unix are created. There is no “spawn new process running this code” operation. Instead: (1) you duplicate yourself with fork() (2) the parent process continues about its business (3) the created process calls execve() to become something else This is the mechanism by which all new processes are created, from init (process 1) to anything you run from your shell. Real life examples here, in this example of a Unix shell: https://github.com/mirror/busybox/blob/master/shell/ash.c https://github.com/mirror/busybox/blob/master/shell/ash.c
  - Icathian 6y ago
    Bash is probably the best example of this style of working. You have a shell process that forks and spawns child processes from itself that are turned into whatever program you've called from your terminal.

deleted 6y ago

[deleted]

ryandrake 6y ago

> (2) everything (ish) is a file I'm not all that knowledgable about Unix history, but one thing that has always puzzled me was that for whatever reason network connections (generally) aren't files. While I can do: cat < /dev/ttyS0 to read from a serial device, I've always wondered why I can't do something like: bind /tmp/mysocket 12.34.56.78 80 cat < /tmp/mysocket It is weird that so many things in Unix are somehow twisted into files (/dev/fb0??), but network connections, to my knowledge, never managed to go that way. I know we have netcat but it's not the same.

moonchild 6y ago
In the c api, you can read/write to sockets. Wouldn't be difficult to implement something like your example with a named pipe. Additionally, there is /dev/tcp in bash.
spijdar 6y ago
The original authors of "UNIX" considered their creation to be fatally flawed, and created a successor OS called "Plan 9", partially to address this specific deficit, among others like the perceived tacked-on nature of GUIs on *nix. That's a whole rabbit hole to go down, but Plan 9 networking is much more like your hypothetical example [0]. Additionally, things like the framebuffer and devices are also exposed as pure files, whereas on Linux and most unix-like systems such devices are just endpoints for making ioctl() calls. [0] http://doc.cat-v.org/plan_9/4th_edition/papers/net/ http://doc.cat-v.org/plan_9/4th_edition/papers/net/
DSMan195276 6y ago
I think you're just seeing the abstraction fall apart a bit - for weirder devices, I would say they are "accessible" via a file, but you likely can't interact with it without special tools/syscalls/ioctls. For example, cat'ing your serial connection won't always work, occasionally you'll need to configure the serial device or tty settings first and things can get pretty messy. That's why programs like `minicom` exist. For networking, I don't think there's a particular reason it doesn't exist, but it's worth noting sockets are a little special and require a bit extra care (Which is why they have special syscalls like `shutdown()` and `send()` and `recv()`). If you did pass a TCP socket to `cat` or other programs (Which you can do! just a bit of fork-exec magic), you'd discover it doesn't always work quite right. And while I don't know the history on why such a feature doesn't exist, with the fact that tools like socat or curl do the job pretty well I don't think it's seen as that necessary.

devchix 6y ago

>(1) everything is text >(2) everything (ish) is a file I'm having a moment. I have to get logs out of AWS, the thing is frustrating to no end. There's a stream, it goes somewhere, it can be written to S3, topics and queues, but it's not like I can jack it to a huge file or many little files and run text processing filters on it. Am I stupid, behind the times, tied to a dead model, just don't get it? There's no "landing" anywhere that I can examine things directly. I miss the accessibility of basic units which I can examine and do things to with simple tools.

macintux 6y ago
I worked around that by writing a Python script to let me pick the groups of interest, download all of the data for a certain time period, and collapse them into a CSV file ordered by time stamp. Yes, it’s very annoying I have to do all that, but I commend Bezos for his foresight in demanding everything be driven by APIs.

dvirsky 6y ago

> Unix is seriously uncool with young people at the moment. Those damn kids with their loud rockn'roll music and their Windows machines. Back in my day we had he vocal stylings of Dean Martin and the verbal stylings of Linus Torvalds let me tell ya. Seriously though, I'm actually seeing younger engineers really taking the time to learn how to do shell magic, using vim, etc. It's like the generation of programmers who started up until the late 90s used those by default, people who like me started 15-20 years ago grew up on IDEs and GUIs, but I've seen a lot of 20-30 something devs who are really into the Unix way of doing things. The really cool kids are in fact doing it.

abnry 6y ago
I guess I am one of the young kids who think the unix command line is wicked cool. It makes the user experience on my laptop feel so much more powerful.
- logicprog 6y ago
  Me too. Especially since I can regularly hit 100 WPM when typing, but I'm a terrible shot with the mouse - not to mention the fact that you can get really nice and satisfying keyboards (I use an IBM Model M at home and a CODE Cherry MX Clear at college) but mice are all kind of the same, and you have to move your hand a lot to get there. On that last point, my mouse hand gets wrist pain consistently way more than my other hand (which I only use for the keyboard) does. Add to all that the fact that the command line allows for a lot of really easy composition and automation, and it's significantly better for me. I can hardly function on a Windows computer!
  - divbzero 6y ago
    Ditto. For projects where I must work in Windows my secret weapon is WSL + tmux + fish + vim for everything without a strict Visual Studio dependency.
    - dvirsky 6y ago

jiggawatts 6y ago

> Pipes are wonderful! It's wonderful only if compared to worse things, pretending that PowerShell is not a thing, and that Python doesn't exist. UNIX pipes are a stringly-typed legacy that we've inherited from the 1970s. The technical constraints of its past have been internalised by its proponents, and lauded as benefits. To put it most succinctly, the "byte stream" nature of UNIX pipes means that any command that does anything high-level such as processing "structured" data must have a parsing step on its input, and then a serialization step after its internal processing. The myth of UNIX pipes is that it works like this: process1 | process2 | process3 | ... The reality is that physically, it actually works like this: (process1,serialize) | (parse,process2,serialize) | ... Where each one of those "parse" and "serialise" steps is unique and special, inflexible, and poorly documented. This is forced by the use of byte streams to connect each step. It cannot be circumvented! It's an inherent limitation of UNIX style pipes. This is not a limitation of PowerShell, which is object oriented, and passes strongly typed, structured objects between processing steps. It makes it much more elegant, flexible, and in effect "more UNIX than UNIX". If you want to see this in action, check out my little "challenge" that I posed in a similar thread on YC recently: https://news.ycombinator.com/item?id=23257776 https://news.ycombinator.com/item?id=23257776 The solution provided by "JoshuaDavid" really impressed me, because I was under the impression that that simple task is actually borderline impossible with UNIX pipes and GNU tools: https://news.ycombinator.com/item?id=23267901 https://news.ycombinator.com/item?id=23267901 Compare that to the much simpler equivalent in PowerShell: https://news.ycombinator.com/item?id=23270291 https://news.ycombinator.com/item?id=23270291 Especially take note half of that script is sample data! > Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material. UNIX is entirely too cool with young people. They have conflated the legacy UNIX design of the 1970s with their preferred open source software, languages, and platforms. There are better things out there, and UNIX isn't the be all and end all of system management.

matheusmoreira 6y ago
> Where each one of those "parse" and "serialise" steps is unique and special, inflexible, and poorly documented. This can also be a security concern. According to research, a great number of defects with security implications occur at the input handling layers: http://langsec.org http://langsec.org

Shared404 6y ago

>Unix is seriously uncool with young people... Not all of us. I much prefer the Unix way of doing things, it just makes more sense then just trying to become another Windows.

pjmlp 6y ago

Everything is a file until one needs to do UNIX IPC, networking or high performance graphics rendering.

lsofzz 6y ago

> Unix is seriously uncool with young people at the moment. I intend to turn that around and articles like this offer good material. I offer you my well wishes on that.

hhas01 6y ago

“(1) everything is text” LOL. What fantasy land are you living in? Pipes are great. Untyped, untagged pipes are not. They’re a frigging disaster in every way imaginable. “Unix is seriously uncool with young people at the moment.” Unix is half a century old. It’s old enough to be your Dad. Hell, it’s old enough to be your Grandad! It’s had 50 years to better itself and it hasn’t done squat. Linux? Please. That’s just a crazy cat lady living in Unix’s trash house. Come back when you’ve a modern successor to Plan 9 that is easy, secure, and doesn’t blow huge UI/UX chunks everywhere.

xkucf03 6y ago

Unix is seriously cool and I consider myself still young :-) But it could be better. See https://relational-pipes.globalcode.info/ https://relational-pipes.globalcode.info/