12 ms·

To play devil's advocate, part of the reason things like The Rule of Silence are talked about is because of the messy unix philosophy of treating everything lik

by borplk 10y ago

To play devil's advocate, part of the reason things like The Rule of Silence are talked about is because of the messy unix philosophy of treating everything like plain text. If structured data was embraced we would have developed appropriate tooling to interact with it in the way that we prefer. This runs very deep in unix and a lot of people are too "brainwashed" to think of other ways. Instead they develop other exotic ways of dealing with the problem. Oh you don't like that output? Easy! pipe that crap into sed then awk then perl then cut then wc and you're golden! When you get tot that point you have to understand that you have chosen to ignore the fact that the data you are dealing with must be represented in something much closer to a relational database than lines of ASCII text. Logging is another area you see the consequences of this. A log is not a line of text. Repeat after me a log entry is not a damn line of text. "Oh but isn't it neat you can pipe it to grep?" NO! No it's not neat, maybe it was neat 20 years ago. Today I want that damn data in a structure. Then you can still print it out in one line and pipe it to grep all you want. Another area that you see the unfortunate side effects of this philosophy is with the mess of file-based software configuration. Yes I get it, you like your SSH session and Emacs/Vim blah blah but that's short-sighted. I want my software configuration stored in a database not in a bunch of fragile files with made up syntax that are always one typo or syntax error away from being potentially silently ignored. The fetish for easily-editable ASCII files and escaping from structure is holding us back. Structured data does not automatically imply hidden and inaccessible, that's a matter of developing appropriate tooling.

skrebbel 10y ago
In 1993, someone made exactly your argument a in Redmond board room, and so many people agreed that what you describe could be adequately called "The Windows Philosophy". All settings in a database and not text files (the registry); a command-line that pipes data, not text (PowerShell). Tailored UIs to change settings, not magic invocations and obsure text file syntaxes. I guess most developers on HN are also aware of the downsides of this philosophy. If not, try configuring IIS.
- rtpg 10y ago
  Surely you can reconcile structured representations and something like the Unix command line. Imagine if the default wasn't bash, but something like Ruby + pipes (or some other terse language). What is the argument for shell scripts not working on typed objects? How much time has been lost, how many bugs have been created because every single interaction between shell scripts has to include its own parser. How many versions of " get file created timestamp from ls" do we need? Something Windows does get right is the clipboard. You place a thing on the clipboard, and when you paste, the receiving program can decide on the best representation. This is why copy-pasting images has worked so magically. I could see an alternative system where such a mechanism exists for shell programs.
  - cyphar 10y ago
    > What is the argument for shell scripts not working on typed objects? How much time has been lost, how many bugs have been created because every single interaction between shell scripts has to include its own parser. How many versions of " get file created timestamp from ls" do we need? Aside: that's what the stat command is for. My big concern with types is how would you make sure that the output of a command will always have the right types? Otherwise you'll have runtime type errors which would be just as bad as runtime parsing errors.
    - junke 10y ago
      > Otherwise you'll have runtime type errors which would be just as bad as runtime parsing errors. Parsing errors in script shells can easily go unnoticed (until you realize your data is corrupted).
  - chei0iaV 10y ago
    The problem is that now simply providing user-friendly output is not enough. For every program or script you throw together you need to provide the text output for the user, and then the type object stream for piping. And then, the user would need to read documentation to see how to access each piece of data, what data type it is, what data type the other command takes, and maybe consider how to convert one to the other. ... at which point you basically have a scripting language, so you could just as well use an existing one (e.g. Ruby).
pwdisswordfish 10y ago
The text-oriented nature of Unix may be a mistake, but the Rule of Silence is not concerned with it. It is about not overwhelming the user with information they are not interested in. GUIs can follow or violate the Rule of Silence just as well.
- SixSigma 10y ago
  Your comment was saved. Click OK to continue.
  - kulu2002 10y ago
    Precisely!
- restalis 10y ago
  "GUIs can follow or violate the Rule of Silence just as well." It would have been nice if this was permitted to be violated only by GUIs. Ask about the first-time *nix experiences before the GUI-embracing era and one of the few things they were noticing was the continuous text-spitting. For instance, that was happening on boot and OS loading sequence (and still happens, now only being hidden by default with splash-screens), a lot of reporting about all the things that were performed successfully. It's funny in this regard seeing Unix' Rule of Silence being respected more... outside Unix, where is just common sense, with no need to be formulated as rule.
  - pwdisswordfish 10y ago
    Booting up is a bit of a special case that I'd actually be willing to carve out an exception for. When the boot-up goes wrong, it's often due to causes so deep that they leave you with very few means of diagnosing it or showing an error message. If the booting process hangs, having a rough idea at what stage it happened can be very helpful. Without these messages, discovering the cause of failure would be much harder. The boot-up splash screen hardly even counts as "GUI" anyway.
- StRoy 10y ago
  Talking about the rule of silence in GUIs, I wish I could slap the eurocrats who decided to force websites to show the cookie warning on a first visit of all websites. Did they even understand the consequence and wasted time of what they were doing? Having to click all the time to get this dumb warning off? Of course some websites had to do it in an even dumber way than the law asks for. Like slashdot: http://i.imgur.com/5Fp0nmo.png http://i.imgur.com/5Fp0nmo.png This is what greets the French every time slashdot decides to forget you agreed to let them put cookies on your computer and you need to click continue before you can get to the actual website. The law actually made it worse for the people it's supposed to protect (those who might refuse cookies for privacy?) because those warning then will stick around like glue if they can't give you a cookie to remember you accepted their existence.
verinus 10y ago
I'm with you when it comes to structured data, but plz no more data bases. these config files do not need to be centralized. I am thinking more in the direction of a parser that could check the validity of a configuration...
- riskneural 10y ago
  So, horribly unreadable xml config files?
  - nailer 10y ago
    CSON or .desktop / .service or something similar is immediately understandable to most people and doesn't waste time with unnecessary tokens like XML does.
  - TeMPOraL 10y ago
    Like S-expressions don't exist. Or even goddamn JSON. Come on, we don't have to jump from one stupidity (unstructured text) to another (using XML as a data representation format).
  - stevedonovan 10y ago
    Not necessarily. JSON is not bad, _if_ you allow comments. Even plain-jane key/value config files can be sanity-checked. I suspect part of the problem is that anything fancy like that is awkward to do in C, so people take the lazy way out.
- karl42 10y ago
  Maybe many tiny sqlite databases? Then you would not need to to centralize your data in a single database, but still do queries across different config files.
  - TeMPOraL 10y ago
    SQLite is a huge win. All the power of SQL, and you don't have to introduce system-wide and non-local dependencies.
Ianvdl 10y ago
> To play devil's advocate, part of the reason things like The Rule of Silence are talked about is because of the messy unix philosophy of treating everything like plain text. The amount of tooling that surrounds text is vast and has evolved over decades. You cannot replace that with a single database and call it better. I can place the majority of my config files in git and version them. I can easily perform a full text search on any log regardless of its syntax. I can effortlessly relocate and distribute logs to different filesystems. > I want my software configuration stored in a database not in a bunch of fragile files with made up syntax that are always one typo or syntax error away from being potentially silently ignored. So you would like your configuration stored as a bunch of made up data structures? Databases do not make you immune from typos and syntax errors, ask anyone who has ever written a delete without a where clause. And what happens when the giant, all-knowing database your system depends on has a bug or vulnerability? When something on my linux box breaks I can drop to a recovery mode or lower runlevel and fix it with a kernel and a shell as dependencies. I think you would be a lot happier with a configuration management system (puppet, ansible et al) and some log infrastructure without having to completely redo the unix philosophy and the years of experience and fine-tuning that comes with it.
- Chris2048 10y ago
  > The amount of tooling Or the amount of cruft. > git Any structured data can still be serialized and diff'd, but it isn't always the clearest. Where is the contrast here? > made up data structures? so standardize the non-text format > Databases do not make you immune from typos Depends on the constraints. There are few on text files, possibly excluding sudoers. If you aren't sticking to good practice you can just as easily rm a text file. > has a bug or vulnerability? What happens when the kernel has a bug or vulnerability? There are quite a few mature db systems. Plus, all text files depend on the file-system, which is why you store root on something stable like ext (still depends on hdd drivers though, unless you have some some ramfs on boot). > years of experience and fine-tuning Can you describe specifically what that experience is, and what the "fine-tuning" is?
  - laumars 10y ago
koralatov 10y ago
I don't disagree with you, but how do we decide which structured data format to use as a replacement for plaintext? I have the sneaking feeling that a large part of why we still use plaintext is because it's established already as the standard, for worse or for better, and replacing it with a standard everyone could agree on proved impossible.
- TeMPOraL 10y ago
  Use whatever and stick to it. The sane structured data formats differ little but by shape of the braces and some punctuation. Were UNIX developers thinking about this back in the days, they'd probably chose s-expressions or something else from that era. Now it may as well be JSON. The thing is, one format should have been picked, and a parser for it should have been available in a system library. Then we wouldn't have to agree on anything, we'd have one system standard to use.
- borplk 10y ago
  Plaintext is the lazy way out. It's not even remotely a standard because everyone does it differently. It's the shortest path from thinking "I need to persist this crap" to getting something working. Write the bytes to a file, sprinkle some separators, read and parse it back.
imagist 10y ago
I can see your argument for consistent structure--I definitely don't love vimscript or nginx's custom configuration language. But that doesn't require we jump out of text--JSON and XML are viable, provided they remain geared toward hand editing. It may be that given proper tooling for database-driven configuration it could be visible and accessible, but the fact is, I haven't seen timing that pulled that off.
- mbrock 10y ago
  JSON and XML are both horrible in their own ways, especially for use as configuration languages.
  - imagist 10y ago
    True, but my point is that the alternatives being proposed are vaporware. JSON works now. IMHO JSON is the least shitty of the shitty solutions that exist.
    - mbrock 10y ago
      I guess my point is that if JSON is the least shitty alternative then I can't really agree that Unix-type environments would be better off using it for everything, because JSON is a pain in the ass to type and edit with a regular text editor... the same with XML... and the same with S-expressions... YAML is a monster, CSV isn't really structured, and so on... Basically there is no universal structural language that's ergonomic and nice for all uses, so it makes sense that the Unix hackers of yore preferred to create tiny custom languages for everything. It's also because "worse is better". https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html
Vogtinator 10y ago
You can just develop alternative tools and pipe JSON. Text and data in one. Pipes can transfer arbitrary data, so it's just the tools that you don't like, not the underlying mechanism.
- TeMPOraL 10y ago
  Yeah, but that's the point. Pipes are fine. The tools suck, though. UNIX would be infinitely better if it defaulted to piping structured text instead of making each tool have to implement its own shotgun parser.
  - qwertyuiop924 10y ago
    ...So you're talking about having text-serialized key-value objects (or any other kind of object), with standard deserializers and tools for manipulation? That's actually a great idea. Better yet, it's actually viable now, unlike many proposals for "fixing" unix.
    - TeMPOraL 10y ago
      > ...So you're talking about having text-serialized key-value objects (or any other kind of object), with standard deserializers and tools for manipulation? Yes, basically. Someone in power should just pick any format - modified JSON (without the integers are really IEEE754 floats stupidity), s-expressions, whatever - and make standard deserializers part of the system library.
      - mbrock 10y ago
        Heh heh, "someone in power." Who would that be? There's no authority to complain to about all this... Structured text formats come and go: S-expressions, SGML, XML, JSON, "modified JSON", etc etc etc... Fortunately the Unix tools work the same with any byte stream, pretty much, so they've survived gloriously since the 1970s. That's kind of the whole deal with "worse is better."
chrisweekly 10y ago
Great points, esp about logfiles. So grateful to have discovered lnav --http://lnav.org http://lnav.org -- an amazing little CLI tool with embedded sqlite engine and regex support. It solved all my logfile parsing / querying problems, and then some.
TeMPOraL 10y ago
That's my primary issue with UNIX culture. It took a huge step backwards by deciding to work with unstructured text. It wasn't a wrong turn, mind you. It was backtracking on known and understood best practices all the way and then picking a wrong turn. And only now people seem to rediscover what was in common use in the era before UNIX - the virtues of structured text. I guess our industry is meant to run in circles, only changing the type of brackets on each loop (from parens to curly on this iteration).
- ArkyBeagle 10y ago
  I can't disagree there. I've used SNMP in anger before. Underlying SNMP is the MIB. People ran away screaming from it :)
- qwertyuiop924 10y ago
  Structured text is good. Very good, in fact. It might even be idea. structured binary data, less so, at least as a storage format. I want to be able to look at your file format using tools that haven't been specialized to the task. Is that so wrong?
  - dagw 10y ago
    Parsing structured text is slow and inefficient. Also reading out just one part of a data structure stored as text often requires either walking through the file character by character or first slurping the whole thing into memory.
    - qwertyuiop924 10y ago
      ...But when your system crashes, having all that data in an easily accessible manner (regardless of what tools you have on hand) is a major win.
      - michrassena 10y ago
        Let's not forget that when your system crashes, all of these easy-to-read text files are actually stored in a binary format, sometimes scattered in pieces, and require special tools to extract.
ArkyBeagle 10y ago
Structured data may or may not imply "hidden and inaccessible", but there's a lot of correlation between the two.
ptero 10y ago
Your complaints are, IMO, orthogonal to the UNIX philosophy (which, IMO, is also pretty orthogonal to the rule of science; now I have upset both sides :) ). UNIX philosophy is, as written in the article, is based on programs that do one thing well and work with other programs. The second part, "work with other programs" is the one that encourages (but not requires) simple, text based I/O. If A, B and C write programs and independently design some custom, structured, binary I/O the chance of them being compatible is nil. If they output text, the UNIX glue of pipes and text conversions makes them cooperate quickly and efficiently. Not elegant? Sure. But working well in no time.
upofadown 10y ago
Our civilization has been using unstructured text for a very long time now... I totally agree that Unix sux. We need a better philosophy. But you eventually have to come up with something that actually works in practice. I am still waiting. I remember my excitement at the idea that things like CP/M and MSDOS running on personal computers were going to free us all from the tyranny of mainframe computers running things like Unix. We all know how that turned out. Everyone eventually just gave up and started emulating the top of a desk. So Unix is good at messing with unstructured text? Good. Get back to me when you have something better that actually works.
yason 10y ago
> If structured data was embraced we would have developed appropriate tooling to interact with it in the way that we prefer. What kind of tooling might work with ad-hoc structured data and still getting all the tools to talk with each other like in Unix? How would it work without having to write input parsing rules, data processing rules, and output formatting/filtering/composing rules for each tool? I suspect that the reason it's not very popular to pass around structured data is that it's damn difficult to make various tools understand arbitrary data streams. Conversely, the power of text is that the tools understand text, only text, and do not understand the context, i.e. the user decides what the text means and how it needs to be processed. Then the tools become generic and the user can apply them in any number of contexts.
- tim333 10y ago
  >it's not very popular to pass around structured data There's an awful lot of JSON that gets passed around. That seems a reasonable compromise between readable text and some sort of structure.
  - yason 10y ago
    So, how do you parse, read, and process JSON without additional instructions of how to interpret the data and what to look for? All right, maybe you do have a common tool that implements a query language so that you can filter out certain paths and objects from the JSON data into a thinner data set with known formatting expected by the next command in the pipeline. Then you need to write that command and you need more instructions, possibly again in another language, to describe what you actually want to do with the data now that you know where to find it. At this point you typically write a separate script file to do this because it's easier to express in full-blown programming language what you want to do with the tree of hashes and lists and values. On the other hand, programs for lines of text are quite short and fit on the command line. I don't see an immediate value in structured data, and especially none that would outweight loss in general applicability and usability in comparison to text based data processing. Don't get me wrong: I would love to see a good prototype or sketch of how such a thing would work, and then try to imagine how I might be able to apply it to similar things for which I use Unix command line today. But I'm sceptical of "how" and also quite sceptical of "why".
xenophonf 10y ago
The fetish for easily-editable ASCII files and escaping from structure is holding us back. Structured data does not automatically imply hidden and inaccessible, that's a matter of developing appropriate tooling. Speaking as someone who's in the process of automating configuration management on Windows, I'll say that this is _much_ easier said than done. Imagine something like Active Directory Federation Services, which stores its configuration in a database (SQL Server) and offers a good API for altering configuration data (Microsoft.Adfs.PowerShell). Instead of using a generic configuration file template---something supported by just about every configuration management system, using a wide variety of templating mechanims---I must instead write a custom interface between the configuration management system and the AD FS configuration management API. Contrast that with Shibboleth, which stores its configuration in a small collection of XML files (i.e., still strongly typed configuration data). These I can manage relatively easily using my configuration management system's existing file templating mechanism---no special adapter required. I can easily keep a log of all changes by storing these XML files in Git. I can put human-readable comments into them using the XML comment syntax. The same goes for configuration files that use Java properties or JSON or YAML or even ini-style syntax, not to mention all the apps that have configuration files that amount to executable code loaded directly into the run-time environment (e.g., amavisd-new's config file is Perl code, SimpleSAMLphp's is PHP, portmaster's is Bourne shell, portupgrade's is Ruby, and so forth). In short, your configuration database scheme is like an executable, whereas text config files are like source code (literally, in some cases). I'd much rather work with source code, as it remains human readable while at the same time amenable to a variety of text manipulation tools. Databases and APIs are more difficult to worth with, especially from the perspective of enterprise configuration management. Edit: See also http://catb.org/jargon/html/S/SMOP.html http://catb.org/jargon/html/S/SMOP.html.
77pt77 10y ago
> This runs very deep in unix and a lot of people are too "brainwashed" to think of other ways. Instead they develop other exotic ways of dealing with the problem. Well, powershell did solve that problem...
emodendroket 10y ago
That's why I think PowerShell is cool.
Rotareti 10y ago
I was on Windows for about 20 years and moved to Linux about a year ago. One thing I happen to like the most in Linux is file-based configuration. If you add a settings-database to Linux you would end up with the same mess that you find in Windows. In Windows you have the Registry (a giant database for OS and app settings) AND you have the config files. Great... settings are spread over a gigantic db AND the file-system. IMHO: a nightmare. If you add a db for settings there will still be config files, they won't go away and you end up with an additional (huge) layer of complexity..
- GrinningFool 10y ago
  Don't forget group policies (2 or 3 layers of those, iirc)!
dalore 10y ago
Agree with log lines. We now output our logs as json format (one message per line but in json format). Often with lots of metadata. Now you can still use something like jq to parse/grep it. But you can also process it into something like elasticsearch and use all the lovely metadata.
pif 10y ago
> ... the messy unix philosophy of treating everything like plain text. If structured data ... You are forgetting a crucial point: plain text is very well defined. Actually, it was already defined when the first Unix tools were being written. Using plain text means that you can use grep to search the logs of your program, even if your program was written yesterday and grep was written 40 years ago. Structured data? In which format? Who will define the UNIQUE format to be used from now on for every tool? The same people who chose Javascript as the web programming language? Do you realize that choosing plain text prevented any IE6-like monstrosity from happening?
- michrassena 10y ago
  You would think plain text was well-defined, but along came Unicode and even that's not true anymore.
- devdas 10y ago
  Clearly, you want to avoid the overloading caused by implementing a half baked version of Common LISP into every utility. Just use s-expressions. Everyone can just boot into emacs :P.
michrassena 10y ago
The thing about today's Unix Philosophy, epitomized by both the BSDs and Linux is that it's backward-looking. There's a Garden of Eden myth and everything. Look at the reaction to systemd. I remember getting fairly excited when Apple OSX first came out and quite a few of the configuration files were XML-based. Finally, a consistent format, but it wasn't pervasive enough. Even Apple couldn't see fit to break with the past. I've even contemplated rewriting some core utils as an experiment to spit out XML (because I didn't know about typed objects at the time), but I lack the skillset. I know we can't (and maybe we shouldn't) change Unix and its derivatives. There's too much invested in the way things work to pull the rug out. But, when a new OS comes along that wants to do something interesting, I hope the authors will take a look at designing the interface between programs rather than just spitting out or consuming whatever semi-structured ball of text that felt right at the time. Wouldn't it be neat, for example, if instead of 'ls' spitting out lines of text, which sometimes fit the terminal window, sometimes not, which contain a date field which is in the local user's locale format, which is in a certain column which is dependent on the switches passed to the command, instead you get structured, typed information, ISO-formatted date and time, etc. On the presentation side, you can make it look like any old listing from ls if you like, rather than mashing the data and presentation layer together. I'd like to imagine such a system would be more robust than one where we could never change the column the date was in for fear of breaking a billion scripts.
- bluetomcat 10y ago
  I would be so excited to see new operating systems which depart from POSIX completely and introduce new abstractions to replace the dated notions of hierarchical filesystems, terminals, shells, shared libraries, IPC, etc. The sad truth is that everyone targets POSIX because there is so much software that can be ported in order to make the system usable.
  - michrassena 10y ago
    I agree. The "Unix compatibility layer" has killed so many interesting projects over the years. If systems research were still an academic pursuit, maybe there would be some interest in bringing a system like this to fruition. It would require a long-term investment in the design of the system foremost.
digi_owl 10y ago
Seem to me that while shell scripts can be used by anyone given a bit of time, wysiwyg style, powershell is made by programmers for programmers.
GrinningFool 10y ago
> When you get tot that point you have to understand that you have chosen to ignore the fact that the data you are dealing with must be represented in something much closer to a relational database than lines of ASCII text. This is true. But when your needs aren't that complex, basic textual output sure is nice. > The fetish for easily-editable ASCII files and escaping from structure is holding us back. Structured data does not automatically imply hidden and inaccessible, that's a matter of developing appropriate tooling. Good plan. I'll set up a schema by which people can exchange data, and wse'll get it standardized. Given the complexity of the relationships involved - and the fact that I really don't know how my data will be used downstream of me - I'd better make it something really robust and extensible. Maybe some kind of markup language? Then we can ensure that everyone follows the same standard. We can write a compatibility layer that wraps the existing text-only commands and transforms the data into this new extensible markup language (what to call it thought? MLX?). Then anyone who has the basic text tools can download the wrappers, learn the schema , and start processing output. Then again, I could just do that grep | cut. The only thing I have to learn is the shape of the thing I'm grepping for, and the way to use cut - the basics take a few seconds, and no additional tooling is required. Best of all, chances are high that it'll work the same way 20 years from now (though likely with expanded options not current available). There's a lot to be said for having simple tools that accept simple input and produce simple output. This doesn't mean it's the only approach - databases and structured data absolutely have their places in modern CLI tooling - but that has no bearing on the value of an ASCII pipeline.
closeparen 10y ago
A somewhat decent middle ground, which my employer uses a lot, is CLIs that output JSON. You pipe them to JQ to do "selects" on the data structures in a reasonably clear and compact way.
jstanley 10y ago
> "Oh but isn't it neat you can pipe it to grep?" NO! No it's not neat, maybe it was neat 20 years ago. Today I want that damn data in a structure. Then you can still print it out in one line and pipe it to grep all you want. The systemd journal works how you describe, and it is very painful to interact with. I'll take plaintext logfiles any day of the week. It's fine if you want to interact with the log in ways that have been designed into it. But: - it's harder to work out what you can delete to free up space in an emergency - it's harder to get logrotate to do what you want - it's harder to use "tac" to search through the log from the bottom up > I want my software configuration stored in a database So now you can't put comments in your config, you can't (as easily) deploy config with puppet, or in RPMs. You can't easily diff separate configs.
- borplk 10y ago
  I get it. All the things that you mention can in theory be fixed over time. The stuff I'm talking about is not for the next 6 months. It's not very meaningful to compare it against the current tools and landscape. I can almost imagine a similar conversation in the past. Someone saying "MAYBE ONE DAY WE CAN FLY!" and everyone's like "BUT OUR HORSES CAN ONLY JUMP 2 METERS HIGH! It would never work." I understand your comment from a pragmatic point of view but none of those problems are big or important enough that we couldn't fix them in other ways. Throwing away a rich structured piece of data and trading that for a dumb line of characters that needs to be re-parsed just so that it's easier to use logrotate and tac with them and so on is a losing trade.
- digi_owl 10y ago
  > You can't easily diff separate configs. Reminding me of one of the little niceties of Gobolinux. You have a pristine pr version config copy sitting in the main dir, and a package-wide settings dir. It also provides a command (implemented as a shell script, as is most of Gobolinux tools) that gets run upon installing a new package version. If said command detects a difference between existing and new config files it gives you various options. You can have it retain all the old files, replace them with the new files, or even bring up a merged file that give you the new lines as comments next to the old ones.
module0000 10y ago
> Yes I get it, you like your SSH session and Emacs/Vim blah blah but that's short-sighted. How so? I'm one of those people that like my SSH session, and in my case vim and "blah blah blah". I've contributed to countless open source software packages that you likely use with this method, and so have tons of other developers. Nothing is broken here, things are working great for everyone who reads the manual and follows it. > I want my software configuration stored in a database not in a bunch of fragile files with made up syntax that are always one typo or syntax error away from being potentially silently ignored. apache, postfix, haproxy, even vim are certainly not prone to silently ignore anything, just to name a few. > The fetish for easily-editable ASCII files and escaping from structure is holding us back. Holding us back from what? I am both a developer and an administrator, and I've had all the fun with solaris/aix configurations that are often not stored in plain text that I care to have. If you also have this experience, and still feel the way you do, then I'd love to hear more. Otherwise, your rant comes off as "your way is hard, and I don't want to learn it!" Look at all the available structures available for that plain text you speak of... XML, JSON, YAML, the list goes on. You are free to use one of those, then you have that structure you crave. There are plenty of areas that could use revolution, but UNIX-like configuration files are not one of them. There is no problem here. If you are making typos or mis-configuring your software, then you have a problem of your own creation.
- borplk 10y ago
  > How so? As I mentioned in another comment in my view the problem is that something like the configuration is shared both by the humans and the computers. Because of this we settle on something that is not optimal for either group. We end up with something that is hostile to both the humans and the computers, just in different ways. In fact the argument of people like you for ASCII config files exactly demonstrates my point. You are fighting for your human-convenience against the machines. > Holding us back from what? By embracing and acknowledging that the humans and computers are not meant to share a language we free ourselves from this push-pull tension between human vs machine convenience. We can develop formats and tooling that respects its human audience, that doesn't punish the human for making small superficial syntax or typo errors and so on. And we can finally step the hell out of the way of computers and let them use what is suitable for them. And at that point you could still have your SSH session and Vim/Emacs and blah blah blah and you could still view and interact with stuff as plaintext if you wanted to. > apache, postfix, haproxy, even vim are certainly not prone to silently ignore anything, just to name a few. It's not always a matter of silently ignoring something but due to the nature of the task it is certainly very easy to shoot yourself in the foot doing something that isn't technically an error but wasn't your intention. For example you can silently break your cron jobs by leaving windows-newlines in them. Perfect example of humans and computers sharing a language that is hostile to the human. BAD BAD human! You stupid human why do you use bad Windows invisible characters? Use good linux invisible characters instead that are more tasty for your almighty lord Linux.
jokoon 10y ago
I'm split, because there was this article saying the opposite, that actually text is a pretty good format because it's simple, compact, and easily debug-able. On the other hand text is awful if you want to serialize data because parsing and searching text is one of the most CPU intensive thing. Now, it's true that we should recognize where text is really inadequate, especially when indexing and searching is needed. Webpages, for example, should not be plain text. I think the problem resides in programmers not being able to properly use and understand how a database works. Databases and their engines are black boxes, so it's normal if fewer developers want to to use DB like you say. Meanwhile dictionaries and B-tree are not very sophisticated algorithms, yet I see almost no programmers using them consciously. The less a programmer know about the tools he has in his hands, the less he will get benefits from it and thus he will start using easier things. So really my thought is that the tools are not accessible enough. The concepts of file and database are so distant that it's completely impossible to work with both, but to me it should.
betenoire 10y ago
It doesn't matter if it is text or not, and if it is structured or not. You missed the whole point about signal vs noise. When I'm ALWAYS presented with a blob of something to decipher, it requires a context switch. Nothing IS something, and it's a structured something.
gaga666 10y ago
First, rule of silence has nothing to do with plain text. It applies to any human-machine interface including GUI and physical knobs. Second, you are right about structured data and all. The only thing is that it's either impossible or extremely hard to achieve. Many have tried, all of them failed. Windows now has a mix of registry, file and database configs which is a nightmare and is much worse than any unix. AIX has smitty and other config management solutions which are a bitch to work with if you want something non-trivial. Solaris is heading this direction (actually it's heading to the grave but it's another story) and it's also not nice. There are a lot of other OSes and programs which tried to do it but failed. This is much like with democracy: it's a terrible form of ruling, too bad we have nothing better. This is exactly what's up with unix configs and data formats. It is possible to make some fancy format and tools which will achieve it's goal for like 80% of the time. But it will cause huge amount of pain in the ass in the rest 20% and this is where it will be ignored and you'll end up with a mix of two words which is worse than one.

by borplk 10y ago

skrebbel 10y ago

In 1993, someone made exactly your argument a in Redmond board room, and so many people agreed that what you describe could be adequately called "The Windows Philosophy". All settings in a database and not text files (the registry); a command-line that pipes data, not text (PowerShell). Tailored UIs to change settings, not magic invocations and obsure text file syntaxes. I guess most developers on HN are also aware of the downsides of this philosophy. If not, try configuring IIS.

rtpg 10y ago
Surely you can reconcile structured representations and something like the Unix command line. Imagine if the default wasn't bash, but something like Ruby + pipes (or some other terse language). What is the argument for shell scripts not working on typed objects? How much time has been lost, how many bugs have been created because every single interaction between shell scripts has to include its own parser. How many versions of " get file created timestamp from ls" do we need? Something Windows does get right is the clipboard. You place a thing on the clipboard, and when you paste, the receiving program can decide on the best representation. This is why copy-pasting images has worked so magically. I could see an alternative system where such a mechanism exists for shell programs.
- cyphar 10y ago
  > What is the argument for shell scripts not working on typed objects? How much time has been lost, how many bugs have been created because every single interaction between shell scripts has to include its own parser. How many versions of " get file created timestamp from ls" do we need? Aside: that's what the stat command is for. My big concern with types is how would you make sure that the output of a command will always have the right types? Otherwise you'll have runtime type errors which would be just as bad as runtime parsing errors.
  - junke 10y ago
    > Otherwise you'll have runtime type errors which would be just as bad as runtime parsing errors. Parsing errors in script shells can easily go unnoticed (until you realize your data is corrupted).
- chei0iaV 10y ago
  The problem is that now simply providing user-friendly output is not enough. For every program or script you throw together you need to provide the text output for the user, and then the type object stream for piping. And then, the user would need to read documentation to see how to access each piece of data, what data type it is, what data type the other command takes, and maybe consider how to convert one to the other. ... at which point you basically have a scripting language, so you could just as well use an existing one (e.g. Ruby).

pwdisswordfish 10y ago

The text-oriented nature of Unix may be a mistake, but the Rule of Silence is not concerned with it. It is about not overwhelming the user with information they are not interested in. GUIs can follow or violate the Rule of Silence just as well.

SixSigma 10y ago
Your comment was saved. Click OK to continue.
- kulu2002 10y ago
  Precisely!
restalis 10y ago
"GUIs can follow or violate the Rule of Silence just as well." It would have been nice if this was permitted to be violated only by GUIs. Ask about the first-time *nix experiences before the GUI-embracing era and one of the few things they were noticing was the continuous text-spitting. For instance, that was happening on boot and OS loading sequence (and still happens, now only being hidden by default with splash-screens), a lot of reporting about all the things that were performed successfully. It's funny in this regard seeing Unix' Rule of Silence being respected more... outside Unix, where is just common sense, with no need to be formulated as rule.
- pwdisswordfish 10y ago
  Booting up is a bit of a special case that I'd actually be willing to carve out an exception for. When the boot-up goes wrong, it's often due to causes so deep that they leave you with very few means of diagnosing it or showing an error message. If the booting process hangs, having a rough idea at what stage it happened can be very helpful. Without these messages, discovering the cause of failure would be much harder. The boot-up splash screen hardly even counts as "GUI" anyway.
StRoy 10y ago
Talking about the rule of silence in GUIs, I wish I could slap the eurocrats who decided to force websites to show the cookie warning on a first visit of all websites. Did they even understand the consequence and wasted time of what they were doing? Having to click all the time to get this dumb warning off? Of course some websites had to do it in an even dumber way than the law asks for. Like slashdot: http://i.imgur.com/5Fp0nmo.png http://i.imgur.com/5Fp0nmo.png This is what greets the French every time slashdot decides to forget you agreed to let them put cookies on your computer and you need to click continue before you can get to the actual website. The law actually made it worse for the people it's supposed to protect (those who might refuse cookies for privacy?) because those warning then will stick around like glue if they can't give you a cookie to remember you accepted their existence.

verinus 10y ago

I'm with you when it comes to structured data, but plz no more data bases. these config files do not need to be centralized. I am thinking more in the direction of a parser that could check the validity of a configuration...

riskneural 10y ago
So, horribly unreadable xml config files?
- nailer 10y ago
  CSON or .desktop / .service or something similar is immediately understandable to most people and doesn't waste time with unnecessary tokens like XML does.
- TeMPOraL 10y ago
  Like S-expressions don't exist. Or even goddamn JSON. Come on, we don't have to jump from one stupidity (unstructured text) to another (using XML as a data representation format).
- stevedonovan 10y ago
  Not necessarily. JSON is not bad, _if_ you allow comments. Even plain-jane key/value config files can be sanity-checked. I suspect part of the problem is that anything fancy like that is awkward to do in C, so people take the lazy way out.
karl42 10y ago
Maybe many tiny sqlite databases? Then you would not need to to centralize your data in a single database, but still do queries across different config files.
- TeMPOraL 10y ago
  SQLite is a huge win. All the power of SQL, and you don't have to introduce system-wide and non-local dependencies.

Ianvdl 10y ago

> To play devil's advocate, part of the reason things like The Rule of Silence are talked about is because of the messy unix philosophy of treating everything like plain text. The amount of tooling that surrounds text is vast and has evolved over decades. You cannot replace that with a single database and call it better. I can place the majority of my config files in git and version them. I can easily perform a full text search on any log regardless of its syntax. I can effortlessly relocate and distribute logs to different filesystems. > I want my software configuration stored in a database not in a bunch of fragile files with made up syntax that are always one typo or syntax error away from being potentially silently ignored. So you would like your configuration stored as a bunch of made up data structures? Databases do not make you immune from typos and syntax errors, ask anyone who has ever written a delete without a where clause. And what happens when the giant, all-knowing database your system depends on has a bug or vulnerability? When something on my linux box breaks I can drop to a recovery mode or lower runlevel and fix it with a kernel and a shell as dependencies. I think you would be a lot happier with a configuration management system (puppet, ansible et al) and some log infrastructure without having to completely redo the unix philosophy and the years of experience and fine-tuning that comes with it.

Chris2048 10y ago
> The amount of tooling Or the amount of cruft. > git Any structured data can still be serialized and diff'd, but it isn't always the clearest. Where is the contrast here? > made up data structures? so standardize the non-text format > Databases do not make you immune from typos Depends on the constraints. There are few on text files, possibly excluding sudoers. If you aren't sticking to good practice you can just as easily rm a text file. > has a bug or vulnerability? What happens when the kernel has a bug or vulnerability? There are quite a few mature db systems. Plus, all text files depend on the file-system, which is why you store root on something stable like ext (still depends on hdd drivers though, unless you have some some ramfs on boot). > years of experience and fine-tuning Can you describe specifically what that experience is, and what the "fine-tuning" is?
- laumars 10y ago

koralatov 10y ago

I don't disagree with you, but how do we decide which structured data format to use as a replacement for plaintext? I have the sneaking feeling that a large part of why we still use plaintext is because it's established already as the standard, for worse or for better, and replacing it with a standard everyone could agree on proved impossible.

TeMPOraL 10y ago
Use whatever and stick to it. The sane structured data formats differ little but by shape of the braces and some punctuation. Were UNIX developers thinking about this back in the days, they'd probably chose s-expressions or something else from that era. Now it may as well be JSON. The thing is, one format should have been picked, and a parser for it should have been available in a system library. Then we wouldn't have to agree on anything, we'd have one system standard to use.
borplk 10y ago
Plaintext is the lazy way out. It's not even remotely a standard because everyone does it differently. It's the shortest path from thinking "I need to persist this crap" to getting something working. Write the bytes to a file, sprinkle some separators, read and parse it back.

imagist 10y ago

I can see your argument for consistent structure--I definitely don't love vimscript or nginx's custom configuration language. But that doesn't require we jump out of text--JSON and XML are viable, provided they remain geared toward hand editing. It may be that given proper tooling for database-driven configuration it could be visible and accessible, but the fact is, I haven't seen timing that pulled that off.

mbrock 10y ago
JSON and XML are both horrible in their own ways, especially for use as configuration languages.
- imagist 10y ago
  True, but my point is that the alternatives being proposed are vaporware. JSON works now. IMHO JSON is the least shitty of the shitty solutions that exist.
  - mbrock 10y ago
    I guess my point is that if JSON is the least shitty alternative then I can't really agree that Unix-type environments would be better off using it for everything, because JSON is a pain in the ass to type and edit with a regular text editor... the same with XML... and the same with S-expressions... YAML is a monster, CSV isn't really structured, and so on... Basically there is no universal structural language that's ergonomic and nice for all uses, so it makes sense that the Unix hackers of yore preferred to create tiny custom languages for everything. It's also because "worse is better". https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html

Vogtinator 10y ago

You can just develop alternative tools and pipe JSON. Text and data in one. Pipes can transfer arbitrary data, so it's just the tools that you don't like, not the underlying mechanism.

TeMPOraL 10y ago
Yeah, but that's the point. Pipes are fine. The tools suck, though. UNIX would be infinitely better if it defaulted to piping structured text instead of making each tool have to implement its own shotgun parser.
- qwertyuiop924 10y ago
  ...So you're talking about having text-serialized key-value objects (or any other kind of object), with standard deserializers and tools for manipulation? That's actually a great idea. Better yet, it's actually viable now, unlike many proposals for "fixing" unix.
  - TeMPOraL 10y ago
    > ...So you're talking about having text-serialized key-value objects (or any other kind of object), with standard deserializers and tools for manipulation? Yes, basically. Someone in power should just pick any format - modified JSON (without the integers are really IEEE754 floats stupidity), s-expressions, whatever - and make standard deserializers part of the system library.
    - mbrock 10y ago
      Heh heh, "someone in power." Who would that be? There's no authority to complain to about all this... Structured text formats come and go: S-expressions, SGML, XML, JSON, "modified JSON", etc etc etc... Fortunately the Unix tools work the same with any byte stream, pretty much, so they've survived gloriously since the 1970s. That's kind of the whole deal with "worse is better."

chrisweekly 10y ago

Great points, esp about logfiles. So grateful to have discovered lnav --http://lnav.org http://lnav.org -- an amazing little CLI tool with embedded sqlite engine and regex support. It solved all my logfile parsing / querying problems, and then some.

TeMPOraL 10y ago

That's my primary issue with UNIX culture. It took a huge step backwards by deciding to work with unstructured text. It wasn't a wrong turn, mind you. It was backtracking on known and understood best practices all the way and then picking a wrong turn. And only now people seem to rediscover what was in common use in the era before UNIX - the virtues of structured text. I guess our industry is meant to run in circles, only changing the type of brackets on each loop (from parens to curly on this iteration).

ArkyBeagle 10y ago
I can't disagree there. I've used SNMP in anger before. Underlying SNMP is the MIB. People ran away screaming from it :)
qwertyuiop924 10y ago
Structured text is good. Very good, in fact. It might even be idea. structured binary data, less so, at least as a storage format. I want to be able to look at your file format using tools that haven't been specialized to the task. Is that so wrong?
- dagw 10y ago
  Parsing structured text is slow and inefficient. Also reading out just one part of a data structure stored as text often requires either walking through the file character by character or first slurping the whole thing into memory.
  - qwertyuiop924 10y ago
    ...But when your system crashes, having all that data in an easily accessible manner (regardless of what tools you have on hand) is a major win.
    - michrassena 10y ago
      Let's not forget that when your system crashes, all of these easy-to-read text files are actually stored in a binary format, sometimes scattered in pieces, and require special tools to extract.

ArkyBeagle 10y ago

Structured data may or may not imply "hidden and inaccessible", but there's a lot of correlation between the two.

ptero 10y ago

Your complaints are, IMO, orthogonal to the UNIX philosophy (which, IMO, is also pretty orthogonal to the rule of science; now I have upset both sides :) ). UNIX philosophy is, as written in the article, is based on programs that do one thing well and work with other programs. The second part, "work with other programs" is the one that encourages (but not requires) simple, text based I/O. If A, B and C write programs and independently design some custom, structured, binary I/O the chance of them being compatible is nil. If they output text, the UNIX glue of pipes and text conversions makes them cooperate quickly and efficiently. Not elegant? Sure. But working well in no time.

upofadown 10y ago

Our civilization has been using unstructured text for a very long time now... I totally agree that Unix sux. We need a better philosophy. But you eventually have to come up with something that actually works in practice. I am still waiting. I remember my excitement at the idea that things like CP/M and MSDOS running on personal computers were going to free us all from the tyranny of mainframe computers running things like Unix. We all know how that turned out. Everyone eventually just gave up and started emulating the top of a desk. So Unix is good at messing with unstructured text? Good. Get back to me when you have something better that actually works.

yason 10y ago

> If structured data was embraced we would have developed appropriate tooling to interact with it in the way that we prefer. What kind of tooling might work with ad-hoc structured data and still getting all the tools to talk with each other like in Unix? How would it work without having to write input parsing rules, data processing rules, and output formatting/filtering/composing rules for each tool? I suspect that the reason it's not very popular to pass around structured data is that it's damn difficult to make various tools understand arbitrary data streams. Conversely, the power of text is that the tools understand text, only text, and do not understand the context, i.e. the user decides what the text means and how it needs to be processed. Then the tools become generic and the user can apply them in any number of contexts.

tim333 10y ago
>it's not very popular to pass around structured data There's an awful lot of JSON that gets passed around. That seems a reasonable compromise between readable text and some sort of structure.
- yason 10y ago
  So, how do you parse, read, and process JSON without additional instructions of how to interpret the data and what to look for? All right, maybe you do have a common tool that implements a query language so that you can filter out certain paths and objects from the JSON data into a thinner data set with known formatting expected by the next command in the pipeline. Then you need to write that command and you need more instructions, possibly again in another language, to describe what you actually want to do with the data now that you know where to find it. At this point you typically write a separate script file to do this because it's easier to express in full-blown programming language what you want to do with the tree of hashes and lists and values. On the other hand, programs for lines of text are quite short and fit on the command line. I don't see an immediate value in structured data, and especially none that would outweight loss in general applicability and usability in comparison to text based data processing. Don't get me wrong: I would love to see a good prototype or sketch of how such a thing would work, and then try to imagine how I might be able to apply it to similar things for which I use Unix command line today. But I'm sceptical of "how" and also quite sceptical of "why".

xenophonf 10y ago

The fetish for easily-editable ASCII files and escaping from structure is holding us back. Structured data does not automatically imply hidden and inaccessible, that's a matter of developing appropriate tooling. Speaking as someone who's in the process of automating configuration management on Windows, I'll say that this is _much_ easier said than done. Imagine something like Active Directory Federation Services, which stores its configuration in a database (SQL Server) and offers a good API for altering configuration data (Microsoft.Adfs.PowerShell). Instead of using a generic configuration file template---something supported by just about every configuration management system, using a wide variety of templating mechanims---I must instead write a custom interface between the configuration management system and the AD FS configuration management API. Contrast that with Shibboleth, which stores its configuration in a small collection of XML files (i.e., still strongly typed configuration data). These I can manage relatively easily using my configuration management system's existing file templating mechanism---no special adapter required. I can easily keep a log of all changes by storing these XML files in Git. I can put human-readable comments into them using the XML comment syntax. The same goes for configuration files that use Java properties or JSON or YAML or even ini-style syntax, not to mention all the apps that have configuration files that amount to executable code loaded directly into the run-time environment (e.g., amavisd-new's config file is Perl code, SimpleSAMLphp's is PHP, portmaster's is Bourne shell, portupgrade's is Ruby, and so forth). In short, your configuration database scheme is like an executable, whereas text config files are like source code (literally, in some cases). I'd much rather work with source code, as it remains human readable while at the same time amenable to a variety of text manipulation tools. Databases and APIs are more difficult to worth with, especially from the perspective of enterprise configuration management. Edit: See also http://catb.org/jargon/html/S/SMOP.html http://catb.org/jargon/html/S/SMOP.html.

77pt77 10y ago

> This runs very deep in unix and a lot of people are too "brainwashed" to think of other ways. Instead they develop other exotic ways of dealing with the problem. Well, powershell did solve that problem...

emodendroket 10y ago

That's why I think PowerShell is cool.

Rotareti 10y ago

I was on Windows for about 20 years and moved to Linux about a year ago. One thing I happen to like the most in Linux is file-based configuration. If you add a settings-database to Linux you would end up with the same mess that you find in Windows. In Windows you have the Registry (a giant database for OS and app settings) AND you have the config files. Great... settings are spread over a gigantic db AND the file-system. IMHO: a nightmare. If you add a db for settings there will still be config files, they won't go away and you end up with an additional (huge) layer of complexity..

GrinningFool 10y ago
Don't forget group policies (2 or 3 layers of those, iirc)!

dalore 10y ago

Agree with log lines. We now output our logs as json format (one message per line but in json format). Often with lots of metadata. Now you can still use something like jq to parse/grep it. But you can also process it into something like elasticsearch and use all the lovely metadata.

pif 10y ago

> ... the messy unix philosophy of treating everything like plain text. If structured data ... You are forgetting a crucial point: plain text is very well defined. Actually, it was already defined when the first Unix tools were being written. Using plain text means that you can use grep to search the logs of your program, even if your program was written yesterday and grep was written 40 years ago. Structured data? In which format? Who will define the UNIQUE format to be used from now on for every tool? The same people who chose Javascript as the web programming language? Do you realize that choosing plain text prevented any IE6-like monstrosity from happening?

michrassena 10y ago
You would think plain text was well-defined, but along came Unicode and even that's not true anymore.
devdas 10y ago
Clearly, you want to avoid the overloading caused by implementing a half baked version of Common LISP into every utility. Just use s-expressions. Everyone can just boot into emacs :P.

michrassena 10y ago

The thing about today's Unix Philosophy, epitomized by both the BSDs and Linux is that it's backward-looking. There's a Garden of Eden myth and everything. Look at the reaction to systemd. I remember getting fairly excited when Apple OSX first came out and quite a few of the configuration files were XML-based. Finally, a consistent format, but it wasn't pervasive enough. Even Apple couldn't see fit to break with the past. I've even contemplated rewriting some core utils as an experiment to spit out XML (because I didn't know about typed objects at the time), but I lack the skillset. I know we can't (and maybe we shouldn't) change Unix and its derivatives. There's too much invested in the way things work to pull the rug out. But, when a new OS comes along that wants to do something interesting, I hope the authors will take a look at designing the interface between programs rather than just spitting out or consuming whatever semi-structured ball of text that felt right at the time. Wouldn't it be neat, for example, if instead of 'ls' spitting out lines of text, which sometimes fit the terminal window, sometimes not, which contain a date field which is in the local user's locale format, which is in a certain column which is dependent on the switches passed to the command, instead you get structured, typed information, ISO-formatted date and time, etc. On the presentation side, you can make it look like any old listing from ls if you like, rather than mashing the data and presentation layer together. I'd like to imagine such a system would be more robust than one where we could never change the column the date was in for fear of breaking a billion scripts.

bluetomcat 10y ago
I would be so excited to see new operating systems which depart from POSIX completely and introduce new abstractions to replace the dated notions of hierarchical filesystems, terminals, shells, shared libraries, IPC, etc. The sad truth is that everyone targets POSIX because there is so much software that can be ported in order to make the system usable.
- michrassena 10y ago
  I agree. The "Unix compatibility layer" has killed so many interesting projects over the years. If systems research were still an academic pursuit, maybe there would be some interest in bringing a system like this to fruition. It would require a long-term investment in the design of the system foremost.

digi_owl 10y ago

Seem to me that while shell scripts can be used by anyone given a bit of time, wysiwyg style, powershell is made by programmers for programmers.

GrinningFool 10y ago

> When you get tot that point you have to understand that you have chosen to ignore the fact that the data you are dealing with must be represented in something much closer to a relational database than lines of ASCII text. This is true. But when your needs aren't that complex, basic textual output sure is nice. > The fetish for easily-editable ASCII files and escaping from structure is holding us back. Structured data does not automatically imply hidden and inaccessible, that's a matter of developing appropriate tooling. Good plan. I'll set up a schema by which people can exchange data, and wse'll get it standardized. Given the complexity of the relationships involved - and the fact that I really don't know how my data will be used downstream of me - I'd better make it something really robust and extensible. Maybe some kind of markup language? Then we can ensure that everyone follows the same standard. We can write a compatibility layer that wraps the existing text-only commands and transforms the data into this new extensible markup language (what to call it thought? MLX?). Then anyone who has the basic text tools can download the wrappers, learn the schema , and start processing output. Then again, I could just do that grep | cut. The only thing I have to learn is the shape of the thing I'm grepping for, and the way to use cut - the basics take a few seconds, and no additional tooling is required. Best of all, chances are high that it'll work the same way 20 years from now (though likely with expanded options not current available). There's a lot to be said for having simple tools that accept simple input and produce simple output. This doesn't mean it's the only approach - databases and structured data absolutely have their places in modern CLI tooling - but that has no bearing on the value of an ASCII pipeline.

closeparen 10y ago

A somewhat decent middle ground, which my employer uses a lot, is CLIs that output JSON. You pipe them to JQ to do "selects" on the data structures in a reasonably clear and compact way.

jstanley 10y ago

> "Oh but isn't it neat you can pipe it to grep?" NO! No it's not neat, maybe it was neat 20 years ago. Today I want that damn data in a structure. Then you can still print it out in one line and pipe it to grep all you want. The systemd journal works how you describe, and it is very painful to interact with. I'll take plaintext logfiles any day of the week. It's fine if you want to interact with the log in ways that have been designed into it. But: - it's harder to work out what you can delete to free up space in an emergency - it's harder to get logrotate to do what you want - it's harder to use "tac" to search through the log from the bottom up > I want my software configuration stored in a database So now you can't put comments in your config, you can't (as easily) deploy config with puppet, or in RPMs. You can't easily diff separate configs.

borplk 10y ago
I get it. All the things that you mention can in theory be fixed over time. The stuff I'm talking about is not for the next 6 months. It's not very meaningful to compare it against the current tools and landscape. I can almost imagine a similar conversation in the past. Someone saying "MAYBE ONE DAY WE CAN FLY!" and everyone's like "BUT OUR HORSES CAN ONLY JUMP 2 METERS HIGH! It would never work." I understand your comment from a pragmatic point of view but none of those problems are big or important enough that we couldn't fix them in other ways. Throwing away a rich structured piece of data and trading that for a dumb line of characters that needs to be re-parsed just so that it's easier to use logrotate and tac with them and so on is a losing trade.
digi_owl 10y ago
> You can't easily diff separate configs. Reminding me of one of the little niceties of Gobolinux. You have a pristine pr version config copy sitting in the main dir, and a package-wide settings dir. It also provides a command (implemented as a shell script, as is most of Gobolinux tools) that gets run upon installing a new package version. If said command detects a difference between existing and new config files it gives you various options. You can have it retain all the old files, replace them with the new files, or even bring up a merged file that give you the new lines as comments next to the old ones.

module0000 10y ago

> Yes I get it, you like your SSH session and Emacs/Vim blah blah but that's short-sighted. How so? I'm one of those people that like my SSH session, and in my case vim and "blah blah blah". I've contributed to countless open source software packages that you likely use with this method, and so have tons of other developers. Nothing is broken here, things are working great for everyone who reads the manual and follows it. > I want my software configuration stored in a database not in a bunch of fragile files with made up syntax that are always one typo or syntax error away from being potentially silently ignored. apache, postfix, haproxy, even vim are certainly not prone to silently ignore anything, just to name a few. > The fetish for easily-editable ASCII files and escaping from structure is holding us back. Holding us back from what? I am both a developer and an administrator, and I've had all the fun with solaris/aix configurations that are often not stored in plain text that I care to have. If you also have this experience, and still feel the way you do, then I'd love to hear more. Otherwise, your rant comes off as "your way is hard, and I don't want to learn it!" Look at all the available structures available for that plain text you speak of... XML, JSON, YAML, the list goes on. You are free to use one of those, then you have that structure you crave. There are plenty of areas that could use revolution, but UNIX-like configuration files are not one of them. There is no problem here. If you are making typos or mis-configuring your software, then you have a problem of your own creation.

borplk 10y ago
> How so? As I mentioned in another comment in my view the problem is that something like the configuration is shared both by the humans and the computers. Because of this we settle on something that is not optimal for either group. We end up with something that is hostile to both the humans and the computers, just in different ways. In fact the argument of people like you for ASCII config files exactly demonstrates my point. You are fighting for your human-convenience against the machines. > Holding us back from what? By embracing and acknowledging that the humans and computers are not meant to share a language we free ourselves from this push-pull tension between human vs machine convenience. We can develop formats and tooling that respects its human audience, that doesn't punish the human for making small superficial syntax or typo errors and so on. And we can finally step the hell out of the way of computers and let them use what is suitable for them. And at that point you could still have your SSH session and Vim/Emacs and blah blah blah and you could still view and interact with stuff as plaintext if you wanted to. > apache, postfix, haproxy, even vim are certainly not prone to silently ignore anything, just to name a few. It's not always a matter of silently ignoring something but due to the nature of the task it is certainly very easy to shoot yourself in the foot doing something that isn't technically an error but wasn't your intention. For example you can silently break your cron jobs by leaving windows-newlines in them. Perfect example of humans and computers sharing a language that is hostile to the human. BAD BAD human! You stupid human why do you use bad Windows invisible characters? Use good linux invisible characters instead that are more tasty for your almighty lord Linux.

jokoon 10y ago

I'm split, because there was this article saying the opposite, that actually text is a pretty good format because it's simple, compact, and easily debug-able. On the other hand text is awful if you want to serialize data because parsing and searching text is one of the most CPU intensive thing. Now, it's true that we should recognize where text is really inadequate, especially when indexing and searching is needed. Webpages, for example, should not be plain text. I think the problem resides in programmers not being able to properly use and understand how a database works. Databases and their engines are black boxes, so it's normal if fewer developers want to to use DB like you say. Meanwhile dictionaries and B-tree are not very sophisticated algorithms, yet I see almost no programmers using them consciously. The less a programmer know about the tools he has in his hands, the less he will get benefits from it and thus he will start using easier things. So really my thought is that the tools are not accessible enough. The concepts of file and database are so distant that it's completely impossible to work with both, but to me it should.

betenoire 10y ago

It doesn't matter if it is text or not, and if it is structured or not. You missed the whole point about signal vs noise. When I'm ALWAYS presented with a blob of something to decipher, it requires a context switch. Nothing IS something, and it's a structured something.

gaga666 10y ago

First, rule of silence has nothing to do with plain text. It applies to any human-machine interface including GUI and physical knobs. Second, you are right about structured data and all. The only thing is that it's either impossible or extremely hard to achieve. Many have tried, all of them failed. Windows now has a mix of registry, file and database configs which is a nightmare and is much worse than any unix. AIX has smitty and other config management solutions which are a bitch to work with if you want something non-trivial. Solaris is heading this direction (actually it's heading to the grave but it's another story) and it's also not nice. There are a lot of other OSes and programs which tried to do it but failed. This is much like with democracy: it's a terrible form of ruling, too bad we have nothing better. This is exactly what's up with unix configs and data formats. It is possible to make some fancy format and tools which will achieve it's goal for like 80% of the time. But it will cause huge amount of pain in the ass in the rest 20% and this is where it will be ignored and you'll end up with a mix of two words which is worse than one.