8 ms·

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wa

The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. People praise perl, sed, awk, cut, etc. for being good at text processing. But the only reason they need these text-processing tools for pipelines is because they are trying to recover structure from the data that was already present before the previous stage of the pipeline threw that structure away by dumping it to flat text! Text is obviously a convenient way for humans to view a program's output, so clearly it's useful that all Unix programs (ls, ps, etc) can dump their output as text. But there's no need to dump to text until the output is being sent to a human. If you're piping "ls | grep" there is no reason for "ls" to dump to text and "grep" to parse it back from text, especially since "grep" doesn't know anything about the format of ls's output. It would be way more convenient if you could say something like: ls | grep 'file.size > 1M' But the only way to do this today is to parse ls's output first. There would be no reason for this if ls could send structured data to grep. What I'm describing is similar to Monad, Microsoft's next-gen shell. AIUI it can send .NET objects between processes instead of flat text. But IMO it's too imposing to mandate a single object representation like .NET objects. I'm experimenting with the idea of letting people specify the output of command-line utilities as a Protocol Buffer schema, for example: message DirectoryEntry { optional uint64 inode = 1; optional string name = 2; optional uint64 size = 3; // etc. } I think this could be a compelling way of making the next generation of usability in command-line pipelines, by saving people from having to write ad-hoc parsers all the time.

anvandare 14y ago
Actually I've been considering/dreaming about a thing like this for a long while now (ever since I got started with jQuery and discovered how smooth I could sail through the DOM with it). The idea I came up with (I'm most likely not the first to think of it, so tell me if someone already implemented it) is a sort of JavaScript shell for *nix that'd work similar to jQuery: passing along (collections of) JavaScript objects with properties and functions. e.g.: var sum = 0; $("~").ls({"type": "file", "size": ">1M"}).each(function () {sum += $(this).size("mb");}); $.echo("total size of home directory: %d MB", sum);
- stcredzero 14y ago
  Or, one could write a library of standard parsers and serializers for the Unix tools that would parse and produce known JSON representations of data that could be passed between scripts.
  - zem 14y ago
    i did something like that at my last job - the cluster manager was designed as a set of command line tools, and every program had a -j flag that would make it parse stdin as json and write json to stdout. we then wrote wrappers around all the linux utilities we were using, to enable the same behaviour. once the basic system was in place it let us experiment with new features very rapidly indeed because everything could be tested in isolation from the command line using automated test scripts. what was especially nice was that we could use the same test framework to chain several utilities together and test the combination, because it was all a black box with json going in and coming out.
  - Evbn 14y ago
    That's recordstream
    - stcredzero 14y ago
      recordstream seems to concentrate on lots of formats. Really, it should concentrate on lots of tools, with a shared repository of parsers for tools. EDIT: Actually, they concentrate on JSON, but they also try to provide a generic set of tools for reading any format. I think this could be better structured to "just work."
- noblethrasher 14y ago
  Powershell on Windows does something like what you want: PS C:\Some Directory $sum = 0 PS C:\Some Directory dir | where { $_Length -gt 1MB } | %{ $sum += $_.Length }
  - mey 14y ago
stcredzero 14y ago
The real problem is that Unix commands produce flat text output without any information about how to parse that text back into structured data. Any user who wants the structured version of the data has to parse it themselves, but these parsers are ad hoc and incomplete by their very nature. There are a variety of serialization schemes that are quite easy to parse and would be suitable for the output of most Unix command-line tools. BEncode from bittorrent would do nicely, and it's quite easy to parse. JSON would do nicely as well. Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects.
- haberman 14y ago
  > JSON would do nicely as well. Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream > Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects. Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dictating representations of objects, you are dictating far too much about the implementation of individual pipeline nodes.
  - nwmcsween 14y ago
    Why does the protocol define the objects? Shouldn't the object metadata should have enough information to transform the output? E.g. object is a URL vs protocol is a URL
  - neutronicus 14y ago
    How can your tools all accept the same kind of structured data without dictating its representation? I don't get it.
    - jcmhn 14y ago
      Allow a shell variable to control output record and field separators. Default to space and newline if nothing is specified. ORS=: OFS=, ls -l # rw-r--r--:1:uname:gname:1025:Jun 1:somefile.txt,...,... Add a specifier for dates, usernames, groupnames, etc. DSF="%Y-%m-%d" ls -l
its_so_on 14y ago
I like your observation but I don't like your conclusion. I think the major thing missing from the world is easy piping a la Unix. You can't pipe your list of paying customer's email addresses to paypal at the command line with some switches - really, no matter how many switches you use - and bill each one the amount stated, because "paypal" is not a commandline app. You can't pipe the results of some long-running analysis to Twitter to announce that you've finished computing it, no matter how many switches you add, because twitter is not a commandline app. The direction you're suggesting we take things is, in fact, a fuller API. These exist. They're slower and worse. The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person. Communication is faster with a mind-meld where you're looking at another person's face and picking up micro-expressions and body language. That's also the easiest thing to misinterpret. When you pipe through such a 'human-readable' lowest common denominator, you're actually setting the ground for a very dynamic and versatile channel. I don't know what the easy solution is to the problem you bring up, but I don't think your proposal is it. Maybe there is no easy solution. Several unicode characters to abstract away the tab character and the newline character to instead n dimensions of characters, and corresponding negative characters so you can put something on a 'line' out of channel, (negative tab comment, tab, text, newline repeat) would probably solve some issues but is too abstract to even discuss. What we have really isn't that bad.
- haberman 14y ago
  > The direction you're suggesting we take things is, in fact, a fuller API. Nothing about my proposal has anything to do with an API. I'm just proposing a structured stream of data instead of an unstructured one. > They're slower and worse. I am proposing avoiding a serialize/parse step between every pair of pipeline elements (or using a more structured/optimized format if a serialization step is desired). Doing less work cannot possibly be slower. > The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person. Communicating with a person is an endless process of content negotiation. What are you and I talking about? Using our interface of plain text, we could be talking about literally anything. I could suddenly start talking about kumquat farming in Russia. At that point you could decide to follow suit and weigh in with your opinions on kumquat farming, or you could stop talking to me altogether because I've gone off-topic. If I start speaking complete jibberish, you could try to learn the language that I'm speaking, or you could start doing something equally nonsensical. You and I can respond to unexpected communications in useful ways because we are fully autonomous, intelligent, sovereign beings that are capable of learning, creativity, and curiosity. I don't know about you, but I don't want my software to be autonomous or react to unexpected situations in unpredictable ways. Data processing software should be as simple, predictable, and deterministic as possible. To use your example, if I somehow got an email address in my list called "send $1000 to Lucy," I don't want PayPal to decide to get smart and interpret the invalid email address as a command to send money to Lucy.
nwmcsween 14y ago
You're being too practical within unix constraints - metadata doesn't need to be tied to a schema or format it only needs to have metadata. Tell me what the metadata _does_ not what it is and let the protocol / format decide how to format the data. E.g. object_1 is HTTP 1.1 is ...
pwpwp 14y ago
Problem with PBs is that the receiver needs a schema to parse. JSON or S-expressions obviate that need.
- Evbn 14y ago
  Sender can send the schema, which it obviously has.
omaranto 14y ago
For people who may not have heard about what happened to Microsoft's experimental Monad shell, you might want to mention it turned into an actual product called Powershell that comes with Windows 7. It's very nice, nice enough I wish there was a good linux port.
deleted 14y ago
[deleted]
oh_sigh 14y ago
find . -depth 1 -type 'f' -size +1M

anvandare 14y ago

Actually I've been considering/dreaming about a thing like this for a long while now (ever since I got started with jQuery and discovered how smooth I could sail through the DOM with it). The idea I came up with (I'm most likely not the first to think of it, so tell me if someone already implemented it) is a sort of JavaScript shell for *nix that'd work similar to jQuery: passing along (collections of) JavaScript objects with properties and functions. e.g.: var sum = 0; $("~").ls({"type": "file", "size": ">1M"}).each(function () {sum += $(this).size("mb");}); $.echo("total size of home directory: %d MB", sum);

stcredzero 14y ago
Or, one could write a library of standard parsers and serializers for the Unix tools that would parse and produce known JSON representations of data that could be passed between scripts.
- zem 14y ago
  i did something like that at my last job - the cluster manager was designed as a set of command line tools, and every program had a -j flag that would make it parse stdin as json and write json to stdout. we then wrote wrappers around all the linux utilities we were using, to enable the same behaviour. once the basic system was in place it let us experiment with new features very rapidly indeed because everything could be tested in isolation from the command line using automated test scripts. what was especially nice was that we could use the same test framework to chain several utilities together and test the combination, because it was all a black box with json going in and coming out.
- Evbn 14y ago
  That's recordstream
  - stcredzero 14y ago
    recordstream seems to concentrate on lots of formats. Really, it should concentrate on lots of tools, with a shared repository of parsers for tools. EDIT: Actually, they concentrate on JSON, but they also try to provide a generic set of tools for reading any format. I think this could be better structured to "just work."
noblethrasher 14y ago
Powershell on Windows does something like what you want: PS C:\Some Directory $sum = 0 PS C:\Some Directory dir | where { $_Length -gt 1MB } | %{ $sum += $_.Length }
- mey 14y ago

stcredzero 14y ago

haberman 14y ago
> JSON would do nicely as well. Yep, some friends of mine did this with JSON, but didn't make the schema explicit like I mean to: https://github.com/benbernard/RecordStream https://github.com/benbernard/RecordStream > Better yet, unify the shell with a virtual machine that is used to implement the OS, and have everything available as 1st class Objects. Please no. This is the Microsoft PowerShell approach, where everything is a .NET object. Once you start dictating representations of objects, you are dictating far too much about the implementation of individual pipeline nodes.
- nwmcsween 14y ago
  Why does the protocol define the objects? Shouldn't the object metadata should have enough information to transform the output? E.g. object is a URL vs protocol is a URL
- neutronicus 14y ago
  How can your tools all accept the same kind of structured data without dictating its representation? I don't get it.
  - jcmhn 14y ago
    Allow a shell variable to control output record and field separators. Default to space and newline if nothing is specified. ORS=: OFS=, ls -l # rw-r--r--:1:uname:gname:1025:Jun 1:somefile.txt,...,... Add a specifier for dates, usernames, groupnames, etc. DSF="%Y-%m-%d" ls -l

its_so_on 14y ago

I like your observation but I don't like your conclusion. I think the major thing missing from the world is easy piping a la Unix. You can't pipe your list of paying customer's email addresses to paypal at the command line with some switches - really, no matter how many switches you use - and bill each one the amount stated, because "paypal" is not a commandline app. You can't pipe the results of some long-running analysis to Twitter to announce that you've finished computing it, no matter how many switches you add, because twitter is not a commandline app. The direction you're suggesting we take things is, in fact, a fuller API. These exist. They're slower and worse. The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person. Communication is faster with a mind-meld where you're looking at another person's face and picking up micro-expressions and body language. That's also the easiest thing to misinterpret. When you pipe through such a 'human-readable' lowest common denominator, you're actually setting the ground for a very dynamic and versatile channel. I don't know what the easy solution is to the problem you bring up, but I don't think your proposal is it. Maybe there is no easy solution. Several unicode characters to abstract away the tab character and the newline character to instead n dimensions of characters, and corresponding negative characters so you can put something on a 'line' out of channel, (negative tab comment, tab, text, newline repeat) would probably solve some issues but is too abstract to even discuss. What we have really isn't that bad.

haberman 14y ago
> The direction you're suggesting we take things is, in fact, a fuller API. Nothing about my proposal has anything to do with an API. I'm just proposing a structured stream of data instead of an unstructured one. > They're slower and worse. I am proposing avoiding a serialize/parse step between every pair of pipeline elements (or using a more structured/optimized format if a serialization step is desired). Doing less work cannot possibly be slower. > The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person. Communicating with a person is an endless process of content negotiation. What are you and I talking about? Using our interface of plain text, we could be talking about literally anything. I could suddenly start talking about kumquat farming in Russia. At that point you could decide to follow suit and weigh in with your opinions on kumquat farming, or you could stop talking to me altogether because I've gone off-topic. If I start speaking complete jibberish, you could try to learn the language that I'm speaking, or you could start doing something equally nonsensical. You and I can respond to unexpected communications in useful ways because we are fully autonomous, intelligent, sovereign beings that are capable of learning, creativity, and curiosity. I don't know about you, but I don't want my software to be autonomous or react to unexpected situations in unpredictable ways. Data processing software should be as simple, predictable, and deterministic as possible. To use your example, if I somehow got an email address in my list called "send $1000 to Lucy," I don't want PayPal to decide to get smart and interpret the invalid email address as a command to send money to Lucy.

nwmcsween 14y ago

You're being too practical within unix constraints - metadata doesn't need to be tied to a schema or format it only needs to have metadata. Tell me what the metadata _does_ not what it is and let the protocol / format decide how to format the data. E.g. object_1 is HTTP 1.1 is ...

pwpwp 14y ago

Problem with PBs is that the receiver needs a schema to parse. JSON or S-expressions obviate that need.

Evbn 14y ago
Sender can send the schema, which it obviously has.

omaranto 14y ago

For people who may not have heard about what happened to Microsoft's experimental Monad shell, you might want to mention it turned into an actual product called Powershell that comes with Windows 7. It's very nice, nice enough I wish there was a good linux port.

deleted 14y ago

[deleted]

oh_sigh 14y ago

find . -depth 1 -type 'f' -size +1M