5 ms·

I like your observation but I don't like your conclusion. I think the major thing missing from the world is easy piping a la Unix. You can't pipe your list of

I like your observation but I don't like your conclusion. I think the major thing missing from the world is easy piping a la Unix. You can't pipe your list of paying customer's email addresses to paypal at the command line with some switches - really, no matter how many switches you use - and bill each one the amount stated, because "paypal" is not a commandline app. You can't pipe the results of some long-running analysis to Twitter to announce that you've finished computing it, no matter how many switches you add, because twitter is not a commandline app. The direction you're suggesting we take things is, in fact, a fuller API. These exist. They're slower and worse. The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person. Communication is faster with a mind-meld where you're looking at another person's face and picking up micro-expressions and body language. That's also the easiest thing to misinterpret. When you pipe through such a 'human-readable' lowest common denominator, you're actually setting the ground for a very dynamic and versatile channel. I don't know what the easy solution is to the problem you bring up, but I don't think your proposal is it. Maybe there is no easy solution. Several unicode characters to abstract away the tab character and the newline character to instead n dimensions of characters, and corresponding negative characters so you can put something on a 'line' out of channel, (negative tab comment, tab, text, newline repeat) would probably solve some issues but is too abstract to even discuss. What we have really isn't that bad.

haberman 14y ago
> The direction you're suggesting we take things is, in fact, a fuller API. Nothing about my proposal has anything to do with an API. I'm just proposing a structured stream of data instead of an unstructured one. > They're slower and worse. I am proposing avoiding a serialize/parse step between every pair of pipeline elements (or using a more structured/optimized format if a serialization step is desired). Doing less work cannot possibly be slower. > The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person. Communicating with a person is an endless process of content negotiation. What are you and I talking about? Using our interface of plain text, we could be talking about literally anything. I could suddenly start talking about kumquat farming in Russia. At that point you could decide to follow suit and weigh in with your opinions on kumquat farming, or you could stop talking to me altogether because I've gone off-topic. If I start speaking complete jibberish, you could try to learn the language that I'm speaking, or you could start doing something equally nonsensical. You and I can respond to unexpected communications in useful ways because we are fully autonomous, intelligent, sovereign beings that are capable of learning, creativity, and curiosity. I don't know about you, but I don't want my software to be autonomous or react to unexpected situations in unpredictable ways. Data processing software should be as simple, predictable, and deterministic as possible. To use your example, if I somehow got an email address in my list called "send $1000 to Lucy," I don't want PayPal to decide to get smart and interpret the invalid email address as a command to send money to Lucy.
- Arelius 14y ago
  > I'm just proposing a structured stream of data instead of an unstructured one. That's basically what an API is.
  - mattdeboard 14y ago
    You are confusing the "how" (the API specification) with the "what" (the data the API exists to provide access to).
    - Arelius 14y ago
      > I'm experimenting with the idea of letting people specify the output of command-line utilities as a Protocol Buffer schema, for example: Looks like an API spec to me
  - darkarmani 14y ago
    So things like CSV, XML, and JSON are APIs now?
    - Arelius 14y ago
      Can any of those not be an interface to an application layer?
      - haberman 14y ago
        Plain text also "can be an interface to an application layer" (see SMTP, IMAP, POP, etc). Your argument is meaningless.
- 205guy 14y ago
  > Communicating with a person is an endless process of content negotiation. What are you and I talking about? Right. So how does grep know it's talking to ls about dir contents with a (disk) size field and not talking to ps about process table contents with a (mem) size field? Some sort of slower content negotiation, I presume. > cannot possibly I suggest that every time you think that, you double-check your assumptions. In this case, I think you're just pushing all the extra hard work off onto some other part of the algorithm.
wendsday 14y ago
I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? They often only achieve making other people's jobs more difficult. Text is what people can read. People do not read binary. When something goes wrong, debugging binary formats becomes insanely cumbersome. The concept of lines is a human one. It is how humans parse. If humans could parse without needing the concept of a "line" then, e.g., there would be no problems with programming in C which has very shaky support for the concept of "lines". But there are problems as we all know. It's proof that people do need to think in terms of "lines". Even though the computer does not need them. The only "problem" with this is that people are not computers. I'm not sure anyone outside of the most unrealistic nerds would agree that this is a "problem". 1. As for the paper, I can think of at least one tool/language to work with text that does keep data in a binary format while one performs a series of transformations. It's not line-based. And I would guess that Mr. Pike does not know how to use it. It's fast and efficient. Probably faster than sam.
- haberman 14y ago
  > I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? My problem is not with text per se, but with unstructured text. I'm fine with JSON in cases where efficiency is not a top concern. Let me ask you this; how would you do the equivalent of this hypothetical command? $ ls | structured-grep 'file.size > 1M' The answer is that you can't in today's world without writing a parser (or some code that calls readdir/stat manually). That is the problem with unstructured text. Text formats like CSV that seem simple actually end up being hugely complicated once you push them to their limits. Nothing is worse than software that breaks once something unexpected happens, like a string that contains an embedded comma.
  - e12e 14y ago
    "ls" prints a list of files. "find" knows about the filesystem and can print a list of files based on file metadata and a path (such as a filename). If you have a list of files (eg from "ls") -- you would need to look up the metadata you want to filter on; it is not part of ls' interface to give them to you directly: ls | find -size +1M On the other hand, if you want to make list of files and their sizes, that can be stored, sent over the network, etc -- and then filer that list, you can do: ls --size | awk 'file_blocks=$1 $file_blocks > 1024 { print $2 }' For most tasks, I think the fact that any human can look at the output from eg: "ls --size" and then produce a valid test dataset in the same format is more valuable than having to explicitly "cast" the metadata while processing.
nnnnni 14y ago
http://www.floodgap.com/software/ttytter/ http://www.floodgap.com/software/ttytter/ http://mike.verdone.ca/twitter/ http://mike.verdone.ca/twitter/ https://launchpad.net/twitter-cmdline/ https://launchpad.net/twitter-cmdline/ http://ruby-paypal.rubyforge.org/ http://ruby-paypal.rubyforge.org/ http://oreilly.com/pub/h/2222 http://oreilly.com/pub/h/2222 Apparently twitter and paypal ARE commandline programs!
- xtractinator 14y ago
  No, they have command line interfaces.

haberman 14y ago

> The direction you're suggesting we take things is, in fact, a fuller API. Nothing about my proposal has anything to do with an API. I'm just proposing a structured stream of data instead of an unstructured one. > They're slower and worse. I am proposing avoiding a serialize/parse step between every pair of pipeline elements (or using a more structured/optimized format if a serialization step is desired). Doing less work cannot possibly be slower. > The amazing thing about text is that it's a lowest-common denominator. Think of communicating with a person. Communicating with a person is an endless process of content negotiation. What are you and I talking about? Using our interface of plain text, we could be talking about literally anything. I could suddenly start talking about kumquat farming in Russia. At that point you could decide to follow suit and weigh in with your opinions on kumquat farming, or you could stop talking to me altogether because I've gone off-topic. If I start speaking complete jibberish, you could try to learn the language that I'm speaking, or you could start doing something equally nonsensical. You and I can respond to unexpected communications in useful ways because we are fully autonomous, intelligent, sovereign beings that are capable of learning, creativity, and curiosity. I don't know about you, but I don't want my software to be autonomous or react to unexpected situations in unpredictable ways. Data processing software should be as simple, predictable, and deterministic as possible. To use your example, if I somehow got an email address in my list called "send $1000 to Lucy," I don't want PayPal to decide to get smart and interpret the invalid email address as a command to send money to Lucy.

Arelius 14y ago
> I'm just proposing a structured stream of data instead of an unstructured one. That's basically what an API is.
- mattdeboard 14y ago
  You are confusing the "how" (the API specification) with the "what" (the data the API exists to provide access to).
  - Arelius 14y ago
    > I'm experimenting with the idea of letting people specify the output of command-line utilities as a Protocol Buffer schema, for example: Looks like an API spec to me
- darkarmani 14y ago
  So things like CSV, XML, and JSON are APIs now?
  - Arelius 14y ago
    Can any of those not be an interface to an application layer?
    - haberman 14y ago
      Plain text also "can be an interface to an application layer" (see SMTP, IMAP, POP, etc). Your argument is meaningless.
205guy 14y ago
> Communicating with a person is an endless process of content negotiation. What are you and I talking about? Right. So how does grep know it's talking to ls about dir contents with a (disk) size field and not talking to ps about process table contents with a (mem) size field? Some sort of slower content negotiation, I presume. > cannot possibly I suggest that every time you think that, you double-check your assumptions. In this case, I think you're just pushing all the extra hard work off onto some other part of the algorithm.

wendsday 14y ago

I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? They often only achieve making other people's jobs more difficult. Text is what people can read. People do not read binary. When something goes wrong, debugging binary formats becomes insanely cumbersome. The concept of lines is a human one. It is how humans parse. If humans could parse without needing the concept of a "line" then, e.g., there would be no problems with programming in C which has very shaky support for the concept of "lines". But there are problems as we all know. It's proof that people do need to think in terms of "lines". Even though the computer does not need them. The only "problem" with this is that people are not computers. I'm not sure anyone outside of the most unrealistic nerds would agree that this is a "problem". 1. As for the paper, I can think of at least one tool/language to work with text that does keep data in a binary format while one performs a series of transformations. It's not line-based. And I would guess that Mr. Pike does not know how to use it. It's fast and efficient. Probably faster than sam.

haberman 14y ago
> I could not agree more with your comment. There's not a problem with text. Why do people pretend there is? My problem is not with text per se, but with unstructured text. I'm fine with JSON in cases where efficiency is not a top concern. Let me ask you this; how would you do the equivalent of this hypothetical command? $ ls | structured-grep 'file.size > 1M' The answer is that you can't in today's world without writing a parser (or some code that calls readdir/stat manually). That is the problem with unstructured text. Text formats like CSV that seem simple actually end up being hugely complicated once you push them to their limits. Nothing is worse than software that breaks once something unexpected happens, like a string that contains an embedded comma.
- e12e 14y ago
  "ls" prints a list of files. "find" knows about the filesystem and can print a list of files based on file metadata and a path (such as a filename). If you have a list of files (eg from "ls") -- you would need to look up the metadata you want to filter on; it is not part of ls' interface to give them to you directly: ls | find -size +1M On the other hand, if you want to make list of files and their sizes, that can be stored, sent over the network, etc -- and then filer that list, you can do: ls --size | awk 'file_blocks=$1 $file_blocks > 1024 { print $2 }' For most tasks, I think the fact that any human can look at the output from eg: "ls --size" and then produce a valid test dataset in the same format is more valuable than having to explicitly "cast" the metadata while processing.

nnnnni 14y ago

http://www.floodgap.com/software/ttytter/ http://www.floodgap.com/software/ttytter/ http://mike.verdone.ca/twitter/ http://mike.verdone.ca/twitter/ https://launchpad.net/twitter-cmdline/ https://launchpad.net/twitter-cmdline/ http://ruby-paypal.rubyforge.org/ http://ruby-paypal.rubyforge.org/ http://oreilly.com/pub/h/2222 http://oreilly.com/pub/h/2222 Apparently twitter and paypal ARE commandline programs!

xtractinator 14y ago
No, they have command line interfaces.