3 ms·

I really like the idea of connecting commands using pipes to do "one-off" commands. But piping strings is dumb and limits what you can reasonably do. It's too h

I really like the idea of connecting commands using pipes to do "one-off" commands. But piping strings is dumb and limits what you can reasonably do. It's too hard extracting what you want from the strings. At my last company, I needed a tool to interact with the nodes of a cluster, including the databases on each node. Putting this all together, I wrote a tool named Object Shell (http://geophile.com/osh http://geophile.com/osh), which is available under the GPL. It takes the idea of piping, but exposes Python language constructs on the command line. Python objects, not strings, are piped between commands. For example, I can write a command line to: execute a database query on each node; bring back the results with each row as a python tuple; combine the stream of tuples from each node into one stream; and then operate on the stream. Or I can write a command line to: get a list of process objects once a second; extract and transform properties of each process; dump the stream of data into a database. The Python objects have Python types, so I can operate directly on files, processes, numbers, times, etc. instead of strings representing those types.

gaius 16y ago
This is why PowerShell rocks - superficially they're text pipes like any shell, but under the hood it's passing COM objects around. Nice.
- DrJokepu 16y ago
  Actually, those are CLR (.NET) objects, not COM objects. Sorry for nitpicking.
  - gaius 16y ago
    Heh, yep, old habits :-)
- CrLf 16y ago
  For certain values of nice. While I find the PowerShell to be more powerful when I'm scripting stuff, I still prefer the unix shell mostly because it passes around and operates over text. Operating over text means you can compose a pipe progressively using what you see, without having to think about how it is structured internally. Operating over text makes the normal usage of the unix shell faster and more efficient than the PowerShell. That and the terser syntax. With the PowerShell I find myself always having to check the type of objects are being passed around and which attributes I should care about, while on unix I just "grep" and "awk" and I'm done. The PowerShell is more of a scripting language than a shell, while the unix shell is more of a shell than a scripting language. Although both of them are both those things.
snprbob86 16y ago
"piping strings is dumb" I used to believe this. I was simultaneously learning Bash and PowerShell. I love PowerShell....(when I am stuck with Windows), but I'd never trade in my Bash shell. Piping objects is cool, but in reality, what is it really? It's just a programming language with a terse syntax for manipulating iterators. Let's compare... Powershell: Get-Process | where { $_.Handles -gt 200 } C# 3: Process.GetProcesses().Where(p => p.Handles > 200) Hmmm... what does piping objects even mean, really? OK, so now why piping strings is interesting: forced serialization. Every component of the transform can be debugged simply by deleting everything after that component. The new output goes to the screen or you can pipe it to a file. You get persistence for free. You can checkpoint your work and continue later. Imagine you are doing some very heavy duty processing that you only really need to do once. If you keep piping to a file along the way, you have backups and checkpoints. What if you want to send the results to another server? You're going to have to serialize your objects. If your objects are arbitrarily complex graphs, how do you serialize them? It can get hairy, fast. In summary: if you need to do arbitrary logic on arbitrary objects, use a programming language. If you need to transform some serialized data into some other serialized data: pipe strings.
- jerf 16y ago
  There's probably a happy middle. Basically what you're saying is that you want a weakly-typed format, which unpacking that statement yet further means that you want a format that lets you do pretty much anything to it. Types boil down to a set of restrictions on what the data can be and what you are permitted to do with it, and a hunk o' text (TM) is just about as untyped as you can get. But it might be nice to pass around JSON or something instead. It isn't quite as untyped, but on the grand scale of things it's not very typed.
  - snprbob86 16y ago
    I'm not saying that Unix's approach is the the final word on the subject. In fact, I quite agree with you: structured formats are a good idea. I, for one, would like to see --format options added to many of the standard Unix tools. I'd probably define --format= as a shortcut for specifying both --informat= and --outformat=. It would be great to have some sort of XPath or dotted object notation on the command line rather than hacking around with awk, sed, cut, etc. Rather than: ps -ef | awk -v p=$PPID 'p==$3{print $2}' Do: ps --format=xml | xquery "/process[ppid=$PPID]/pid" Or something like that. A bit more verbose, but far more understandable. If I were to design the future, every tool would have metadata which describes which input and output formats it supports and prefers, so that most --format arguments would be optional: ps | xquery "/process[ppid=$PPID]/pid" I'd prefer that 100 times over object piping.

gaius 16y ago

This is why PowerShell rocks - superficially they're text pipes like any shell, but under the hood it's passing COM objects around. Nice.

DrJokepu 16y ago
Actually, those are CLR (.NET) objects, not COM objects. Sorry for nitpicking.
- gaius 16y ago
  Heh, yep, old habits :-)
CrLf 16y ago
For certain values of nice. While I find the PowerShell to be more powerful when I'm scripting stuff, I still prefer the unix shell mostly because it passes around and operates over text. Operating over text means you can compose a pipe progressively using what you see, without having to think about how it is structured internally. Operating over text makes the normal usage of the unix shell faster and more efficient than the PowerShell. That and the terser syntax. With the PowerShell I find myself always having to check the type of objects are being passed around and which attributes I should care about, while on unix I just "grep" and "awk" and I'm done. The PowerShell is more of a scripting language than a shell, while the unix shell is more of a shell than a scripting language. Although both of them are both those things.

snprbob86 16y ago

"piping strings is dumb" I used to believe this. I was simultaneously learning Bash and PowerShell. I love PowerShell....(when I am stuck with Windows), but I'd never trade in my Bash shell. Piping objects is cool, but in reality, what is it really? It's just a programming language with a terse syntax for manipulating iterators. Let's compare... Powershell: Get-Process | where { $_.Handles -gt 200 } C# 3: Process.GetProcesses().Where(p => p.Handles > 200) Hmmm... what does piping objects even mean, really? OK, so now why piping strings is interesting: forced serialization. Every component of the transform can be debugged simply by deleting everything after that component. The new output goes to the screen or you can pipe it to a file. You get persistence for free. You can checkpoint your work and continue later. Imagine you are doing some very heavy duty processing that you only really need to do once. If you keep piping to a file along the way, you have backups and checkpoints. What if you want to send the results to another server? You're going to have to serialize your objects. If your objects are arbitrarily complex graphs, how do you serialize them? It can get hairy, fast. In summary: if you need to do arbitrary logic on arbitrary objects, use a programming language. If you need to transform some serialized data into some other serialized data: pipe strings.

jerf 16y ago
There's probably a happy middle. Basically what you're saying is that you want a weakly-typed format, which unpacking that statement yet further means that you want a format that lets you do pretty much anything to it. Types boil down to a set of restrictions on what the data can be and what you are permitted to do with it, and a hunk o' text (TM) is just about as untyped as you can get. But it might be nice to pass around JSON or something instead. It isn't quite as untyped, but on the grand scale of things it's not very typed.
- snprbob86 16y ago
  I'm not saying that Unix's approach is the the final word on the subject. In fact, I quite agree with you: structured formats are a good idea. I, for one, would like to see --format options added to many of the standard Unix tools. I'd probably define --format= as a shortcut for specifying both --informat= and --outformat=. It would be great to have some sort of XPath or dotted object notation on the command line rather than hacking around with awk, sed, cut, etc. Rather than: ps -ef | awk -v p=$PPID 'p==$3{print $2}' Do: ps --format=xml | xquery "/process[ppid=$PPID]/pid" Or something like that. A bit more verbose, but far more understandable. If I were to design the future, every tool would have metadata which describes which input and output formats it supports and prefers, so that most --format arguments would be optional: ps | xquery "/process[ppid=$PPID]/pid" I'd prefer that 100 times over object piping.