In 2003 I started working on a distributed file store for archival, (an "object store"). I was responsible for the metadata management component, which was impl

In 2003 I started working on a distributed file store for archival, (an "object store"). I was responsible for the metadata management component, which was implemented by a Java distribution layer over a postgres database on each node. I needed to operate on all nodes of the cluster. cssh opens a console on each node, and echoes commands to each. But that doesn't scale well past about 12 nodes, doesn't combine results from nodes, it really didn't work for me. I wrote a tool I called Object Shell, or osh. The idea is to combine commands by passing data over pipe-like connectors. Osh is written in Python, and the commands are customized by Python functions. E.g., to find all the files recursively inside the current directory, and add up all the sizes: osh ls -rf ^ f 'file: file.size' ^ red + $ osh: Invokes the tool ls -rf: List files (f) recursively (r), yielding a stream of Python file objects. ^: Pipe the stream to the next command. f: Apply a function to each item in the input stream, writing function output to the output stream. 'file: file.size': A function that returns the size of a file. red +: Reduction using +, combining all the numbers coming in. $: Print result to stdout. And then to do this for all files under /foo/bar, in all nodes of a cluster named mycluster, and to do a cluster-wide sum: osh @mycluster [ ls -rf ^ f 'file: file.size' ^ red + ] ^ f 'host, sum: sum' ^ red + $ @mycluster [ ... ]: Distribute the bracketed code to each node of mycluster. f 'host, sum: sum': The distributed command returns results that include the host name; drop the host name. red +: Sum the sums from each node. Osh is available here: https://github.com/geophile/osh https://github.com/geophile/osh