3 ms·

Dgsh – Directed Graph Shell

135 points by pcr910303 7y ago

dang 7y ago
A thread from 2017: https://news.ycombinator.com/item?id=13352659 https://news.ycombinator.com/item?id=13352659
- IggleSniggle 7y ago
  Thanks for posting that. There is good additional discussion there. Edit: SciPipe looked particularly interesting. https://github.com/scipipe/scipipe https://github.com/scipipe/scipipe
j88439h84 7y ago
Python's mario is another terminal-oriented tool with concurrent task execution, but with async concurrency rather than multiprocess parallelism. For example, to get a bunch of urls concurrently $ mario async-map 'await asks.get ! x.text' < urls.txt https://github.com/python-mario/mario https://github.com/python-mario/mario
- IggleSniggle 7y ago
  Was mario inspired by spotify's luigi? Coincidence? https://github.com/spotify/luigi https://github.com/spotify/luigi
Legogris 7y ago
Thanks for sharing - I was previously pondering a bit how a generic graph pipe shell syntax would look - I might try this out myself soon :)
sdan 7y ago
Looks pretty interesting (automatically deleting duplicate files). Bookmarking to see if I can utilize this!
SeriousM 7y ago
Dgsh sounds like dogshit... Sorry
- sdegutis 7y ago
  Weird, my first thought when reading it was a character from Morrowind...
keid 7y ago
1980, operating systems class, write a shell for UNIX. I thought I'd be clever and do more than simple pipes, so I created some syntax to enable the specification of a directed graph. It's pretty neat to see that someone has now done this industrial strength.
- kedean 7y ago
  I think it's a pretty natural idea. According to Brian Kernighan's memoir, the original vision for pipes was more like a mesh than the omnidirectional flow we ended up with: "Doug wanted to allow arbitrary connections in a sort of mesh of programs, but it was not obvious how to describe an unconstrained graph in a natural way, and there were semantic problems as well: data that flowed between programs would have to be queued properly, and queues could explode in an anarchic connection of programs. And Ken couldn't think of any real applications anyway." My first thought when reading over the docs for dgsh is always something similar...what's the real application? What does it gain me other than some mathematical elegance that I don't lose in harder to read scripts? The examples all pretty much collapse to doing things in parallel and collecting them via cat, which seems a little basic for something as abstract as a DAG.
elmolino89 7y ago
Interesting effort. On the other hand there is a bunch of other more popular and battle tested tools from Apache Airflow to a bioinformatics pipelines managers, such as Nextflow, Snakemake etc.

135 points by pcr910303 7y ago

dang 7y ago

A thread from 2017: https://news.ycombinator.com/item?id=13352659 https://news.ycombinator.com/item?id=13352659

IggleSniggle 7y ago
Thanks for posting that. There is good additional discussion there. Edit: SciPipe looked particularly interesting. https://github.com/scipipe/scipipe https://github.com/scipipe/scipipe

j88439h84 7y ago

Python's mario is another terminal-oriented tool with concurrent task execution, but with async concurrency rather than multiprocess parallelism. For example, to get a bunch of urls concurrently $ mario async-map 'await asks.get ! x.text' < urls.txt https://github.com/python-mario/mario https://github.com/python-mario/mario

IggleSniggle 7y ago
Was mario inspired by spotify's luigi? Coincidence? https://github.com/spotify/luigi https://github.com/spotify/luigi

Legogris 7y ago

Thanks for sharing - I was previously pondering a bit how a generic graph pipe shell syntax would look - I might try this out myself soon :)

sdan 7y ago

Looks pretty interesting (automatically deleting duplicate files). Bookmarking to see if I can utilize this!

SeriousM 7y ago

Dgsh sounds like dogshit... Sorry

sdegutis 7y ago
Weird, my first thought when reading it was a character from Morrowind...

keid 7y ago

1980, operating systems class, write a shell for UNIX. I thought I'd be clever and do more than simple pipes, so I created some syntax to enable the specification of a directed graph. It's pretty neat to see that someone has now done this industrial strength.

kedean 7y ago
I think it's a pretty natural idea. According to Brian Kernighan's memoir, the original vision for pipes was more like a mesh than the omnidirectional flow we ended up with: "Doug wanted to allow arbitrary connections in a sort of mesh of programs, but it was not obvious how to describe an unconstrained graph in a natural way, and there were semantic problems as well: data that flowed between programs would have to be queued properly, and queues could explode in an anarchic connection of programs. And Ken couldn't think of any real applications anyway." My first thought when reading over the docs for dgsh is always something similar...what's the real application? What does it gain me other than some mathematical elegance that I don't lose in harder to read scripts? The examples all pretty much collapse to doing things in parallel and collecting them via cat, which seems a little basic for something as abstract as a DAG.

elmolino89 7y ago

Interesting effort. On the other hand there is a bunch of other more popular and battle tested tools from Apache Airflow to a bioinformatics pipelines managers, such as Nextflow, Snakemake etc.