7 ms·
Dgsh – Directed graph shell
- uncletaco 1y agoHello. In English this makes me think of the phrase “dog shit”. Not sure if that’s intentional or not.
- pentaphobe 1y agoSecond English speaker here who didn't make that connection at all
- nasretdinov 1y agoEnglish is my third language and I can confirm I didn't even think about this
- rirze 1y agoSame english is my fourth language and it didn't even appear to me
- DSpinellis 1y agoAuthor of dgsh here. This is definitely not what I had in mind.
- DonHopkins 1y agoYeah, and the Slack logo designers claim they didn't have penises or swastikas on their mind, but here we are. https://www.buzzfeednews.com/article/nicolenguyen/slack-new-logo-design-backlash https://www.buzzfeednews.com/article/nicolenguyen/slack-new-...
- lucideer 1y agoAnother English speaker data point here & I actually read dogshit before I read dgsh.
- DonHopkins 1y agoThat's what I think when I hear "bash".
- kbr2000 1y agobatshit?
- dotnetcarpenter 1y agoThat's literally the word for shit in Norwegian: bæsj https://upload.wikimedia.org/wikipedia/commons/6/69/Nb-b%C3%A6sj.ogg https://upload.wikimedia.org/wikipedia/commons/6/69/Nb-b%C3%...
- cestith 1y agoThe word “dash” is a word for shit in English (as in dashboard - literally the board on a buggy or wagon to deflect the horse droppings). That doesn’t keep a shell from being named that. Of course, dash also means to move quickly so it’s not the only meaning. Moving quickly seems to be the inspiration for the shell’s name.
- nasretdinov 1y agoWell, bash as a shell is ok. Writing complex scripts on the other hand...
- goldenCeasar 1y agoNow I can't unconnect this, I hope OP was aware because now he wont forget too.
- em-bee 1y agoand no matter how much i try, i can't make the connection. best i can come up with is dogshell, and even that is a stretch. phew...
- AdieuToLogic 1y ago> In English this makes me think of the phrase “dog shit”. In English, this makes me think of the phrase "dig shell". I guess we just have different things on our minds... :-p
- DonHopkins 1y agoOr shoes.
- SoftTalker 1y agoFirst thing I thought of also.
- jimbokun 1y agoThis is very interesting, but I'm wondering how it compares to just using a dynamic language like Python or Ruby for the same tasks. Curious how the line count to express the same tasks would come out.
- PaulHoule 1y agoThere is a lot of stuff for Python which follows the "express computation as a dag" approach, especially Apache Airflow https://airflow.apache.org/ https://airflow.apache.org/
- DSpinellis 1y agoApache Airflow solves a very different problem. Its DAGs are static dependencies between sequentially executed processing steps, whereas the DAGs of dgsh express live direct data flows.
- PaulHoule 1y agoYeah, there are also the boxes and lines tools like https://www.knime.com/ https://www.knime.com/ which have their own subculture. You could solve the same problems they do with pandas and scikit-learn but people who use those tools would never use pandas and scikit-learn and vice versa. Circa 2015 I was thinking those tools all had the architectural flaw that they pass relational rows over the lines as opposed to JSON objects (or equivalent) which means you had to realize joins as highly complex graphs where things that seem like local concerns to me require a global structure and where what seems like a little change to management changes the whole graph in a big way. I found the people who were buying up that sort of tools didn’t give a damn because they thought customers demanded the speed of columnar execution which our way couldn’t deliver. I made a prototype that gave the right answers every time and then went to work for a place which had some luck selling their own version that didn’t always give the right answers because: they didn’t know what algebra it supported, didn’t believe something like that had an algebra, and didn’t properly tear the pipeline down at the end.
- jpitz 1y ago
- esafak 1y agoThis would have been great 10-20 years ago, or even at the coining of Unix pipes. By today's standards, however, the syntax feels clunky and dated. I'd like to see contemporary shells like nushell and elvish copy these ideas, with attribution of course, in a more modern way. That is the best way I can see to honor this stagnant project: https://github.com/dspinellis/dgsh https://github.com/dspinellis/dgsh
- hnlmorg 1y agoMurex has had this capability for years. (https://github.com/lmorg/murex https://github.com/lmorg/murex) I’m on my phone at the moment and cooking so cannot type any examples, but if I get time, I’ll throw together some comparisons later tonight
- esafak 1y agoI could not find any mention of DAGs or directed acyclic graphs in the documentation.
- hnlmorg 1y agoYeah it’s not technically DAG since it uses iteration, but then dgsh will use iteration under the hood too. However Murex does support CSP-style concurrency. So while there’s no syntax sugar for writing graphs, you can very easily create adhoc pipes and pass them around instead of using stdout / stderr. So it wouldn’t actually take much to refine that with some DAG-friendly syntax. In fact maybe that can be my next project…
- DSpinellis 1y agoI'm curious: what do you mean by "dgsh will use iteration under the hood too"? Dgsh does several things under the hood, but I wouldn't characterize any of them as iteration.
- hnlmorg 1y ago
- politician 1y agoA solution to the One Billion Row Challenge (1brc.dev) written in dgsh would be a interesting as a benchmark.
- DSpinellis 1y agoNice benchmark! This is a (not at all efficient) awk one-liner. awk -F\; ' $2 > max[$1] { max[$1] = $2 } !($1 in min) || $2 < min[$1] { min[$1] = $2 } { sum[$1] += $2; count[$1]++} END { for (n in sum) printf("%s=%.1f/%.1f/%.1f, ", n, min[n], sum[n] / count[n], max[n])}' Can't see how dgsh could be applied to it.
- dang 1y agoRelated. Others? Dgsh – Directed Graph Shell - https://news.ycombinator.com/item?id=21700014 https://news.ycombinator.com/item?id=21700014 - Dec 2019 (11 comments) Dgsh – Directed graph shell - https://news.ycombinator.com/item?id=13352659 https://news.ycombinator.com/item?id=13352659 - Jan 2017 (51 comments)
- byearthithatius 1y agoInteresting. What are the benefits of thinking of data pipelines in terms of a DAG? Why cant it be cyclical with exit conditions?
- DSpinellis 1y agoA nicer syntax and a lower probability of deadlocks.
- UltraSane 1y agoAs someone who loves graphs and Neo4j I wish I had though of this.
- willjp 1y agoThis is the nerdiest thing I've ever seen, and I absolutely adore it.
- gigatexal 1y agoSame! As a data engineer I have long wished we could get away from airflow and move back to Unix pipelines. This is really cool.
- reddit_clone 1y agoDoes the build work on Mac OSX?