3 ms·
On the pipability front of software, I think a lot of that happens through APIs now, either directly linked or through various network APIs. The gotcha is that
by dhosek 4y ago
On the pipability front of software, I think a lot of that happens through APIs now, either directly linked or through various network APIs. The gotcha is that few APIs can handle the kind of asynchronicity that a pipe can. With pipes I can so something like:
oaktree | nutgrinder | cookiemaker | cookieeater
and as soon as each nut is ready from the oaktree, it can be consumed by the nutgrinder and it can send nut flour to the cookiemaker as its ready and when the cookiemaker has enough nut flour to make a cookie, it can send a cookie to the cookieeater. Four processes, acting in parallel¹ with little friction to make it happen.
Now, try to architect this in your favorite programming language.² I did see where Google proposed something (almost a decade ago) for C++ to allow this https://isocpp.org/files/papers/n3534.html https://isocpp.org/files/papers/n3534.html although I don’t know where it stands now. I’ve been thinking along something similar for finl for each of its stages so that, e.g., parsing and tokenization doesn’t need to be complete to start formatting, but it would be nice if there were a simple language facility to allow this.
REST APIs also fail at this insofar as you need to wait for a whole request to complete before you get a response rather than being able to send the request and get each chunk of data back as appropriate. We seem to have forgotten so much that we knew thirty years ago.
⸻
1. Although not necessarily. IIRC, good old fashioned DOS pipes would wait until each process was done before opening the next application in the chain.
2. Assuming, of course, that shell scripting isn’t your favorite programming language.
- pvg 4y agoTangential but just curious, do people make cookies out of acorns? Beside squirrel people that is.
- tonto 4y agoNative Americans as well as others have used acorns for flour and such https://en.m.wikipedia.org/wiki/Acorn https://en.m.wikipedia.org/wiki/Acorn
- charcircuit 4y ago>REST APIs also fail at this insofar as you need to wait for a whole request to complete before you get a response This isn't a restriction from REST. Nothing stops you from implementing an API that starts responding before your request is finished.
- ColonelPhantom 4y agoThe main problem I imagine is that you do not have a real way to deal with errors; you need to reply with 200 OK before you send anything else. There's also no separate channel for errors analogous to stderr.
- ahtihn 4y ago> Now, try to architect this in your favorite programming language. Isn't this trivial in java with streams? Or in C# with linq? Maybe I'm missing something.
- qsort 4y agoOr in C++20 with range adaptors. for (const auto &nut: oaktree | std::views::transform(nutgrinder) | std::views::transform(cookiemaker) | std::views::transform(cookieeater)) { // do something }
- throw_a_grenade 4y ago...or in Python with generators/iterators. Or in Lua with coroutines. Or in Rust with iterators... I guess most higher-ish languages at some point gain support/framework for something like this. But the point stands: there's something both simple and powerful with pipeline abstraction. I think of pipeline as a way of thought. Something that bridges the gap between "user" and "programmer". It's no wonder some of that "way of thinking" was implemented into programming languages (with obvious and non-obvious implications, not limited to memory requirements).
- dhosek 4y agoThese are all syntax though (except perhaps Lua’s coroutines—I don’t know enough to say about those). It’s not the syntax that’s the wonderful part about pipes, it’s the parallelism. See my other post with some Java code in this thread.
- dhosek 4y agoThe individual steps in a java stream don’t run in separate threads/processes. So, for example. given the following code: IntStream.of(1,2,3,4,5,6,7,8,9) .peek(System.out::println) .boxed() .map(val -> { try { Thread.sleep(1000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "a" + val; }) .peek(System.out::println) .map(val -> { try { Thread.sleep(3000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "b" + val; }) .forEach(System.out::println); the forEach essentially acts as the driver for each step so that you’ll see everything happening in sequence. The equivalent shell script version of this would output in the order (approximately) of 1, 2, 3, 4, 5, 6, 7, 8, 9, a1, a2, a3, ab1, a4, a5, a6, ab2, a7, a8, a9, ab3, ab4, ab5, ab6, ab7, ab8, ab9 and finish in roughly 28 seconds instead of the 36 seconds that the streams version does. Streams are syntactic sugar and do not provide parallelism on their own.¹ ⸻ 1. And don’t get me started on the hot mess that’s the Java parallelStream
- 314 4y agoGo handles this very cleanly with channels and goroutines.
- dhosek 4y agoI literally read this chapter in The Go Programming Language last night after posting this and thought to myself, well, shit, there it is.