8 ms·

> Now, try to architect this in your favorite programming language. Isn't this trivial in java with streams? Or in C# with linq? Maybe I'm missing something.

• qsort 4y ago
  Or in C++20 with range adaptors. for (const auto &nut: oaktree | std::views::transform(nutgrinder) | std::views::transform(cookiemaker) | std::views::transform(cookieeater)) { // do something }
  • throw_a_grenade 4y ago
    ...or in Python with generators/iterators. Or in Lua with coroutines. Or in Rust with iterators... I guess most higher-ish languages at some point gain support/framework for something like this. But the point stands: there's something both simple and powerful with pipeline abstraction. I think of pipeline as a way of thought. Something that bridges the gap between "user" and "programmer". It's no wonder some of that "way of thinking" was implemented into programming languages (with obvious and non-obvious implications, not limited to memory requirements).
    • dhosek 4y ago
      These are all syntax though (except perhaps Lua’s coroutines—I don’t know enough to say about those). It’s not the syntax that’s the wonderful part about pipes, it’s the parallelism. See my other post with some Java code in this thread.
• dhosek 4y ago
  The individual steps in a java stream don’t run in separate threads/processes. So, for example. given the following code: IntStream.of(1,2,3,4,5,6,7,8,9) .peek(System.out::println) .boxed() .map(val -> { try { Thread.sleep(1000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "a" + val; }) .peek(System.out::println) .map(val -> { try { Thread.sleep(3000); } catch (InterruptedException e) { throw new RuntimeException(e); } return "b" + val; }) .forEach(System.out::println); the forEach essentially acts as the driver for each step so that you’ll see everything happening in sequence. The equivalent shell script version of this would output in the order (approximately) of 1, 2, 3, 4, 5, 6, 7, 8, 9, a1, a2, a3, ab1, a4, a5, a6, ab2, a7, a8, a9, ab3, ab4, ab5, ab6, ab7, ab8, ab9 and finish in roughly 28 seconds instead of the 36 seconds that the streams version does. Streams are syntactic sugar and do not provide parallelism on their own.¹ ⸻ 1. And don’t get me started on the hot mess that’s the Java parallelStream
  • krackers 4y ago
    But don't pipes also block when the buffer (~1-4kb) is filled until the output can be consumed? So as I understand it's still sequential in the bash version, only there's a bit of buffering at play.