14 ms·

Teeing, a hidden gem in the Java API

242 points by sidcool 5y ago

cma 5y ago
Would the Python3 version be: map(merge_func, zip(X,Y))?
- lovasoa 5y ago
  No. In idiomatic python, you would not use a functional approach for something like this, but if you really wanted to, you could do: def teeing(reducer1, reducer2): return lambda acc,elem: ( reducer1(acc[0],elem), reducer2(acc[1],elem) ) functools.reduce( teeing( (lambda l,elem: [elem] + l), (lambda l,elem: elem.price + l) ), cart.getProducts(), ([], 0) )
  - mjburgess 5y ago
    It is quaint how there is supposed to be an "idiomatic python" just as mypy, "pattern statements", walrus operators, etc. are being added. "Idomatic python" is dead, and lives now only as a pretty dumb ideology which states: everything must be phrased as a naively-typed naively-imperative program. Python is being made, retroactively, a bad imperative programming language: largely because its creator always thought that's what he'd made (he is wrong). It's a great tragedy "pythonic" has become this: a back-reaction against the times (of increasing adoption of functional programming driven by increasing data-transformation needs).
    - eskaytwo 5y ago
      That seems a very broad statement. Are there any such articles on this topic? I don’t see how Pythonic as a concept has to be mutually exclusive with new language features. To me, the features you cite seem to fit the Pythonic approach intuitively.
  - cma 5y ago
    Ah I didn't read their description well and didn't realize it was doing reductions.
- Doxin 5y ago
  It looks to be the inverse of that. i.e. split a single stream of stuff so two different things can consume it. In python you'd do something like that using itertools.tee, though I think the java API goes about it quite differently. https://docs.python.org/3/library/itertools.html#itertools.tee https://docs.python.org/3/library/itertools.html#itertools.t...
- deleted 5y ago
  [deleted]
danidiaz 5y ago
In Haskell terms, Collectors have an Applicative instance, and teeing corresponds to the liftA2 function: > liftA2 :: (x -> y -> z) -> Fold a x -> Fold a y -> Fold a z The "teeing" functionality is actually one of the "selling points" of the Collector-like library, not a hidden gem: > This module provides efficient and streaming left folds that you can combine using Applicative style. It's curious how the same abstraction can have different emphases across languages. Collectors are also Comonads: - You can always extract a value of the type that parameterizes the Collector, just by "closing" it. > extract :: Fold a x -> x - You could, in theory, "duplicate" a Collector<X> and get a Collector<Collector<X>>. This seems like a dumb function, but it would allow you for example to feed different Streams to the "same" collector, by duplicating it before consuming a Stream, the taking the result Collector, duplicating it again, passing it to another Stream... > duplicate :: Fold a x -> Fold a (Fold a x) http://hackage.haskell.org/package/foldl-1.4.11/docs/Control-Foldl.html http://hackage.haskell.org/package/foldl-1.4.11/docs/Control...
- cole-k 5y ago
  I imagine part of the reason is because Java has the idiomatic usage of list.stream().functionalOperations.collect(Collectors.toList()) whereas in Haskell you can just do whatever on your lists and it should fuse... or so I thought. Clearly I don't know too much about which way is the "right way" in Haskell since I haven't used Control.Foldl before and I just sort of assumed fusion would happen at least for most list operations.
- lmm 5y ago
  You should look into iteratees. It's the same insight: if you have the concept of something that receives values and eventually yields a value of a given type, that's a structure that has some very nice algebraic properties (e.g. they're monads).
  - danidiaz 5y ago
    I'm not sure you could implement a useful flatMap() for Collector-like types, at least without forcing the collector to hold all the received values in memory, which would defeat the purpose. It would be like a function that takes - a Collector that produces an X - a function that takes an X and returns a Collector that produces an Y and returns a Collector that produces an Y. The thing is: while being fed, the result Collector should first feed the initial Collector and, at some time, "switch" to the Collector produced by the function. But when to perform the switch?
ed_blackburn 5y ago
C# Aggregate(..) extension method (linq)?
- kaba0 5y ago
  That’s just reduce. Teeing is basically creating multiple streams from a single one and after doing something with those “branches”, aggregating them.
  - jayd16 5y ago
    Aggregate does the job just fine in this case. Multiple streams are not needed. .Aggregate(new PriceAndRows(), //here result is the PriceAndRows instance and next is a cart row. (result,next) => { result.Price+=next.Price; result.Rows.Add(next); return result; }) I'm trying to think of a better use case where multiple collectors would be cleaner than LINQ but LINQ has a lot of tools in the toolbox, SelectMany, Aggregate, temporary anonymous types, etc. However, even in the Java side, the example could be done with reduce alone, I think.
    - kaba0 5y ago
      > However, even in the Java side, the example could be done with reduce alone, I think. That’s my point: teeing itself is not aggregate/reduce. If after the branching the streams differ in size, reduce no longer applies, for example.
      - jayd16 5y ago
        But my point is that Aggregate and reduce can handle that in this case because they both can simply add to the PriceAndRows instance incrementally. You can sum the total and build the list in the body of the Aggregate/reduce method. Teeing is pointless here. There's no need to use another collector.
_old_dude_ 5y ago
There is a talk from Venkat Subramaniam, if you want to explore Collectors to death, Here is the part about teeing https://youtu.be/pGroX3gmeP8?t=5499 https://youtu.be/pGroX3gmeP8?t=5499
- vips7L 5y ago
  I don't think I've watched this one! Thanks!
Laremere 5y ago
I prefer imperative code over collector like APIs for two reasons, both visible in this example: 1. The intention of the code is obscured by more layers of abstraction, increasing complexity. 2. Small changes to what the code is conceptually doing tend to lead to larger changes to the actual code than with imperative code. For the first point, reading through this post and the previous post linked, the code is doing the following: Taking the entries of a map of product to count. Turning those entries into a different class, a row of each product and count. Then it collects this list on one hand. On the other hand, it sums the total, based on the per product cost and the count of the product in the cart. Then it combines the row list and the total cost into one object, returning it. Deconstructed, half of this code is just unnecessary complexity. A map of products to count, and a list of unique product/count tuples are theoretically identical. You can iterate over them, find specific products, etc. There /might/ be a reason to specifically desire a list; However why would that code be coupled with code to sum the cost of the cart? All in all, why is the code not just: public BigDecimal sumPrice(Cart cart) { BigDecimal sum = BigDecimal.ZERO; for (Map.Entry<Product,Integer> entry : cart.getProducts.entrySet()) { sum = sum.add(entry.getKey().getPrice().multiply(new BigInteger(entry.getValue()) } return sum; } For the second point, briefly: Consider how code would have to change to calculate a deal, such as Buy One Get One Free. Such code to do this calculation would add another layer to the collector with another function defined somewhere (or hidden in some other existing abstraction, such as OP's CartRow::getRowPrice) instead of just visible in the function that calculates a cart subtotal. If the deal relied upon concepts not limited to one row at a time, eg buy any two flavors of chips for %25 off, the proposed solution would have to be completely rewritten.
- gnud 5y ago
  You don't want the "buy one get one free" to apply directly to the cart subtotal. You want to show the customer which entry is free (or discounted), so they can see that the rebate is applied correctly. You probably want to check all 'rebate rules' attached to all products in the cart whenever one is added/removed. An active rebate will then modify the price of the cart item, and the sum method can be stupidly simple.
- hathym 5y ago
unwind 5y ago
Just in case it's not obvious to everyone, the name is a reference to the tee(1) [1] shell command, part of "coreutils" [2]. The manual page begins: tee - read from standard input and write to standard output and files It's a proper vintage tool, listing RMS as an author. Edit: less repetitive repetition, added coreutils link. [1]: https://man7.org/linux/man-pages/man1/tee.1.html https://man7.org/linux/man-pages/man1/tee.1.html [2]: https://www.gnu.org/software/coreutils/ https://www.gnu.org/software/coreutils/
- chriswarbo 5y ago
  The name "tee" is also a reference to a physical piece of plumbing which looks like the letter T, and allows connecting one "input" pipe to two "output" pipes.
throwaway4good 5y ago
I think this is made more complex and confusing than it should be. Notice that the author uses the word price (or row price) for two different things: the price of a product and the total (price * quantity) in a shopping cart line (cart row). The set of CartRow can be calculated as a straight forward entrySet().map(...) of the shopping cart in products map form (Map<Product, Integer>). The PriceAndRows object is really the total for all the cart rows and the union of all the cart rows. Both things can be calculated as a straight forward map / reduce.
kleiba 5y ago
Of course, the command evokes an association with the Unix tee command which fulfills a similar purpose: you can use it to pipe the output of some other command into a file while having it be printed to stdout at the same time. So, for instance: grep "error\|warning" log.txt | tee /tmp/issues.txt would find mentions of the terms "error" and "warning" in a file and both print them to the terminal window as well as write them to the file /tmp/issues.txt. This can be quite handy at times. According to Wikipedia, the name "tee" is a reference to a T-splitter used in plumbing, which makes sense.
BiteCode_dev 5y ago
At first, I though "this article just made me realized I didn't use the python itertools.tee to its full potential". But then I tried to think of a code where I would rather use this than a list comprehension or yield and a more manual control flow. And I couldn't. Those streams are elegant when the business logic flows perfectly like a river. Unfortunately, reality is messy, and production code will have matching, conditions, casting, extractions and transformations all over the place, leading to: - very long chains of calls - hard to change code when feedback pushes for it - limiting your tooling (specially debuggers) to stuff that have exceptional support of chaining, and it's rare
- PedroBatista 5y ago
  Completely agree, streams are good for when you have a ( you guess it ) stream of things where you need to do a limited and well defined amount of things.
- mrw 5y ago
  This article made me realise that I don't know how to perform step-through debugging on chained of operations on Streams in Java, so I looked it up. I found out that there's Stream Trace Dialog in IntelliJ IDEA [0]. I guess it proves your point that working with Stream API requires additional tooling in cases when the code doesn't 'just work'. [0]: https://www.jetbrains.com/help/idea/analyze-java-stream-operations.html https://www.jetbrains.com/help/idea/analyze-java-stream-oper...
  - chii 5y ago
    the ideal scenario with functional programming like this is to reason about the code, and may be algebraically model it mentally so that you "know" it works. But i find a lot of programmers don't do that - but instead write a first version which they don't truly understand (or understand completely), and then use stepped debugging to tweak the program until they get to a verison that works to their desired outcome.
    - BiteCode_dev 5y ago
mt5918 5y ago
Can somebody provide a link what this is all about? What's the objective?
galaxyLogic 5y ago
You can write either imperative for-loops or a set of connected stream-processors. An arbitrary set of connected "streamers" can always be converted to imperative code and I assume that is what is happening under the covers. But the reverse is not true, an arbitrary set of for-loops can not be translated into a set of streamers. Right? That means that the structure of your program is much more constrained when you compose it out of streams, than if you compose it out of arbitrary for-loops. And if you know that your program obeys a set of constraints imposed on it by the connected streams, the program becomes easier to understand, because you can RE-use your knowledge of how those streams always work to understand every component of the system, meaning every stream-component of it.
- rdsubhas 5y ago
  Multiple leaps of faith. "More constraints mean more easier to understand" is a little too reductive. Replace streams with "operator". All your program code can be written as operators like dot or plus, and since those operators gave constraints, then the whole program will be simpler, right? If that were true, we'll still be writing code in assembly, managing registers by hand. Because hey, just 256 registers means that you can reuse your knowledge of registers everywhere, which means programs will be easier to understand again, right? There are examples in threads which show real examples where plain code is better.
  - galaxyLogic 5y ago
    All streams work very similarly. Much more similarly than if you mix and match all kinds of operators together. Think Unix pipes they are easy to understand because they all behave similarly. Yes sometimes plain code is better sure but in cases where streams fit the job they are better. I guess the main point is that streams operate on multiple elements and they operate the same on every element. Therefore you don't need to reason what happens to every element that goes through the pipe. It is a bit like adding and multiplying matrices, you can understand the calculation without having to mentally follow how each matrix element is processed.
titzer 5y ago
In theory, the streaming APIs are meant to make it possible to operate on streaming database queries, where the entire result set is simply too large to fit into memory. A single, unified API over both in-memory datastructures and streaming queries is supposed to make it easy to do both with the same set of APIs without needing reason about the details, nor materialize the entire intermediate result in memory [1]. But I find the opposite to be true; it's hard to reason about where/when the entire result might need to be collected, and the streaming APIs cannot really match the query language underneath (SQL, e.g.). Instead, streaming APIs are frankly confusing, and less efficient than just doing the straightforward for loops, IMHO. Especially when things get complicated, with multiple joins and map/reduce. [1] Another way to achieve the incremental streaming result effect is to write everything in terms of generators. It is sooo much clearer to see a loop over a data structure and a yield to know how much the computation is actually incrementalized, IMHO.
dfee 5y ago
As an aside, I’ve recently been onboarded into a heavy Java ecosystem (backend, Java 8). What are the best resources to follow for growing in this language - and not just mapping patterns from one language to another?

242 points by sidcool 5y ago

cma 5y ago

Would the Python3 version be: map(merge_func, zip(X,Y))?

lovasoa 5y ago
No. In idiomatic python, you would not use a functional approach for something like this, but if you really wanted to, you could do: def teeing(reducer1, reducer2): return lambda acc,elem: ( reducer1(acc[0],elem), reducer2(acc[1],elem) ) functools.reduce( teeing( (lambda l,elem: [elem] + l), (lambda l,elem: elem.price + l) ), cart.getProducts(), ([], 0) )
- mjburgess 5y ago
  It is quaint how there is supposed to be an "idiomatic python" just as mypy, "pattern statements", walrus operators, etc. are being added. "Idomatic python" is dead, and lives now only as a pretty dumb ideology which states: everything must be phrased as a naively-typed naively-imperative program. Python is being made, retroactively, a bad imperative programming language: largely because its creator always thought that's what he'd made (he is wrong). It's a great tragedy "pythonic" has become this: a back-reaction against the times (of increasing adoption of functional programming driven by increasing data-transformation needs).
  - eskaytwo 5y ago
    That seems a very broad statement. Are there any such articles on this topic? I don’t see how Pythonic as a concept has to be mutually exclusive with new language features. To me, the features you cite seem to fit the Pythonic approach intuitively.
- cma 5y ago
  Ah I didn't read their description well and didn't realize it was doing reductions.
Doxin 5y ago
It looks to be the inverse of that. i.e. split a single stream of stuff so two different things can consume it. In python you'd do something like that using itertools.tee, though I think the java API goes about it quite differently. https://docs.python.org/3/library/itertools.html#itertools.tee https://docs.python.org/3/library/itertools.html#itertools.t...
deleted 5y ago
[deleted]

danidiaz 5y ago

In Haskell terms, Collectors have an Applicative instance, and teeing corresponds to the liftA2 function: > liftA2 :: (x -> y -> z) -> Fold a x -> Fold a y -> Fold a z The "teeing" functionality is actually one of the "selling points" of the Collector-like library, not a hidden gem: > This module provides efficient and streaming left folds that you can combine using Applicative style. It's curious how the same abstraction can have different emphases across languages. Collectors are also Comonads: - You can always extract a value of the type that parameterizes the Collector, just by "closing" it. > extract :: Fold a x -> x - You could, in theory, "duplicate" a Collector<X> and get a Collector<Collector<X>>. This seems like a dumb function, but it would allow you for example to feed different Streams to the "same" collector, by duplicating it before consuming a Stream, the taking the result Collector, duplicating it again, passing it to another Stream... > duplicate :: Fold a x -> Fold a (Fold a x) http://hackage.haskell.org/package/foldl-1.4.11/docs/Control-Foldl.html http://hackage.haskell.org/package/foldl-1.4.11/docs/Control...

cole-k 5y ago
I imagine part of the reason is because Java has the idiomatic usage of list.stream().functionalOperations.collect(Collectors.toList()) whereas in Haskell you can just do whatever on your lists and it should fuse... or so I thought. Clearly I don't know too much about which way is the "right way" in Haskell since I haven't used Control.Foldl before and I just sort of assumed fusion would happen at least for most list operations.
lmm 5y ago
You should look into iteratees. It's the same insight: if you have the concept of something that receives values and eventually yields a value of a given type, that's a structure that has some very nice algebraic properties (e.g. they're monads).
- danidiaz 5y ago
  I'm not sure you could implement a useful flatMap() for Collector-like types, at least without forcing the collector to hold all the received values in memory, which would defeat the purpose. It would be like a function that takes - a Collector that produces an X - a function that takes an X and returns a Collector that produces an Y and returns a Collector that produces an Y. The thing is: while being fed, the result Collector should first feed the initial Collector and, at some time, "switch" to the Collector produced by the function. But when to perform the switch?

ed_blackburn 5y ago

C# Aggregate(..) extension method (linq)?

kaba0 5y ago
That’s just reduce. Teeing is basically creating multiple streams from a single one and after doing something with those “branches”, aggregating them.
- jayd16 5y ago
  Aggregate does the job just fine in this case. Multiple streams are not needed. .Aggregate(new PriceAndRows(), //here result is the PriceAndRows instance and next is a cart row. (result,next) => { result.Price+=next.Price; result.Rows.Add(next); return result; }) I'm trying to think of a better use case where multiple collectors would be cleaner than LINQ but LINQ has a lot of tools in the toolbox, SelectMany, Aggregate, temporary anonymous types, etc. However, even in the Java side, the example could be done with reduce alone, I think.
  - kaba0 5y ago
    > However, even in the Java side, the example could be done with reduce alone, I think. That’s my point: teeing itself is not aggregate/reduce. If after the branching the streams differ in size, reduce no longer applies, for example.
    - jayd16 5y ago
      But my point is that Aggregate and reduce can handle that in this case because they both can simply add to the PriceAndRows instance incrementally. You can sum the total and build the list in the body of the Aggregate/reduce method. Teeing is pointless here. There's no need to use another collector.

_old_dude_ 5y ago

There is a talk from Venkat Subramaniam, if you want to explore Collectors to death, Here is the part about teeing https://youtu.be/pGroX3gmeP8?t=5499 https://youtu.be/pGroX3gmeP8?t=5499

vips7L 5y ago
I don't think I've watched this one! Thanks!

Laremere 5y ago

I prefer imperative code over collector like APIs for two reasons, both visible in this example: 1. The intention of the code is obscured by more layers of abstraction, increasing complexity. 2. Small changes to what the code is conceptually doing tend to lead to larger changes to the actual code than with imperative code. For the first point, reading through this post and the previous post linked, the code is doing the following: Taking the entries of a map of product to count. Turning those entries into a different class, a row of each product and count. Then it collects this list on one hand. On the other hand, it sums the total, based on the per product cost and the count of the product in the cart. Then it combines the row list and the total cost into one object, returning it. Deconstructed, half of this code is just unnecessary complexity. A map of products to count, and a list of unique product/count tuples are theoretically identical. You can iterate over them, find specific products, etc. There /might/ be a reason to specifically desire a list; However why would that code be coupled with code to sum the cost of the cart? All in all, why is the code not just: public BigDecimal sumPrice(Cart cart) { BigDecimal sum = BigDecimal.ZERO; for (Map.Entry<Product,Integer> entry : cart.getProducts.entrySet()) { sum = sum.add(entry.getKey().getPrice().multiply(new BigInteger(entry.getValue()) } return sum; } For the second point, briefly: Consider how code would have to change to calculate a deal, such as Buy One Get One Free. Such code to do this calculation would add another layer to the collector with another function defined somewhere (or hidden in some other existing abstraction, such as OP's CartRow::getRowPrice) instead of just visible in the function that calculates a cart subtotal. If the deal relied upon concepts not limited to one row at a time, eg buy any two flavors of chips for %25 off, the proposed solution would have to be completely rewritten.

gnud 5y ago
You don't want the "buy one get one free" to apply directly to the cart subtotal. You want to show the customer which entry is free (or discounted), so they can see that the rebate is applied correctly. You probably want to check all 'rebate rules' attached to all products in the cart whenever one is added/removed. An active rebate will then modify the price of the cart item, and the sum method can be stupidly simple.
hathym 5y ago

unwind 5y ago

Just in case it's not obvious to everyone, the name is a reference to the tee(1) [1] shell command, part of "coreutils" [2]. The manual page begins: tee - read from standard input and write to standard output and files It's a proper vintage tool, listing RMS as an author. Edit: less repetitive repetition, added coreutils link. [1]: https://man7.org/linux/man-pages/man1/tee.1.html https://man7.org/linux/man-pages/man1/tee.1.html [2]: https://www.gnu.org/software/coreutils/ https://www.gnu.org/software/coreutils/

chriswarbo 5y ago
The name "tee" is also a reference to a physical piece of plumbing which looks like the letter T, and allows connecting one "input" pipe to two "output" pipes.

throwaway4good 5y ago

I think this is made more complex and confusing than it should be. Notice that the author uses the word price (or row price) for two different things: the price of a product and the total (price * quantity) in a shopping cart line (cart row). The set of CartRow can be calculated as a straight forward entrySet().map(...) of the shopping cart in products map form (Map<Product, Integer>). The PriceAndRows object is really the total for all the cart rows and the union of all the cart rows. Both things can be calculated as a straight forward map / reduce.

kleiba 5y ago

Of course, the command evokes an association with the Unix tee command which fulfills a similar purpose: you can use it to pipe the output of some other command into a file while having it be printed to stdout at the same time. So, for instance: grep "error\|warning" log.txt | tee /tmp/issues.txt would find mentions of the terms "error" and "warning" in a file and both print them to the terminal window as well as write them to the file /tmp/issues.txt. This can be quite handy at times. According to Wikipedia, the name "tee" is a reference to a T-splitter used in plumbing, which makes sense.

BiteCode_dev 5y ago

At first, I though "this article just made me realized I didn't use the python itertools.tee to its full potential". But then I tried to think of a code where I would rather use this than a list comprehension or yield and a more manual control flow. And I couldn't. Those streams are elegant when the business logic flows perfectly like a river. Unfortunately, reality is messy, and production code will have matching, conditions, casting, extractions and transformations all over the place, leading to: - very long chains of calls - hard to change code when feedback pushes for it - limiting your tooling (specially debuggers) to stuff that have exceptional support of chaining, and it's rare

PedroBatista 5y ago
Completely agree, streams are good for when you have a ( you guess it ) stream of things where you need to do a limited and well defined amount of things.
mrw 5y ago
This article made me realise that I don't know how to perform step-through debugging on chained of operations on Streams in Java, so I looked it up. I found out that there's Stream Trace Dialog in IntelliJ IDEA [0]. I guess it proves your point that working with Stream API requires additional tooling in cases when the code doesn't 'just work'. [0]: https://www.jetbrains.com/help/idea/analyze-java-stream-operations.html https://www.jetbrains.com/help/idea/analyze-java-stream-oper...
- chii 5y ago
  the ideal scenario with functional programming like this is to reason about the code, and may be algebraically model it mentally so that you "know" it works. But i find a lot of programmers don't do that - but instead write a first version which they don't truly understand (or understand completely), and then use stepped debugging to tweak the program until they get to a verison that works to their desired outcome.
  - BiteCode_dev 5y ago

mt5918 5y ago

Can somebody provide a link what this is all about? What's the objective?

galaxyLogic 5y ago

You can write either imperative for-loops or a set of connected stream-processors. An arbitrary set of connected "streamers" can always be converted to imperative code and I assume that is what is happening under the covers. But the reverse is not true, an arbitrary set of for-loops can not be translated into a set of streamers. Right? That means that the structure of your program is much more constrained when you compose it out of streams, than if you compose it out of arbitrary for-loops. And if you know that your program obeys a set of constraints imposed on it by the connected streams, the program becomes easier to understand, because you can RE-use your knowledge of how those streams always work to understand every component of the system, meaning every stream-component of it.

rdsubhas 5y ago
Multiple leaps of faith. "More constraints mean more easier to understand" is a little too reductive. Replace streams with "operator". All your program code can be written as operators like dot or plus, and since those operators gave constraints, then the whole program will be simpler, right? If that were true, we'll still be writing code in assembly, managing registers by hand. Because hey, just 256 registers means that you can reuse your knowledge of registers everywhere, which means programs will be easier to understand again, right? There are examples in threads which show real examples where plain code is better.
- galaxyLogic 5y ago
  All streams work very similarly. Much more similarly than if you mix and match all kinds of operators together. Think Unix pipes they are easy to understand because they all behave similarly. Yes sometimes plain code is better sure but in cases where streams fit the job they are better. I guess the main point is that streams operate on multiple elements and they operate the same on every element. Therefore you don't need to reason what happens to every element that goes through the pipe. It is a bit like adding and multiplying matrices, you can understand the calculation without having to mentally follow how each matrix element is processed.

titzer 5y ago

In theory, the streaming APIs are meant to make it possible to operate on streaming database queries, where the entire result set is simply too large to fit into memory. A single, unified API over both in-memory datastructures and streaming queries is supposed to make it easy to do both with the same set of APIs without needing reason about the details, nor materialize the entire intermediate result in memory [1]. But I find the opposite to be true; it's hard to reason about where/when the entire result might need to be collected, and the streaming APIs cannot really match the query language underneath (SQL, e.g.). Instead, streaming APIs are frankly confusing, and less efficient than just doing the straightforward for loops, IMHO. Especially when things get complicated, with multiple joins and map/reduce. [1] Another way to achieve the incremental streaming result effect is to write everything in terms of generators. It is sooo much clearer to see a loop over a data structure and a yield to know how much the computation is actually incrementalized, IMHO.

dfee 5y ago

As an aside, I’ve recently been onboarded into a heavy Java ecosystem (backend, Java 8). What are the best resources to follow for growing in this language - and not just mapping patterns from one language to another?