4 ms·

Java's parallel stream is a cool feature, but it's suitable only for small batches of data. Once i tried streams with lambdas to process a large corpus of data

by haddr 12y ago

Java's parallel stream is a cool feature, but it's suitable only for small batches of data. Once i tried streams with lambdas to process a large corpus of data (processing many items in a multi-stage manner). While the code was clean and nice, the performance was horrible. A lot of thread waits, etc. Much worse than handling the whole process on my own with thread pools. Also there are many catches: for instance, when you start your stream with 1 item there will be only max 1 parallel thread to handle the data, and it doesn't matter if you do some fancy data split on some stage of the stream.

LoSboccacc 12y ago
Same experiences here. Had to process large number of texts to extract chains, said why not and used java streams. Total failure, could not even get one core feed consistently because they where all fighting for resources. Maybe could work for Non local or in memory data, but I rebuilt it with queues and a thread pool and it was easy way faster.
- PaulHoule 12y ago
  In terms of performance you really need to minimize contention. If you build something that sends things round-robin to different spliterators you are going to have horrible performance, you really need to batch 10,000 or so lines and send those to different spliterators. Also for anything that is pipelined there is ultimately going to be a bottleneck at the weakest point, which could be the spliterator factory.
smoyer 12y ago
I don't necessarily agree ... parallel streams can handle immense amounts of data if they're processing a suitable data set. The problem that's being described was prevalent when threads were being created manually too. Like anything else, the tool needs to be matched to the job. We processed about 32E9 pieces of telemetry per day ... our parallel threads were normalizing and aggregating that data as well as preparing it for storage. If you have too many threads waiting, you've got too many threads. I did a significant amount of testing using threads (on a project I can't share) and found that generically, you should have n + 1 threads when you have n processor cores. If your processor I/O bandwidth is limited, you may find better performance with less processors. If you're creating something like a tick-store (Michael Stonebraker - newest Turing award winner), you'll need to tune the rest of the system to feed the processors data at the maximum rate too. We were using E15K (and another bigger machine I can't remember) and used processor affinity to make sure all but 3 of the processors were dedicated to parallel threads. The thread count was the number of dedicated processors plus one. Having too many threads leads to context churn ... think about what you're trying to accomplish and design the system appropriately. And of course, some problems need other solutions.

by haddr 12y ago

LoSboccacc 12y ago

Same experiences here. Had to process large number of texts to extract chains, said why not and used java streams. Total failure, could not even get one core feed consistently because they where all fighting for resources. Maybe could work for Non local or in memory data, but I rebuilt it with queues and a thread pool and it was easy way faster.

PaulHoule 12y ago
In terms of performance you really need to minimize contention. If you build something that sends things round-robin to different spliterators you are going to have horrible performance, you really need to batch 10,000 or so lines and send those to different spliterators. Also for anything that is pipelined there is ultimately going to be a bottleneck at the weakest point, which could be the spliterator factory.

smoyer 12y ago

I don't necessarily agree ... parallel streams can handle immense amounts of data if they're processing a suitable data set. The problem that's being described was prevalent when threads were being created manually too. Like anything else, the tool needs to be matched to the job. We processed about 32E9 pieces of telemetry per day ... our parallel threads were normalizing and aggregating that data as well as preparing it for storage. If you have too many threads waiting, you've got too many threads. I did a significant amount of testing using threads (on a project I can't share) and found that generically, you should have n + 1 threads when you have n processor cores. If your processor I/O bandwidth is limited, you may find better performance with less processors. If you're creating something like a tick-store (Michael Stonebraker - newest Turing award winner), you'll need to tune the rest of the system to feed the processors data at the maximum rate too. We were using E15K (and another bigger machine I can't remember) and used processor affinity to make sure all but 3 of the processors were dedicated to parallel threads. The thread count was the number of dedicated processors plus one. Having too many threads leads to context churn ... think about what you're trying to accomplish and design the system appropriately. And of course, some problems need other solutions.