3 ms·
> In my case, I was reading a bunch of data off disk. maybe 10GB of data. And I was submitting all of that data into the ExecutorService work queue. Unsurprisin
by Terr_ 11y ago
> In my case, I was reading a bunch of data off disk. maybe 10GB of data. And I was submitting all of that data into the ExecutorService work queue. Unsurprisingly, the queue exploded and crashed my program.
In the spirit of brainstorming:
1) Abstract out the I/O step so that you can pass around a bunch of lightweight and lazy "I'll get the lines for you when you ask" objects. Then you can safely queue up a large number of them.
2) Choose to break the "job" apart into two parts, and submit the "get lines from filename" jobs into a different, smaller pool, with the results flowing into the larger (CPU-core-limited) pool.