2 ms·
Java or Scala? list.par().map(f);
by eternalban 15y ago
Java or Scala?
list.par().map(f);
- jebberjeb 15y agoyep, that's what I did!
- dxbydt 15y agoI've found that if the list is very large, using list.par.map(...) hits the memory limit & then simply hangs. I process upwards of a million facilities ( a facility is the exposure on a commercial loan a bank makes to a client ), trying to forecast their expected loss over 12 future quarters. I rewrote my code like so : val facs:List[Facility] = ...populated via jdbc query /// val N = 1000 facs.grouped(N).foreach( group => { group.par.map( fac => expectedloss(fac) }) Now the outer loop is sequential but the inner loop is parallel. It chugs along, and processes the entire million plus facilities! I get some control via the N. Currently my N is 1000 and all 8 cores running at 100% capacity, but I can slow down the workload with smaller N.