3 ms·
> However, to our surprise, we soon ran into out-of-memory issues, even when using utilities like p-limit to limit ourselves to just two jobs in parallel. If o
by leononame 3y ago
> However, to our surprise, we soon ran into out-of-memory issues, even when using utilities like p-limit to limit ourselves to just two jobs in parallel.
If only two async jobs already run OOM, your issue is probably bad code, not node.js. From the code snippet posted, it looks like they do a `records.push(...)`, whereas it probably would've been more efficient to just continue using streams downstream as well. Additionally, using a transform stream to split the string rather than doing it in a loop might've yielded better results because the previous row could be cleaned up. Obviously, this is just poking in the dark, but such stark difference in numbers in the Rust implementation leads me to believe that they changed some other fundamental part of the design (e.g. streaming the results to the database rather than inserting everything in one big batch at the end).