3 ms·
> "The intriguing part here is that line and fields won't be cleared from memory until the garbage collector determines it's time to do so. We find ourselves wi
by franciscop 3y ago
> "The intriguing part here is that line and fields won't be cleared from memory until the garbage collector determines it's time to do so. We find ourselves with RAM filled with redundant strings and arrays that need cleaning. It's hardly surprising that our computer wasn't thrilled with this situation."
I don't have enough experience with super-optimizing Javascript, but wouldn't this be an easy solution? Instead of creating new variables in each loop and let the engine clear them, reuse the variables to avoid using more memory:
let line, fields, httpStatus;
for await (line of readlineStream) {
fields = line.split('\t');
httpStatus = Number(fields[5]);
if (httpStatus < 200 || httpStatus > 299) continue;
await dbWrite({
pathname: fields[7],
referrer: fields[8],
// ...
});
}
Edit: and as others suggest, keep streaming those records, instead of just pushing them into an array
- pkolaczk 3y agoNo, this still allocates a new string array per each line.split call. It's not really a problem with variables. The variables are likely on the stack anyways, but they only store pointers to data on the heap. If you call any function that allocates data on the heap you cannot get away with that by moving the variable declarations.