4 ms·

Following you in the abstract, but still a bit confused. * "unbounded array" - are you referring to `records`? * If so, is the memory of `records` too large?

Following you in the abstract, but still a bit confused. * "unbounded array" - are you referring to `records`? * If so, is the memory of `records` too large? Or is it that `records[i].pathname` has a reference to `fields[7]`, AND v8 will retain all copies of `fields` until the loop is done? Do you think this code would be more GC-friendly? ``` let count = 0; const MAX = 1000; const flushedRecords = []; for await (const line of readlineStream) { const fields = line.split('\t'); const httpStatus = Number(fields[5]); if (httpStatus < 200 || httpStatus > 299) continue; records.push({ pathname: fields[7], referrer: fields[8], // ... }); if(count == MAX || isLastLine(line)){ flushedRecords.push([...records]); records = []; count = 0; } else { count++; } } ``` Or were you thinking something totally different?

orf 3y ago
We know a few concrete things: 1. It parses 200gb at an average rate of 700kb/s 2. The job splits strings into a format that is loaded into mongodb 3. There where OOM errors that went away with a smaller workload None of this points to an inherent issue with NodeJS and the way it references memory. The speed issue alone points to a different problem root cause, and the post is completely lacking in concrete analysis. If v8 is being dumb and retains references to huge strings, then understanding this is the case and forcing a copy (and thus enabling the gc to reclaim data) is quicker and easier than any of their attempts at fixing the issue. Splitting a small TSV file and loading it into mongo can be done with even the most memory inefficient language without issue.
- smaudet 3y ago
  "and the post is completely lacking in concrete analysis." I do agree here, but I think it's telling that the rust solution is, without analysis, just better. Is this the best developer or best case study? Perhaps not, but it is interesting in that little analysis was needed. In the case where you have some constraint placed on you (running in the browser, company policy forbids rust e.g.), the analysis is worthwhile. Otherwise, assuming the fix didn't take long (relative to the requisite analysis and dev training), the decision to switch to a less shoot-yourself-in-the-foot language seems reasonable to me.
  - tomhallett 3y ago
    I think we can all agree that the blog article, while interesting and tells a story about the way the author solved a problem, does not have high enough rigor to serve as a benchmark for "don't do this, do this instead", or "this language is bad at this, and this one is better". Those types of statements would require much more rigor and A vs B comparisons where code A and code B are the same, just in different languages. BUT, what I am interested in - if I'm using javascript to parse a large file line by line, plucking off specific fields like the example does, what can I try to see if it's more GC-friendly? Obviously the answer is always some flavor of "profile it" and "it depends". If the blog author could have gotten a big boost in performance by adding 10 lines of js code to their loop, that is something all readers of the article could benefit from. I will play around with it and see what I can learn about the V8 GC. For reference: here is a talk about profiling the ruby gc, which helped me fix a memory leak at work (https://www.youtube.com/watch?v=kZcqyuPeDao https://www.youtube.com/watch?v=kZcqyuPeDao).
  - Karrot_Kream 3y ago
    There's a lot of other considerations when adding a new language. Questions come up such as: * How long does it take to ramp up on the new code base (author notwithstanding as a user of Rust outside of the company)? * If the author of the code is sick and there's a bug, or logic needs to be added, how hard would it be for someone else to do this work? * What is the build story for the Rust code like? How does it integrate with existing CI? * How do you store the Rust library? Some artifact store that Node can interact with? Without more details, unless the company was already writing Rust, this doesn't seem like a reasonable choice at all. Doing some profiling and experimenting with allocation, along with comments as to why you're doing something that's not naive (e.g. appending to a large array) seems much cheaper than doing all this Rust work. I've been in situations at $WORK where we were locked into using Python for certain code. Rewriting portions of it in Rust (if it needed running in the runtime) or Java (if it was across a service boundary) would have been enjoyable. But answering all those questions made us stick to Python. And surely, in most of those instances, we never really came back to those hot loops again so it never needed subsequent work. There were a few cases where we did find ourselves constantly trying to eke out more performance and in most of those cases a rewrite did end up going through, but those are rare and part of the craft of software engineering is to identify these areas and get ahead of rewrites early enough to not waste cycles.

orf 3y ago

We know a few concrete things: 1. It parses 200gb at an average rate of 700kb/s 2. The job splits strings into a format that is loaded into mongodb 3. There where OOM errors that went away with a smaller workload None of this points to an inherent issue with NodeJS and the way it references memory. The speed issue alone points to a different problem root cause, and the post is completely lacking in concrete analysis. If v8 is being dumb and retains references to huge strings, then understanding this is the case and forcing a copy (and thus enabling the gc to reclaim data) is quicker and easier than any of their attempts at fixing the issue. Splitting a small TSV file and loading it into mongo can be done with even the most memory inefficient language without issue.

smaudet 3y ago
"and the post is completely lacking in concrete analysis." I do agree here, but I think it's telling that the rust solution is, without analysis, just better. Is this the best developer or best case study? Perhaps not, but it is interesting in that little analysis was needed. In the case where you have some constraint placed on you (running in the browser, company policy forbids rust e.g.), the analysis is worthwhile. Otherwise, assuming the fix didn't take long (relative to the requisite analysis and dev training), the decision to switch to a less shoot-yourself-in-the-foot language seems reasonable to me.
- tomhallett 3y ago
  I think we can all agree that the blog article, while interesting and tells a story about the way the author solved a problem, does not have high enough rigor to serve as a benchmark for "don't do this, do this instead", or "this language is bad at this, and this one is better". Those types of statements would require much more rigor and A vs B comparisons where code A and code B are the same, just in different languages. BUT, what I am interested in - if I'm using javascript to parse a large file line by line, plucking off specific fields like the example does, what can I try to see if it's more GC-friendly? Obviously the answer is always some flavor of "profile it" and "it depends". If the blog author could have gotten a big boost in performance by adding 10 lines of js code to their loop, that is something all readers of the article could benefit from. I will play around with it and see what I can learn about the V8 GC. For reference: here is a talk about profiling the ruby gc, which helped me fix a memory leak at work (https://www.youtube.com/watch?v=kZcqyuPeDao https://www.youtube.com/watch?v=kZcqyuPeDao).
- Karrot_Kream 3y ago
  There's a lot of other considerations when adding a new language. Questions come up such as: * How long does it take to ramp up on the new code base (author notwithstanding as a user of Rust outside of the company)? * If the author of the code is sick and there's a bug, or logic needs to be added, how hard would it be for someone else to do this work? * What is the build story for the Rust code like? How does it integrate with existing CI? * How do you store the Rust library? Some artifact store that Node can interact with? Without more details, unless the company was already writing Rust, this doesn't seem like a reasonable choice at all. Doing some profiling and experimenting with allocation, along with comments as to why you're doing something that's not naive (e.g. appending to a large array) seems much cheaper than doing all this Rust work. I've been in situations at $WORK where we were locked into using Python for certain code. Rewriting portions of it in Rust (if it needed running in the runtime) or Java (if it was across a service boundary) would have been enjoyable. But answering all those questions made us stick to Python. And surely, in most of those instances, we never really came back to those hot loops again so it never needed subsequent work. There were a few cases where we did find ourselves constantly trying to eke out more performance and in most of those cases a rewrite did end up going through, but those are rare and part of the craft of software engineering is to identify these areas and get ahead of rewrites early enough to not waste cycles.