3 ms·
> But is the code shitty? It looks like boring normal code to me. I'm not sure what point you're making here. 1. It should never be that stupidly slow 2. That
by orf 3y ago
> But is the code shitty? It looks like boring normal code to me. I'm not sure what point you're making here.
1. It should never be that stupidly slow
2. That should be obvious to anyone with a napkin and some math
3. Excessive allocations are highly unlikely to be the root cause of the slowness
4. Instead of debugging, they seemingly spent a lot of time putting it on Kubernetes, scaling it out to 25 instances and then finally rewriting a portion of it in Rust.
The whole thing smells of confusion and multiple issues being conflated together, with the rewrite in rust fixing these by virtue of deletion.
And if the root is was the GC, then some `--trace-gc` output in the post would be nice at the absolute minimum. Showing a for loop with a string split doesn't cut it.
- maurice2k 3y agoExactly. I'm not a super-expert in Node.js but worked a lot with it and also know a lot of internals. And I can definitely say: IO speed is pretty good. To be honest, I've never come across a language with bad IO speed and I've seen quite a lot of languages. I also did some benchmarks (like orf did) and speed is pretty good. Maybe Rust is a little bit better because it's nearer to syscalls than Node.js is but we're not talking about 760kb/s vs. 250mb/s. 200GB of logs and 25 instances means 8GB per instance. Not sure what they're doing, but processing of 8GB taking 3h is like doing it with my 80286 processor ;-) Even with inserting stuff in MongoDB this seems way to slow. And they only replaced the file IO part with Rust... I guess the records array is getting pretty huge which might result in swapping to disk. But then: don't let it happen, just flush it to the database or whatever. I believe the problem could have been solved with Node.js without issues.