15 ms·

My Node.js is a bit Rusty

176 points by qsantos 3y ago

jasfi 3y ago
Related: you can write a Node.js module in Nim with Denim: https://github.com/openpeeps/denim https://github.com/openpeeps/denim
skybrian 3y ago
Blink and you missed the interesting bit, which is how they hooked them together. Looks like it's with this project: https://napi.rs/ https://napi.rs/ That project seems to be creating platform-specific DLL's, which I guess doesn't matter too much if it's wrapped up nicely in an npm package and you're deploying Docker files on Linux anyway? But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. Alternatively, writing very C-like code using a Uint8Array and not allocating in the loop might have fixed it too? But if you don't use JavaScript string libraries, it's not much like writing JavaScript anymore.
- leononame 3y ago
  Yeah, I think Uint8Array might've been a strong solution. But I've got the feeling this isn't the whole story. Unless the TSV is extremely wide, this alone feels a bit insignificant. I don't know the details so this is pure speculation, but using a transform stream on each row to split it up might've been enough to reduce memory pressure. I've done a bit of data manipulation in node and while imo js is not the ideal language for the job, it can be very fast. Strings are hard though, I'll admit that
- eyelidlessness 3y ago
  > But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. This is true, but N-API will generally perform better (sometimes significantly, usually in exactly the same CPU-bound scenarios where WASM will perform better than V8). And NAPI-RS is quite good at paving the happy path for multi-target packages.
- pimeys 3y ago
  > But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. If the database IO happens in Rust, the option with WebAssembly is fetch, meaning HTTP and the Atlas-only Data API. If the IO is done in JS, you pay some extra CPU time transferring UTF-16 strings over the border and copy them as UTF-8 in Rust.
bubblydoops 3y ago
Batch maintenance is a drag. However even just from the snippet of the purported memory/slow issue, I think theres a smell that was not revealed. If your constantly running out of memory you have a bad design. This seems like a native code agenda type post which I have a problem with. I will not code in native code ever again. Its not worth it anymore, its a horrible practice and it just enables bad code to pretend its good.
- satvikpendem 3y ago
  What do you mean "native code" and why will you not program in such code?
warkanlock 3y ago
In my experience, Node.js generally performs very well for most use cases although when it comes to data manipulation (or similar use cases), performance often goes down the hill, becoming the primary bottleneck nearly every time. It's simply not the right tool for the job. The beauty of this is that it allows teams to isolate and optimize just the problematic portion of the code, preventing them from going down a rabbit hole of extensive changes that can be more harmful than beneficial. NAPI is such a good tool for this; that I've been advocating the use of it to my team since forever.
- maccard 3y ago
  > In my experience, Node.js generally performs very well Hard disagree here. Node is extremely fast to get up and running with, has as close to 0 barrier to entry as possible, and doesn't suffer from slow startup times that plague JVM languages. It can also be fast for tight loops. But in the experience I've had with it"generally", it's orders of magnitude slower than other languages. A hello world with express.js and gin in go are a full order of magnitude apart, and the express app with default settings dies under a load test that I would describe as "very light". Stories like this thread are common in the JS world where technically it can be rewritten to be faster, but like with Bun, esbuild, and many other examples we've seen here on HN, a better language choice makes an enormous difference. And to me, the only time node is the right tool for the job is if you have a team of JavaScript developers who can't learn another language
  - nicoburns 3y ago
    Depends what you're comparing it to though. That same Node.js application will likely be an order of magnitude faster than a similar one in Python or Ruby.
  - warkanlock 3y ago
    So, when it comes to a technology that works well, I think it should be easy to adopt for the whole team, Node.js won here even against Go. In my experience, Go seems to fit quite well there, although there are plenty of developers out there without Go skills. But let's face it: In most of the projects we work on, clients don't need to handle millions of requests from distributed systems and geolocated servers all over the world. Generally, our clients just want a CRUD that is a little more elegant than Excel and can help them solve their real-world problems. IMO if someone from your team tells you that you cannot do something with an average of ~10,000 req/s [1] (around that number is the consensus nowadays for Javascript whereas Bun is highly superior to everyone here), you may be solving the problem in the wrong way. [1] That's around 36 million requests per hour!
Klaster_1 3y ago
Leveraging Rust to speed Node.js is cool and all, but why the author didn't attempt to optimize the JS code first? If too much intermediate objects negatively impact memory consumption, surely re-using the same structure or variables to hold the parsed result would have improved the performance? The benchmark tests not only the speedup brought up by Rust, but an optimization too, so it's unclear which impacts the performance the most.
- pkolaczk 3y ago
  That approach may work in short term but in long term you'd likely end up with unreadable and brittle code that is often still not as fast as it would be after the rewrite. You'd introduce additional complexity into your product to overcome limitations of the platform. And tricks like reusing objects can turn into really nasty bugs, especially in languages that don't come with any tools to control aliasing.
  - beezlewax 3y ago
    What does aliasing mean in this context?
    - pkolaczk 3y ago
      If you want to reuse an object to avoid allocation, you better not have a reference to that object from somewhere else. In JS AFAIK there is no way to detect such situation automatically.
      - Etheryte 3y ago
        Yes there is, it's called scope and it's super easy to keep track of? In fact, you have to go out of your way to share a reference if you scope it.
        pkolaczk 3y ago
        In languages like JS, Java, Python is trivial to accidentally make an object escape the scope by storing a reference in a field of another object that outlives the scope. And it can be another developer 3 months later that can introduce a bug, not knowing about your clever object reuse.
devjab 3y ago
> Promise.all, isn't that delightful?). However, to our surprise, we soon ran into out-of-memory issues, I think I must be misunderstanding something, but isn’t that exactly what you should expect? We use node.js to process solar plant data. Which is massive amounts of data as each individual solar panel will log information about its power generation and status every few seconds. We can then use this to figure out which solar panels have a birds nest on them, and so on. Node performs this task well enough, but it obviously can’t be aggregated. Even if you were to spread it out on 25 node instances “concurrently” with control shift you’d run into out-of-memory issues extremely fast. I guess you could up the upper limit on when node.js stops being cool with its own memory usage, but honestly we tend to simply avoid the use promise aggregation exactly because it tends to ”break” node. I think moving to Rust was probably a good idea for this. Go might’ve been a better option if you were going for a “modern” concurrent language, but whatever works! We tend to use C ourselves when node needs a little boost. But often Node can handle these things on its own. With the solar data we use worker threads to collect and process the data in real time, which is really parallel-execution and not concurrency, but it works well and it doesn’t get you into performance issues.
- go_prodev 3y ago
  That sounds really interesting. Out of curiosity, how do you use Node for this purpose? Is it building a real time dashboard? My company has possibly similar challenges. We have lots of sensors on equipment that streams information every few ms through a time series database from the vendor. It's used to build monitoring dashboards, trigger alarms and help engineers to prioritize and optimize equipment settings, and look for trends over time. It's such high volumes that capturing it and analyzing can be difficult, so we use a combination of vendor systems, Azure Synapse and Power BI. There are still lots of opportunities to improve, and I'd be interested to hear how you landed on your setup and what you do with the data.
  - devjab 3y ago
    It’s mainly for data transformation and transport. Solar inverters don’t deliver data in the same formats, there are no protocols and apparently most engineers in the solar panel business still think FTP (not SFTP) is fine but some will deliver a file per reading while others will aggregate and so on. We use node to gather the data and transform it into consumptive formats that our 3rd party AI vendor can work with, as well as store it in a range of SQL databases for our own PowerBI consumption. We do no frontend work. We don’t use node because it’s a good idea to use node for this. We use node because we try to use TypeScript as much as possible because we’re a small team and having a single language covering as much of our stack as possible makes it easier to work together.
leononame 3y ago
> However, to our surprise, we soon ran into out-of-memory issues, even when using utilities like p-limit to limit ourselves to just two jobs in parallel. If only two async jobs already run OOM, your issue is probably bad code, not node.js. From the code snippet posted, it looks like they do a `records.push(...)`, whereas it probably would've been more efficient to just continue using streams downstream as well. Additionally, using a transform stream to split the string rather than doing it in a loop might've yielded better results because the previous row could be cleaned up. Obviously, this is just poking in the dark, but such stark difference in numbers in the Rust implementation leads me to believe that they changed some other fundamental part of the design (e.g. streaming the results to the database rather than inserting everything in one big batch at the end).
lemper 3y ago
it's as if using fitting tool for solving problem's domain is a magic. good for you for saving that much of resource, though.
franciscop 3y ago
> "The intriguing part here is that line and fields won't be cleared from memory until the garbage collector determines it's time to do so. We find ourselves with RAM filled with redundant strings and arrays that need cleaning. It's hardly surprising that our computer wasn't thrilled with this situation." I don't have enough experience with super-optimizing Javascript, but wouldn't this be an easy solution? Instead of creating new variables in each loop and let the engine clear them, reuse the variables to avoid using more memory: let line, fields, httpStatus; for await (line of readlineStream) { fields = line.split('\t'); httpStatus = Number(fields[5]); if (httpStatus < 200 || httpStatus > 299) continue; await dbWrite({ pathname: fields[7], referrer: fields[8], // ... }); } Edit: and as others suggest, keep streaming those records, instead of just pushing them into an array
- pkolaczk 3y ago
  No, this still allocates a new string array per each line.split call. It's not really a problem with variables. The variables are likely on the stack anyways, but they only store pointers to data on the heap. If you call any function that allocates data on the heap you cannot get away with that by moving the variable declarations.
  - getcrunk 3y ago
    Okay, but it still gets rid of two other allocations (line and http)
    - pkolaczk 3y ago
      Line is a string. It allocates a fresh one on each line even if the variable is reused.
      - getcrunk 3y ago
        Lol shows how much I know!
orf 3y ago
> having 25 Node.js instances running for three hours to parse 200GB of data seems like an indication that something isn't quite right. That’s about 700kb/s. I’m all for playing with rust, but this post strikes me as engineers being distracted by shiny new things because they are only able to parse a TSV at 700kb/s? You should be able to understand why your current system performs so absolutely terribly before you decide to rewrite it in any language. Hand-waving at the garbage collector whilst seemingly doing very little to understand what’s happening doesn’t convince me that there isn’t another, possibly pretty obvious issue going on.
- pkolaczk 3y ago
  They debugged the problem to be excessive allocations in the tight loop of the TSV parser. If you allocate a separate object per each field, I can imagine how it can result in throughput counted in kB/s instead of GB/s. We have had very similar issues with Java - excessive heap allocations can bring the system to a crawl and there is often no easy way out.
  - DanielHB 3y ago
    A common technique (at least in Java, never seen it in nodejs) is to have a pool of pre-allocated objects and just reuse them as they are discarded. Essentially manual GC
    - DrDroop 3y ago
      Happens JS world aswel SimonDev did a video about it, this one I think: https://www.youtube.com/watch?v=easvMCCBFkQ https://www.youtube.com/watch?v=easvMCCBFkQ This channel is a real gem, show what talent really is: https://www.youtube.com/watch?v=a1L7k35EHIc https://www.youtube.com/watch?v=a1L7k35EHIc
      - KeyXiote 3y ago
        Thanks for the vid links, always looking for new channels with informative content.
mrighele 3y ago
I guess there is something I am missing here. The original code should consume not much more memory than the optimized one. Yes you're going to allocate a lot of more memory, but whatever doesn't finish in the record becomes garbage after every single line, so I can expect worse performance, but not using two order of magnitude more memory. I would see more problematic the fact that you're collecting all the records in a single array. Depending on how you use them it can be beneficial to use a generator and consume them incrementally.
- didntcheck 3y ago
  Agreed in terms of live memory, but if it's purely measuring heap size then it makes more sense. So it possibly could run with a much stricter memory limit, but with the tradeoff of more GC (and for a compacting GC, this would mean lower throughout, since the GC time is proportional only to live objects. May as well let the garbage pile up as much as you can and therefore do fewer collections)
kohlerm 3y ago
This is probably fixable in Node.js, just preallocate objects and do not excessively create new ones. But yeah using Node.js is probably not the best choice for this. As others have mentioned a JVM could easily handle this with some pre-allocation strategy.
getcrunk 3y ago
The author gives no info about how much memory these node instances had. I think that context is very important to know. Also as others have mentioned the js code could have been optimized. Id be very interested in a part two testing out some of the suggestions and sharing the benchmarks!
winrid 3y ago
Let me know if you want some help optimizing that aggregation query :) looks like a fun challenge.
brownfox 3y ago
I believe that the memory management aspect alone doesn't provide sufficient justification for rewriting the code in Rust. While it is possible to achieve reduced memory consumption through techniques like reusing a backing buffer and using allocUnsafe(), which require developers to take on memory management responsibilities, it might result in slightly more verbose code compared to a straightforward line-by-line split() approach. I am eager to explore the possibility of enhancing the line parsing process using Node.js. If you could provide a runnable example on a GitHub repository, it would enable me to compare its memory efficiency with the Rust rewrite version.
ifdefdebug 3y ago
I just skimmed through the article but it seems that they refactored from allocating and copying substring to pointing at the substrings within the original string (inside a hot path). So 25x performance boost should be expected just from that and has nothing to do with GC or the language used.
- sebstefan 3y ago
  200gB in 2.5 hours is still only 22mB/s, I feel like I've seen blog posts of people going several orders of magnitude faster than that when the bottleneck is the sequential read of a big file on an SSD
  - xmcqdpt2 3y ago
    I've written similar code before using absolutely bog standard idiomatic OOP Java. For a single threaded program I'd expect at least 1GB/s before any optimization. Most of the allocations in the loop that aren't the final records objects will get erased by JIT. They are doing something very wrong.
    - sebstefan 3y ago
      Let me rephrase this for you They're doing something slightly wrong that's making a non-bottleneck part of their code less optimal than it should be.
  - Kwpolska 3y ago
    Millibytes per second?
    - sebstefan 3y ago
      See in french we don't even have the possibility for that kind of confusion because we named a "byte" an "octet" which also reflects its relation with the number 8 That way they don't get gigabytes and gigabits abbreviated with the same letters and we don't have to rely on a capitalization convention Join the enlightened gigaoctet convention
philipwhiuk 3y ago
Why only do a daily import of CDN logs? If this process was an on-demand pipeline they wouldn't have to deal with such a storm of data every day. It would also mean people got stats 'closer-to-live' which feels like it would be useful.
andrewstuart 3y ago
Nodejs modules should be written in javascript/typescript.
o1y32 3y ago
Honestly doing line.split('\t') seems a newbie mistake. I don't know how many columns each line has but it could be a bottleneck. Wouldn't manually processing tab and non-tab character help, in the same way you don't load the entire log into memory but only read by line? When you are dealing with huge amount of data, you have to be extra careful, regardless of which language you use. That plus other optimization could make Node.js run very efficiently. Otherwise, I still need to be convinced that the "line" array is causing trouble but not anything else -- the article definitely doesn't provide detail about how they found this problem. It is almost funny that the author tried to put this on Kubernetes instead of even attempting to optimize the JS code. I wish I had that luxury and tell my boss I should use more resources instead of fixing my code.
- didntcheck 3y ago
  Yep. I've written code like this before for small inputs or one-off scripts, but I always thought it was obvious that doing it like this would be slow and produce lots of garbage. It's very nice and readable code, but for the hundreds of GBs they mention, it's clearly not ideal
jongjong 3y ago
About Node.js running out of memory, I wonder if they tried setting the max-old-space-size flag to increase the memory limit. Based on the descriptions, I wonder if the real problem was not Node.js but that all these solutions were implemented inside a constrained environment.
thefilmore 3y ago
Before going to Rust, run `node --cpu-prof main.js`, and open the profile in chrome://inspect -> DevTools for Node -> Performance tab. It will tell you what's taking time and it may not be what you expect.
liveoneggs 3y ago
If only they had heard of perl..
shrubble 3y ago
If they are using Docker anyways and thus able to use any command line tools they want, I can't help but wonder how fast 'awk' or even a Perl script would have given them what they need... I have seen convoluted stuff done in JS when a few shell commands would have taken care of things, before.
datadrivenangel 3y ago
Author probably could have saved all the trouble if they just asked one of their data engineers for advice...
platz 3y ago
I thought node was supposed to be really good at streaming. How could this be written in node to use streams more effectively (at whatever stage necessary, parsing, reading, etc ..) to limit memory usage?
- deleted 3y ago
  [deleted]
swader999 3y ago
For the love of all that is good, profile it.
pleoxy 3y ago
So you hang onto your allocated strings in a record array and wonder why you run out of memory? You should be saving the strings to the database so the GC has something to reclaim. It doesn't work when you keep fields 5, 7, 8 around in your records array. Bulk insert should probably be used and bulk write should be called periodically. You should pause processing while it saves so you keep total active allocations nice and compact. Otherwise the process will slow to a crawl when you get near your memory limit and all the time is spent in the GC trying to find ways to reclaim memory that you are still hanging onto. You see massive swings in processing time by chunking the data because you are handing memory very poorly. This should be bounded by database writes, not your language. Sigh.
deleted 3y ago
[deleted]

176 points by qsantos 3y ago

Related: you can write a Node.js module in Nim with Denim: https://github.com/openpeeps/denim https://github.com/openpeeps/denim

skybrian 3y ago

Blink and you missed the interesting bit, which is how they hooked them together. Looks like it's with this project: https://napi.rs/ https://napi.rs/ That project seems to be creating platform-specific DLL's, which I guess doesn't matter too much if it's wrapped up nicely in an npm package and you're deploying Docker files on Linux anyway? But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. Alternatively, writing very C-like code using a Uint8Array and not allocating in the loop might have fixed it too? But if you don't use JavaScript string libraries, it's not much like writing JavaScript anymore.

leononame 3y ago
Yeah, I think Uint8Array might've been a strong solution. But I've got the feeling this isn't the whole story. Unless the TSV is extremely wide, this alone feels a bit insignificant. I don't know the details so this is pure speculation, but using a transform stream on each row to split it up might've been enough to reduce memory pressure. I've done a bit of data manipulation in node and while imo js is not the ideal language for the job, it can be very fast. Strings are hard though, I'll admit that
eyelidlessness 3y ago
> But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. This is true, but N-API will generally perform better (sometimes significantly, usually in exactly the same CPU-bound scenarios where WASM will perform better than V8). And NAPI-RS is quite good at paving the happy path for multi-target packages.
pimeys 3y ago
> But conceptually, it seems like it would be more portable to generate a single WebAssembly binary instead of a bunch of DLL's. If the database IO happens in Rust, the option with WebAssembly is fetch, meaning HTTP and the Atlas-only Data API. If the IO is done in JS, you pay some extra CPU time transferring UTF-16 strings over the border and copy them as UTF-8 in Rust.

bubblydoops 3y ago

Batch maintenance is a drag. However even just from the snippet of the purported memory/slow issue, I think theres a smell that was not revealed. If your constantly running out of memory you have a bad design. This seems like a native code agenda type post which I have a problem with. I will not code in native code ever again. Its not worth it anymore, its a horrible practice and it just enables bad code to pretend its good.

satvikpendem 3y ago
What do you mean "native code" and why will you not program in such code?

warkanlock 3y ago

In my experience, Node.js generally performs very well for most use cases although when it comes to data manipulation (or similar use cases), performance often goes down the hill, becoming the primary bottleneck nearly every time. It's simply not the right tool for the job. The beauty of this is that it allows teams to isolate and optimize just the problematic portion of the code, preventing them from going down a rabbit hole of extensive changes that can be more harmful than beneficial. NAPI is such a good tool for this; that I've been advocating the use of it to my team since forever.

maccard 3y ago
> In my experience, Node.js generally performs very well Hard disagree here. Node is extremely fast to get up and running with, has as close to 0 barrier to entry as possible, and doesn't suffer from slow startup times that plague JVM languages. It can also be fast for tight loops. But in the experience I've had with it"generally", it's orders of magnitude slower than other languages. A hello world with express.js and gin in go are a full order of magnitude apart, and the express app with default settings dies under a load test that I would describe as "very light". Stories like this thread are common in the JS world where technically it can be rewritten to be faster, but like with Bun, esbuild, and many other examples we've seen here on HN, a better language choice makes an enormous difference. And to me, the only time node is the right tool for the job is if you have a team of JavaScript developers who can't learn another language
- nicoburns 3y ago
  Depends what you're comparing it to though. That same Node.js application will likely be an order of magnitude faster than a similar one in Python or Ruby.
- warkanlock 3y ago
  So, when it comes to a technology that works well, I think it should be easy to adopt for the whole team, Node.js won here even against Go. In my experience, Go seems to fit quite well there, although there are plenty of developers out there without Go skills. But let's face it: In most of the projects we work on, clients don't need to handle millions of requests from distributed systems and geolocated servers all over the world. Generally, our clients just want a CRUD that is a little more elegant than Excel and can help them solve their real-world problems. IMO if someone from your team tells you that you cannot do something with an average of ~10,000 req/s [1] (around that number is the consensus nowadays for Javascript whereas Bun is highly superior to everyone here), you may be solving the problem in the wrong way. [1] That's around 36 million requests per hour!

Klaster_1 3y ago

Leveraging Rust to speed Node.js is cool and all, but why the author didn't attempt to optimize the JS code first? If too much intermediate objects negatively impact memory consumption, surely re-using the same structure or variables to hold the parsed result would have improved the performance? The benchmark tests not only the speedup brought up by Rust, but an optimization too, so it's unclear which impacts the performance the most.

pkolaczk 3y ago
That approach may work in short term but in long term you'd likely end up with unreadable and brittle code that is often still not as fast as it would be after the rewrite. You'd introduce additional complexity into your product to overcome limitations of the platform. And tricks like reusing objects can turn into really nasty bugs, especially in languages that don't come with any tools to control aliasing.
- beezlewax 3y ago
  What does aliasing mean in this context?
  - pkolaczk 3y ago
    If you want to reuse an object to avoid allocation, you better not have a reference to that object from somewhere else. In JS AFAIK there is no way to detect such situation automatically.
    - Etheryte 3y ago
      Yes there is, it's called scope and it's super easy to keep track of? In fact, you have to go out of your way to share a reference if you scope it.
      - pkolaczk 3y ago
        In languages like JS, Java, Python is trivial to accidentally make an object escape the scope by storing a reference in a field of another object that outlives the scope. And it can be another developer 3 months later that can introduce a bug, not knowing about your clever object reuse.

devjab 3y ago

> Promise.all, isn't that delightful?). However, to our surprise, we soon ran into out-of-memory issues, I think I must be misunderstanding something, but isn’t that exactly what you should expect? We use node.js to process solar plant data. Which is massive amounts of data as each individual solar panel will log information about its power generation and status every few seconds. We can then use this to figure out which solar panels have a birds nest on them, and so on. Node performs this task well enough, but it obviously can’t be aggregated. Even if you were to spread it out on 25 node instances “concurrently” with control shift you’d run into out-of-memory issues extremely fast. I guess you could up the upper limit on when node.js stops being cool with its own memory usage, but honestly we tend to simply avoid the use promise aggregation exactly because it tends to ”break” node. I think moving to Rust was probably a good idea for this. Go might’ve been a better option if you were going for a “modern” concurrent language, but whatever works! We tend to use C ourselves when node needs a little boost. But often Node can handle these things on its own. With the solar data we use worker threads to collect and process the data in real time, which is really parallel-execution and not concurrency, but it works well and it doesn’t get you into performance issues.

go_prodev 3y ago
That sounds really interesting. Out of curiosity, how do you use Node for this purpose? Is it building a real time dashboard? My company has possibly similar challenges. We have lots of sensors on equipment that streams information every few ms through a time series database from the vendor. It's used to build monitoring dashboards, trigger alarms and help engineers to prioritize and optimize equipment settings, and look for trends over time. It's such high volumes that capturing it and analyzing can be difficult, so we use a combination of vendor systems, Azure Synapse and Power BI. There are still lots of opportunities to improve, and I'd be interested to hear how you landed on your setup and what you do with the data.
- devjab 3y ago
  It’s mainly for data transformation and transport. Solar inverters don’t deliver data in the same formats, there are no protocols and apparently most engineers in the solar panel business still think FTP (not SFTP) is fine but some will deliver a file per reading while others will aggregate and so on. We use node to gather the data and transform it into consumptive formats that our 3rd party AI vendor can work with, as well as store it in a range of SQL databases for our own PowerBI consumption. We do no frontend work. We don’t use node because it’s a good idea to use node for this. We use node because we try to use TypeScript as much as possible because we’re a small team and having a single language covering as much of our stack as possible makes it easier to work together.

leononame 3y ago

> However, to our surprise, we soon ran into out-of-memory issues, even when using utilities like p-limit to limit ourselves to just two jobs in parallel. If only two async jobs already run OOM, your issue is probably bad code, not node.js. From the code snippet posted, it looks like they do a `records.push(...)`, whereas it probably would've been more efficient to just continue using streams downstream as well. Additionally, using a transform stream to split the string rather than doing it in a loop might've yielded better results because the previous row could be cleaned up. Obviously, this is just poking in the dark, but such stark difference in numbers in the Rust implementation leads me to believe that they changed some other fundamental part of the design (e.g. streaming the results to the database rather than inserting everything in one big batch at the end).

lemper 3y ago

it's as if using fitting tool for solving problem's domain is a magic. good for you for saving that much of resource, though.

franciscop 3y ago

> "The intriguing part here is that line and fields won't be cleared from memory until the garbage collector determines it's time to do so. We find ourselves with RAM filled with redundant strings and arrays that need cleaning. It's hardly surprising that our computer wasn't thrilled with this situation." I don't have enough experience with super-optimizing Javascript, but wouldn't this be an easy solution? Instead of creating new variables in each loop and let the engine clear them, reuse the variables to avoid using more memory: let line, fields, httpStatus; for await (line of readlineStream) { fields = line.split('\t'); httpStatus = Number(fields[5]); if (httpStatus < 200 || httpStatus > 299) continue; await dbWrite({ pathname: fields[7], referrer: fields[8], // ... }); } Edit: and as others suggest, keep streaming those records, instead of just pushing them into an array

pkolaczk 3y ago
No, this still allocates a new string array per each line.split call. It's not really a problem with variables. The variables are likely on the stack anyways, but they only store pointers to data on the heap. If you call any function that allocates data on the heap you cannot get away with that by moving the variable declarations.
- getcrunk 3y ago
  Okay, but it still gets rid of two other allocations (line and http)
  - pkolaczk 3y ago
    Line is a string. It allocates a fresh one on each line even if the variable is reused.
    - getcrunk 3y ago
      Lol shows how much I know!

orf 3y ago

> having 25 Node.js instances running for three hours to parse 200GB of data seems like an indication that something isn't quite right. That’s about 700kb/s. I’m all for playing with rust, but this post strikes me as engineers being distracted by shiny new things because they are only able to parse a TSV at 700kb/s? You should be able to understand why your current system performs so absolutely terribly before you decide to rewrite it in any language. Hand-waving at the garbage collector whilst seemingly doing very little to understand what’s happening doesn’t convince me that there isn’t another, possibly pretty obvious issue going on.

pkolaczk 3y ago
They debugged the problem to be excessive allocations in the tight loop of the TSV parser. If you allocate a separate object per each field, I can imagine how it can result in throughput counted in kB/s instead of GB/s. We have had very similar issues with Java - excessive heap allocations can bring the system to a crawl and there is often no easy way out.
- DanielHB 3y ago
  A common technique (at least in Java, never seen it in nodejs) is to have a pool of pre-allocated objects and just reuse them as they are discarded. Essentially manual GC
  - DrDroop 3y ago
    Happens JS world aswel SimonDev did a video about it, this one I think: https://www.youtube.com/watch?v=easvMCCBFkQ https://www.youtube.com/watch?v=easvMCCBFkQ This channel is a real gem, show what talent really is: https://www.youtube.com/watch?v=a1L7k35EHIc https://www.youtube.com/watch?v=a1L7k35EHIc
    - KeyXiote 3y ago
      Thanks for the vid links, always looking for new channels with informative content.

mrighele 3y ago

I guess there is something I am missing here. The original code should consume not much more memory than the optimized one. Yes you're going to allocate a lot of more memory, but whatever doesn't finish in the record becomes garbage after every single line, so I can expect worse performance, but not using two order of magnitude more memory. I would see more problematic the fact that you're collecting all the records in a single array. Depending on how you use them it can be beneficial to use a generator and consume them incrementally.

didntcheck 3y ago
Agreed in terms of live memory, but if it's purely measuring heap size then it makes more sense. So it possibly could run with a much stricter memory limit, but with the tradeoff of more GC (and for a compacting GC, this would mean lower throughout, since the GC time is proportional only to live objects. May as well let the garbage pile up as much as you can and therefore do fewer collections)

kohlerm 3y ago

This is probably fixable in Node.js, just preallocate objects and do not excessively create new ones. But yeah using Node.js is probably not the best choice for this. As others have mentioned a JVM could easily handle this with some pre-allocation strategy.

getcrunk 3y ago

The author gives no info about how much memory these node instances had. I think that context is very important to know. Also as others have mentioned the js code could have been optimized. Id be very interested in a part two testing out some of the suggestions and sharing the benchmarks!

winrid 3y ago

Let me know if you want some help optimizing that aggregation query :) looks like a fun challenge.

brownfox 3y ago

I believe that the memory management aspect alone doesn't provide sufficient justification for rewriting the code in Rust. While it is possible to achieve reduced memory consumption through techniques like reusing a backing buffer and using allocUnsafe(), which require developers to take on memory management responsibilities, it might result in slightly more verbose code compared to a straightforward line-by-line split() approach. I am eager to explore the possibility of enhancing the line parsing process using Node.js. If you could provide a runnable example on a GitHub repository, it would enable me to compare its memory efficiency with the Rust rewrite version.

ifdefdebug 3y ago

I just skimmed through the article but it seems that they refactored from allocating and copying substring to pointing at the substrings within the original string (inside a hot path). So 25x performance boost should be expected just from that and has nothing to do with GC or the language used.

sebstefan 3y ago
200gB in 2.5 hours is still only 22mB/s, I feel like I've seen blog posts of people going several orders of magnitude faster than that when the bottleneck is the sequential read of a big file on an SSD
- xmcqdpt2 3y ago
  I've written similar code before using absolutely bog standard idiomatic OOP Java. For a single threaded program I'd expect at least 1GB/s before any optimization. Most of the allocations in the loop that aren't the final records objects will get erased by JIT. They are doing something very wrong.
  - sebstefan 3y ago
    Let me rephrase this for you They're doing something slightly wrong that's making a non-bottleneck part of their code less optimal than it should be.
- Kwpolska 3y ago
  Millibytes per second?
  - sebstefan 3y ago
    See in french we don't even have the possibility for that kind of confusion because we named a "byte" an "octet" which also reflects its relation with the number 8 That way they don't get gigabytes and gigabits abbreviated with the same letters and we don't have to rely on a capitalization convention Join the enlightened gigaoctet convention

philipwhiuk 3y ago

Why only do a daily import of CDN logs? If this process was an on-demand pipeline they wouldn't have to deal with such a storm of data every day. It would also mean people got stats 'closer-to-live' which feels like it would be useful.

andrewstuart 3y ago

Nodejs modules should be written in javascript/typescript.

o1y32 3y ago

Honestly doing line.split('\t') seems a newbie mistake. I don't know how many columns each line has but it could be a bottleneck. Wouldn't manually processing tab and non-tab character help, in the same way you don't load the entire log into memory but only read by line? When you are dealing with huge amount of data, you have to be extra careful, regardless of which language you use. That plus other optimization could make Node.js run very efficiently. Otherwise, I still need to be convinced that the "line" array is causing trouble but not anything else -- the article definitely doesn't provide detail about how they found this problem. It is almost funny that the author tried to put this on Kubernetes instead of even attempting to optimize the JS code. I wish I had that luxury and tell my boss I should use more resources instead of fixing my code.

didntcheck 3y ago
Yep. I've written code like this before for small inputs or one-off scripts, but I always thought it was obvious that doing it like this would be slow and produce lots of garbage. It's very nice and readable code, but for the hundreds of GBs they mention, it's clearly not ideal

jongjong 3y ago

About Node.js running out of memory, I wonder if they tried setting the max-old-space-size flag to increase the memory limit. Based on the descriptions, I wonder if the real problem was not Node.js but that all these solutions were implemented inside a constrained environment.

thefilmore 3y ago

Before going to Rust, run `node --cpu-prof main.js`, and open the profile in chrome://inspect -> DevTools for Node -> Performance tab. It will tell you what's taking time and it may not be what you expect.

liveoneggs 3y ago

If only they had heard of perl..

shrubble 3y ago

If they are using Docker anyways and thus able to use any command line tools they want, I can't help but wonder how fast 'awk' or even a Perl script would have given them what they need... I have seen convoluted stuff done in JS when a few shell commands would have taken care of things, before.

datadrivenangel 3y ago

Author probably could have saved all the trouble if they just asked one of their data engineers for advice...

platz 3y ago

I thought node was supposed to be really good at streaming. How could this be written in node to use streams more effectively (at whatever stage necessary, parsing, reading, etc ..) to limit memory usage?

deleted 3y ago
[deleted]

swader999 3y ago

For the love of all that is good, profile it.

pleoxy 3y ago

So you hang onto your allocated strings in a record array and wonder why you run out of memory? You should be saving the strings to the database so the GC has something to reclaim. It doesn't work when you keep fields 5, 7, 8 around in your records array. Bulk insert should probably be used and bulk write should be called periodically. You should pause processing while it saves so you keep total active allocations nice and compact. Otherwise the process will slow to a crawl when you get near your memory limit and all the time is spent in the GC trying to find ways to reclaim memory that you are still hanging onto. You see massive swings in processing time by chunking the data because you are handing memory very poorly. This should be bounded by database writes, not your language. Sigh.

deleted 3y ago

[deleted]