4 ms·
> Promise.all, isn't that delightful?). However, to our surprise, we soon ran into out-of-memory issues, I think I must be misunderstanding something, but isn’
by devjab 3y ago
> Promise.all, isn't that delightful?). However, to our surprise, we soon ran into out-of-memory issues,
I think I must be misunderstanding something, but isn’t that exactly what you should expect?
We use node.js to process solar plant data. Which is massive amounts of data as each individual solar panel will log information about its power generation and status every few seconds. We can then use this to figure out which solar panels have a birds nest on them, and so on. Node performs this task well enough, but it obviously can’t be aggregated. Even if you were to spread it out on 25 node instances “concurrently” with control shift you’d run into out-of-memory issues extremely fast. I guess you could up the upper limit on when node.js stops being cool with its own memory usage, but honestly we tend to simply avoid the use promise aggregation exactly because it tends to ”break” node.
I think moving to Rust was probably a good idea for this. Go might’ve been a better option if you were going for a “modern” concurrent language, but whatever works! We tend to use C ourselves when node needs a little boost. But often Node can handle these things on its own. With the solar data we use worker threads to collect and process the data in real time, which is really parallel-execution and not concurrency, but it works well and it doesn’t get you into performance issues.
- go_prodev 3y agoThat sounds really interesting. Out of curiosity, how do you use Node for this purpose? Is it building a real time dashboard? My company has possibly similar challenges. We have lots of sensors on equipment that streams information every few ms through a time series database from the vendor. It's used to build monitoring dashboards, trigger alarms and help engineers to prioritize and optimize equipment settings, and look for trends over time. It's such high volumes that capturing it and analyzing can be difficult, so we use a combination of vendor systems, Azure Synapse and Power BI. There are still lots of opportunities to improve, and I'd be interested to hear how you landed on your setup and what you do with the data.
- devjab 3y agoIt’s mainly for data transformation and transport. Solar inverters don’t deliver data in the same formats, there are no protocols and apparently most engineers in the solar panel business still think FTP (not SFTP) is fine but some will deliver a file per reading while others will aggregate and so on. We use node to gather the data and transform it into consumptive formats that our 3rd party AI vendor can work with, as well as store it in a range of SQL databases for our own PowerBI consumption. We do no frontend work. We don’t use node because it’s a good idea to use node for this. We use node because we try to use TypeScript as much as possible because we’re a small team and having a single language covering as much of our stack as possible makes it easier to work together.
- nesarkvechnep 3y agoModern concurrent language, suitable for processing massive amounts of data, is Elixir.
- synthc 3y agoIsn't the BEAM VM on which it runs rather slow?
- nesarkvechnep 3y agoRather slow means what?
- Alifatisk 3y agoSlow or not, it’s extremely concurrent!
- pkolaczk 3y ago> Go might’ve been a better option if you were going for a “modern” concurrent language, but whatever works! Go doesn't embed in other runtimes as nicely as Rust, nor it cannot call foreign code (including even C) as efficiently as Rust does. Rust strength is that it needs virtually no runtime and there is no magic needed to cross the ffi boundary.
- winrid 3y agoThe issue isn't Node. The issue is a poor understanding of memory requirements of each task and trying to do too many at once. You need to throttle your concurrency.
- pookha 3y agoWhy not just use the JVM and concurrency? I routinely parse 300+ GB of geospatial data without breaking a sweat using a very simple combination of concurrent linked queues and cached thread pools. I couldn't imagine diving head long into some new esoteric language just to do something as simple as what this author is describing. And things are only going to get easier with Virtual Threads...If I'm doing something boring (parsing data) I'd just want to use a boring standard language I guess (Java).
- devjab 3y agoJava isn’t really something we want to work with. So I guess that’s the quick answer. There is nothing wrong with Java, I think projects like quarkus is pretty cool and the JVM is a battle tested machine. Our issue is more process related. We’re a small team so we try to use as few languages as possible, and since we can’t realistically avoid using TypeScript we tend to use it everywhere we can. Then when we need some extra juice we turn to C. We even PoC’ed Rust but decided it wasn’t “boring” enough for us yet. Aside from that it can be hard to hire people for Java around here. It’s not a language a lot of people want to work with, and it’s common enough that the ones who do already have jobs.