4 ms·

Sure but that's the whole premise of Node, there doesn't _need_ to be multithreading. Whats wrong with spinning up multiple workers (in something like Docker/Sw

Sure but that's the whole premise of Node, there doesn't _need_ to be multithreading. Whats wrong with spinning up multiple workers (in something like Docker/Swarm/K8S) that split up the work and report back to whatever collector or service that wants the end result? So many problems have spawned out of the desire to thread everywhere that the safety of single-threadedness is a feature not a bug.

zaarn 7y ago
Multithreaded (or Multiprocess) can be done safely. I write a lot of stuff in Rust these days, where (unless you use unsafe) it doesn't even let you do anything that would break under multi-threading, even if you don't use it. As a result, it was incredibly easy to write a light-weight crawler that delivery results into a main worker queue and the data is processed by <cpu_cores> threads. The crawler largely uses async for IO advantages but the worker threads rely heavily on data crunching where your throughput is limited by CPU not by IO and async suffers a bit. And since crawler and worker are on the same process, the latency between a page being grabbed and starting to be processed is incredibly low, any kind of networking would likely cut throughput by a factor of 10 or more. Not even mentioning having to serialize the data instead of relying on references to get read-only copies and only making minimal copies on the stack.
- halfmatthalfcat 7y ago
  Ok, fair point Rust makes multithreading easier but one could easily make the argument that fighting Rust's borrow checker is a huge burden to development. My point is Node isn't as bad as a server-side language as people claim. There are certain use cases where you can leverage Node's strengths and use another language somewhere else where its shines more. I use Node heavily in my backend stack and have been extremely pleased with it's performance and library ecosystem.
  - zaarn 7y ago
    You can do the things you do in Node, Async, in Rust too. Futures and friends are in nightly and soon in Stable, which enable you to mix the best of both world; Multithreaded and Async. Once you get used to Rust, you fight the borrow checker a lot less.
    - hombre_fatal 7y ago
      You can level "just use Rust" at any non-Rust ecosystem. It's kind of tired advice. You might as well be recommending Haskell or any other hobby horse language and then argue from a standpoint of technical superiority as if that's the only concern when choosing tools. I use Rust all the time yet I'll still use Node for most HTTP-based applications and tools I start, and not because I'm an idiot or beginner, as many comments here will suggest. They are completely different languages and ecosystems and workflows that satisfy different goals.
      - zaarn 7y ago
        This isn't "just use rust". This is "just use something more mature". And there are plenty of more mature ecosystems developing Async capability or that have a good Threading story.
marktangotango 7y ago
Whats wrong with spinning up multiple workers (in something like Docker/Swarm/K8S) that split up the work and report back to whatever collector or service that wants the end result? The "spinning up multiple workers" is what's wrong with that. Specifically a language/runtime that does support multi-processing handles it for you; ie gomaxprocs. Another example would be servers like apache forking off several workers during start up (depending on the mpm module). With node this has to be managed manually, as you describe, with Docker or whatever. With a node async server, you're generally not cpu bound, so there's really no reason to try to utilize all the cores on the machine. Except if you are cpu bound, then you'd want to do that.
HereBeBeasties 7y ago
If you think there's nothing wrong with all the complexity that comes with running a distributed system for something that could be done trivially in a single process in a sensible language, I would like to sell you this hammer - you will find it very useful when you next need to screw something together.

zaarn 7y ago

Multithreaded (or Multiprocess) can be done safely. I write a lot of stuff in Rust these days, where (unless you use unsafe) it doesn't even let you do anything that would break under multi-threading, even if you don't use it. As a result, it was incredibly easy to write a light-weight crawler that delivery results into a main worker queue and the data is processed by <cpu_cores> threads. The crawler largely uses async for IO advantages but the worker threads rely heavily on data crunching where your throughput is limited by CPU not by IO and async suffers a bit. And since crawler and worker are on the same process, the latency between a page being grabbed and starting to be processed is incredibly low, any kind of networking would likely cut throughput by a factor of 10 or more. Not even mentioning having to serialize the data instead of relying on references to get read-only copies and only making minimal copies on the stack.

halfmatthalfcat 7y ago
Ok, fair point Rust makes multithreading easier but one could easily make the argument that fighting Rust's borrow checker is a huge burden to development. My point is Node isn't as bad as a server-side language as people claim. There are certain use cases where you can leverage Node's strengths and use another language somewhere else where its shines more. I use Node heavily in my backend stack and have been extremely pleased with it's performance and library ecosystem.
- zaarn 7y ago
  You can do the things you do in Node, Async, in Rust too. Futures and friends are in nightly and soon in Stable, which enable you to mix the best of both world; Multithreaded and Async. Once you get used to Rust, you fight the borrow checker a lot less.
  - hombre_fatal 7y ago
    You can level "just use Rust" at any non-Rust ecosystem. It's kind of tired advice. You might as well be recommending Haskell or any other hobby horse language and then argue from a standpoint of technical superiority as if that's the only concern when choosing tools. I use Rust all the time yet I'll still use Node for most HTTP-based applications and tools I start, and not because I'm an idiot or beginner, as many comments here will suggest. They are completely different languages and ecosystems and workflows that satisfy different goals.
    - zaarn 7y ago
      This isn't "just use rust". This is "just use something more mature". And there are plenty of more mature ecosystems developing Async capability or that have a good Threading story.

marktangotango 7y ago

Whats wrong with spinning up multiple workers (in something like Docker/Swarm/K8S) that split up the work and report back to whatever collector or service that wants the end result? The "spinning up multiple workers" is what's wrong with that. Specifically a language/runtime that does support multi-processing handles it for you; ie gomaxprocs. Another example would be servers like apache forking off several workers during start up (depending on the mpm module). With node this has to be managed manually, as you describe, with Docker or whatever. With a node async server, you're generally not cpu bound, so there's really no reason to try to utilize all the cores on the machine. Except if you are cpu bound, then you'd want to do that.

HereBeBeasties 7y ago

If you think there's nothing wrong with all the complexity that comes with running a distributed system for something that could be done trivially in a single process in a sensible language, I would like to sell you this hammer - you will find it very useful when you next need to screw something together.