5 ms·

Is parcel the first node module bundler to run multiple cores? I’ve been waiting for this forever! Definitely going to take a look at the code and try this out

Is parcel the first node module bundler to run multiple cores? I’ve been waiting for this forever! Definitely going to take a look at the code and try this out soon.

chmln 9y ago
I believe webpack can run using multiple threads at least [0] [0] https://github.com/webpack-contrib/thread-loader https://github.com/webpack-contrib/thread-loader
chrismorgan 9y ago
There’s a reason why it’s not generally done: there are no good tools for doing this in Node, nor is JavaScript particularly conducive to it. The state of the art in such matters is Rust with its type system-assisted fearless concurrency; my favourite demonstration of how it can work is Rayon, where introducing data parallelism is typically as simple as replacing a .iter() with a .par_iter(). I’d love it if something like Rayon could be ported to Node, somehow. (By virtue of being excellent at the low-level and having zero-cost abstractions, Rust actually ends up being a really good high-level language.) In Node, if you want to process a zillion files in the same way, there are no good solutions. I recently needed to improve the performance of a manually-crafted, Node-centred build process; in Rust, it would have been a matter of: add Rayon, replace a .iter() with .par_iter() and we’re done—and what’s more, it’ll only flex its parallelism muscles where it’s worthwhile. But in Node it took a several hours of trouble (after research, I settled on using worker-farm to help) to produce a strictly inferior result. Rayon: https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon worker-farm: https://www.npmjs.com/package/worker-farm https://www.npmjs.com/package/worker-farm
- chickenfries 9y ago
  I've also tried to naively improve a node build process and given up after realizing that, for the particular project I was working on, splitting up the build into multiple processes actually slowed things down compared to the single threaded build process Node's single threaded event loop works well for some use cases but makes it hard to parallelize build process. I would absolutely use a module bundler developed in a better suited language and distributed on npm as a binary like flow.
  - thangngoc89 9y ago
    Then you should consider fastpack. It’s written in OCaml and use Flow parser. Also, it should be distributed as a binary Fastpack: https://github.com/fastpack/fastpack https://github.com/fastpack/fastpack
    - beckler 9y ago
      I would also consider bazel. https://bazel.build https://bazel.build
      - lambdadmitry 9y ago
        Have you tried it to build web projects? Unfortunately there are very few experience reports out there.
  - chrismorgan 9y ago
    I’ve thought in concrete terms about porting postcss to Rust before. It’s the sort of tool that is stuck with slow while it’s in JavaScript, but could be amazingly fast in Rust, for which it’s a really good match (especially algebraic data types). (Seriously, the sort of thing that typically takes a second or so in JavaScript with things like postcss I would expect to take under five milliseconds in well-written Rust, and under a hundred in badly-written Rust.) The real problem with all these sorts of things is network effects. People aren’t going to use the Rust port of postcss unless the plugin they want to use works in it, and it can be conveniently tied into their existing toolchain, and—— and—— This is why we can’t have fast, efficient things. I’m really looking forward to getting something of the pthread style in WebAssembly, because then, finally, blending the two should be easily feasible. You can already use things like Neon (https://www.neon-bindings.com https://www.neon-bindings.com) to write Node modules in Rust, but it’s not as easy as it should eventually be a few years down the track.
    - chickenfries 9y ago
      > The real problem with all these sorts of things is network effects. People aren’t going to use the Rust port of postcss unless the plugin they want to use works in it, and it can be conveniently tied into their existing toolchain, and—— and—— I've never used postcss, but I don't think this is strictly for all js build tools. There's libsass, written in C++. I've mentioned flow. I'm sure there are others. I think you're right though that webassembly is going to be big for this kind of thing.

Is parcel the first node module bundler to run multiple cores? I’ve been waiting for this forever! Definitely going to take a look at the code and try this out soon.

chmln 9y ago

I believe webpack can run using multiple threads at least [0] [0] https://github.com/webpack-contrib/thread-loader https://github.com/webpack-contrib/thread-loader

chrismorgan 9y ago

There’s a reason why it’s not generally done: there are no good tools for doing this in Node, nor is JavaScript particularly conducive to it. The state of the art in such matters is Rust with its type system-assisted fearless concurrency; my favourite demonstration of how it can work is Rayon, where introducing data parallelism is typically as simple as replacing a .iter() with a .par_iter(). I’d love it if something like Rayon could be ported to Node, somehow. (By virtue of being excellent at the low-level and having zero-cost abstractions, Rust actually ends up being a really good high-level language.) In Node, if you want to process a zillion files in the same way, there are no good solutions. I recently needed to improve the performance of a manually-crafted, Node-centred build process; in Rust, it would have been a matter of: add Rayon, replace a .iter() with .par_iter() and we’re done—and what’s more, it’ll only flex its parallelism muscles where it’s worthwhile. But in Node it took a several hours of trouble (after research, I settled on using worker-farm to help) to produce a strictly inferior result. Rayon: https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon worker-farm: https://www.npmjs.com/package/worker-farm https://www.npmjs.com/package/worker-farm

chickenfries 9y ago
I've also tried to naively improve a node build process and given up after realizing that, for the particular project I was working on, splitting up the build into multiple processes actually slowed things down compared to the single threaded build process Node's single threaded event loop works well for some use cases but makes it hard to parallelize build process. I would absolutely use a module bundler developed in a better suited language and distributed on npm as a binary like flow.
- thangngoc89 9y ago
  Then you should consider fastpack. It’s written in OCaml and use Flow parser. Also, it should be distributed as a binary Fastpack: https://github.com/fastpack/fastpack https://github.com/fastpack/fastpack
  - beckler 9y ago
    I would also consider bazel. https://bazel.build https://bazel.build
    - lambdadmitry 9y ago
      Have you tried it to build web projects? Unfortunately there are very few experience reports out there.
- chrismorgan 9y ago
  I’ve thought in concrete terms about porting postcss to Rust before. It’s the sort of tool that is stuck with slow while it’s in JavaScript, but could be amazingly fast in Rust, for which it’s a really good match (especially algebraic data types). (Seriously, the sort of thing that typically takes a second or so in JavaScript with things like postcss I would expect to take under five milliseconds in well-written Rust, and under a hundred in badly-written Rust.) The real problem with all these sorts of things is network effects. People aren’t going to use the Rust port of postcss unless the plugin they want to use works in it, and it can be conveniently tied into their existing toolchain, and—— and—— This is why we can’t have fast, efficient things. I’m really looking forward to getting something of the pthread style in WebAssembly, because then, finally, blending the two should be easily feasible. You can already use things like Neon (https://www.neon-bindings.com https://www.neon-bindings.com) to write Node modules in Rust, but it’s not as easy as it should eventually be a few years down the track.
  - chickenfries 9y ago
    > The real problem with all these sorts of things is network effects. People aren’t going to use the Rust port of postcss unless the plugin they want to use works in it, and it can be conveniently tied into their existing toolchain, and—— and—— I've never used postcss, but I don't think this is strictly for all js build tools. There's libsass, written in C++. I've mentioned flow. I'm sure there are others. I think you're right though that webassembly is going to be big for this kind of thing.