15 ms·

Keep Webpack Fast: A Field Guide for Better Build Performance

214 points by firloop 9y ago

acemarke 9y ago
Excellent summary of some of the techniques you can use to speed up Webpack builds. It'll be interesting to see how the ongoing work on performance in Webpack 4 relates to this [0] [1], and I'd guess that the just-announced Rust->WASM sourcemap lib work [2] will help too. If anyone's looking for more info on this topic, my React/Redux links list has a large section of additional posts on improving Webpack build perf, code splitting, bundle size optimization, and other related techniques [3]. [0] https://github.com/webpack/webpack/issues/6244 https://github.com/webpack/webpack/issues/6244 [1] https://twitter.com/TheLarkInn/status/945486181575340032 https://twitter.com/TheLarkInn/status/945486181575340032 [2] https://twitter.com/TheLarkInn/status/954053251854344192 https://twitter.com/TheLarkInn/status/954053251854344192 [3] https://github.com/markerikson/react-redux-links/blob/master/webpack-advanced-techniques.md https://github.com/markerikson/react-redux-links/blob/master...
simooooo 9y ago
I use webpack every day. But the config for it scares the crap out of me.
- osteele 9y ago
  I recently discover Poi https://poi.js.org/ https://poi.js.org/. It’s built on top of webpack. If your issues with webpack are configuration complexity, not speed, then Poi may be appealing to you.
  - edmorley 9y ago
    Another option similar to Poi, that I'd recommend adding to any comparison, is https://neutrino.js.org/ https://neutrino.js.org/
shawn 9y ago
If anyone is tired of Webpack configuration issues / slow performance, give Parcel a try: https://parceljs.org/ https://parceljs.org/ The project is less than two months old, so Parcel might not be a good fit for you. But if it is, it's usually a drop-in replacement. Zero configuration. Just run it and go. We're working hard to add the most requested features: source map support is coming very soon, for example. But if you can put up with a few rough edges, Parcel promises to give you a ~5x performance increase over webpack. (We kick off multiple processes to do the actual bundling work, and we have an asset cache, which accounts for most of the speedups.) If you'd like to participate or have any questions, hop on our Slack! https://slack.parceljs.org/ https://slack.parceljs.org/ We have about 500 devs ready to help you out and chat with you (about anything, really). Feel free to snag an issue and start working on it: https://github.com/parcel-bundler/parcel/issues?utf8=%E2%9C%93&q=is%3Aissue+sort%3Aupdated-desc+ https://github.com/parcel-bundler/parcel/issues?utf8=%E2%9C%... There's plenty of work to go around. It's been really cool to see so many people pitching in. We recently had a Japanese contributor help get Parcel up and running for multi-language projects: https://qiita.com/zacky1972/items/0ce05454b67506edc634 https://qiita.com/zacky1972/items/0ce05454b67506edc634
- chickenfries 9y ago
  Is parcel the first node module bundler to run multiple cores? I’ve been waiting for this forever! Definitely going to take a look at the code and try this out soon.
  - chmln 9y ago
    I believe webpack can run using multiple threads at least [0] [0] https://github.com/webpack-contrib/thread-loader https://github.com/webpack-contrib/thread-loader
  - chrismorgan 9y ago
    There’s a reason why it’s not generally done: there are no good tools for doing this in Node, nor is JavaScript particularly conducive to it. The state of the art in such matters is Rust with its type system-assisted fearless concurrency; my favourite demonstration of how it can work is Rayon, where introducing data parallelism is typically as simple as replacing a .iter() with a .par_iter(). I’d love it if something like Rayon could be ported to Node, somehow. (By virtue of being excellent at the low-level and having zero-cost abstractions, Rust actually ends up being a really good high-level language.) In Node, if you want to process a zillion files in the same way, there are no good solutions. I recently needed to improve the performance of a manually-crafted, Node-centred build process; in Rust, it would have been a matter of: add Rayon, replace a .iter() with .par_iter() and we’re done—and what’s more, it’ll only flex its parallelism muscles where it’s worthwhile. But in Node it took a several hours of trouble (after research, I settled on using worker-farm to help) to produce a strictly inferior result. Rayon: https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon worker-farm: https://www.npmjs.com/package/worker-farm https://www.npmjs.com/package/worker-farm
jamesmcintyre 9y ago
From someone who has experienced the dev flow on apps with very large, gnarly webpack configs 17 seconds is extremely impressive and sounds like heaven! What a lot of devs may not be aware of, if their projects aren't of a certain scale & complexity (support for legacy code/library-choices/extreme-modularity/etc.) is that the magical nearly-instant hot-module reloading you experience with a clean, "normal" project should not be taken for granted because webpack builds/reloading can become a monster! Not to say you can't scale up good webpack performance (obviously slack has) just that it's not an automatic thing but something that requires some TLC. Thanks for taking the time to share slack team!
Game_Ender 9y ago
Can anyone explain how someone can make a build system and not make correctness and speed the top priorities from the start? Once you do that parallelism is third feature you implement, the first being running an action, and the second being a hash based dependency graph of those actions inputs & outputs. Once you have that dependency graph parallelism and caching just appear pretty much for free. It looks like google uses bazel (a parallel dependency graph based build) internally and they have 1-2 second rebuild times on apps much larger then Slack [0]. 0 - https://medium.com/@Jakeherringbone/what-angular-is-doing-with-bazel-and-closure-21f526f64a34 https://medium.com/@Jakeherringbone/what-angular-is-doing-wi...
- swsieber 9y ago
  See also, pants and buck. I think it's a story of PostgreSQL vs Mongo - do you start out with conservative correctness and slowly expand your feature set, or do you advertise a lot of (poorly imolemented) features, and with the large userbase slowly increase your quality.
jsmeaton 9y ago
Hoping that someone might have some recommendations for a problem that we have in particular. We have multiple sites, each of which has it's own theme. Our various sass files check the theme variable passed into the sass loader, and decide on the correct values for particular styles. Our react components import the stylesheet and reference values accordingly. We also have a spattering of old backbone on some pages, so it's not just react involved. Unfortunately, this means that we have to build our entire application for each theme. This takes an excruciating amount of time. Parallel Webpack makes it somewhat manageable, but I'd like to eliminate building for each theme altogether. Any ideas on paths we can look into? If our application was all React, I'd consider styled components or a theme HOC. It's not though, so I've put those options on the back burner for the moment.
mbrumlow 9y ago
I hate webpack, it has been a hack from the start. I can't wait till HTTP2 is more widely supported. My latest projects use polymer and a custom built service (in go) to automatically parse entry points to my application and index the required files. When a HTTP2 connection is made -- and the browser supports push, I will automatically push all the required files. With browsers like chrome the results are near instant and I have full control over what gets pushed or not. Personally I think we need to stop hacking the web together and properly solve these problems. I think HTTP2 with push is a good step in that direction.
- jaredklewis 9y ago
  I don't see how HTTP2 can replace webkack. Even if websites stop making single file bundles of JS, I think we'll still need something to: - Transpile from latest ES standards, to browser supported code - Compile from non-JS languages like TypeScript, Flow, and so on - Do dead code elimination, minification, and tree shaking - Bundle code shared by other bundles - Optimize images And on and on... I think webpack is a mess, but I think it really is just a reflection of the complexity of the web platform.
  - paulddraper 9y ago
    HTTP/2 can't. But it can't make builds less complex. You're not longer bundling every script, inlining every image, etc.
    - always_good 9y ago
      Still just a drop in the bucket of what Webpack can do, so the HTTP/2 post above shows a major misunderstanding of what people are using Webpack for, and probably what HTTP/2 actually is. I don't see how you can work on a modern JS application, understand HTTP/2, and think HTTP/2 replaces Webpack. Pick two...
      - mbrumlow 9y ago
        Yes, you don't agree with what I said so you attack the author and not the content. Webpack is used for a lot of things -- that it should not be used for. As I posted above many of the things it is used for should out of band of your blunder. HTTP2 removes the need for bundling by allowing a smart web server either learn, or be told what files will be required for rendering to be pushed on the first request from the client. This eliminates the multiple request / and connections required in HTTP1 -- even with 4 pipelines that can be a lot of round trip request. If your application was not smart enough to know what to push to the client HTTP2 can allow multiple request over the same connection again negating the need for bundling.

214 points by firloop 9y ago

Excellent summary of some of the techniques you can use to speed up Webpack builds. It'll be interesting to see how the ongoing work on performance in Webpack 4 relates to this [0] [1], and I'd guess that the just-announced Rust->WASM sourcemap lib work [2] will help too. If anyone's looking for more info on this topic, my React/Redux links list has a large section of additional posts on improving Webpack build perf, code splitting, bundle size optimization, and other related techniques [3]. [0] https://github.com/webpack/webpack/issues/6244 https://github.com/webpack/webpack/issues/6244 [1] https://twitter.com/TheLarkInn/status/945486181575340032 https://twitter.com/TheLarkInn/status/945486181575340032 [2] https://twitter.com/TheLarkInn/status/954053251854344192 https://twitter.com/TheLarkInn/status/954053251854344192 [3] https://github.com/markerikson/react-redux-links/blob/master/webpack-advanced-techniques.md https://github.com/markerikson/react-redux-links/blob/master...

simooooo 9y ago

I use webpack every day. But the config for it scares the crap out of me.

osteele 9y ago
I recently discover Poi https://poi.js.org/ https://poi.js.org/. It’s built on top of webpack. If your issues with webpack are configuration complexity, not speed, then Poi may be appealing to you.
- edmorley 9y ago
  Another option similar to Poi, that I'd recommend adding to any comparison, is https://neutrino.js.org/ https://neutrino.js.org/

shawn 9y ago

If anyone is tired of Webpack configuration issues / slow performance, give Parcel a try: https://parceljs.org/ https://parceljs.org/ The project is less than two months old, so Parcel might not be a good fit for you. But if it is, it's usually a drop-in replacement. Zero configuration. Just run it and go. We're working hard to add the most requested features: source map support is coming very soon, for example. But if you can put up with a few rough edges, Parcel promises to give you a ~5x performance increase over webpack. (We kick off multiple processes to do the actual bundling work, and we have an asset cache, which accounts for most of the speedups.) If you'd like to participate or have any questions, hop on our Slack! https://slack.parceljs.org/ https://slack.parceljs.org/ We have about 500 devs ready to help you out and chat with you (about anything, really). Feel free to snag an issue and start working on it: https://github.com/parcel-bundler/parcel/issues?utf8=%E2%9C%93&q=is%3Aissue+sort%3Aupdated-desc+ https://github.com/parcel-bundler/parcel/issues?utf8=%E2%9C%... There's plenty of work to go around. It's been really cool to see so many people pitching in. We recently had a Japanese contributor help get Parcel up and running for multi-language projects: https://qiita.com/zacky1972/items/0ce05454b67506edc634 https://qiita.com/zacky1972/items/0ce05454b67506edc634

chickenfries 9y ago
Is parcel the first node module bundler to run multiple cores? I’ve been waiting for this forever! Definitely going to take a look at the code and try this out soon.
- chmln 9y ago
  I believe webpack can run using multiple threads at least [0] [0] https://github.com/webpack-contrib/thread-loader https://github.com/webpack-contrib/thread-loader
- chrismorgan 9y ago
  There’s a reason why it’s not generally done: there are no good tools for doing this in Node, nor is JavaScript particularly conducive to it. The state of the art in such matters is Rust with its type system-assisted fearless concurrency; my favourite demonstration of how it can work is Rayon, where introducing data parallelism is typically as simple as replacing a .iter() with a .par_iter(). I’d love it if something like Rayon could be ported to Node, somehow. (By virtue of being excellent at the low-level and having zero-cost abstractions, Rust actually ends up being a really good high-level language.) In Node, if you want to process a zillion files in the same way, there are no good solutions. I recently needed to improve the performance of a manually-crafted, Node-centred build process; in Rust, it would have been a matter of: add Rayon, replace a .iter() with .par_iter() and we’re done—and what’s more, it’ll only flex its parallelism muscles where it’s worthwhile. But in Node it took a several hours of trouble (after research, I settled on using worker-farm to help) to produce a strictly inferior result. Rayon: https://github.com/rayon-rs/rayon https://github.com/rayon-rs/rayon worker-farm: https://www.npmjs.com/package/worker-farm https://www.npmjs.com/package/worker-farm

jamesmcintyre 9y ago

From someone who has experienced the dev flow on apps with very large, gnarly webpack configs 17 seconds is extremely impressive and sounds like heaven! What a lot of devs may not be aware of, if their projects aren't of a certain scale & complexity (support for legacy code/library-choices/extreme-modularity/etc.) is that the magical nearly-instant hot-module reloading you experience with a clean, "normal" project should not be taken for granted because webpack builds/reloading can become a monster! Not to say you can't scale up good webpack performance (obviously slack has) just that it's not an automatic thing but something that requires some TLC. Thanks for taking the time to share slack team!

Game_Ender 9y ago

Can anyone explain how someone can make a build system and not make correctness and speed the top priorities from the start? Once you do that parallelism is third feature you implement, the first being running an action, and the second being a hash based dependency graph of those actions inputs & outputs. Once you have that dependency graph parallelism and caching just appear pretty much for free. It looks like google uses bazel (a parallel dependency graph based build) internally and they have 1-2 second rebuild times on apps much larger then Slack [0]. 0 - https://medium.com/@Jakeherringbone/what-angular-is-doing-with-bazel-and-closure-21f526f64a34 https://medium.com/@Jakeherringbone/what-angular-is-doing-wi...

swsieber 9y ago
See also, pants and buck. I think it's a story of PostgreSQL vs Mongo - do you start out with conservative correctness and slowly expand your feature set, or do you advertise a lot of (poorly imolemented) features, and with the large userbase slowly increase your quality.

jsmeaton 9y ago

Hoping that someone might have some recommendations for a problem that we have in particular. We have multiple sites, each of which has it's own theme. Our various sass files check the theme variable passed into the sass loader, and decide on the correct values for particular styles. Our react components import the stylesheet and reference values accordingly. We also have a spattering of old backbone on some pages, so it's not just react involved. Unfortunately, this means that we have to build our entire application for each theme. This takes an excruciating amount of time. Parallel Webpack makes it somewhat manageable, but I'd like to eliminate building for each theme altogether. Any ideas on paths we can look into? If our application was all React, I'd consider styled components or a theme HOC. It's not though, so I've put those options on the back burner for the moment.

mbrumlow 9y ago

I hate webpack, it has been a hack from the start. I can't wait till HTTP2 is more widely supported. My latest projects use polymer and a custom built service (in go) to automatically parse entry points to my application and index the required files. When a HTTP2 connection is made -- and the browser supports push, I will automatically push all the required files. With browsers like chrome the results are near instant and I have full control over what gets pushed or not. Personally I think we need to stop hacking the web together and properly solve these problems. I think HTTP2 with push is a good step in that direction.

jaredklewis 9y ago
I don't see how HTTP2 can replace webkack. Even if websites stop making single file bundles of JS, I think we'll still need something to: - Transpile from latest ES standards, to browser supported code - Compile from non-JS languages like TypeScript, Flow, and so on - Do dead code elimination, minification, and tree shaking - Bundle code shared by other bundles - Optimize images And on and on... I think webpack is a mess, but I think it really is just a reflection of the complexity of the web platform.
- paulddraper 9y ago
  HTTP/2 can't. But it can't make builds less complex. You're not longer bundling every script, inlining every image, etc.
  - always_good 9y ago
    Still just a drop in the bucket of what Webpack can do, so the HTTP/2 post above shows a major misunderstanding of what people are using Webpack for, and probably what HTTP/2 actually is. I don't see how you can work on a modern JS application, understand HTTP/2, and think HTTP/2 replaces Webpack. Pick two...
    - mbrumlow 9y ago
      Yes, you don't agree with what I said so you attack the author and not the content. Webpack is used for a lot of things -- that it should not be used for. As I posted above many of the things it is used for should out of band of your blunder. HTTP2 removes the need for bundling by allowing a smart web server either learn, or be told what files will be required for rendering to be pushed on the first request from the client. This eliminates the multiple request / and connections required in HTTP1 -- even with 4 pipelines that can be a lot of round trip request. If your application was not smart enough to know what to push to the client HTTP2 can allow multiple request over the same connection again negating the need for bundling.