7 ms·

The “Why Rolldown” section cites two reasons: 1) esbuild isn’t good at splitting 2) rollup is too slow Why wasn’t it an option to improve esbuild’s splitting

The “Why Rolldown” section cites two reasons: 1) esbuild isn’t good at splitting 2) rollup is too slow Why wasn’t it an option to improve esbuild’s splitting functionality or to improve rollup performance? Why is it the best option to introduce yet another tool?

truckerbill 3y ago
A lot of the time the politics gets in the way. ‘It’s broken on purpose’ etc
danielheath 3y ago
> Why wasn’t it an option to improve esbuild’s splitting functionality or to improve rollup performance? Rollup is written in javascript; not only is the runtime slow, the multicore story is terrible. Improving esbuilds splitting functionality is hard because splitting is full of hard tradeoffs; the spec allows module imports to have side-effects and requires that they be evaluated in order. It's therefore impossible to do optimal code-splitting without violating the esmodule spec (potentially introducing subtle errors), and the `esbuild` project values correctness even more highly than performance. Personally, I'd be very happy with "If you enable the 'make it fast' option and have modules with top level side-effects, you will get weird bugs", but I respect where they're coming from in not wanting to do that.
- VincentEvans 3y ago
  Speaking of things “written in javascript” for the longest time all I kept hearing is how incredibly fast things written js are… Is it fair to say that it was a bit of a sales pitch? When key components of the ecosystem that were written in the language of the ecosystem itself are being rewritten in other languages - I think it’s hard to find a more compelling evidence that all is not as great as it was advertised.
  - steveklabnik 3y ago
    Everything is relative! Moreover, just because something is fast doesn’t mean it’s the fastest.
    - 9question1 3y ago
      I don't know what OP is referring to here, but from discussions like this: https://stackoverflow.com/a/4417485 https://stackoverflow.com/a/4417485 It seems like circa 2010 folks were referring to JavaScript as "fast" because it was more performant in some ways than other interpreted languages like Ruby and Python and those were the only major popular options at the time that people would consider for some use cases. The event loop model of JavaScript led to some creative use cases for having a single-threaded application handle many highly parallel requests that require very little CPU usage or computation but a large amount of blocking I/O, which I think was also pitched as "fast". But I think it's always been uncontroversial that interpreted languages like JS, Ruby, and Python are all "slow" compared to compiled ones for CPU heavy use cases.
      - steveklabnik 3y ago
        Not just that, but Google pouring way more money and engineering effort into V8 than either the Ruby or Python teams could.
  - azangru 3y ago
    > for the longest time all I kept hearing is how incredibly fast things written js are… I am sure no-one ever claimed that things (properly) written in JS are as fast as things (properly) written in C, Go, or Rust, and running in a native environment. JS on the server may have been faster than php or python, due to the non-blocking nature of the event loop; but even that may not be the case anymore. Generally, the pitch has been that js is "fast enough" for most tasks.
  - solumunus 3y ago
    Node/JS (really any scripting language) is fast enough in scenarios where the productivity gains are worth the performance trade off (example: many web backends). Tooling is not one of those use cases.
    - pdimitar 3y ago
      Golang has the same or better productivity than JS and is objectively running faster. So JS/Node don't have any real edge here.
    - neonsunset 3y ago
      Productivity gains of using Node.js are a myth. You will be better served by C# + ASP.NET Core with massive performance gains while writing about the same amount of code. Or sometimes a little more but cutting down 10 times on ensuring it works correctly.
  - 0x457 3y ago
    Things in JS (node.js) are fast when: a) It's actually calling some native code internally (i.e. rust, c/c++ or V8 internals) b) JIT'ed Path "b" is very hard to achieve in CLI tool
    - robocat 3y ago
      > Path "b" is very hard to achieve in CLI tool Why say that? Yes, perhaps if the CLI tool is being invoked many many times in succession. But otherwise JITing isn't an issue: either (a) your program runs quickly in which case the Ignition interpreter is used and the JIT is not used, or (b) your program runs slowly in which case JIT (Sparkplug/Maglev/Turbofan) has time to warm up.
      - 0x457 3y ago
        > Yes, perhaps if the CLI tool is being invoked many many times in succession. Each execution will be a fresh VM though. A CLI tool isn't a long-running daemon. There is also the whole lack of parallelism without forking in this ecosystem.
  - wredue 3y ago
    JS IS fast compared to many languages. In general use, it’s about as fast as Haskell. Yeah. It’s in the order of 4x slower than Java for most things, but it’s also like 100x faster than Python. It’s all a matter of perspective, but in general, JS is probably about the slowest language I’d be inclined to reach for for long term solutions.
  - nox101 3y ago
    I find VSCode is fast enough for me. I work on a project with ~800k files. VSCode launches fast, finds files fast, greps fast enough, etc... So to me, JS is working as advertised. Choose the right solutions and algorithms and it works fast. I get that VSCode might not be performant for others. I'm on a M1 Mac with 32gig of ram. I never notice any slowdown compared to any other editor I've used. I'm not saying other editors aren't measurably faster. I'm saying I don't notice any issues personally.
- kjksf 3y ago
  It'll be equally hard to write it from scratch in rust as it would be to add this to esbuild. Maybe there are some considerations I don't know about but it does seem like hard forking esbuild would be much faster way to get there (hard fork so that they are not blocked or slowed down by whatever esbuild owner wants).
  - pas 3y ago
    It really depends on who (which team) is doing it. I would almost surely give up before understanding what esbuild does and how, and what needs to be modified. (Because Go basically gives me the creeps.) So someone with a lot of Rust experience probably simply opts for the rewrite. That said, it seems folks on the Vite team decided to give this a try because of oxc_parser (a JS parser written in Rust), so probably it's not because of Rust-Go sentimentalism. // Though ... there's also otto[1] (a JS parser in Go) ... so who knows! :)) // And of course even more context here[2] and here[3] (this one is from Evan You) // [1] https://github.com/robertkrimen/otto https://github.com/robertkrimen/otto [2] https://news.ycombinator.com/item?id=39641634 https://news.ycombinator.com/item?id=39641634 [3] https://news.ycombinator.com/item?id=39648399 https://news.ycombinator.com/item?id=39648399
pjmlp 3y ago
Both of those options imply not using Rust, hence why. I tend to think Rust isn't suitable to rewrite JavaScript tooling, any compiled language would do, or even native addons, but apparently dealing with borrow checker for userspace software that doesn't require 1us performance within 512 KB is appealing. Currently watching https://www.infoq.com/presentations/rust-javascript/ https://www.infoq.com/presentations/rust-javascript/
- michaelmior 3y ago
  > I tend to think Rust isn't suitable to rewrite JavaScript tooling > any compiled language would do So why not Rust?
  - RamblingCTO 3y ago
    Wrong question. Why rust?
    - seanhunter 3y ago
      Because the people who actually want to do the work want to do it in rust.
      - todotask 3y ago
        With both Vite and TailwindCSS v4 written in Tailwind, web development is going to be faster for Astro and other frameworks.
        no_wizard 3y ago
        >TailwindCSS v4 written in Tailwind Didn't realize it was a self bootstrapping language now. Thats awesome
      - RamblingCTO 3y ago
        That's a fair answer. Although I still don't agree with rust being particularly fitting for that problem.
sondr3 3y ago
I can't speak for the authors of Rolldown, but esbuild is essentially a one-man show and everything is bespoke and written for esbuild. Not saying this as a bad thing, Evan single-handedly improved bundling performance for Node and started us down this path of tools in Rust. The libraries and ecosystem for writing tooling for other languages is really taking off with projects like oxc, SWC, LightningCSS, Biome, rslint, Deno, rspack, napi-rs/Neon for JS/TS/Node etc plus ruff, uv, rattler, pixi for Python and so on, so you get a proliferation of libraries and authors that can share and help each other. Plus, on a personal level, using a language with pattern matching and algebraic data types makes writing tooling for parsers and such much more ergonomic than in languages without.
- cAtte_ 3y ago
  note that esbuild is written in go, not rust
  - sondr3 3y ago
    Yep, forgot to mention that in my post, thanks.
  - dunham 3y ago
    Also, it was originally written in both Go and Rust, and the author decided to continue in Go. Both are fine languages, but it makes an interesting case study: https://news.ycombinator.com/item?id=22336284 https://news.ycombinator.com/item?id=22336284
    - pcthrowaway 3y ago
      Wow, thank you for sharing this. The author echoes a lot of thoughts I've had working with rust. However, doesn't the explanation given for why the Go version was 10% faster mean that esbuild was built to take advantage of a fixed number of cores rather than all of them? Sorry if this is a dumb question, I'm not really that experienced with parallel computation
      - 3y ago
swsieber 3y ago
I mean... if it was me: 1) I know Rust, not Go (esbuild is written in go-lang) 2) I don't have to worry as much about breaking Rollup, and I'm not nearly as confident in my ability to write performant JS as I am in writing performant Rust.
- Tade0 3y ago
  My experience is that the main reason why Rust is faster than JS in some applications is that the latter tends to do a lot of copying and other memory-intensive tasks - especially those related to running the JS virtual machine. If you can reduce the problem to a series of operations on pre-allocated typed arrays, JS is fast enough. Unfortunately in the case of Rollup that would be hard to do.
no_wizard 3y ago
esbuild is and has always been governed strongly by Evan Wallace. I can't vouch for whether or not the Evans (both Wallace and You) talked to each other about it, but I'm almost 100% certain Wallaces' vision for the esbuild project is different than what rolldown is ultimately trying to achieve. In some respects, rolldown is also the next generation of Rollup[0] and is a direct answer to rspack[1], which is a webpack compatible rust based toolchain from ByteDance [0]: Which switched to SWC as its underlying parser from acorn, which makes me wonder if that influenced the decision to use OXC, perhaps they found SWC to be troublesome. I know the Rollup and Vite teams are extremely close, and wouldn't be shocked if any troubles they had with SWC weren't shared with the Vite team when considering underlying technologies [1]: https://www.rspack.dev/ https://www.rspack.dev/
Thaxll 3y ago
The real reason is that author wanted to use Rust ( as he previously done ) and does not really care about improving existing solution. From the start you want to use x.y.z tech then the justification is meaningless.
WorldMaker 3y ago
> 1) esbuild isn’t good at splitting I've found esbuild is pretty good at splitting when source formats and outputs are ESM. (Given the name esbuild a focus on/priority for ESM input/output makes sense to me.) Is this maybe Vite team saying that they still have too many brownfield CJS dependencies and still don't believe in ESM output for production bundles in 2024?
- rk06 3y ago
  Even if their dependencies were esm only, Vote may be used in projects which have cjs dependencies. Hence need to support it
  - WorldMaker 3y ago
    esbuild handles CJS dependencies just fine and had been what Vite was using in development for a long time anyway. It doesn't split CJS dependencies or tree-shake them as automatically/smartly as it could, but it still supports splitting them and there are manual workarounds. I think the bigger problem, from my experience here, is still the question of not moving to ESM as the default production output. That seems out of date.
nox101 3y ago
agreed... Personally I feel like moving a JS tool to rust means 99% of JS/TS programmers are unlikely to be able to supply a patch to fix a bug or participate in most ways. I suspect some people will consider that a good thing though.
EvanYou 3y ago
For esbuild: We know other teams that have attempted to improve code splitting based on esbuild and found it very difficult. A big part of it is that in order to be fast, esbuild applies multiple features (bundle, treeshaking, transforms) in as few AST visits possible, but that comes at the cost of the logic of different features not being layered / decoupled nicely. It is difficult for anyone other than Evan Wallace himself to add non-trivial new mechanisms to esbuild, and although we didn't directly talk to Evan about this, we felt it would be too much to ask for a substantial refactor of esbuild in order to get what we need. In addition, the people interested in making Rolldown happen has much more experience in Rust than in Go - and there is a lot more to leverage (e.g. napi-rs & Oxc) in the Rust-for-JS ecosystem. For Rollup: the Rollup team itself has been trying to incrementally improve Rollup's performance, e.g. by swapping acorn with a Rust-based parser. But there's only so much you can gain starting from a pure JavaScript base, especially considering multicore utilization. Another aspect of the performance is in the back-and-forth between Rollup (on the JS side) and native transforms (swc, esbuild) - there is a lot of overhead repeatedly parsing / serializing ASTs and then passing strings across JS/native. By building on top of Oxc (which will ship transforms in the future) we hope to be able to stay on the native-side as much as possible to avoid such overhead.
- catlifeonmars 3y ago
  Thanks for the detailed explanation. It’s really interesting how much of an impact language choice has on outcomes. Given your description of esbuild, it almost feels like there is space for a bundler compiler that can generate an optimized bundler from a pipeline.

truckerbill 3y ago

A lot of the time the politics gets in the way. ‘It’s broken on purpose’ etc

danielheath 3y ago

> Why wasn’t it an option to improve esbuild’s splitting functionality or to improve rollup performance? Rollup is written in javascript; not only is the runtime slow, the multicore story is terrible. Improving esbuilds splitting functionality is hard because splitting is full of hard tradeoffs; the spec allows module imports to have side-effects and requires that they be evaluated in order. It's therefore impossible to do optimal code-splitting without violating the esmodule spec (potentially introducing subtle errors), and the `esbuild` project values correctness even more highly than performance. Personally, I'd be very happy with "If you enable the 'make it fast' option and have modules with top level side-effects, you will get weird bugs", but I respect where they're coming from in not wanting to do that.

VincentEvans 3y ago
Speaking of things “written in javascript” for the longest time all I kept hearing is how incredibly fast things written js are… Is it fair to say that it was a bit of a sales pitch? When key components of the ecosystem that were written in the language of the ecosystem itself are being rewritten in other languages - I think it’s hard to find a more compelling evidence that all is not as great as it was advertised.
- steveklabnik 3y ago
  Everything is relative! Moreover, just because something is fast doesn’t mean it’s the fastest.
  - 9question1 3y ago
    I don't know what OP is referring to here, but from discussions like this: https://stackoverflow.com/a/4417485 https://stackoverflow.com/a/4417485 It seems like circa 2010 folks were referring to JavaScript as "fast" because it was more performant in some ways than other interpreted languages like Ruby and Python and those were the only major popular options at the time that people would consider for some use cases. The event loop model of JavaScript led to some creative use cases for having a single-threaded application handle many highly parallel requests that require very little CPU usage or computation but a large amount of blocking I/O, which I think was also pitched as "fast". But I think it's always been uncontroversial that interpreted languages like JS, Ruby, and Python are all "slow" compared to compiled ones for CPU heavy use cases.
    - steveklabnik 3y ago
      Not just that, but Google pouring way more money and engineering effort into V8 than either the Ruby or Python teams could.
- azangru 3y ago
  > for the longest time all I kept hearing is how incredibly fast things written js are… I am sure no-one ever claimed that things (properly) written in JS are as fast as things (properly) written in C, Go, or Rust, and running in a native environment. JS on the server may have been faster than php or python, due to the non-blocking nature of the event loop; but even that may not be the case anymore. Generally, the pitch has been that js is "fast enough" for most tasks.
- solumunus 3y ago
  Node/JS (really any scripting language) is fast enough in scenarios where the productivity gains are worth the performance trade off (example: many web backends). Tooling is not one of those use cases.
  - pdimitar 3y ago
    Golang has the same or better productivity than JS and is objectively running faster. So JS/Node don't have any real edge here.
  - neonsunset 3y ago
    Productivity gains of using Node.js are a myth. You will be better served by C# + ASP.NET Core with massive performance gains while writing about the same amount of code. Or sometimes a little more but cutting down 10 times on ensuring it works correctly.
- 0x457 3y ago
  Things in JS (node.js) are fast when: a) It's actually calling some native code internally (i.e. rust, c/c++ or V8 internals) b) JIT'ed Path "b" is very hard to achieve in CLI tool
  - robocat 3y ago
    > Path "b" is very hard to achieve in CLI tool Why say that? Yes, perhaps if the CLI tool is being invoked many many times in succession. But otherwise JITing isn't an issue: either (a) your program runs quickly in which case the Ignition interpreter is used and the JIT is not used, or (b) your program runs slowly in which case JIT (Sparkplug/Maglev/Turbofan) has time to warm up.
    - 0x457 3y ago
      > Yes, perhaps if the CLI tool is being invoked many many times in succession. Each execution will be a fresh VM though. A CLI tool isn't a long-running daemon. There is also the whole lack of parallelism without forking in this ecosystem.
- wredue 3y ago
  JS IS fast compared to many languages. In general use, it’s about as fast as Haskell. Yeah. It’s in the order of 4x slower than Java for most things, but it’s also like 100x faster than Python. It’s all a matter of perspective, but in general, JS is probably about the slowest language I’d be inclined to reach for for long term solutions.
- nox101 3y ago
  I find VSCode is fast enough for me. I work on a project with ~800k files. VSCode launches fast, finds files fast, greps fast enough, etc... So to me, JS is working as advertised. Choose the right solutions and algorithms and it works fast. I get that VSCode might not be performant for others. I'm on a M1 Mac with 32gig of ram. I never notice any slowdown compared to any other editor I've used. I'm not saying other editors aren't measurably faster. I'm saying I don't notice any issues personally.
kjksf 3y ago
It'll be equally hard to write it from scratch in rust as it would be to add this to esbuild. Maybe there are some considerations I don't know about but it does seem like hard forking esbuild would be much faster way to get there (hard fork so that they are not blocked or slowed down by whatever esbuild owner wants).
- pas 3y ago
  It really depends on who (which team) is doing it. I would almost surely give up before understanding what esbuild does and how, and what needs to be modified. (Because Go basically gives me the creeps.) So someone with a lot of Rust experience probably simply opts for the rewrite. That said, it seems folks on the Vite team decided to give this a try because of oxc_parser (a JS parser written in Rust), so probably it's not because of Rust-Go sentimentalism. // Though ... there's also otto[1] (a JS parser in Go) ... so who knows! :)) // And of course even more context here[2] and here[3] (this one is from Evan You) // [1] https://github.com/robertkrimen/otto https://github.com/robertkrimen/otto [2] https://news.ycombinator.com/item?id=39641634 https://news.ycombinator.com/item?id=39641634 [3] https://news.ycombinator.com/item?id=39648399 https://news.ycombinator.com/item?id=39648399

pjmlp 3y ago

Both of those options imply not using Rust, hence why. I tend to think Rust isn't suitable to rewrite JavaScript tooling, any compiled language would do, or even native addons, but apparently dealing with borrow checker for userspace software that doesn't require 1us performance within 512 KB is appealing. Currently watching https://www.infoq.com/presentations/rust-javascript/ https://www.infoq.com/presentations/rust-javascript/

michaelmior 3y ago
> I tend to think Rust isn't suitable to rewrite JavaScript tooling > any compiled language would do So why not Rust?
- RamblingCTO 3y ago
  Wrong question. Why rust?
  - seanhunter 3y ago
    Because the people who actually want to do the work want to do it in rust.
    - todotask 3y ago
      With both Vite and TailwindCSS v4 written in Tailwind, web development is going to be faster for Astro and other frameworks.
      - no_wizard 3y ago
        >TailwindCSS v4 written in Tailwind Didn't realize it was a self bootstrapping language now. Thats awesome
    - RamblingCTO 3y ago
      That's a fair answer. Although I still don't agree with rust being particularly fitting for that problem.

sondr3 3y ago

I can't speak for the authors of Rolldown, but esbuild is essentially a one-man show and everything is bespoke and written for esbuild. Not saying this as a bad thing, Evan single-handedly improved bundling performance for Node and started us down this path of tools in Rust. The libraries and ecosystem for writing tooling for other languages is really taking off with projects like oxc, SWC, LightningCSS, Biome, rslint, Deno, rspack, napi-rs/Neon for JS/TS/Node etc plus ruff, uv, rattler, pixi for Python and so on, so you get a proliferation of libraries and authors that can share and help each other. Plus, on a personal level, using a language with pattern matching and algebraic data types makes writing tooling for parsers and such much more ergonomic than in languages without.

cAtte_ 3y ago
note that esbuild is written in go, not rust
- sondr3 3y ago
  Yep, forgot to mention that in my post, thanks.
- dunham 3y ago
  Also, it was originally written in both Go and Rust, and the author decided to continue in Go. Both are fine languages, but it makes an interesting case study: https://news.ycombinator.com/item?id=22336284 https://news.ycombinator.com/item?id=22336284
  - pcthrowaway 3y ago
    Wow, thank you for sharing this. The author echoes a lot of thoughts I've had working with rust. However, doesn't the explanation given for why the Go version was 10% faster mean that esbuild was built to take advantage of a fixed number of cores rather than all of them? Sorry if this is a dumb question, I'm not really that experienced with parallel computation
    - 3y ago

swsieber 3y ago

I mean... if it was me: 1) I know Rust, not Go (esbuild is written in go-lang) 2) I don't have to worry as much about breaking Rollup, and I'm not nearly as confident in my ability to write performant JS as I am in writing performant Rust.

Tade0 3y ago
My experience is that the main reason why Rust is faster than JS in some applications is that the latter tends to do a lot of copying and other memory-intensive tasks - especially those related to running the JS virtual machine. If you can reduce the problem to a series of operations on pre-allocated typed arrays, JS is fast enough. Unfortunately in the case of Rollup that would be hard to do.

no_wizard 3y ago

esbuild is and has always been governed strongly by Evan Wallace. I can't vouch for whether or not the Evans (both Wallace and You) talked to each other about it, but I'm almost 100% certain Wallaces' vision for the esbuild project is different than what rolldown is ultimately trying to achieve. In some respects, rolldown is also the next generation of Rollup[0] and is a direct answer to rspack[1], which is a webpack compatible rust based toolchain from ByteDance [0]: Which switched to SWC as its underlying parser from acorn, which makes me wonder if that influenced the decision to use OXC, perhaps they found SWC to be troublesome. I know the Rollup and Vite teams are extremely close, and wouldn't be shocked if any troubles they had with SWC weren't shared with the Vite team when considering underlying technologies [1]: https://www.rspack.dev/ https://www.rspack.dev/

Thaxll 3y ago

The real reason is that author wanted to use Rust ( as he previously done ) and does not really care about improving existing solution. From the start you want to use x.y.z tech then the justification is meaningless.

WorldMaker 3y ago

> 1) esbuild isn’t good at splitting I've found esbuild is pretty good at splitting when source formats and outputs are ESM. (Given the name esbuild a focus on/priority for ESM input/output makes sense to me.) Is this maybe Vite team saying that they still have too many brownfield CJS dependencies and still don't believe in ESM output for production bundles in 2024?

rk06 3y ago
Even if their dependencies were esm only, Vote may be used in projects which have cjs dependencies. Hence need to support it
- WorldMaker 3y ago
  esbuild handles CJS dependencies just fine and had been what Vite was using in development for a long time anyway. It doesn't split CJS dependencies or tree-shake them as automatically/smartly as it could, but it still supports splitting them and there are manual workarounds. I think the bigger problem, from my experience here, is still the question of not moving to ESM as the default production output. That seems out of date.

nox101 3y ago

agreed... Personally I feel like moving a JS tool to rust means 99% of JS/TS programmers are unlikely to be able to supply a patch to fix a bug or participate in most ways. I suspect some people will consider that a good thing though.

EvanYou 3y ago

For esbuild: We know other teams that have attempted to improve code splitting based on esbuild and found it very difficult. A big part of it is that in order to be fast, esbuild applies multiple features (bundle, treeshaking, transforms) in as few AST visits possible, but that comes at the cost of the logic of different features not being layered / decoupled nicely. It is difficult for anyone other than Evan Wallace himself to add non-trivial new mechanisms to esbuild, and although we didn't directly talk to Evan about this, we felt it would be too much to ask for a substantial refactor of esbuild in order to get what we need. In addition, the people interested in making Rolldown happen has much more experience in Rust than in Go - and there is a lot more to leverage (e.g. napi-rs & Oxc) in the Rust-for-JS ecosystem. For Rollup: the Rollup team itself has been trying to incrementally improve Rollup's performance, e.g. by swapping acorn with a Rust-based parser. But there's only so much you can gain starting from a pure JavaScript base, especially considering multicore utilization. Another aspect of the performance is in the back-and-forth between Rollup (on the JS side) and native transforms (swc, esbuild) - there is a lot of overhead repeatedly parsing / serializing ASTs and then passing strings across JS/native. By building on top of Oxc (which will ship transforms in the future) we hope to be able to stay on the native-side as much as possible to avoid such overhead.

catlifeonmars 3y ago
Thanks for the detailed explanation. It’s really interesting how much of an impact language choice has on outcomes. Given your description of esbuild, it almost feels like there is space for a bundler compiler that can generate an optimized bundler from a pipeline.