18 ms·

PR that converts the TypeScript repo from namespaces to modules

338 points by Kyza 4y ago

Kyza 4y ago
> TL;DR: The TypeScript compiler is now implemented internally with modules, not namespaces. The compiler is now 10-25% faster. tsc is now 30% faster to start. Our npm package is now 43% smaller. More improvements are in the works.
- deleted 4y ago
  [deleted]
KingOfCoders 4y ago
- svnpenn 4y ago
  thats pretty harsh. I dont even like TypeScript/JavaScript, but these are some pretty good numbers: > The compiler is now 10-25% faster. `tsc` is now 30% faster to start. Our npm package is now 43% smaller. More improvements are in the works.
- TAForObvReasons 4y ago
  In every single one of these cases, it begs the question whether "JS" is the problem or "backwards compatibility". There are a huge number of inefficiencies that can be fixed with a full rewrite. There was a recent conversation over ViteJS (a pure JS bundler) vs rust-based tooling, and when you dig into the numbers the real difference is SWC vs Babel. It raises the question whether a new transpiler written in JS can be competitive with Rust, but it's unclear if anyone tried.
  - ppseafield 4y ago
    SWC is written in rust, and Babel is written in javascript, so you've proven the OP's point. Vite can be configured to use SWC. The recent conversation was about Vercel benchmarking a tuned Turbopack+SWC vs. a default Vite+Babel, not really an apples to apples comparison. When Vite is configured to use SWC as a compiler, Vite's HMR gets faster; but it's not the default for compatibility reasons.
    - longrod 4y ago
      Sucrase is faster or really close to SWC (see rhe benchmarks https://github.com/alangpierce/sucrase https://github.com/alangpierce/sucrase). Everyone still uses Babel because of the transforms. And yes, Babel can also be made faster if enough effort is dedicated into it. it's not an impossible feat.
      - ppseafield 4y ago
        For single core benchmarks only, correct?
  - RyanCavanaugh 4y ago
    Sucrase is written in JS and boasts the highest line throughput of any competing transpiler https://github.com/alangpierce/sucrase https://github.com/alangpierce/sucrase Time Speed Sucrase 0.57 seconds 636975 lines per second swc 1.19 seconds 304526 lines per second esbuild 1.45 seconds 248692 lines per second TypeScript 8.98 seconds 40240 lines per second Babel 9.18 seconds 39366 lines per second
    - merb 4y ago
      but the benchmark is stupid: https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts https://github.com/alangpierce/sucrase/blob/main/benchmark/b... > Like all JavaScript code run in V8, Sucrase runs more slowly at first, then gets faster as the just-in-time compiler applies more optimizations. From a rough measurement, Sucrase is about 2x faster after running for 3 seconds than after running for 1 second. swc (written in Rust) and esbuild (written in Go) don't have this effect because they're pre-compiled, so comparing them with Sucrase gets significantly different results depending on how large of a codebase is being tested and whether each compiler is allowed a "warm up" period before the benchmark is run. (worse it disables esbuild and swc's multi-threading... https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts#L257 https://github.com/alangpierce/sucrase/blob/main/benchmark/b... https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts#L231 https://github.com/alangpierce/sucrase/blob/main/benchmark/b...) fake it till ya make it. it's like saying "if I disable everything and wait for 5 minutes it's faster"
      - RyanCavanaugh 4y ago
        I don't see how this is a rebuttal to the claim. 636,975 is more than 2x 304,526, so assuming the quoted paragraph is correct, sucrase is still the highest-throughput transpiler even during its warm-up phase. Probably this isn't true for the first 100 milliseconds of execution during the very first warm-up stages, but if the transpile phase is that short, it's basically irrelevant anyway.
        merb 4y ago
- speedgoose 4y ago
  I think tooling of one programming language should be written in the programming language. Otherwise the community will hardly be involved. All these rust tools may be slight faster, but typescript developers will not learn rust to improve the typescript compiler.
  - llanowarelves 4y ago
    What's the conversion rate on that though? For smaller languages that are much less supported that makes much more sense. For bigger languages we treat it much more as a black box.
  - berkes 4y ago
    If you think of typescript as "tooling" that makes sense. But if you think of it as compiler, or transpiler, not so much. Python, Ruby etc "compilers" (runtimes, really) aren't written in Python or Ruby. Or rather, there are alternatives around, but they aren't the fastest nor the best. Such "tools" are built in C. As are compilers for C++, Rust or most languages really.
    - speedgoose 4y ago
      I understand your point. Compilers and interpreters are traditionally written in C/C++, but we don't have to do the same than before.
  - satvikpendem 4y ago
    > but typescript developers will not learn rust to improve the typescript compiler. Why not? TS has much more in common with Rust than with JS, in terms of typing. Compiler engineers are not that prevalent in the community, but they are much more prevalent in the systems engineering world which Rust is a part of, so I'd assume it's actually more beneficial to have a compiler in Rust than in TS because it would mean more people interested in contributing.
    - shepherdjerred 4y ago
      > TS has much more in common with Rust than with JS, in terms of typing. TS has much more in common with JS than it does with Rust in every other comparison Every TS dev will know at least a little bit of JS. Not every TS dev will know Rust at all. They're very different languages.
- Kyza 4y ago
  I think a performance improvement in anything existing is great. Especially for something as big as TSC. It has such a large amount of usage that many people will appreciate. While it would be great to have it reimplemented effectively in something inherently faster, not improving what we have while it's clearly still the best option wouldn't be helpful.
- shepherdjerred 4y ago
  > I don't care. I care when it's implemented in Rust. What does this add to the discussion? This comment serves nothing except to be negative to an author who has made a great improvement.
longrod 4y ago
Sucrase is proof that JS is not the problem when it comes to slow performance. JS is not slow. NodeJS is not slow. It's the code that is slow. All these people wanting to write it in Rust or Go or XYZ programming language need to acknowledge this. Yes, multithreading is awesome and really helpful but it's the cherry on top, not the whole thing. If the same amount of effort was put into optimizing the TSC codebase as it is being spent to rewrite it in Rust, I have no doubt that it can become faster. Perhaps it'll require some big changes but it won't create compatibility concerns and it won't be a cat-and-mouse race between the Rust version and the TS version. I don't think "Write it in Rust" is always the solution to fast programs. Rust itself can be pretty damn slow if you don't keep performance in mind. That is why you have to optimize and profile and optimize over and over again. Can't the same be done for TSC? I think the biggest reason devs don't do this is because no one likes profiling and optimizing since it is a slow and boring task. Rewriting is so exciting! It's the thing you do when you are tired of maintaining the old codebase. So just ditch it and rewrite it in Rust. I have nothing against Rust, mind you. I love what it has done but I don't think rewriting everything is either feasible or even the solution. And waiting for that to happen for every slow tool out there is utter foolishness.
- Analemma_ 4y ago
  Sucrase consciously and deliberately breaks compatibility. Which, to be clear, isn't necessarily a bad thing for some use cases. But you can't really generalize from that to a tool like tsc where this isn't an option. There might be a performance ceiling here that can only be surpassed with a different language.
  - swsieber 4y ago
    I suspect you have a point, given this line from the Sucrase readme: > Because of this smaller scope, Sucrase can get away with an architecture that is much more performant but less extensible and maintainable. Sucrase's parser is forked from Babel's parser (so Sucrase is indebted to Babel and wouldn't be possible without it) and trims it down to a focused subset of what Babel solves. If it fits your use case, hopefully Sucrase can speed up your development experience!
simlevesque 4y ago
The headline is pretty misleading. The TypeScript compiler isn't implemented with modules. This is an open pull request which would make it that way.
- RyanCavanaugh 4y ago
  It's a PR that will be merged within the next week or so, and has been scheduled in the TS roadmap for quite a while. It's happening.
  - simlevesque 4y ago
    Hi Ryan, thanks for chiming in ! I just wanted to make it clear that the current HEAD doesn't implement this but maybe I'm just nitpicking.
- Kyza 4y ago
  Oops, I didn't mean to mislead. I updated the title to be more clear.
  - simlevesque 4y ago
    Thank you
swsieber 4y ago
> Finally, as a result of both of the previous performance improvements (faster code and less of it), tsc.js is 30% faster to start and typescript.js (our public API) is 10% faster to import. As we improve performance and code size, these numbers are likely to improve. We now include these metrics in our benchmarks to track over time and in relevant PRs. > [...] > The TypeScript package now targets ES2018. Prior to 5.0, our package targeted ES5 syntax and the ES2015 library, however, esbuild has a hard minimum syntax target of ES2015 (aka ES6). ES2018 was chosen as a balance between compatibility with older environments and access to more modern syntax and library features I'd be curious as to what percentage of the improvement comes from modules vs comes from a different target.
- berkes 4y ago
  I'm curious about that too. From my superficial knowledge of compilers, "modularization" itself should not make code faster, if anything slower. There'll always be some overhead of loading modules and communicating between them, not? I presume, from my own experience when building software (not compilers), that modules allow for a much easier to reason about, much better isolated (cohesion, loose coupling). And therefore for much easier improvements inside the module. I would presume that, here too, modules allowed them to improve the inner workings much better, allowing for the performance increase. Or am I completely misunderstanding this feature?
  - akiselev 4y ago
    For ES6 modules, the exports object is frozen (made read-only) so the JIT can make some extra assumptions and optimizations. With bundles, unless the bundler inserts `Object.freeze` around `module.exports`, they have to be treated as dynamic objects.
  - jakebailey 4y ago
    There are some key things here that maybe weren't clearly stated in my writeup. Firstly, the old codebase is TS namespaces, which compile down to IIFEs that push properties onto objects. Each file that declares that namespace is its own IIFE, and so every access to other files incurs the overhead of a property access. With modules, tooling like esbuild, rollup, can now actually see those dependencies (now they are standard ES module imports) and optimize access to them. In this PR's case, the main boost comes from scope hoisting. For example, in one file, we may declare the helper `isIdentifier`. In namespaces, we would write `isIdentifier` in another file, but this would at emit time turn into `ts.isIdentifier`, which is slower. Now, we import that helper, and then esbuild (or rollup) can see that exact symbol. All of the helpers get pulled to the top of the output bundle, and calls to those helpers are direct. That's why modules gives us a boost. There's also more (modules means we can use tooling to tree shake the output, and smaller bundles are faster to load), but the hoisting is the big thing.
MitchellCash 4y ago
> a change in the indentation used in our bundle files (4 spaces -> 2 spaces) I find it interesting that one of the reasons given for the reduction in package size is due to such a simple indentation change from 4 spaces to 2 spaces. Not interesting that 2 bytes are less than 4 bytes, rather, TypeScript is a large project and it would be interesting to know how much size was saved from this one specific change? Seems like a trivial change, so why not do it sooner? And assuming readability isn't required in the bundle output why not bundle with no indentation at all and put everything on a single line, would this not be even smaller again?
- RyanCavanaugh 4y ago
  > Seems like a trivial change, so why not do it sooner? Re: indentation: Literally, no one thought of it, as far as anyone can tell. Linus's law appears to have its limits.
  - robertlagrant 4y ago
    Most minifiers already put things on one line, though.
    - jakebailey 4y ago
      TS has some unique restrictions due to downstream patching of our package; my PR description briefly talks about this as something we can try to improve in the future. Minification absolutely would save a lot more size out of our package, but I was not willing to change that in this PR.
- IshKebab 4y ago
  Kind of funny because one of the benefits of tabs vs spaces that people laugh off is that it saves space. I think it's probably correct to laugh this off though. Why would you care about the non-minified/gzipped size this much?
  - yrgulation 4y ago
msoad 4y ago
This opens up the door to going in there and replacing the slow parts with Rust. There is a lot of interest in making typescript fast but all of them want to do a rewrite. Swapping the slow parts seems a lot more viable
zebracanevra 4y ago
After this change, the TypeScript compiler will now be compiled with esbuild. I feel like thats probably the best endorsement esbuild could get, hah. Surprising they call out the 2 space indent level that esbuild is hardcoded[1] to use as a benefit. Why not save even more bytes and re-format the output to single tab indentation? I wrote a simple script to replace the indentation with tabs. 2 indent size: 29.2MB, tabbed size: 27.3MB. 2MB more of indentation saved! Not significant after compression, but parsing time over billions of starts? Definitely worth it. [1] https://github.com/evanw/esbuild/issues/1126 https://github.com/evanw/esbuild/issues/1126
- quickthrower2 4y ago
  Reminds me of Silicon Valley (HBO) where Richard uses “we are a compression company” to justify using tabs over spaces. Ironically once gzip compressed I doubt it would make any difference.
- Krizzu 4y ago
  Why not suggest this in the PR? Seems like it could be used in pipeline to decrease size further down
- CodeWriter23 4y ago
  > Why not save even more bytes and re-format the output to single tab indentation? For your answer search "programmers who use spaces make more money"
  - z3c0 4y ago
    TLDR "Programmers who use spaces are more likely to respond to StackOverflow surveys soliciting information about their income."
    - shynrou 4y ago
      That's not how statistics work, unless you assume there is a correlation between tab users salary and their willingness to respond to surveys. Which would also be an interesting find
redox99 4y ago
I absolutely hate how with Typescript and ES Modules, if you have a file utils/foo.ts you have to import it as import Foo from "utils/foo.js" Even though there is no .js file on disk, and you might be running ts-node or whatever that doesn't build a .js file. Importing a file that "doesn't exist" is so counterintuitive. In addition all code breaks because you have to change all your imports, and /index.ts or /index.js won't work either.
- kuramitropolis 4y ago
  A thousand times this. It's not only the dumbest thing I've seen a programming language do, it's also dumbest thing I've seen in the JS ecosystem. Ended up having to implement an AST-based post-processor to fix packages before publishing them.
- bigyikes 4y ago
  Every TypeScript project I have worked on either: 1) enforces no extension, e.g. “utils/foo”, or 2) allows TS extensions, e.g. “utils/foo.ts” I have never imported a TS file using a JS extension. Maybe your woes could be fixed with a configuration change?
  - dickfickling 4y ago
    have you worked on Typescript projects using ES modules? What you’ve described is the status quo for CommonJS modules, but doesn’t work when you switch to ESM (afaik, at least)
  - sebdufbeau 4y ago
    From my understanding, this is a fairly new predicament, for projects that target ESM (type module in package.json) instead of the default CJS
  - cypress66 4y ago
    And that's how it should work imo. But if you enable esm (which you might need in the future because of packages being esm only) you can't use those, only .js. That's because typescript developers are dead set that they don't want to transpile the imports, they just want to copy paste them into the resulting file when running tsc.
darig 4y ago
breck 4y ago
Bravo. This must have been painful. Super excited to use a faster tsc. That will make a huge difference in our products. Thank you.
siftrics 4y ago
Jake Bailey was one of the best TAs I ever had in college. It's great to see his name behind this.
- jakebailey 4y ago
  I miss the Piazza days... Thank you for the kind words!
throwup 4y ago
Does anyone have any insight about how to coordinate this kind of change to a large project? This kind of change touches literally every file, so every branch will have merge conflicts. The best idea I can think of is to announce the date ahead of time and make every contributor rebase their branches on the day of the merge. But there has to be a better way. > | edit [–] | on: PR that converts the TypeScript repo from namespac...
- jakebailey 4y ago
  Surprisingly, at least for this PR, solving merge conflicts turns out to not be too hard. By not squash merging it, we can have a single commit that unindents the codebase all in one go (and the commit is in the tree), which means that every line has a clear path back to the current state of the main branch. (And crucially, we can make git blame not point every line to me...) Potentially, an approach like this might be applicable to other changes; I have a commit in my stack which moves the old build system config to the new build system config's path (even though it's wrong), as git does a much better job understanding where the code is going if you help it.
tantalor 4y ago
500k lines changed, oof. Imagine the merge conflicts. Could this not have been done incrementally?
- jakebailey 4y ago
  This is as incremental as it really could be; the entire build had to change, all of the code needed to be unindented one level, etc. I have tested the merge conflict problem, and thanks to the way the PR is constructed (in steps), git actually does a good job figuring things out.
  - nikeee 4y ago
    I'm curious on how to make such a change in such a large and living code base. Did you consider _not_ re-indenting the entire code base and rely on the auto formatters of contributing people? Did you consider re-formatting parts of the code step-by-step, beginning with code that's not frequently changed? Didn't know about the feature that ignores commits in git-blame, thank you for that.
    - jakebailey 4y ago
      No formatter would support having code indented like that at the top level for no reason, so that's not an option. Though, I have been looking into getting us to use a formatter, period (right now we don't). I didn't consider doing it step by step, no, but I'm not sure how I would really achieve that effectively. The reality is that the bulk of PRs are submitted by the team, and it's straightforward to get everyone to get their code in before this change, pause merges, rebase, and then continue as normal. I'd rather go for the "rip the band-aid off" method.
      - tantalor 4y ago
        > pause merges That's what I was worried about, but I guess you can't have merge conflicts if you freeze the code. Suppose you couldn't freeze the code, then what would you have done?
iainmerrick 4y ago
A little while ago I asked (https://news.ycombinator.com/item?id=33051021 https://news.ycombinator.com/item?id=33051021): I’m curious, how many people are using TSC only for type-checking, and a different system (eg esbuild or ts-node) to actually compile/bundle/execute their code? Looks like my suspicion was correct; not even tsc uses tsc!
- presentation 4y ago
  Probably the majority since popular frameworks like NextJS do it with SWC now.
  - WorldMaker 4y ago
    The default configurations for Create-React-App and others use babel for type stripping today. This seems to me like a great "win" for Typescript that so many tools just natively handle TS type stripping and that so much Typescript today only needs type stripping and doesn't need other parts of TS emit processes (or tslib).
styfle 4y ago
Thanks for doing this! I reported typescript install size issue back in 2018 and changing to modules seemed to have the biggest impact here! https://github.com/microsoft/TypeScript/issues/23339 https://github.com/microsoft/TypeScript/issues/23339 For anyone curious, TS 1.0 was 7MB and today it’s 65MB. https://packagephobia.com/result?p=typescript%401.0.1%2Ctypescript%402.0.2%2Ctypescript%403.0.1%2Ctypescript%404.0.2%2Ctypescript%404.8.4 https://packagephobia.com/result?p=typescript%401.0.1%2Ctype... Really excited to see this number move in the downward direction for a change :)
- jakebailey 4y ago
  These days, it's tracked at https://github.com/microsoft/TypeScript/issues/27891 https://github.com/microsoft/TypeScript/issues/27891. I have a gameplan to drop this by another 7 MB (by turning our executables into ESM), probably for 5.0 as well if we decide that Node versions older than 12 are worth dropping.

338 points by Kyza 4y ago

Kyza 4y ago

> TL;DR: The TypeScript compiler is now implemented internally with modules, not namespaces. The compiler is now 10-25% faster. tsc is now 30% faster to start. Our npm package is now 43% smaller. More improvements are in the works.

deleted 4y ago
[deleted]

KingOfCoders 4y ago

svnpenn 4y ago
thats pretty harsh. I dont even like TypeScript/JavaScript, but these are some pretty good numbers: > The compiler is now 10-25% faster. `tsc` is now 30% faster to start. Our npm package is now 43% smaller. More improvements are in the works.
TAForObvReasons 4y ago
In every single one of these cases, it begs the question whether "JS" is the problem or "backwards compatibility". There are a huge number of inefficiencies that can be fixed with a full rewrite. There was a recent conversation over ViteJS (a pure JS bundler) vs rust-based tooling, and when you dig into the numbers the real difference is SWC vs Babel. It raises the question whether a new transpiler written in JS can be competitive with Rust, but it's unclear if anyone tried.
- ppseafield 4y ago
  SWC is written in rust, and Babel is written in javascript, so you've proven the OP's point. Vite can be configured to use SWC. The recent conversation was about Vercel benchmarking a tuned Turbopack+SWC vs. a default Vite+Babel, not really an apples to apples comparison. When Vite is configured to use SWC as a compiler, Vite's HMR gets faster; but it's not the default for compatibility reasons.
  - longrod 4y ago
    Sucrase is faster or really close to SWC (see rhe benchmarks https://github.com/alangpierce/sucrase https://github.com/alangpierce/sucrase). Everyone still uses Babel because of the transforms. And yes, Babel can also be made faster if enough effort is dedicated into it. it's not an impossible feat.
    - ppseafield 4y ago
      For single core benchmarks only, correct?
- RyanCavanaugh 4y ago
  Sucrase is written in JS and boasts the highest line throughput of any competing transpiler https://github.com/alangpierce/sucrase https://github.com/alangpierce/sucrase Time Speed Sucrase 0.57 seconds 636975 lines per second swc 1.19 seconds 304526 lines per second esbuild 1.45 seconds 248692 lines per second TypeScript 8.98 seconds 40240 lines per second Babel 9.18 seconds 39366 lines per second
  - merb 4y ago
    but the benchmark is stupid: https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts https://github.com/alangpierce/sucrase/blob/main/benchmark/b... > Like all JavaScript code run in V8, Sucrase runs more slowly at first, then gets faster as the just-in-time compiler applies more optimizations. From a rough measurement, Sucrase is about 2x faster after running for 3 seconds than after running for 1 second. swc (written in Rust) and esbuild (written in Go) don't have this effect because they're pre-compiled, so comparing them with Sucrase gets significantly different results depending on how large of a codebase is being tested and whether each compiler is allowed a "warm up" period before the benchmark is run. (worse it disables esbuild and swc's multi-threading... https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts#L257 https://github.com/alangpierce/sucrase/blob/main/benchmark/b... https://github.com/alangpierce/sucrase/blob/main/benchmark/benchmark.ts#L231 https://github.com/alangpierce/sucrase/blob/main/benchmark/b...) fake it till ya make it. it's like saying "if I disable everything and wait for 5 minutes it's faster"
    - RyanCavanaugh 4y ago
      I don't see how this is a rebuttal to the claim. 636,975 is more than 2x 304,526, so assuming the quoted paragraph is correct, sucrase is still the highest-throughput transpiler even during its warm-up phase. Probably this isn't true for the first 100 milliseconds of execution during the very first warm-up stages, but if the transpile phase is that short, it's basically irrelevant anyway.
      - merb 4y ago
speedgoose 4y ago
I think tooling of one programming language should be written in the programming language. Otherwise the community will hardly be involved. All these rust tools may be slight faster, but typescript developers will not learn rust to improve the typescript compiler.
- llanowarelves 4y ago
  What's the conversion rate on that though? For smaller languages that are much less supported that makes much more sense. For bigger languages we treat it much more as a black box.
- berkes 4y ago
  If you think of typescript as "tooling" that makes sense. But if you think of it as compiler, or transpiler, not so much. Python, Ruby etc "compilers" (runtimes, really) aren't written in Python or Ruby. Or rather, there are alternatives around, but they aren't the fastest nor the best. Such "tools" are built in C. As are compilers for C++, Rust or most languages really.
  - speedgoose 4y ago
    I understand your point. Compilers and interpreters are traditionally written in C/C++, but we don't have to do the same than before.
- satvikpendem 4y ago
  > but typescript developers will not learn rust to improve the typescript compiler. Why not? TS has much more in common with Rust than with JS, in terms of typing. Compiler engineers are not that prevalent in the community, but they are much more prevalent in the systems engineering world which Rust is a part of, so I'd assume it's actually more beneficial to have a compiler in Rust than in TS because it would mean more people interested in contributing.
  - shepherdjerred 4y ago
    > TS has much more in common with Rust than with JS, in terms of typing. TS has much more in common with JS than it does with Rust in every other comparison Every TS dev will know at least a little bit of JS. Not every TS dev will know Rust at all. They're very different languages.
Kyza 4y ago
I think a performance improvement in anything existing is great. Especially for something as big as TSC. It has such a large amount of usage that many people will appreciate. While it would be great to have it reimplemented effectively in something inherently faster, not improving what we have while it's clearly still the best option wouldn't be helpful.
shepherdjerred 4y ago
> I don't care. I care when it's implemented in Rust. What does this add to the discussion? This comment serves nothing except to be negative to an author who has made a great improvement.

longrod 4y ago

Sucrase is proof that JS is not the problem when it comes to slow performance. JS is not slow. NodeJS is not slow. It's the code that is slow. All these people wanting to write it in Rust or Go or XYZ programming language need to acknowledge this. Yes, multithreading is awesome and really helpful but it's the cherry on top, not the whole thing. If the same amount of effort was put into optimizing the TSC codebase as it is being spent to rewrite it in Rust, I have no doubt that it can become faster. Perhaps it'll require some big changes but it won't create compatibility concerns and it won't be a cat-and-mouse race between the Rust version and the TS version. I don't think "Write it in Rust" is always the solution to fast programs. Rust itself can be pretty damn slow if you don't keep performance in mind. That is why you have to optimize and profile and optimize over and over again. Can't the same be done for TSC? I think the biggest reason devs don't do this is because no one likes profiling and optimizing since it is a slow and boring task. Rewriting is so exciting! It's the thing you do when you are tired of maintaining the old codebase. So just ditch it and rewrite it in Rust. I have nothing against Rust, mind you. I love what it has done but I don't think rewriting everything is either feasible or even the solution. And waiting for that to happen for every slow tool out there is utter foolishness.

Analemma_ 4y ago
Sucrase consciously and deliberately breaks compatibility. Which, to be clear, isn't necessarily a bad thing for some use cases. But you can't really generalize from that to a tool like tsc where this isn't an option. There might be a performance ceiling here that can only be surpassed with a different language.
- swsieber 4y ago
  I suspect you have a point, given this line from the Sucrase readme: > Because of this smaller scope, Sucrase can get away with an architecture that is much more performant but less extensible and maintainable. Sucrase's parser is forked from Babel's parser (so Sucrase is indebted to Babel and wouldn't be possible without it) and trims it down to a focused subset of what Babel solves. If it fits your use case, hopefully Sucrase can speed up your development experience!

simlevesque 4y ago

The headline is pretty misleading. The TypeScript compiler isn't implemented with modules. This is an open pull request which would make it that way.

RyanCavanaugh 4y ago
It's a PR that will be merged within the next week or so, and has been scheduled in the TS roadmap for quite a while. It's happening.
- simlevesque 4y ago
  Hi Ryan, thanks for chiming in ! I just wanted to make it clear that the current HEAD doesn't implement this but maybe I'm just nitpicking.
Kyza 4y ago
Oops, I didn't mean to mislead. I updated the title to be more clear.
- simlevesque 4y ago
  Thank you

swsieber 4y ago

> Finally, as a result of both of the previous performance improvements (faster code and less of it), tsc.js is 30% faster to start and typescript.js (our public API) is 10% faster to import. As we improve performance and code size, these numbers are likely to improve. We now include these metrics in our benchmarks to track over time and in relevant PRs. > [...] > The TypeScript package now targets ES2018. Prior to 5.0, our package targeted ES5 syntax and the ES2015 library, however, esbuild has a hard minimum syntax target of ES2015 (aka ES6). ES2018 was chosen as a balance between compatibility with older environments and access to more modern syntax and library features I'd be curious as to what percentage of the improvement comes from modules vs comes from a different target.

berkes 4y ago
I'm curious about that too. From my superficial knowledge of compilers, "modularization" itself should not make code faster, if anything slower. There'll always be some overhead of loading modules and communicating between them, not? I presume, from my own experience when building software (not compilers), that modules allow for a much easier to reason about, much better isolated (cohesion, loose coupling). And therefore for much easier improvements inside the module. I would presume that, here too, modules allowed them to improve the inner workings much better, allowing for the performance increase. Or am I completely misunderstanding this feature?
- akiselev 4y ago
  For ES6 modules, the exports object is frozen (made read-only) so the JIT can make some extra assumptions and optimizations. With bundles, unless the bundler inserts `Object.freeze` around `module.exports`, they have to be treated as dynamic objects.
- jakebailey 4y ago
  There are some key things here that maybe weren't clearly stated in my writeup. Firstly, the old codebase is TS namespaces, which compile down to IIFEs that push properties onto objects. Each file that declares that namespace is its own IIFE, and so every access to other files incurs the overhead of a property access. With modules, tooling like esbuild, rollup, can now actually see those dependencies (now they are standard ES module imports) and optimize access to them. In this PR's case, the main boost comes from scope hoisting. For example, in one file, we may declare the helper `isIdentifier`. In namespaces, we would write `isIdentifier` in another file, but this would at emit time turn into `ts.isIdentifier`, which is slower. Now, we import that helper, and then esbuild (or rollup) can see that exact symbol. All of the helpers get pulled to the top of the output bundle, and calls to those helpers are direct. That's why modules gives us a boost. There's also more (modules means we can use tooling to tree shake the output, and smaller bundles are faster to load), but the hoisting is the big thing.

MitchellCash 4y ago

> a change in the indentation used in our bundle files (4 spaces -> 2 spaces) I find it interesting that one of the reasons given for the reduction in package size is due to such a simple indentation change from 4 spaces to 2 spaces. Not interesting that 2 bytes are less than 4 bytes, rather, TypeScript is a large project and it would be interesting to know how much size was saved from this one specific change? Seems like a trivial change, so why not do it sooner? And assuming readability isn't required in the bundle output why not bundle with no indentation at all and put everything on a single line, would this not be even smaller again?

RyanCavanaugh 4y ago
> Seems like a trivial change, so why not do it sooner? Re: indentation: Literally, no one thought of it, as far as anyone can tell. Linus's law appears to have its limits.
- robertlagrant 4y ago
  Most minifiers already put things on one line, though.
  - jakebailey 4y ago
    TS has some unique restrictions due to downstream patching of our package; my PR description briefly talks about this as something we can try to improve in the future. Minification absolutely would save a lot more size out of our package, but I was not willing to change that in this PR.
IshKebab 4y ago
Kind of funny because one of the benefits of tabs vs spaces that people laugh off is that it saves space. I think it's probably correct to laugh this off though. Why would you care about the non-minified/gzipped size this much?
- yrgulation 4y ago

msoad 4y ago

This opens up the door to going in there and replacing the slow parts with Rust. There is a lot of interest in making typescript fast but all of them want to do a rewrite. Swapping the slow parts seems a lot more viable

zebracanevra 4y ago

After this change, the TypeScript compiler will now be compiled with esbuild. I feel like thats probably the best endorsement esbuild could get, hah. Surprising they call out the 2 space indent level that esbuild is hardcoded[1] to use as a benefit. Why not save even more bytes and re-format the output to single tab indentation? I wrote a simple script to replace the indentation with tabs. 2 indent size: 29.2MB, tabbed size: 27.3MB. 2MB more of indentation saved! Not significant after compression, but parsing time over billions of starts? Definitely worth it. [1] https://github.com/evanw/esbuild/issues/1126 https://github.com/evanw/esbuild/issues/1126

quickthrower2 4y ago
Reminds me of Silicon Valley (HBO) where Richard uses “we are a compression company” to justify using tabs over spaces. Ironically once gzip compressed I doubt it would make any difference.
Krizzu 4y ago
Why not suggest this in the PR? Seems like it could be used in pipeline to decrease size further down
CodeWriter23 4y ago
> Why not save even more bytes and re-format the output to single tab indentation? For your answer search "programmers who use spaces make more money"
- z3c0 4y ago
  TLDR "Programmers who use spaces are more likely to respond to StackOverflow surveys soliciting information about their income."
  - shynrou 4y ago
    That's not how statistics work, unless you assume there is a correlation between tab users salary and their willingness to respond to surveys. Which would also be an interesting find

redox99 4y ago

I absolutely hate how with Typescript and ES Modules, if you have a file utils/foo.ts you have to import it as import Foo from "utils/foo.js" Even though there is no .js file on disk, and you might be running ts-node or whatever that doesn't build a .js file. Importing a file that "doesn't exist" is so counterintuitive. In addition all code breaks because you have to change all your imports, and /index.ts or /index.js won't work either.

kuramitropolis 4y ago
A thousand times this. It's not only the dumbest thing I've seen a programming language do, it's also dumbest thing I've seen in the JS ecosystem. Ended up having to implement an AST-based post-processor to fix packages before publishing them.
bigyikes 4y ago
Every TypeScript project I have worked on either: 1) enforces no extension, e.g. “utils/foo”, or 2) allows TS extensions, e.g. “utils/foo.ts” I have never imported a TS file using a JS extension. Maybe your woes could be fixed with a configuration change?
- dickfickling 4y ago
  have you worked on Typescript projects using ES modules? What you’ve described is the status quo for CommonJS modules, but doesn’t work when you switch to ESM (afaik, at least)
- sebdufbeau 4y ago
  From my understanding, this is a fairly new predicament, for projects that target ESM (type module in package.json) instead of the default CJS
- cypress66 4y ago
  And that's how it should work imo. But if you enable esm (which you might need in the future because of packages being esm only) you can't use those, only .js. That's because typescript developers are dead set that they don't want to transpile the imports, they just want to copy paste them into the resulting file when running tsc.

darig 4y ago

breck 4y ago

Bravo. This must have been painful. Super excited to use a faster tsc. That will make a huge difference in our products. Thank you.

siftrics 4y ago

Jake Bailey was one of the best TAs I ever had in college. It's great to see his name behind this.

jakebailey 4y ago
I miss the Piazza days... Thank you for the kind words!

throwup 4y ago

Does anyone have any insight about how to coordinate this kind of change to a large project? This kind of change touches literally every file, so every branch will have merge conflicts. The best idea I can think of is to announce the date ahead of time and make every contributor rebase their branches on the day of the merge. But there has to be a better way. > | edit [–] | on: PR that converts the TypeScript repo from namespac...

jakebailey 4y ago
Surprisingly, at least for this PR, solving merge conflicts turns out to not be too hard. By not squash merging it, we can have a single commit that unindents the codebase all in one go (and the commit is in the tree), which means that every line has a clear path back to the current state of the main branch. (And crucially, we can make git blame not point every line to me...) Potentially, an approach like this might be applicable to other changes; I have a commit in my stack which moves the old build system config to the new build system config's path (even though it's wrong), as git does a much better job understanding where the code is going if you help it.

tantalor 4y ago

500k lines changed, oof. Imagine the merge conflicts. Could this not have been done incrementally?

jakebailey 4y ago
This is as incremental as it really could be; the entire build had to change, all of the code needed to be unindented one level, etc. I have tested the merge conflict problem, and thanks to the way the PR is constructed (in steps), git actually does a good job figuring things out.
- nikeee 4y ago
  I'm curious on how to make such a change in such a large and living code base. Did you consider _not_ re-indenting the entire code base and rely on the auto formatters of contributing people? Did you consider re-formatting parts of the code step-by-step, beginning with code that's not frequently changed? Didn't know about the feature that ignores commits in git-blame, thank you for that.
  - jakebailey 4y ago
    No formatter would support having code indented like that at the top level for no reason, so that's not an option. Though, I have been looking into getting us to use a formatter, period (right now we don't). I didn't consider doing it step by step, no, but I'm not sure how I would really achieve that effectively. The reality is that the bulk of PRs are submitted by the team, and it's straightforward to get everyone to get their code in before this change, pause merges, rebase, and then continue as normal. I'd rather go for the "rip the band-aid off" method.
    - tantalor 4y ago
      > pause merges That's what I was worried about, but I guess you can't have merge conflicts if you freeze the code. Suppose you couldn't freeze the code, then what would you have done?

iainmerrick 4y ago

A little while ago I asked (https://news.ycombinator.com/item?id=33051021 https://news.ycombinator.com/item?id=33051021): I’m curious, how many people are using TSC only for type-checking, and a different system (eg esbuild or ts-node) to actually compile/bundle/execute their code? Looks like my suspicion was correct; not even tsc uses tsc!

presentation 4y ago
Probably the majority since popular frameworks like NextJS do it with SWC now.
- WorldMaker 4y ago
  The default configurations for Create-React-App and others use babel for type stripping today. This seems to me like a great "win" for Typescript that so many tools just natively handle TS type stripping and that so much Typescript today only needs type stripping and doesn't need other parts of TS emit processes (or tslib).

styfle 4y ago

Thanks for doing this! I reported typescript install size issue back in 2018 and changing to modules seemed to have the biggest impact here! https://github.com/microsoft/TypeScript/issues/23339 https://github.com/microsoft/TypeScript/issues/23339 For anyone curious, TS 1.0 was 7MB and today it’s 65MB. https://packagephobia.com/result?p=typescript%401.0.1%2Ctypescript%402.0.2%2Ctypescript%403.0.1%2Ctypescript%404.0.2%2Ctypescript%404.8.4 https://packagephobia.com/result?p=typescript%401.0.1%2Ctype... Really excited to see this number move in the downward direction for a change :)

jakebailey 4y ago
These days, it's tracked at https://github.com/microsoft/TypeScript/issues/27891 https://github.com/microsoft/TypeScript/issues/27891. I have a gameplan to drop this by another 7 MB (by turning our executables into ESM), probably for 5.0 as well if we decide that Node versions older than 12 are worth dropping.