22 ms·

Speeding up the JavaScript ecosystem – Polyfills gone rogue

234 points by jviide 3y ago

azemetre 3y ago
This series is great! You should seriously think about consolidating them into a book. Something I notice other engineers struggle with is how to properly assess performance, read heap snapshots, or even understand how to read a flamegraph for stack tracing tools. It would be nice to point, or buy, them a resource showing this. I'd definitely buy a copy.
- bjnewman85 3y ago
  Second that, I would definitely buy a book based on this series.
  - mhagemeister 3y ago
    Thanks for the kind feedback! It's definitely something in the back of my mind. I feel like I need to collect a little more content to fill a whole book, but I'm enticed by the thought of writing one nonetheless.
    - leipert 3y ago
      Part 3 send me down a debugging route of eslint performance in the GitLab project and we were able to move from 25 min of listing time in CI, down to 5. So, thanks for the inspiration!
cxr 3y ago
You'll never get NPM apologists to acknowledge this. One of their only skills is making non-specific appeals to the necessity of it all (as essential infrastructure) and vague arguments that boil down to "you need to trust the wisdom of the crowds" (and e.g. the fact that it exists and everyone else is using it means that anyone who disputes its value just doesn't understand it—bonus points for them if they manage to work in a slight that's designed to paint you, implicitly or explicitly, as a junior), despite not being able to attest to any firsthand knowledge of the real why of anything they're defending. > The new dependencies were all polyfills for JavaScript functions that have long been supported everywhere. The Object.defineProperties method for example was shipped as part of the very first public Node 0.10.0 release dating back to 2013. Heck, even Internet Explorer 9 supported that. And yet there were numerous packages in that dependend on a polyfill for it.
- CharlesW 3y ago
  > You'll never get NPM apologists to acknowledge this. How should NPM prevent archaic dependencies, or the "even more bizarre" (author's words) problem of developers calling polyfills directly instead of the function that the polyfill fills?
  - whstl 3y ago
    The parent is not criticising NPM the tool/registry, but rather the ecosystem and culture.
    - cxr 3y ago
      I'm happy to criticize NPM the tool. The whole thing is designed as a second, crummier version control system that lives in disharmony with and on top of your base-level version control system (so it can subvert it). It's a terrible design. There's basically at most one reasonable use for npm: as a glorified download manager, i.e. to quickly fetch a module by name (right before you check it in to version control with Git). This differs wildly, of course, from how it's actually used, which is as a drug that sweeps mountains of unaudited code under the rug so people can trick themselves and others into thinking that none of it's really there on the basis that it's not visible when anyone first clones the repo. To answer the other commenter's question, "How should NPM prevent archaic dependencies": it shouldn't; it's okay for programmers to be responsible for their work.
      - spankalee 3y ago
        How is npm a version control system? npm is a fairly standard package manager, much like many others, pretty good even. I don't know of anyone who says that package managers and source control solve the same problems. They both happen to use the word and concept of "version" but to mean different things. Yes, some projects vendor in their dependencies into their source control system, but they must either manually verify package version compatibility or use a package manager like npm to help them do it. And vendoring doesn't work for actual packages published to the package repo. If they vendored dependencies then every dependency would be duplicated always, defeating the very purpose of a package manager!
no_wizard 3y ago
Marvin is doing wonderful work in the JS ecosystem around performance. It has been largely focused on tools and node, however, he did have an interesting set of things to say about how they optimized performance in Preact as well on his website that was really interesting too. One thing I've noticed is the rampant duplication of polyfills and babel helpers. To the point that I now have overrides setup via pnpm and I re-write many imports of polyfills to point at my own shims, which simply re-export existing functionality native to the language, most of the time. For smaller utility packages, I often simply clone the repo and copy things over that we need, or copy the src right out of the node_modules folder if possible, then I strip away all the superfluous imports (and often convert from commonjs to ESM if needed) Saves so much headache, its better for users, smaller builds etc.
- Dextro 3y ago
  That sounds awesome. I've dabled with something like that but only for lodash (makings sure all different flavours get aliased to a single thing) but I never went too far with all the other stuff. You wouldn't happen to have an example of what you're doing laying around would you? I'd be genuinely curious to try stuff like that out.
- mhagemeister 3y ago
  Author here. Thanks for the kind words! It's feedback like this that encourages me to keep writing about it. I share your experiences regarding babel helpers and haven't found a good solution myself. Similar to you, I often patch unnecessary stuff out via patch-package, but that approach doesn't scale well.
  - no_wizard 3y ago
    I use path resolution, it tends to scale better (not great but better) for Babel helpers because ‘Babel-runtime’ can 100% be re-mapped to ‘@babel/runtime’. Same with corejs 2 -> 3, they just need path mapping. Patching packages is definitely something I still have to do to strip polyfills and convert CJS to ESM if I can’t simply re-compile source
jluxenberg 3y ago
For what it's worth; `eslint-plugin-react` has been around for a long time and seems to support running in very old versions of Node.JS (back to v4[1] apparently! tho I can't find anything documenting that for sure.) I was surprised to learn that Object.values is only supported in Node >v7, Object.fronEntries was added in v12, etc. So for this project maybe the polyfills are needed. [1] https://github.com/jsx-eslint/eslint-plugin-react/pull/1038 https://github.com/jsx-eslint/eslint-plugin-react/pull/1038
- mhagemeister 3y ago
  Yeah, engines are a moving target. I'm all for backwards compatibility, but I'm worried about promoting old node versions with known unpatched security issues. Given that eslint itself only supports node >= 12.22.0 it seems like it's time to get rid of the polyfills. I wish we as in the industry would find a better solution to adapt to this. It's a bit unfortunate that the polyfills as part of the library code itself, which makes it difficult to get rid of them once they're not needed anymore.
- gigel82 3y ago
  Genuinely curious, are there people out there using newer versions of this package with old / unsupported versions of Node (in production)?
  - silverwind 3y ago
    Not really. Adoption of new Node versions is quite quick, given their short support periods. https://nodejs.org/metrics/summaries/version.png https://nodejs.org/metrics/summaries/version.png
  - phire 3y ago
    It probably happens, but not really on purpose. If the package.lock file gets deleted or someone runs a global npm-update then npm will update any packages while respecting semantic versioning. It's possible an organisation forgot to include the package.lock file in their deployment image and they get updated npm packages every time they redeploy. It's also possible a developer making minor changes to a legacy system triggers packages to be updated, perhaps without even noticing.
e_y_ 3y ago
> Polyfills that don’t polyfill This is what's sometimes called a "ponyfill". The idea is to avoid messing with global scope (monkeypatching), which could be problematic if you have multiple polyfills for the same API or polyfills that don't perfectly match the native behavior. This can be a good thing in some situations, but in general it's probably best to leave polyfill decisions to the bundler so you can decide which browsers you want to support. Or even produce multiple versions, a lightweight one for modern browsers and one with tons of polyfills that gets served to ancient ones.
- mhagemeister 3y ago
  Author here. Good point. Agree that the ideal scenario would be that the end user (or the tools they use) have the final say in which polyfills to load. It's a bit of a bummer that they are shipped as part of npm packages without an easy way to get rid of them. I wonder if our industry will move to publishing the original source files to npm in the long run. Only the last piece of the chain, the developer using these dependencies, knows what their target environments are. So the bundler could then downlevel or polyfill the code for the specified targets.
  - bsimpson 3y ago
    I've always been kind of surprised that original sources aren't part of the NPM culture. For a long time, I included "typescript:main" in my package.jsons and configured my tools to prefer that to "main" (and now "module").
- MBCook 3y ago
  I get not defining it yourself, especially if your polyfill is limited to the sliver of a feature you use, but why not check if the feature is there first? Ok, maybe someone else monkeypatched it. But at least you’d end up using the native functionality if it was there.
petetnt 3y ago
I don't really care to comment about the practice itself, but the "Polyfills that don’t polyfill" section is missing the point: the function is called directly instead of patching the global object so that the global object is not polluted by an possibly non-standard implementation. Additionally it does use Object.defineProperty if available - furthermore it doesn't even call itself a polyfill in the first place. If it's needed in 2023 is a valid point however.
- fiddlerwoaroof 3y ago
  I think it would be better to just expect the standardized functions to be present and then document that the project needs them (e.g. via peer dependencies), allowing users to install them themselves as needed.
  - shpx 3y ago
    That's a lot more work than your library just working in more places.
    - fiddlerwoaroof 3y ago
      I don’t think it’s all that much more: basically every bundler I’ve used uses browserslist to include polyfills for the developer’s target audience. But, also, I think this sort of easy path inflicts a huge cost on the ecosystem as a whole: writing to the standards and expecting your users to supply a compliant environment solves a lot of N*M problems in the dev process.
ComputerGuru 3y ago
I maintain a few JavaScript libraries that I manually verify compatibility against IE6 (and have lints to catch violations). I manually polyfill a few necessities and quality-of-life improvements up top in the script. Out of curiosity, I removed my polyfills and tried swc and babel both, followed by an eslint pass, and the results were absolutely atrocious. Everything gets polyfilled, even stuff that has been supported in every IE version ever. The usage-based detection is completely borked, and is completely based off string searching property/function names. Using toString() anywhere pulls in the polyfills for Date to string, Regex to string, and Object to string. Using regex anywhere pulls in a bunch of regex polyfills. It was a nightmare and the size of my library increased by orders of magnitude! (I tried opening an swc issue about optionally using typescript ast info (via a plugin, not in swc core) to have more correct usage-based polyfill detection, but that was closed as unlikely to be acted upon.)
- mhagemeister 3y ago
  Author here. That mirrors my experience too on working in various projects. The automatic polyfilling story is such a good thing in theory, but reality isn't as rosy and much more polyfills than necessary are included.
  - ComputerGuru 3y ago
    Thanks for writing your article and sharing! One thing I do on my other libraries is also to polyfill a few heavier things asynchronously instead of making everyone pay the price upfront, so for example I detect if a JSON polyfill is required before asynchronously loading that polyfill. I think the expectation for most things is that the browser will support them by default, and it’s ok if all/any polyfills are loaded separately and asynchronously.
- spankalee 3y ago
  Why are you supporting IE6?
  - ComputerGuru 3y ago
deleted 3y ago
[deleted]
romellem 3y ago
There is some interesting [drama][1] with this, since this article noticeably doesn't mention any PRs they opened to remove some of these older polyfills. The reason those PRs were never opened/merged is the maintainer of many of those libraries [has a strong stance on "breaking" changes][2] in software: > I have developed an intense avoidance for breaking changes in all of my packages, because I don't want to inflict hundreds of millions of dollars of person-hour cost on the entire industry unnecessarily. IMO this argument avoids the opposite claim, that people then spend a ton of time (and money) trying to make old tech work with newer tech since not everyone maintains to the same standards of backwards compatibility. But regardless, no one is required to stick to a particular way of creating open source software, so the one benefit here is that you are free to [fork the library][3] (assuming its license allows for that) to remove some backwards compatibility that isn't relevant to you. [1]: https://twitter.com/ljharb/status/1704912065486618915 https://twitter.com/ljharb/status/1704912065486618915 [2]: https://github.com/import-js/eslint-plugin-import/pull/2447#issuecomment-1119864501 https://github.com/import-js/eslint-plugin-import/pull/2447#... [3]: https://www.npmjs.com/package/react-outside-click-handler-lite https://www.npmjs.com/package/react-outside-click-handler-li...
- potsandpans 3y ago
  Ljharb is harmful to the ECMAScript community
  - silverwind 3y ago
    I'm thankful for his work, but I do agree, this polyfill madness has to stop.
    - potsandpans 3y ago
      I am _appreciative_ that people like him exist, because -- personal feelings aside -- we do need people who have his level of dedication in the open source space. My only interactions with ljharb have been in TC39. More often than not when looking at issues he's active in, I find myself wanting a level of maturity and thoughtfulness that just isn't there. His contributions are neither novel or very compelling. He's more of an open-source bureaucrat than an engineer or a developer, yet he is currently steering several major TC39 proposals. I've seen him shutdown or sidetrack valid input and criticisms in all of them. The vibes I get from his behavior are akin to him being a member of "the cool kid club" -- wanting to maintain that ingroup/outgroup boundary very tightly. He really should just get out of the standards/committee space entirely for the time being, and get some coaching on leadership skills before reentering. I realize this is super negative. Still on the ropes whether or not this feedback belongs in the public space.
smallnamespace 3y ago
The reason libraries call polyfills directly is because it's impolite for a library to change global scope underneath you. Usually it's the top-level application's author who chooses and configures polyfills. Now one may reasonably ask, why doesn't the library just call Object.defineProperties directly, and tell the user to install the appropriate polyfill? I'm going to guess that a library that Just Works after an npm install will see much better adoption than one that requires each user to configure their babel/swc/etc. correctly, especially since the library can be a dependency of another library. There's currently no standardized mechanism in the npm ecosystem to do the equivalent of "Install this library, and also configure your environment to pull in all required polyfills" so that the required functionality is available in global scope. One reason is because the transpilers that automatically polyfill into global scope are third-party tools. Maybe a standard mechanism like this should exist, but it doesn't today, hence the quite reasonable choice of library authors to directly use polyfills because doing so: 1. Avoids pollute the global namespace by avoiding applying a polyfill globally 2. Works as a dependency without additional configuration by the user 3. Preserve backwards compatibility A somewhat cheap fix to at least reduce duplication of polyfills would be for libraries that need polyfills to accept a wide version range. That would give the package manager room to pick a version that's compatible across call sites.
- tedunangst 3y ago
  But why are we polyfilling a function that exists in every version of node? When did this code not just work after installation that it required a polyfill?
  - vinnymac 3y ago
    My best guess is that the code was used in non-node environments. It wasn’t and isn’t uncommon to pull down a dependency from npm and expect it to work in multiple runtimes.
- samus 3y ago
  Modifying global scope is the whole point of a polyfill though. And polyfills check themselves whether they were already applied or not. Maybe a step to a more sane situation would be reducing redundancies between polyfill libraries to ensure they don't step on each other's toes.
chatmasta 3y ago
Sometimes it can be a security vulnerability to call a polyfill instead of the now available default implementation. For example, this 2018 bug [0] in the Grammarly Chrome Extension had a much wider impact due to its reliance on a fetch polyfill that was able to make requests (via XHR) to origins that native fetch could not. I suppose in that case you could argue the real bug is in the XHR API, but it only affected the extension because the extension was using a fetch polyfill that relied on it in functions that could be triggered by an external page. [0] https://hackerone.com/reports/389108 https://hackerone.com/reports/389108
- mhagemeister 3y ago
  That's a very good point. Didn't know about the grammarly incident. I could definitely see this happening again with the amount of polyfills in npm packages. Polyfills are usually frozen in time and not developed further after they are released.
jongjong 3y ago
In the early days of the Node.js ecosystem, there was a trend which was all about 'tiny modules'; many developers published tiny 10-line or so modules and people in the community were promoting these tiny modules really hard. It went a bit out of control and a lot of higher level modules were including many of those tiny modules, then those modules were themselves included into other, even higher level modules, etc... The number of modules used by some of these higher level modules/tools/frameworks grew exponentially and we ended up with a lot of unnecessary dependencies. Each tiny module did just a bit more than it should have done or included just one more dependency than was necessary, sometimes the scope of the module would grow over time and all this added up. Also, different sub-modules used different sub-sub-modules for the same functionality so this caused a lot of duplication in the higher level modules. For my own open source project, I've always been very careful about which dependencies I use. I favor module authors who try to keep their number of dependencies to a minimum. A lot of times, it comes down to figuring out the correct scope of the module... Most low level libraries should not need to do their own logging; therefore, they should not need to include sub-modules to colorize the bash output; instead, they should just emit events and let higher level modules handle the logging. Anyway there are many cases like that where modules give themselves too much scope.
- DanielHB 3y ago
  I think a lot of the sentiment behind this was that the module system and bundlers didn't really support tree shaking so bringing in a big library with a lot of utility functions brought in a ton of code you didn't need.
arthur2e5 3y ago
These polyfills aren't just large, they're also slow because they never use the native implementation. The good news is that you can replace them all with "overrides" in package.json. That's what nolyfill (https://github.com/SukkaW/nolyfill https://github.com/SukkaW/nolyfill) does. Oh, and of course the README mentions ljharb.
- scns 3y ago
  I'd upvote you a 1000 times if i could. To lazy to mail dang if it's possible to pin it on top.
deevus 3y ago
I haven't got much experience with WASM, but is dependency hell something that WASM completely solves? All of the cruft that you don't use will get optimised away by the compiler, right? I'm not aware of any production ready WASM frameworks, but I'm ready for it.
- corbezzoli 3y ago
  1. WASM is non-JavaScript code, so it's unrelated to npm 2. Whatever dependency hell exists in the source language still exists at WASM compilation time 3. There will never be WASM frameworks because they're generally not the bottleneck. The closest to WASM framework was Cappuccino, which let you compose a whole application in a language close to Objective-C
  - deevus 3y ago
    1. Not related to npm, but related to the web. 2. True, but compilers are generally better than transpilers. 3. Have you seen https://yew.rs/ https://yew.rs/ ?
wonderfuly 3y ago
Checkout https://github.com/SukkaW/nolyfill https://github.com/SukkaW/nolyfill
pbowyer 3y ago
This is a really nice post and series. I'm curious how you're doing the profiling and then generating the flame graphs e.g. in https://marvinh.dev/blog/speeding-up-javascript-ecosystem/ https://marvinh.dev/blog/speeding-up-javascript-ecosystem/. Is this Chrome's built-in devtools being used or something else?
- mhagemeister 3y ago
  Glad to hear you like it! Those flame graph screenshots are taken from https://www.speedscope.app/ https://www.speedscope.app/ .
gryzzly 3y ago
I think /vendor/ folder should make a come back. This is how I’ve been doing all my side projects for a while now. People really have no mercy upon themselves, to deal with the bloated crap of the [struggle-stack™](https://twitter.com/brianleroux/status/1643337745463644160 https://twitter.com/brianleroux/status/1643337745463644160)
plugin-baby 3y ago
Are 73% of devs choosing to use typescript? Or is it just the default setting for popular frontend project setup scripts?
synergy20 3y ago
will these unneeded polyfills etc be removed by treeshaking and code splitting in the production build via bundlers? are Bun and Deno solving this problem to some extent? node.js/bun/deno need a battery-included stdlib to me like what python provides.
- mhagemeister 3y ago
  They are unfortunately not removed, because the way they are used makes it difficult for bundlers to detect them. Deno encourages you to submit the original sources which can be even in TypeScript if you want. The users are very close to the newest Deno release and there is barely anyone staying on old versions. This works because Deno takes semver very seriously, which in turn encourages folks to upgrade. It removes the need for polyfills and allows you to always use the latest JS features. Disclaimer: I work at Deno

234 points by jviide 3y ago

This series is great! You should seriously think about consolidating them into a book. Something I notice other engineers struggle with is how to properly assess performance, read heap snapshots, or even understand how to read a flamegraph for stack tracing tools. It would be nice to point, or buy, them a resource showing this. I'd definitely buy a copy.

bjnewman85 3y ago
Second that, I would definitely buy a book based on this series.
- mhagemeister 3y ago
  Thanks for the kind feedback! It's definitely something in the back of my mind. I feel like I need to collect a little more content to fill a whole book, but I'm enticed by the thought of writing one nonetheless.
  - leipert 3y ago
    Part 3 send me down a debugging route of eslint performance in the GitLab project and we were able to move from 25 min of listing time in CI, down to 5. So, thanks for the inspiration!

cxr 3y ago

You'll never get NPM apologists to acknowledge this. One of their only skills is making non-specific appeals to the necessity of it all (as essential infrastructure) and vague arguments that boil down to "you need to trust the wisdom of the crowds" (and e.g. the fact that it exists and everyone else is using it means that anyone who disputes its value just doesn't understand it—bonus points for them if they manage to work in a slight that's designed to paint you, implicitly or explicitly, as a junior), despite not being able to attest to any firsthand knowledge of the real why of anything they're defending. > The new dependencies were all polyfills for JavaScript functions that have long been supported everywhere. The Object.defineProperties method for example was shipped as part of the very first public Node 0.10.0 release dating back to 2013. Heck, even Internet Explorer 9 supported that. And yet there were numerous packages in that dependend on a polyfill for it.

CharlesW 3y ago
> You'll never get NPM apologists to acknowledge this. How should NPM prevent archaic dependencies, or the "even more bizarre" (author's words) problem of developers calling polyfills directly instead of the function that the polyfill fills?
- whstl 3y ago
  The parent is not criticising NPM the tool/registry, but rather the ecosystem and culture.
  - cxr 3y ago
    I'm happy to criticize NPM the tool. The whole thing is designed as a second, crummier version control system that lives in disharmony with and on top of your base-level version control system (so it can subvert it). It's a terrible design. There's basically at most one reasonable use for npm: as a glorified download manager, i.e. to quickly fetch a module by name (right before you check it in to version control with Git). This differs wildly, of course, from how it's actually used, which is as a drug that sweeps mountains of unaudited code under the rug so people can trick themselves and others into thinking that none of it's really there on the basis that it's not visible when anyone first clones the repo. To answer the other commenter's question, "How should NPM prevent archaic dependencies": it shouldn't; it's okay for programmers to be responsible for their work.
    - spankalee 3y ago
      How is npm a version control system? npm is a fairly standard package manager, much like many others, pretty good even. I don't know of anyone who says that package managers and source control solve the same problems. They both happen to use the word and concept of "version" but to mean different things. Yes, some projects vendor in their dependencies into their source control system, but they must either manually verify package version compatibility or use a package manager like npm to help them do it. And vendoring doesn't work for actual packages published to the package repo. If they vendored dependencies then every dependency would be duplicated always, defeating the very purpose of a package manager!

no_wizard 3y ago

Marvin is doing wonderful work in the JS ecosystem around performance. It has been largely focused on tools and node, however, he did have an interesting set of things to say about how they optimized performance in Preact as well on his website that was really interesting too. One thing I've noticed is the rampant duplication of polyfills and babel helpers. To the point that I now have overrides setup via pnpm and I re-write many imports of polyfills to point at my own shims, which simply re-export existing functionality native to the language, most of the time. For smaller utility packages, I often simply clone the repo and copy things over that we need, or copy the src right out of the node_modules folder if possible, then I strip away all the superfluous imports (and often convert from commonjs to ESM if needed) Saves so much headache, its better for users, smaller builds etc.

Dextro 3y ago
That sounds awesome. I've dabled with something like that but only for lodash (makings sure all different flavours get aliased to a single thing) but I never went too far with all the other stuff. You wouldn't happen to have an example of what you're doing laying around would you? I'd be genuinely curious to try stuff like that out.
mhagemeister 3y ago
Author here. Thanks for the kind words! It's feedback like this that encourages me to keep writing about it. I share your experiences regarding babel helpers and haven't found a good solution myself. Similar to you, I often patch unnecessary stuff out via patch-package, but that approach doesn't scale well.
- no_wizard 3y ago
  I use path resolution, it tends to scale better (not great but better) for Babel helpers because ‘Babel-runtime’ can 100% be re-mapped to ‘@babel/runtime’. Same with corejs 2 -> 3, they just need path mapping. Patching packages is definitely something I still have to do to strip polyfills and convert CJS to ESM if I can’t simply re-compile source

jluxenberg 3y ago

For what it's worth; `eslint-plugin-react` has been around for a long time and seems to support running in very old versions of Node.JS (back to v4[1] apparently! tho I can't find anything documenting that for sure.) I was surprised to learn that Object.values is only supported in Node >v7, Object.fronEntries was added in v12, etc. So for this project maybe the polyfills are needed. [1] https://github.com/jsx-eslint/eslint-plugin-react/pull/1038 https://github.com/jsx-eslint/eslint-plugin-react/pull/1038

mhagemeister 3y ago
Yeah, engines are a moving target. I'm all for backwards compatibility, but I'm worried about promoting old node versions with known unpatched security issues. Given that eslint itself only supports node >= 12.22.0 it seems like it's time to get rid of the polyfills. I wish we as in the industry would find a better solution to adapt to this. It's a bit unfortunate that the polyfills as part of the library code itself, which makes it difficult to get rid of them once they're not needed anymore.
gigel82 3y ago
Genuinely curious, are there people out there using newer versions of this package with old / unsupported versions of Node (in production)?
- silverwind 3y ago
  Not really. Adoption of new Node versions is quite quick, given their short support periods. https://nodejs.org/metrics/summaries/version.png https://nodejs.org/metrics/summaries/version.png
- phire 3y ago
  It probably happens, but not really on purpose. If the package.lock file gets deleted or someone runs a global npm-update then npm will update any packages while respecting semantic versioning. It's possible an organisation forgot to include the package.lock file in their deployment image and they get updated npm packages every time they redeploy. It's also possible a developer making minor changes to a legacy system triggers packages to be updated, perhaps without even noticing.

e_y_ 3y ago

> Polyfills that don’t polyfill This is what's sometimes called a "ponyfill". The idea is to avoid messing with global scope (monkeypatching), which could be problematic if you have multiple polyfills for the same API or polyfills that don't perfectly match the native behavior. This can be a good thing in some situations, but in general it's probably best to leave polyfill decisions to the bundler so you can decide which browsers you want to support. Or even produce multiple versions, a lightweight one for modern browsers and one with tons of polyfills that gets served to ancient ones.

mhagemeister 3y ago
Author here. Good point. Agree that the ideal scenario would be that the end user (or the tools they use) have the final say in which polyfills to load. It's a bit of a bummer that they are shipped as part of npm packages without an easy way to get rid of them. I wonder if our industry will move to publishing the original source files to npm in the long run. Only the last piece of the chain, the developer using these dependencies, knows what their target environments are. So the bundler could then downlevel or polyfill the code for the specified targets.
- bsimpson 3y ago
  I've always been kind of surprised that original sources aren't part of the NPM culture. For a long time, I included "typescript:main" in my package.jsons and configured my tools to prefer that to "main" (and now "module").
MBCook 3y ago
I get not defining it yourself, especially if your polyfill is limited to the sliver of a feature you use, but why not check if the feature is there first? Ok, maybe someone else monkeypatched it. But at least you’d end up using the native functionality if it was there.

petetnt 3y ago

I don't really care to comment about the practice itself, but the "Polyfills that don’t polyfill" section is missing the point: the function is called directly instead of patching the global object so that the global object is not polluted by an possibly non-standard implementation. Additionally it does use Object.defineProperty if available - furthermore it doesn't even call itself a polyfill in the first place. If it's needed in 2023 is a valid point however.

fiddlerwoaroof 3y ago
I think it would be better to just expect the standardized functions to be present and then document that the project needs them (e.g. via peer dependencies), allowing users to install them themselves as needed.
- shpx 3y ago
  That's a lot more work than your library just working in more places.
  - fiddlerwoaroof 3y ago
    I don’t think it’s all that much more: basically every bundler I’ve used uses browserslist to include polyfills for the developer’s target audience. But, also, I think this sort of easy path inflicts a huge cost on the ecosystem as a whole: writing to the standards and expecting your users to supply a compliant environment solves a lot of N*M problems in the dev process.

ComputerGuru 3y ago

I maintain a few JavaScript libraries that I manually verify compatibility against IE6 (and have lints to catch violations). I manually polyfill a few necessities and quality-of-life improvements up top in the script. Out of curiosity, I removed my polyfills and tried swc and babel both, followed by an eslint pass, and the results were absolutely atrocious. Everything gets polyfilled, even stuff that has been supported in every IE version ever. The usage-based detection is completely borked, and is completely based off string searching property/function names. Using toString() anywhere pulls in the polyfills for Date to string, Regex to string, and Object to string. Using regex anywhere pulls in a bunch of regex polyfills. It was a nightmare and the size of my library increased by orders of magnitude! (I tried opening an swc issue about optionally using typescript ast info (via a plugin, not in swc core) to have more correct usage-based polyfill detection, but that was closed as unlikely to be acted upon.)

mhagemeister 3y ago
Author here. That mirrors my experience too on working in various projects. The automatic polyfilling story is such a good thing in theory, but reality isn't as rosy and much more polyfills than necessary are included.
- ComputerGuru 3y ago
  Thanks for writing your article and sharing! One thing I do on my other libraries is also to polyfill a few heavier things asynchronously instead of making everyone pay the price upfront, so for example I detect if a JSON polyfill is required before asynchronously loading that polyfill. I think the expectation for most things is that the browser will support them by default, and it’s ok if all/any polyfills are loaded separately and asynchronously.
spankalee 3y ago
Why are you supporting IE6?
- ComputerGuru 3y ago

deleted 3y ago

[deleted]

romellem 3y ago

There is some interesting [drama][1] with this, since this article noticeably doesn't mention any PRs they opened to remove some of these older polyfills. The reason those PRs were never opened/merged is the maintainer of many of those libraries [has a strong stance on "breaking" changes][2] in software: > I have developed an intense avoidance for breaking changes in all of my packages, because I don't want to inflict hundreds of millions of dollars of person-hour cost on the entire industry unnecessarily. IMO this argument avoids the opposite claim, that people then spend a ton of time (and money) trying to make old tech work with newer tech since not everyone maintains to the same standards of backwards compatibility. But regardless, no one is required to stick to a particular way of creating open source software, so the one benefit here is that you are free to [fork the library][3] (assuming its license allows for that) to remove some backwards compatibility that isn't relevant to you. [1]: https://twitter.com/ljharb/status/1704912065486618915 https://twitter.com/ljharb/status/1704912065486618915 [2]: https://github.com/import-js/eslint-plugin-import/pull/2447#issuecomment-1119864501 https://github.com/import-js/eslint-plugin-import/pull/2447#... [3]: https://www.npmjs.com/package/react-outside-click-handler-lite https://www.npmjs.com/package/react-outside-click-handler-li...

potsandpans 3y ago
Ljharb is harmful to the ECMAScript community
- silverwind 3y ago
  I'm thankful for his work, but I do agree, this polyfill madness has to stop.
  - potsandpans 3y ago
    I am _appreciative_ that people like him exist, because -- personal feelings aside -- we do need people who have his level of dedication in the open source space. My only interactions with ljharb have been in TC39. More often than not when looking at issues he's active in, I find myself wanting a level of maturity and thoughtfulness that just isn't there. His contributions are neither novel or very compelling. He's more of an open-source bureaucrat than an engineer or a developer, yet he is currently steering several major TC39 proposals. I've seen him shutdown or sidetrack valid input and criticisms in all of them. The vibes I get from his behavior are akin to him being a member of "the cool kid club" -- wanting to maintain that ingroup/outgroup boundary very tightly. He really should just get out of the standards/committee space entirely for the time being, and get some coaching on leadership skills before reentering. I realize this is super negative. Still on the ropes whether or not this feedback belongs in the public space.

smallnamespace 3y ago

The reason libraries call polyfills directly is because it's impolite for a library to change global scope underneath you. Usually it's the top-level application's author who chooses and configures polyfills. Now one may reasonably ask, why doesn't the library just call Object.defineProperties directly, and tell the user to install the appropriate polyfill? I'm going to guess that a library that Just Works after an npm install will see much better adoption than one that requires each user to configure their babel/swc/etc. correctly, especially since the library can be a dependency of another library. There's currently no standardized mechanism in the npm ecosystem to do the equivalent of "Install this library, and also configure your environment to pull in all required polyfills" so that the required functionality is available in global scope. One reason is because the transpilers that automatically polyfill into global scope are third-party tools. Maybe a standard mechanism like this should exist, but it doesn't today, hence the quite reasonable choice of library authors to directly use polyfills because doing so: 1. Avoids pollute the global namespace by avoiding applying a polyfill globally 2. Works as a dependency without additional configuration by the user 3. Preserve backwards compatibility A somewhat cheap fix to at least reduce duplication of polyfills would be for libraries that need polyfills to accept a wide version range. That would give the package manager room to pick a version that's compatible across call sites.

tedunangst 3y ago
But why are we polyfilling a function that exists in every version of node? When did this code not just work after installation that it required a polyfill?
- vinnymac 3y ago
  My best guess is that the code was used in non-node environments. It wasn’t and isn’t uncommon to pull down a dependency from npm and expect it to work in multiple runtimes.
samus 3y ago
Modifying global scope is the whole point of a polyfill though. And polyfills check themselves whether they were already applied or not. Maybe a step to a more sane situation would be reducing redundancies between polyfill libraries to ensure they don't step on each other's toes.

chatmasta 3y ago

Sometimes it can be a security vulnerability to call a polyfill instead of the now available default implementation. For example, this 2018 bug [0] in the Grammarly Chrome Extension had a much wider impact due to its reliance on a fetch polyfill that was able to make requests (via XHR) to origins that native fetch could not. I suppose in that case you could argue the real bug is in the XHR API, but it only affected the extension because the extension was using a fetch polyfill that relied on it in functions that could be triggered by an external page. [0] https://hackerone.com/reports/389108 https://hackerone.com/reports/389108

mhagemeister 3y ago
That's a very good point. Didn't know about the grammarly incident. I could definitely see this happening again with the amount of polyfills in npm packages. Polyfills are usually frozen in time and not developed further after they are released.

jongjong 3y ago

In the early days of the Node.js ecosystem, there was a trend which was all about 'tiny modules'; many developers published tiny 10-line or so modules and people in the community were promoting these tiny modules really hard. It went a bit out of control and a lot of higher level modules were including many of those tiny modules, then those modules were themselves included into other, even higher level modules, etc... The number of modules used by some of these higher level modules/tools/frameworks grew exponentially and we ended up with a lot of unnecessary dependencies. Each tiny module did just a bit more than it should have done or included just one more dependency than was necessary, sometimes the scope of the module would grow over time and all this added up. Also, different sub-modules used different sub-sub-modules for the same functionality so this caused a lot of duplication in the higher level modules. For my own open source project, I've always been very careful about which dependencies I use. I favor module authors who try to keep their number of dependencies to a minimum. A lot of times, it comes down to figuring out the correct scope of the module... Most low level libraries should not need to do their own logging; therefore, they should not need to include sub-modules to colorize the bash output; instead, they should just emit events and let higher level modules handle the logging. Anyway there are many cases like that where modules give themselves too much scope.

DanielHB 3y ago
I think a lot of the sentiment behind this was that the module system and bundlers didn't really support tree shaking so bringing in a big library with a lot of utility functions brought in a ton of code you didn't need.

arthur2e5 3y ago

These polyfills aren't just large, they're also slow because they never use the native implementation. The good news is that you can replace them all with "overrides" in package.json. That's what nolyfill (https://github.com/SukkaW/nolyfill https://github.com/SukkaW/nolyfill) does. Oh, and of course the README mentions ljharb.

scns 3y ago
I'd upvote you a 1000 times if i could. To lazy to mail dang if it's possible to pin it on top.

deevus 3y ago

I haven't got much experience with WASM, but is dependency hell something that WASM completely solves? All of the cruft that you don't use will get optimised away by the compiler, right? I'm not aware of any production ready WASM frameworks, but I'm ready for it.

corbezzoli 3y ago
1. WASM is non-JavaScript code, so it's unrelated to npm 2. Whatever dependency hell exists in the source language still exists at WASM compilation time 3. There will never be WASM frameworks because they're generally not the bottleneck. The closest to WASM framework was Cappuccino, which let you compose a whole application in a language close to Objective-C
- deevus 3y ago
  1. Not related to npm, but related to the web. 2. True, but compilers are generally better than transpilers. 3. Have you seen https://yew.rs/ https://yew.rs/ ?

wonderfuly 3y ago

Checkout https://github.com/SukkaW/nolyfill https://github.com/SukkaW/nolyfill

pbowyer 3y ago

This is a really nice post and series. I'm curious how you're doing the profiling and then generating the flame graphs e.g. in https://marvinh.dev/blog/speeding-up-javascript-ecosystem/ https://marvinh.dev/blog/speeding-up-javascript-ecosystem/. Is this Chrome's built-in devtools being used or something else?

mhagemeister 3y ago
Glad to hear you like it! Those flame graph screenshots are taken from https://www.speedscope.app/ https://www.speedscope.app/ .

gryzzly 3y ago

I think /vendor/ folder should make a come back. This is how I’ve been doing all my side projects for a while now. People really have no mercy upon themselves, to deal with the bloated crap of the [struggle-stack™](https://twitter.com/brianleroux/status/1643337745463644160 https://twitter.com/brianleroux/status/1643337745463644160)

plugin-baby 3y ago

Are 73% of devs choosing to use typescript? Or is it just the default setting for popular frontend project setup scripts?

synergy20 3y ago

will these unneeded polyfills etc be removed by treeshaking and code splitting in the production build via bundlers? are Bun and Deno solving this problem to some extent? node.js/bun/deno need a battery-included stdlib to me like what python provides.

mhagemeister 3y ago
They are unfortunately not removed, because the way they are used makes it difficult for bundlers to detect them. Deno encourages you to submit the original sources which can be even in TypeScript if you want. The users are very close to the newest Deno release and there is barely anyone staying on old versions. This works because Deno takes semver very seriously, which in turn encourages folks to upgrade. It removes the need for polyfills and allows you to always use the latest JS features. Disclaimer: I work at Deno