4 ms·

Require(esm) in Node.js

34 points by tilt 3y ago

jauntywundrkind 3y ago
It's all just so extra crazy because https://github.com/standard-things/esm https://github.com/standard-things/esm has just worked for so so long. The intense module discussions kept happening, about why it wouldn't work or how it would blow up, but standard-things/esm was already out & working great.
lloydatkinson 3y ago
> Recently I landed experimental support for require()-ing synchronous ES modules in Node.js, a feature that has been long overdue. Oh come on, overdue for what purpose? Digging Node out of another Node created hole?
- dcgudeman 3y ago
  Did you read the article? He explains the motivation for the change pretty clearly.
  - WorldMaker 3y ago
    The motivation seems to be "many people have transpilers configured wrong" and this seems an extreme solution to fixing people's transpiler configurations. Especially because it will still fail for modules using top-level await (anywhere in their dependency graph), so it is at best a band-aid. Beyond that it does seem to be missing real motivating use cases. What real reason in 2024 is there to use CommonJS at the top level in NPM? How hard is it to rewrite `require()` statements to `await import()` ones, especially when the fix is a transpiler configuration change away?
    - pquki4 3y ago
      I don't know how much experience you have, but in the JS world there are band-aids everywhere. They "fix" a lot of real-world problems, help push things to production, and are more important than you think. Just look at the typing gymnastics in TypeScript.
      - lloydatkinson 3y ago
        The solution to band aids existing isn’t to introduce more, especially not more that are useless.
        pquki4 3y ago
        1. I don't agree this is useless. You may very likely never need to use it, but not for others. 2. Until every single JavaScript file in the world is in the ESM format, anything that happens before that will be a band aid of some sort.
      - WorldMaker 3y ago
        > I don't know how much experience you have Try better to avoid ad hominem attacks, please. > but in the JS world there are band-aids everywhere I apologize, "band-aid" was the wrong word to use for this specific case, because that unintentionally disparages "band-aids". You are right there are "band-aids" that fix real-world problems, and help heal accidental wounds. This experiment does neither and seems to me worse than a "band-aid". It's a fragile broken and fake bridge that looks like it can get you to your destination but will collapse right under you if you try to use it (rather than switching to what already works today).
        pquki4 3y ago
        Sorry to disappoint you, half of the things in the JS world are half baked, broken, and are fake bridges. That's why I need to mention experience -- it is hard to explain all of this within a few sentences if you don't already have the context of what I am trying to say.
    - paulddraper 3y ago
      > many people have transpilers configured wrong ? It's so that CommonJS modules can use ESM modules. --- Like suppose examplelib uses otherlib, both in CommonJS. The examplelib wants to upgrade otherlib, but the new version uses ESM. Now, examplelib is stuck either on the old version, or has to undertake a rewrite (and propagate this situation downstream). Or adopt a transpiler and force downstream users to do the same. Some would say "just rewrite in ESM" and that is a nice idea, but there are lots of tools with caveats, TypeScript, Jest, anything that depends on require-in-the-middle, etc. It's hard to boil the ocean. (Just look at Python 3.)
      - WorldMaker 3y ago
        CommonJS modules (and anybody else that prefers or needs "require-in-the-middle") can already import ESM. They need to use `await import()` or `import().then()` if they prefer callback waterfalls to async/await syntax. That works everywhere in Node today. Sure, it's a pain when upstream switches to ESM that you have to use more Promises, but that the price you pay and it is certainly a lot less work than a full rewrite even for the most brownfield CommonJS library. You don't have to rewrite everything to ESM all at once. (You should want to, but don't have to.)
        paulddraper 3y ago
        Exchanging one function coloring problem for another isn't what I would call "less work."
    - wdb 3y ago
      It's quite difficult to write require() to `await import()` due to the introducing of `await`... Now everything needs to become async
      - WorldMaker 3y ago
        If your upstream has switched to ESM and expects to do async things, then yes, everywhere you depend on that upstream you need to consider those imports async. Yes, it's contagious, but your upstream already caught that contagion. Trying to keep smashing it into a synchronous box seems like a recipe for unexpected bugs. Especially in the case of this experiment where the difference between "works" and "gives the same error it already gives today" is a one keyword change anywhere in the upstream file. That sounds brittle to me and less useful than "everything needs to become async".
        wdb 3y ago
        Just not upgrading and use patch-package to apply security fixes work so far.
dimgl 3y ago
Appreciate this change. The ESM stuff is nightmarishly esoteric. Most recently I ran into ERR_REQUIRE_ESM while trying to write Playwright E2E tests that import from other packages in our monorepo. I really think the way ESM was handled hurt Node.js a lot. Couple this with the introduction of TypeScript and complexity in the Node.js ecosystem skyrocketed. I've lost so many hours on this. Thankfully libraries like esbuild exist (and by extension `tsx` which uses esbuild under the hood). https://github.com/privatenumber/tsx https://github.com/privatenumber/tsx
- DanielHB 3y ago
  > Couple this with the introduction of TypeScript and complexity in the Node.js ecosystem skyrocketed I remember the days of AMD, UMD and so on and so on. The complexity just got reduced during CommonJS days and then came back with a vengeance.
paulddraper 3y ago
> After dwelling on this a bit, I think the reason why the synchronicity of ESM didn’t lead to a synchronous require(esm) in Node.js sooner was more cultural than technical. There seemed to be a silo problem between those who worked on the ESM implementation in Node.js/communication with the standard bodies and those who didn’t. Must be something like that. This never made a lick of sense. (At the very least for synchronous modules.)
sam_goody 3y ago
Awesome that this is in Node. I created require-esm-in-cjs[1] two years ago when I needed to require ESM modules in a commonJS (regular legacy) page. It is only a few bytes long and is simple - it effectively causes ESM modules to load synchronously by checking every 100ms if the JS has loaded before allowing the page to continue. I have been meaning to improve it, such as adding a timeout and whatnot but never got around to it (it is good enough for my needs) - but always thought it was odd that something so (relatively) simple to hack around and so low in the stack can not be supported natively. It gets a fair amount of downloads even though it doesn't get much visibility so I guess I am not the only one who felt the need. But native is MUCH better! [1]: https://www.npmjs.com/package/require-esm-in-cjs https://www.npmjs.com/package/require-esm-in-cjs
- paulddraper 3y ago
  I read your readme, clever
xeckr 3y ago
I really wish ECMA just ran with the Node.js require instead of unnecessarily creating a new standard. Maybe one day we'll see interchangeable use of require/import. One has to wonder about the total number of hours wasted debugging issues that trace their origin to ECMA's decision.
- spankalee 3y ago
  Not possible. require() is broken by design by being synchronous.
  - zarzavat 3y ago
    It’s not though. You can return an awaitable object.
- paulddraper 3y ago
  No, async module loading or statically analyzable module loading is necessary for web. AMD wasn't invented on a whim
dimgl 3y ago
> After dwelling on this a bit, I think the reason why the synchronicity of ESM didn’t lead to a synchronous require(esm) in Node.js sooner was more cultural than technical. There seemed to be a silo problem between those who worked on the ESM implementation in Node.js/communication with the standard bodies and those who didn’t. By the way, we've been saying this for years. And anyone who pointed out how ridiculous this was got ostracized by Node devs. It's just baffling how this was handled.

34 points by tilt 3y ago

It's all just so extra crazy because https://github.com/standard-things/esm https://github.com/standard-things/esm has just worked for so so long. The intense module discussions kept happening, about why it wouldn't work or how it would blow up, but standard-things/esm was already out & working great.

lloydatkinson 3y ago

> Recently I landed experimental support for require()-ing synchronous ES modules in Node.js, a feature that has been long overdue. Oh come on, overdue for what purpose? Digging Node out of another Node created hole?

dcgudeman 3y ago
Did you read the article? He explains the motivation for the change pretty clearly.
- WorldMaker 3y ago
  The motivation seems to be "many people have transpilers configured wrong" and this seems an extreme solution to fixing people's transpiler configurations. Especially because it will still fail for modules using top-level await (anywhere in their dependency graph), so it is at best a band-aid. Beyond that it does seem to be missing real motivating use cases. What real reason in 2024 is there to use CommonJS at the top level in NPM? How hard is it to rewrite `require()` statements to `await import()` ones, especially when the fix is a transpiler configuration change away?
  - pquki4 3y ago
    I don't know how much experience you have, but in the JS world there are band-aids everywhere. They "fix" a lot of real-world problems, help push things to production, and are more important than you think. Just look at the typing gymnastics in TypeScript.
    - lloydatkinson 3y ago
      The solution to band aids existing isn’t to introduce more, especially not more that are useless.
      - pquki4 3y ago
        1. I don't agree this is useless. You may very likely never need to use it, but not for others. 2. Until every single JavaScript file in the world is in the ESM format, anything that happens before that will be a band aid of some sort.
    - WorldMaker 3y ago
      > I don't know how much experience you have Try better to avoid ad hominem attacks, please. > but in the JS world there are band-aids everywhere I apologize, "band-aid" was the wrong word to use for this specific case, because that unintentionally disparages "band-aids". You are right there are "band-aids" that fix real-world problems, and help heal accidental wounds. This experiment does neither and seems to me worse than a "band-aid". It's a fragile broken and fake bridge that looks like it can get you to your destination but will collapse right under you if you try to use it (rather than switching to what already works today).
      - pquki4 3y ago
        Sorry to disappoint you, half of the things in the JS world are half baked, broken, and are fake bridges. That's why I need to mention experience -- it is hard to explain all of this within a few sentences if you don't already have the context of what I am trying to say.
  - paulddraper 3y ago
    > many people have transpilers configured wrong ? It's so that CommonJS modules can use ESM modules. --- Like suppose examplelib uses otherlib, both in CommonJS. The examplelib wants to upgrade otherlib, but the new version uses ESM. Now, examplelib is stuck either on the old version, or has to undertake a rewrite (and propagate this situation downstream). Or adopt a transpiler and force downstream users to do the same. Some would say "just rewrite in ESM" and that is a nice idea, but there are lots of tools with caveats, TypeScript, Jest, anything that depends on require-in-the-middle, etc. It's hard to boil the ocean. (Just look at Python 3.)
    - WorldMaker 3y ago
      CommonJS modules (and anybody else that prefers or needs "require-in-the-middle") can already import ESM. They need to use `await import()` or `import().then()` if they prefer callback waterfalls to async/await syntax. That works everywhere in Node today. Sure, it's a pain when upstream switches to ESM that you have to use more Promises, but that the price you pay and it is certainly a lot less work than a full rewrite even for the most brownfield CommonJS library. You don't have to rewrite everything to ESM all at once. (You should want to, but don't have to.)
      - paulddraper 3y ago
        Exchanging one function coloring problem for another isn't what I would call "less work."
  - wdb 3y ago
    It's quite difficult to write require() to `await import()` due to the introducing of `await`... Now everything needs to become async
    - WorldMaker 3y ago
      If your upstream has switched to ESM and expects to do async things, then yes, everywhere you depend on that upstream you need to consider those imports async. Yes, it's contagious, but your upstream already caught that contagion. Trying to keep smashing it into a synchronous box seems like a recipe for unexpected bugs. Especially in the case of this experiment where the difference between "works" and "gives the same error it already gives today" is a one keyword change anywhere in the upstream file. That sounds brittle to me and less useful than "everything needs to become async".
      - wdb 3y ago
        Just not upgrading and use patch-package to apply security fixes work so far.

dimgl 3y ago

Appreciate this change. The ESM stuff is nightmarishly esoteric. Most recently I ran into ERR_REQUIRE_ESM while trying to write Playwright E2E tests that import from other packages in our monorepo. I really think the way ESM was handled hurt Node.js a lot. Couple this with the introduction of TypeScript and complexity in the Node.js ecosystem skyrocketed. I've lost so many hours on this. Thankfully libraries like esbuild exist (and by extension `tsx` which uses esbuild under the hood). https://github.com/privatenumber/tsx https://github.com/privatenumber/tsx

DanielHB 3y ago
> Couple this with the introduction of TypeScript and complexity in the Node.js ecosystem skyrocketed I remember the days of AMD, UMD and so on and so on. The complexity just got reduced during CommonJS days and then came back with a vengeance.

paulddraper 3y ago

> After dwelling on this a bit, I think the reason why the synchronicity of ESM didn’t lead to a synchronous require(esm) in Node.js sooner was more cultural than technical. There seemed to be a silo problem between those who worked on the ESM implementation in Node.js/communication with the standard bodies and those who didn’t. Must be something like that. This never made a lick of sense. (At the very least for synchronous modules.)

sam_goody 3y ago

Awesome that this is in Node. I created require-esm-in-cjs[1] two years ago when I needed to require ESM modules in a commonJS (regular legacy) page. It is only a few bytes long and is simple - it effectively causes ESM modules to load synchronously by checking every 100ms if the JS has loaded before allowing the page to continue. I have been meaning to improve it, such as adding a timeout and whatnot but never got around to it (it is good enough for my needs) - but always thought it was odd that something so (relatively) simple to hack around and so low in the stack can not be supported natively. It gets a fair amount of downloads even though it doesn't get much visibility so I guess I am not the only one who felt the need. But native is MUCH better! [1]: https://www.npmjs.com/package/require-esm-in-cjs https://www.npmjs.com/package/require-esm-in-cjs

paulddraper 3y ago
I read your readme, clever

xeckr 3y ago

I really wish ECMA just ran with the Node.js require instead of unnecessarily creating a new standard. Maybe one day we'll see interchangeable use of require/import. One has to wonder about the total number of hours wasted debugging issues that trace their origin to ECMA's decision.

spankalee 3y ago
Not possible. require() is broken by design by being synchronous.
- zarzavat 3y ago
  It’s not though. You can return an awaitable object.
paulddraper 3y ago
No, async module loading or statically analyzable module loading is necessary for web. AMD wasn't invented on a whim

dimgl 3y ago

> After dwelling on this a bit, I think the reason why the synchronicity of ESM didn’t lead to a synchronous require(esm) in Node.js sooner was more cultural than technical. There seemed to be a silo problem between those who worked on the ESM implementation in Node.js/communication with the standard bodies and those who didn’t. By the way, we've been saying this for years. And anyone who pointed out how ridiculous this was got ostracized by Node devs. It's just baffling how this was handled.