6 ms·

<quote> I’ll conclude with three guidelines for library authors to follow: Provide a CJS version of your library Provide a thin ESM wrapper for your CJS Add

by rjeli 6y ago

<quote> I’ll conclude with three guidelines for library authors to follow: Provide a CJS version of your library Provide a thin ESM wrapper for your CJS Add an exports map to your package.json </quote> This seems like the same fallacy of python’s 2to3, and why they should have written 3to2 instead. If ESM is the new way, library authors should instead have a CJS shim to prevent fossilization of cruft.

nurettin 6y ago
PSF wanted people to migrate to python3 for reasons including not wanting to support multiple languages. Why would they write some tool to convert new python3 code back to python2 that would ultimately work against their agenda? It doesn't make sense to me.
1337shadow 6y ago
2to3 was still useful because all code was in python 2 when python 3 first came out.
- kqr 6y ago
  Sure, but maybe upgrading the existing Python 2 code wasn't the problem: the vast majority of the Python 2 code alive when Python 3 was released was probably scrapped by the time Python 2 was EOL'd. Over a decade is a very long time for a codebase to live. (Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so.) The (comparatively) few Python 2 projects that were alive by the release of Python 3 and still alive by the EOL of Python 2 could probably have been upgraded manually, rather than with a tool. On the other hand, people not wanting to write new code in Python 3 because it would be incompatible with the Python 2 ecosystem – that was a real problem. That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem. If only people would have written new code in Python 3 right away, we would have been in a far better position by the time Python 2 would have been EOL'd, purely due to natural turnaround of code bases. Porting Python 2 code to Python 3 was never the problem. Writing new code in Python 3 was.
  - masklinn 6y ago
    > Porting Python 2 code to Python 3 was never the problem. You're, frankly, high as a kite. As the author of one such port on a large codebase, porting Python 2 code to Python 3 was absolutely a problem. The issue with 2to3 is that it assumed you'd do the conversion as a one-shot and straight flip over with entirely separate codebases (or dropping Python 2 entirely) from there on, which is completely insane. What the vast majority of transitioning projects needed was a way to migrate progressively through cross-version codebases: * libraries weren't going to drop older Python versions, the very few which attempted that (I'm aware of dateutil at least) were extremely painful to users and IIRC ended up rolling back to a more progressive codebase * programs being distributed were not going to drop Python 2 until EOF at least * large codebases simply could not afford to perform a completely untested and uncheckable one-shot transition That's why the community created cross-version support tools like six and friends, and lobbied hard for the reintroduction of cross-version features into Python 3 e.g. reintroduction of `callable` (3.2), reintroduction of the "u" prefix (3.3), reintroduction of % on bytes (3.5) and I'm sure others. > Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so. […] Python 3 because it would be incompatible with the Python 2 ecosystem What do you think "the python 2 ecosystem is" except for a ton of projects which needed to be ported from Python 2 to Python 3 exactly? "the vast majority of websites or data processing scripts" is not an ecosystem, tooling and libraries are. Do you somehow think numpy and lxml and django just scrapped their entire Python 2 codebase and rewrote the entire thing from scratch? Of course not. > That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem. 3to2 would not have worked any better better than 2to3. The syntactic incompatibilities between the version were not the hard part, the semantic ones were, and 3to2 would not have succeeded any better than 2to3 there because it simply couldn't. All 3to2 would have given you is a broken Python 2 codebase from a Python 3 codebase you could not even run.
    - kqr 6y ago
      Isn't "migrating progressively through cross-version codebases" exactly what a 3to2 compiler would allow you to do? You'd write new code (and incrementally port old code) to Python 3, and then when you build the project you'd compile the Python 3 code down to Python 2 before building the entire project as Python 2, including all dependencies. So for over a decade, you'd be writing Python 3 code but building a Python 2 project. After a decade, one would hopefully have managed to port everything over to Python 3, and can then drop the "compile to Python 2" step. A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed! ---- I don't see why 3to2 would not have worked. It's a compiler. We have written compilers before. We can write compilers. Compilers work. Both languages are Turing complete and about the same in expressiveness, so clearly compiling one to the other is something we can do. What we cannot (reasonably) do is produce nice, human-looking target code from a compiler. This is what 2to3 attempted to do, and what a 3to2 would never need to do. Who cares if the Python 2 code it produced looked a little wonky, as long as it was correct and roughly recogniseable as being built from the original Python 3 code? The Python 3 code is the one humans write and edit. The Python 2 code is just compiler output to make it compatible with all of the Python 2 ecosystem until one is ready to go fully Python 3. ---- This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that. Slowly but surely, more and more things are being written in the modern fashion, and eventually, when everything has caught up, the compatibility conversion to the legacy code can be dropped.
kqr 6y ago
> This seems like the same fallacy of python’s 2to3, and why they should have written 3to2 instead. This is incredibly insightful. I'm frankly a bit stunned. I would never have thought of this (and clearly the Python people also didn't) but it is absolutely true. It's also a very transferable strategy for any time compatibility is to be broken. Is this an original thought of yours or do you have references to further reading?
- rjeli 6y ago
  It's a common refrain when the py2/3 discussions get going. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=3to2&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- danmur 6y ago
  This is something like what the future package and tools do, though I can say from experience it's not without its issues.
desmap 6y ago
> should instead have a CJS shim OP wrote, note that it’s easy to write an ESM wrapper for CJS libraries, but it’s not possible to write a CJS wrapper for ESM libraries.
- Macha 6y ago
  It's not possible to write a nice automated transparent one for arbitrary libraries because of top level await, but it's totally possible for the 95%.
  - desmap 6y ago
    Re your point OP also commented/quoted someone with “I don’t think designing a system with the blanket assumption that some feature just won’t get used is a viable path.”
    - nightski 6y ago
      Yea they are looking at the ecosystem as a whole. But if you are a library author you have the power to control these things.
    - Macha 6y ago
      Yes, someone cannot design a system to handle arbitrary libraries with the assumption. A library author can provide cjs bindings to their library for as long as cjs remains relevant with the assumption that they won't use top level await.
  - jkrems 6y ago
    You cannot write a CJS shim for any library because the CJS shim cannot expose any ESM file with a synchronous API because importing ESM always returns a Promise. You can - and people widely do - compile a CJS version of your library from ESM sources. You'll accept that there may be multiple copies of your library in the program but it does work. But in any case there's no reason to actually author in CJS, everything including the ESM shim can be generated from ESM.

by rjeli 6y ago

nurettin 6y ago

PSF wanted people to migrate to python3 for reasons including not wanting to support multiple languages. Why would they write some tool to convert new python3 code back to python2 that would ultimately work against their agenda? It doesn't make sense to me.

1337shadow 6y ago

2to3 was still useful because all code was in python 2 when python 3 first came out.

kqr 6y ago
Sure, but maybe upgrading the existing Python 2 code wasn't the problem: the vast majority of the Python 2 code alive when Python 3 was released was probably scrapped by the time Python 2 was EOL'd. Over a decade is a very long time for a codebase to live. (Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so.) The (comparatively) few Python 2 projects that were alive by the release of Python 3 and still alive by the EOL of Python 2 could probably have been upgraded manually, rather than with a tool. On the other hand, people not wanting to write new code in Python 3 because it would be incompatible with the Python 2 ecosystem – that was a real problem. That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem. If only people would have written new code in Python 3 right away, we would have been in a far better position by the time Python 2 would have been EOL'd, purely due to natural turnaround of code bases. Porting Python 2 code to Python 3 was never the problem. Writing new code in Python 3 was.
- masklinn 6y ago
  > Porting Python 2 code to Python 3 was never the problem. You're, frankly, high as a kite. As the author of one such port on a large codebase, porting Python 2 code to Python 3 was absolutely a problem. The issue with 2to3 is that it assumed you'd do the conversion as a one-shot and straight flip over with entirely separate codebases (or dropping Python 2 entirely) from there on, which is completely insane. What the vast majority of transitioning projects needed was a way to migrate progressively through cross-version codebases: * libraries weren't going to drop older Python versions, the very few which attempted that (I'm aware of dateutil at least) were extremely painful to users and IIRC ended up rolling back to a more progressive codebase * programs being distributed were not going to drop Python 2 until EOF at least * large codebases simply could not afford to perform a completely untested and uncheckable one-shot transition That's why the community created cross-version support tools like six and friends, and lobbied hard for the reintroduction of cross-version features into Python 3 e.g. reintroduction of `callable` (3.2), reintroduction of the "u" prefix (3.3), reintroduction of % on bytes (3.5) and I'm sure others. > Again, there are exceptions, but the vast majority of websites or data processing scripts or whatnot are significantly re-written or decommissioned in – pulling a number out of my arse – five years or so. […] Python 3 because it would be incompatible with the Python 2 ecosystem What do you think "the python 2 ecosystem is" except for a ton of projects which needed to be ported from Python 2 to Python 3 exactly? "the vast majority of websites or data processing scripts" is not an ecosystem, tooling and libraries are. Do you somehow think numpy and lxml and django just scrapped their entire Python 2 codebase and rewrote the entire thing from scratch? Of course not. > That problem could have been prevented with a 3to2 tool, which would compile the Python 3 code to Python 2, allowing it to be used with the existing ecosystem. 3to2 would not have worked any better better than 2to3. The syntactic incompatibilities between the version were not the hard part, the semantic ones were, and 3to2 would not have succeeded any better than 2to3 there because it simply couldn't. All 3to2 would have given you is a broken Python 2 codebase from a Python 3 codebase you could not even run.
  - kqr 6y ago
    Isn't "migrating progressively through cross-version codebases" exactly what a 3to2 compiler would allow you to do? You'd write new code (and incrementally port old code) to Python 3, and then when you build the project you'd compile the Python 3 code down to Python 2 before building the entire project as Python 2, including all dependencies. So for over a decade, you'd be writing Python 3 code but building a Python 2 project. After a decade, one would hopefully have managed to port everything over to Python 3, and can then drop the "compile to Python 2" step. A 3to2 tool would solve exactly the problems you're bringing up, with no intoxication needed! ---- I don't see why 3to2 would not have worked. It's a compiler. We have written compilers before. We can write compilers. Compilers work. Both languages are Turing complete and about the same in expressiveness, so clearly compiling one to the other is something we can do. What we cannot (reasonably) do is produce nice, human-looking target code from a compiler. This is what 2to3 attempted to do, and what a 3to2 would never need to do. Who cares if the Python 2 code it produced looked a little wonky, as long as it was correct and roughly recogniseable as being built from the original Python 3 code? The Python 3 code is the one humans write and edit. The Python 2 code is just compiler output to make it compatible with all of the Python 2 ecosystem until one is ready to go fully Python 3. ---- This is also not something completely foreign and unheard of. This is exactly how I've been porting legacy JavaScript applications to a more modern ECMAScript approach: write new code in the modern way, compile it down to legacy code compatible with the existing legacy code, and then run that. Slowly but surely, more and more things are being written in the modern fashion, and eventually, when everything has caught up, the compatibility conversion to the legacy code can be dropped.

kqr 6y ago

> This seems like the same fallacy of python’s 2to3, and why they should have written 3to2 instead. This is incredibly insightful. I'm frankly a bit stunned. I would never have thought of this (and clearly the Python people also didn't) but it is absolutely true. It's also a very transferable strategy for any time compatibility is to be broken. Is this an original thought of yours or do you have references to further reading?

rjeli 6y ago
It's a common refrain when the py2/3 discussions get going. https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=3to2&sort=byPopularity&type=comment https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
danmur 6y ago
This is something like what the future package and tools do, though I can say from experience it's not without its issues.

desmap 6y ago

> should instead have a CJS shim OP wrote, note that it’s easy to write an ESM wrapper for CJS libraries, but it’s not possible to write a CJS wrapper for ESM libraries.

Macha 6y ago
It's not possible to write a nice automated transparent one for arbitrary libraries because of top level await, but it's totally possible for the 95%.
- desmap 6y ago
  Re your point OP also commented/quoted someone with “I don’t think designing a system with the blanket assumption that some feature just won’t get used is a viable path.”
  - nightski 6y ago
    Yea they are looking at the ecosystem as a whole. But if you are a library author you have the power to control these things.
  - Macha 6y ago
    Yes, someone cannot design a system to handle arbitrary libraries with the assumption. A library author can provide cjs bindings to their library for as long as cjs remains relevant with the assumption that they won't use top level await.
- jkrems 6y ago
  You cannot write a CJS shim for any library because the CJS shim cannot expose any ESM file with a synchronous API because importing ESM always returns a Promise. You can - and people widely do - compile a CJS version of your library from ESM sources. You'll accept that there may be multiple copies of your library in the program but it does work. But in any case there's no reason to actually author in CJS, everything including the ESM shim can be generated from ESM.