4 ms·
I built a tool relevant to this the other day. I was fed up of how hard it was to download ES modules from a CDN and store them locally so I could build apps wi
by simonw 4y ago
I built a tool relevant to this the other day. I was fed up of how hard it was to download ES modules from a CDN and store them locally so I could build apps without that CDN dependency, so I made this:
pip install download-esm
download-esm @observablehq/plot
This downloads the ESM version of Observable Plot from the jsDelivr CDN, figures out its dependencies, downloads them as well (40 files total!) and rewrites their imports to be local, not CDN paths.
More details here: https://simonwillison.net/2023/May/2/download-esm/ https://simonwillison.net/2023/May/2/download-esm/
I'm now considering adding import maps support: https://github.com/simonw/download-esm/issues/4 https://github.com/simonw/download-esm/issues/4
- softfalcon 4y agoOh no! You've gone through dependency hell so many times you've started writing your own package management! I'm only half kidding, this is pretty cool, but I'm also really sorry that it has come to this for you and your tooling!
- zeven7 4y agoThis is JS. Dependency hell is par for the course.
- jakelazaroff 4y agoGenuinely curious — what language do you think has good/better package management? Every time I start a Python project, for example, there's a bunch of time and frustration spent getting pip and virtual envs working.
- moron4hire 4y agoI haven't had any package management troubles with .NET in over a decade.
- HideousKojima 4y agoOnly times I've run into dependency issues with .NET are when a library expects certain native DLLs to already be present on the system, but that's pretty rare (and fairly easy to solve).
- zovin 4y agoEspecially since moving to SDK-style projects, I haven't even thought about assembly bindings in a long time. I've come to really like nuget.
- frizkie 4y agoYou didn't ask me this question, but I work in Ruby, and off the top of my head, have no complaints about Bundler. Switching Ruby versions between projects is not something that is handled natively in any way (unsurprisingly?) and I know that I've struggled a lot with `rvm`, but since switching to *env (rbenv, pyenv, nodenv) I can safely say I do not have any struggles on that front either.
- salzig 4y agoYou should give asdf-vm a chance. Like rbenv, but for almost everything. Switch is easy, using legacyfile support almost unnoticeable.
- technion 4y agoI feel ruby's whole problem in this space is that everyone recommends a different tool. I've used several. None are terrible. But walking into an environment and getting told everyone uses a different tool is
- BiteCode_dev 4y agoRust is the golden standard. python and js are on part with each other, they just have different problems: - tooling and dependencies are hell in js - bootstrapping is half the battle in python (see https://bitecode.substack.com/p/relieving-your-python-packaging-pain https://bitecode.substack.com/p/relieving-your-python-packag...)
- k__ 4y agoPython devs always told me npm was better than pip
- xeromal 4y agoAs a python and node dev, I think NPM wins out just because I don't have to "activate" an environment when I want to use it. The environment is always right there where my code is.
- BiteCode_dev 4y agoPip and npm are pretty much equivalent, but node_modules makes for an easier setup, while venv makes for easier tooling configuration.
- hombre_fatal 4y agoThey are talking about client development so none of the answers are going to compare. You aren’t grabbing ESMs from CDNs for a Node server app. And even if you are, that’s just not a situation that’s happening outside of an ecosystem that straddles client development.
- tnorthcutt 4y agoOf the three languages that I deal with semi-frequently (JS, Python, and PHP), I have by far the least trouble with PHP's package management.
- noir_lord 4y agocomposer is remarkable in that it causes very few problems in practice. It’s an excellent package manager in a mainstream language that just does its job. It helps that the PHP community mostly just saw PSR4 and went, yeah, that’ll do. Meanwhile over in JS land issues are still caused because different modules use different module formats and it’s all just…fragile somehow. Ironic give the reputation of PHP that it nailed its package manager.
- jamesfinlayson 4y agoI know - PHP cops a lot of flack but composer just works and has never given me any grief.
- dingledork69 4y agoJava/Maven
- chao- 4y agoIn my mind, anyone setting out to create a package manager, or a new language that will need one, needs to at least match the functionality of Gems/Bundler or Hex/Mix. They've been around long enough that it feels like table stakes at this point. Mix does more than just dependencies, however I am referring the table stakes as being the more narrow scope of dependency management.
- sodapopcan 4y agoElixir’s tooling, Mix, is really great.
- rad_gruchalski 4y agoFortran: fpm is really nice. Go: modules.
- simplotek 4y ago> This is JS. Dependency hell is par for the course. Is there any programming language that does not suffer from dependency hell?
- pier25 4y agoObviously it's a matter of degree, but languages with a better standard library seem less affected by dependency hell.
- simplotek 4y ago> (...) but languages with a better standard library seem less affected by dependency hell. Let's ignore the hand-waving over what "better standard library" means. Dependency hell has zero to do with standard libraries, and is exclusively related to modularity. If your language supports third party modules that can depend on third party modules, you have dependency management. Managing dependencies so far is hell, no matter how you go about it.
- pier25 4y agoOf course it's related. The more the standard library provides for you, the less you need to rely on third party dependencies.
- cxr 4y agoThis particular incarnation of the problem is a thing because this is actually not (standards-compliant) JS. This is a direct consequence of people infecting everything with non-standard NodeJS-isms (or NodeJS-inspired isms).
- randall 4y agoDeno has a similar thing... fwiw. esm.sh can do some of it but it ends up being kind of a shit show.
- triyambakam 4y agoWhat has been your bad experience with it?
- benatkin 4y agoYou should download from npm. Neither UNPKG nor jsDelivr is considered authoritative.
- simonw 4y agoThe problem I'm trying to solve is "give me an ECMAScript module version of this package, and fetch all of the dependencies too, in a way that I can use in a browser without needing a build system". If you know of the npm recipe for doing that I'd love to learn it - I'm fine running npm once to grab the module versions of things (which I intend to vendor in my repo), as long as I don't have to use any build tooling after that point.
- spankalee 4y agonpm + an import map generator should work. I think you might be able to use the jspm import map generator via the `jspm link` command: https://jspm.org/docs/jspm-cli/stable/#link https://jspm.org/docs/jspm-cli/stable/#link
- benatkin 4y agoA wrapper bundle is what I prefer right now, here is a wrapper bundle I made for CodeMirror 6. It just provides the needed exports from the library. It can also be verified on the client side with UNPKG or jsDelivr. My client side build prints out the integrity attribute. I still need to write the npm bundling code so at the moment I am relying on UNPKG. https://codeberg.org/macchiato/editor-lib-codemirror/src/branch/pages/app.js https://codeberg.org/macchiato/editor-lib-codemirror/src/bra... The rollup output is nice. I don't have terser, though I may add it later. When I added a couple exports the diff was just the change in my code. The library using it can include the integrity attribute and a can go here and run the build after checking the html and JavaScript file and check the output (codeberg.page is like GitHub pages in that it the repo contents controls what it serves): https://macchiato.codeberg.page/editor-lib-codemirror/ https://macchiato.codeberg.page/editor-lib-codemirror/
- triyambakam 4y agoCheck out https://esm.sh/#cli https://esm.sh/#cli for a minimalist yet full featured solution that amends your import map for you. It does require Deno.
- chatmasta 4y agoThanks for the tip about esm.sh. For some reason I hadn't heard of it before, or maybe I had but it was before I fixed issues with ES module exports in my library. Previously I was using Skypack, but it didn't bundle dependencies (at least as far as I could tell). Thanks to your tip, I can query public datasets with SQL with a one line import in an observable notebook [0] :) [0] https://observablehq.com/@milesrichardson/madatdata-esm-splitgraph-client https://observablehq.com/@milesrichardson/madatdata-esm-spli...
- Scarbutt 4y agoOr you just can download/install with npm and create a bundle with esbuild.
- simonw 4y agoI don't want to do any bundling - I want to use modules in my browser, at runtime.
- wruza 4y agoI understand there may be various reasons to avoid js bundling, but in case you avoid complexity, check out Vite. It’s zero config and can work on a regular directory (no node_modules, only index.html, styles, scripts). `vite` runs devserver with hmr, `vite build` generates dist. It may be useful to you even if you stick to esm downloading due to out of box support for all tech like ts, sass, less, etc. It’s basically an http server like npm/http-server, but with all modern tools automagically included.
- eyelidlessness 4y agoSo, the thing with Vite is that it’s great as long as your use case is in the happy paths it caters to. But if you diverge from them, its config story rapidly devolves to: - A very large subset of Rollup, which is so large that the docs seem to be self-mocking about it - Another very large subset of ESBuild, which is smaller but still non-trivial - A whole bunch of Vite-specific stuff that’s only partly (and incompletely/inconsistently) abstracting the omitted portions of those subsets - An enormous amount of terminology, some very poorly described in any docs if you even know which to consult - Plugins you’ll definitely need and which are a superset of Rollup’s, and the abstraction is so leaky that you’re often better off skipping the Vite keyword in your searches I mean, even in the worst case it’s not Webpack, but it’s astonishing how close it can feel very easily. And I don’t mean this nearly as harshly as it probably sounds. Vite really is great if it’s designed for your use case. But it’s far from the no-headaches panacea even I thought it was a few months ago.
- 4y ago
- laurencerowe 4y agoIf you're happy with a single bundled file, another option is: deno bundle 'https://esm.sh/@observablehq/plot' > plot.js My browser development workflow looks somewhat like the following: Imports / versions in importmap.json: { "imports": { "plot": "https://esm.sh/@observablehq/plot@0.6.6" } } Code in app.js: import * as Plot from "plot"; const plot = Plot.rectY({length: 10000}, Plot.binX({y: "count"}, {x: Math.random})).plot(); const div = document.querySelector("#myplot"); div.append(plot); For development use dev.html: <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <script type="importmap" src="./importmap.json"></script> <script type="module" src="./app.js"></script> </head> <body> <div id="myplot"></div> </body> </html> Then you can then bundle with: deno bundle --import-map=importmap.json app.js > bundle.js And have as your prod.html: <!DOCTYPE html> <html> <head> <meta charset="utf-8" /> <script type="module" src="./bundle.js"></script> </head> <body> <div id="myplot"></div> </body> </html> Note that `deno bundle` is deprecated. You can almost replace it with esbuild but it currently lacks builtin support for import maps: deno run --allow-all https://deno.land/x/esbuild@v0.17.18/mod.js --bundle app.js --outfile=bundle.js. # errors, see: https://github.com/evanw/esbuild/issues/2230 I'm really looking forward to the ECMAScript proposal on Type Annotations making it into browsers some days as this would effectively unlock transpiler free typescript development, leaving just a simple optional bundling step for production. https://devblogs.microsoft.com/typescript/a-proposal-for-type-syntax-in-javascript/ https://devblogs.microsoft.com/typescript/a-proposal-for-typ... https://github.com/tc39/proposal-type-annotations https://github.com/tc39/proposal-type-annotations
- easrng 4y agoi did more or less the same thing but running in the browser https://easrng.github.io/modbundler/ https://easrng.github.io/modbundler/ you give it an import map and a list of urls and it either bundles + minifies to a single .js file or rewrites the paths and saves everything as a .zip
- mikojan 4y agoI "just" reverse-proxy and http-cache the CDN using Varnish (i.a.). There is no build-step, no download- or installation-step, no nothing. Just... import x from "https://packages.my-domain.net/x@v1.2.3 https://packages.my-domain.net/x@v1.2.3" in JavaScript and it works. All files served locally. I always assumed, worst that could happen is somebody decides to download all of that CDN's library to my RAM by simply requesting the appropriate URLs until Varnish starts dropping the modules I wanted to cache in the first place.