9 ms·

My failed attempt to shrink all NPM packages by 5%

365 points by todsacerdoti 2y ago

huqedato 2y ago
Try to use this https://github.com/xthezealot/npmprune https://github.com/xthezealot/npmprune
- phdelightful 2y ago
  My reading of OP is that it’s less about whether zopfli is technically the best way to achieve a 5% reduction in package size, and more about how that relatively simple proposal interacted with the NPM committee. Do you think something like this would fare better or differently for some reason?
hartator 2y ago
I mean 4-5% the size for 10-100x the time is not worth it.
- swiftcoder 2y ago
  That's not actually so straightforward. You pay the 10-100x slowdown once on the compressing side, to save 4-5% on every download - which for a popular package one would expect downloads to be in the millions.
  - philipwhiuk 2y ago
    The downloads are cached. The build happens on every publish for every CI build.
    - realityking 2y ago
      Most packages don‘t publish every CI build. Some packages, such as Typescript, publish a nightly build once a day. Even then a longer compression time dosen‘t seem too bad.
    - swiftcoder 2y ago
      Caching downloads on a CDN helps offload work from the main server, it doesn't meaningfully change the bandwidth picture from the client's perspective.
- inglor_cz 2y ago
  As the author himself said, just React was downloaded half a billion times; that is a lot of saved bandwidth on both sides, but especially so for the server. Maybe it would make sense to only apply this improvement in images that are a) either very big or b) get downloaded at least million times each year or so. That would cover most of the savings while leaving most packages and developers out of it.
- bonzini 2y ago
  Assuming download and decompression cost to be proportional to the size of the incoming compressed stream, it would break even at 2000 downloads. A big assumption I know, but 2000 is a really small number.
- liontwist 2y ago
  It absolutely is. Packages are zipped once and downloaded thousands of times.
sd9 2y ago
The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't closed the RFC themselves. It could have been the sort of thing that eventually happens after 2 years, but then quietly makes everybody's lives better.
- n4r9 2y ago
  I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.
  - IshKebab 2y ago
    > The proposal wasn't rejected! They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider <extremely unlikely issue>? Please go and run more tests." It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares if it takes an extra 2 minutes to publish Typescript. Kind of a wild decision considering NPM uses 4.5 PB of traffic per week. 5% of that is 225 TB/week, which according to my brief checks costs around $10k/week! I guess this is a "not my money" problem fundamentally.
    - lyu07282 2y ago
      > which according to my brief checks costs around $10k/week That's the market price though, for Microsoft its a tiny fraction of that.
    - johnfn 2y ago
      This doesn't seem quite correct to me. They weren't asking for "more validation than was reasonable". They were asking for literally any proof that users would benefit from the proposal. That seems like an entirely reasonable thing to ask before changing the way every single NPM package gets published, ever. I do agree that 10k/week is non-negligible. Perhaps that means the people responsible for the 10k weren't in the room?
orta 2y ago
Congrats on a great write-up. Sometimes trying to ship something at that sorta scale turns out to just not really make sense in a way that is hard to see at the beginning. Another personal win is that you got a very thorough understanding of the people involved and how the outreach parts of the RFC process works. I've also had a few fail, but I've also had a few pass! Always easier to do the next time
stabbles 2y ago
One thing that's excellent about zopfli (apart from being gzip compatible) is how easy it is to bootstrap: git clone https://github.com/google/zopfli.git cc -O2 zopfli/src/zopfli/*.c -lm It just requires a C compiler and linker.
- stabbles 2y ago
  The main downside though, it's impressively slow. Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`. For example, gzipping llvm.tar (624MB) takes less than a second for me: $ time /home/harmen/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-13.2.0/pigz-2.8-5ptdjrmudifhjvhb757ym2bzvgtcsoqc/bin/pigz -k hello.tar real 0m0.779s user 0m11.126s sys 0m0.460s At the same time, zopfli compiled with -O3 -march=native takes 35 minutes. No wonder it's not popular. It is almost 2700x slower than the state of the art for just 6.8% bytes saved.
  - Levitating 2y ago
    > 2700x slower That is impressively slow. In my opinion even the 28x decrease in performance mentioned would be a no-go. Sure the package saves a few bytes but I don't need my entire pc to grind to a halt every time I publish a package. Besides, storage is cheap but CPU power draw is not. Imagine the additional CO2 that would have to be produced if this RFC was merged. > 2 gigabytes of bandwidth per year across all installations This must be a really rough estimate and I am curious how it was calculated. In any case 2 gigabytes over a year is absolutely nothing. Just my home network can produce a terabyte a day.
    - bonzini 2y ago
      2 GB for the author's package which is neither extremely common nor large; it would be 2 TB/year just for react core.
      - Levitating 2y ago
Demlmlm 2y ago
He should not have been closing this tbh. 5% sounds like a good deal
ape4 2y ago
Usually people require more than 5% to make a big change
- hinkley 2y ago
  That’s why our code is so slow. Dozens of poor decisions that each account for 2-4% of overall time lost, but 30-60% in aggregate.
fergie 2y ago
Props to anyone who tries to make the world a better place. Its not always obvious who has the most important use cases. In the case of NPM they are prioritizing the user experience of module authors. I totally see how this change would be great for module consumers, yet create potentially massive inconvenience for module authors. Interesting write-up
- atiedebee 2y ago
  I think "massive" is overstating it. I don't think deploying a new version of a package is something that happens many times a day, so it wouldn't be a constant pain point. Also, since this is a case of having something compressed once and decompressed potentially thousands of times, it seems like the perfect tool for the job.
  - philipwhiuk 2y ago
    Every build in a CI system would probably create the package. This is changing every build in every CI system to make it slower.
    - mkesper 2y ago
      Just use it on the release build.
  - fergie 2y ago
    Module authors generally have fairly large test suites which are run often- sometimes on each file save. If you have a 1 or 2 second build script its not a huge deal. If that script starts taking 30-60 seconds- you have just hosed productivity. Also you have massively increased the load on your CI server- possibly bumping you out of a free tier. The fix would then have to be some variation of: a) Stop testing (so often) b) Stop bundling before testing c) Publish to a different package manager - all of which would affect the overall quality and quantity npm modules.
    - atiedebee 2y ago
avodonosov 2y ago
My experiment on how to reduce javascript size of every web app by 30-50% : https://github.com/avodonosov/pocl https://github.com/avodonosov/pocl Working approach, but in the end I abandoned the project - I doubt people care about such js size savings.
- dagelf 2y ago
  Wdym?? 50% is a big deal
  - bluGill 2y ago
    50% size savings isn't important to the people who pay for it. They pay at most pennies for 100% savings (that is somehow all the functionality in zero bytes - not worth anything to those paying the bills)
    - tyre 2y ago
      Size savings translates to latency improvements which directly affects conversion rates. Smaller size isn’t about reducing costs but increased revenue. People care.
      - soared 2y ago
        Agreed - often a CTO of an ecom site is very very focused on site speed and has it as their #1 priority since it directly increases revenue.
      - fwip 2y ago
        Note that this proof-of-concept implementation saves latency on first load, but may add latency at surprising points while using the website. Any user invoking a rarely-used function would see a delay before the javascript executes, without the traditional UI affordances (spinners etc) to indicate that the application was waiting on the network. Further, these secretly-slow paths may change from visit to visit. Many users know how to "wait for the app to be ready," but the traditional expectation is that once it's loaded, the page/app will work, and any further delays will be signposted. I'm sure it works great when you've got high-speed internet, but might break things unacceptably for users on mobile or satellite connections.
jsheard 2y ago
I wonder if it would make more sense to pursue Brotli at this point, Node has had it built-in since 10.x so it should be pretty ubiquitous by now. It would require an update to NPM itself though.
- silverwind 2y ago
  +1 to brotli. Newly published packages could use brotli by default, so old ones stay compatible.
  - canucker2016 2y ago
    Here's the Brotli supporter's blog post about adding Brotli support to NPM packages. https://jamiemagee.co.uk/blog/honey-i-shrunk-the-npm-package/ https://jamiemagee.co.uk/blog/honey-i-shrunk-the-npm-package... and the related HN discussion from that time: https://news.ycombinator.com/item?id=37754489 https://news.ycombinator.com/item?id=37754489
pornel 2y ago
It only doesn't apply to existing versions of existing packages. Newer releases would apply Zopfli, so over time likely the majority of actively used/maintained packages would be recompressed.
choobacker 2y ago
Nice write up! > When it was finally my turn, I stammered. > Watching it back, I cringe a bit. I was wordy, unclear, and unconvincing. > You can watch my mumbling in the recording I watched this, and the author was articulate and presented well. The author is too harsh! Good job for trying to push the boundaries.
cedws 2y ago
Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.
- MortyWaves 2y ago
  Totally agree with you. I wish npm did a better job of filtering the crap files out of packages.
- Alifatisk 2y ago
  At least, switch to pnpm minimize the bloat
  - jefozabuss 2y ago
    I just installed a project with pnpm about 120 packages mostly react/webpack/eslint/redux related with prod env: 700MB without prod env: 900MB sadly the bloat cannot be avoided that well :/
    - jeffhuys 2y ago
      pnpm stores them in a central place and symlinks them. You’ll see the benefits when you have multiple projects with a lot of the same packages.
      - syncsynchalt 2y ago
        You'll also see the benefit when `rm -rf`ing a `node_modules` and re-installing, as pnpm still has a local copy that it can re-link after validating its integrity.
  - MBCook 2y ago
    As someone who mostly works in Java it continues to floor me that this isn’t the default. Why does every project I work on need an identical copy of possibly hundreds of packages if they’re the same version? I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping. Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.
glenjamin 2y ago
This strikes me as something that could be done for the highest traffic packages at the backend, rather than be driven by the client at pubish-time.
- fastest963 2y ago
  The article talks about this. There are hashes that are generated for the tarball so the backend can't recompress anything.
deleted 2y ago
[deleted]
BonoboIO 2y ago
In summary: It’s a nice feature, which gives nice benefits for often downloaded packages, but nobody at npm cares for the bandwidth?
nikeee 2y ago
I don't see why it wouldn't be possible to hide behind a flag once Node.js supports zopfli natively. In case of CI/CD, it's totally feasible to just add a --strong-compression flag. In that case, the user expects it to take its time. TS releases a non-preview version every few months, so using 2.5 minutes for compression would work.
- jopsen 2y ago
  Think of the complexity involved, think of having to fix bugs because something went wrong. Such effort would be better spent preparing npm/node for a future where packages with a lower-bound npm version constraint can be compressed with zstd or similar. Every feature you add to a program is complexity, you need to really decide if you want it.
abound 2y ago
A few people have mentioned the environmental angle, but I'd care more about if/how much this slows down decompression on the client. Compressing React 20x slower once is one thing, but 50 million decompressions being even 1% slower is likely net more energy intensive, even accounting for the saved energy transmitting 4-5% fewer bits on the wire.
- web007 2y ago
  It's very likely zero or positive impact on the decompression side of things. Starting with smaller data means everything ends up smaller. It's the same decompression algorithm in all cases, so it's not some special / unoptimized branch of code. It's yielding the same data in the end, so writes equal out plus or minus disk queue fullness and power cycles. It's _maybe_ better for RAM and CPU because more data fits in cache, so less memory is used and the compute is idle less often. It's relatively easy to test decompression efficiency if you think CPU time is a good proxy for energy usage: go find something like React and test the decomp time of gzip -9 vs zopfli. Or even better, find something similar but much bigger so you can see the delta and it's not lost in rounding errors.
- sltkr 2y ago
  For formats like deflate, decompression time doesn't generally depend on compressed size. (zstd is similar, though memory use can depend on the compression level used). This means an optimization like this is virtually guaranteed to be a net positive on the receiving end, since you always save a bit of time/energy when downloading a smaller compressed file.
- DannyBee 2y ago
  I can speak to this - there is no meaningful decompression effect across an insane set of tested data at Google and elsewhere. Zopfli was invented prior to brotli Zopfli is easiest to think of as something that just tries harder than gzip to find matches and better encodings. Much harder. decompression speed is linear either way. It's easiest to think of decompression as a linear time vm executor[1], where the bytecoded instructions are basically go back <distance> bytes, output the next <length> bytes you see, then output character <c> (outputting literal data is the instruction <0,0,{character to output}>) Assuming you did not output a file larger than the original uncompressed file (why would you bother?), you will, worst case, process N bytes during decompression, where N is the size of the original input file. The practical decompression speed is driven by cache behavior, but it thrashes the cache no matter what. In practice, reduction of size vs gzip occurs by either finding larger runs, or encodings that are smaller than the existing ones. After all, if you want the compressed file to shrink, you need output less instructions somehow, or make more of the instructions identical (so they can be represented in less bits by later huffman coding). In practice, this has almost exclusively positive effects on decompression speed - either the vm has less things to process (which is faster), or more of the things it does look the same (which has better cache behavior). [1] this is one way archive formats will sometimes choose to deal with multiple compression method support - encode them all to the same kind of bytecode (usually some form of copy + literal instruction set), and then decoding is the same for all of them. ~all compression algorithms output some bytecode like the above on their own already, so it's not a lot of work. This doesn't help you support other archive formats, but if you want to have a bunch of per-file compression options that you pick from based on what works best, this enables you to still only have to have one decoder.
adgjlsfhk1 2y ago
This seems like a place where the more ambitious version that switches to ZSTD might have better tradeoffs. You would get similar or better compression, with faster decompression and recompression than zstd.It would lose backward compatibility though...
- bufferoverflow 2y ago
  Brotli and lzo1b have good compression ratios and pretty fast decompression speeds. Compression speed should not matter that much, since you only do it once. https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ There even more obscure options: https://www.mattmahoney.net/dc/text.html https://www.mattmahoney.net/dc/text.html
- vlovich123 2y ago
  Not necessarily - could retain backward compat by publishing both gzip and zstd variants and having downloaders with newer npm’s prefer to download zstd. Over time, you could require packages only upload zstd going forward and either generate zstd versions of the backlog of unmaintained packages or at least those that see some amount of traffic over some time period if you’re willing to drop very old packages. The ability to install arbitrary versions of packages probably means you’re probably better off reprocessing the backlog although that may cost more than doing nothing. The package lock checksum is probably a more solvable issue with some coordination. The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).
- MBCook 2y ago
  Thats a much higher hurdle to jump. I don’t blame the author for trying this first. If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think. So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.
chuckadams 2y ago
Switching to a shared cache in the fashion of pnpm would eliminate far more redundant downloads than a compression algorithm needing 20x more CPU.
liontwist 2y ago
The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And I would have quit at the first roadblock. What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line.
- dr-detroit 2y ago
  [dead]
- coliveira 2y ago
  The problem is that this guy is treating open source as if it was a company where you need to direct a project to completion. Nobody in open source wants to be told what to do. Just release your work, if it is useful, the community will pick it up and everybody will benefit. You cannot force your improvement into the whole group, even if it is beneficial in your opinion.
  - deleted 2y ago
    [deleted]
  - liontwist 2y ago
    > where you need to direct a project to completion Do you want to get a change in, or not? Is this a project working with the community or not? > Just release your work What motivation exists to optimize package formats if nobody uses that package format? There are no benefits unless it’s in mainline. > Nobody in open source wants to be told what to do He’s not telling anybody to do work. He is sharing an optimization with clear tradeoffs - not a new architecture. > You cannot force your improvement into the whole group Nope, but communication is key. “put up a PR and we will let you know whether it’s something we want to pursue”. Instead they put him through several levels of gatekeepers where each one gave him incorrect feedback. “Why do we want to optimize bandwidth” is a question they should have the answer to. If this PR showed up on my project I would say “I’m worried about X,Y,Z” we will set up a test for X and Y and get back to you. Could you look into Z?
liontwist 2y ago
Why would a more complex zip slow down decompress? This comment seems to misunderstand how these formats work. OP is right.
dvh 2y ago
Imagine being in the middle of nowhere, in winter, on Saturday night, on some farm, knee deep in a cow piss, servicing some 3rd party feed dispenser, only to discover that you have possible solution but it's in some obscure format instead of .tar.gz. Nearest internet 60 miles away. This is what I always imagine happening when some new obscure format come into play, imagine the poor fella, alone, cold, screaming. So incredibly close to his goal, but ultimately stopped by some artificial unnecessary made-up bullshit.
- tehjoker 2y ago
  I believe zopfli compression is backwards compatible with DEFLATE, it just uses more CPU during the compression phase.
  - DannyBee 2y ago
    Correct - it is just looking for matches harder, and encoding harder.
PaulHoule 2y ago
Years back I came to the conclusion that conda using bzip2 for compression was a big mistake. Back then if you wanted to use a particular neural network it was meant for a certain version of Tensorflow which expected you to have a certain version of the CUDA libs. If you had to work with multiple models the "normal" way to do things was use the developer unfriendly [1][2] installers from NVIDIA to install a single version of the libs at a time. Turned out you could have many versions of CUDA installed as long as you kept them in different directories and set the library path accordingly, it made sense to pack them up for conda and install them together with everything else. But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time. If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression. [1] click... click... click... [2] like they're really going to do something useful with my email address
- zahlman 2y ago
  >But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time. For Paper, I'm planning to cache both the wheel archives (so that they're available without recompressing on demand) and unpacked versions (installing into new environments will generally use hard links to the unpacked cache, where possible). > If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression. FWIW, in my testing LZMA is a big win (and I'm sure zstd would be as well, but LZMA has standard library support already). But there are serious roadblocks to adopting a change like that in the Python ecosystem. This sort of idea puts them several layers deep in meta-discussion - see for example https://discuss.python.org/t/pep-777-how-to-re-invent-the-wheel/67484 https://discuss.python.org/t/pep-777-how-to-re-invent-the-wh... . In general, progress on Python packaging gets stuck in a double-bind: try to change too little and you won't get any buy-in that it's worthwhile, but try to change too much and everyone will freak out about backwards compatibility.
  - PaulHoule 2y ago
kittikitti 2y ago
Thank you so much for posting this. The original logic was clear and it had me excited! I believe this is useful because compression is very common and although it might not fit perfectly in this scenario, it could very well be a breakthrough in another. If I come across a framework that could also benefit from this compression algorithm, I'll be sure to give you credit.
bhouston 2y ago
What about a different approach - an optional npm proxy that recompresses popular packages with 7z/etc in the background? Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready. Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packages. Main tradeoff is additional verification steps, but they could be optional given a digital signature approach. Curious if others see major security holes in this approach?
- ndriscoll 2y ago
  This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller than a .gz.
  - cesarb 2y ago
    > hash a .tar file, not a .tar.gz file For security reasons, it's usually better to hash the compressed file, since it reduces the attack surface: the decompressor is not exposed to unverified data. There have already been vulnerabilities in decompressor implementations which can be exploited through malformed compressed data (and this includes IIRC at least one vulnerability in zlib, which is the standard decompressor for .gz).
    - bhouston 2y ago
      This suggests one should just upload a tar rather than a compressed file. Makes sense because one can scan the contents for malicious files without risking a decompressor bug. BTW npm decompressed all packages anyhow because it lets you view the contents these days on its website.
  - bhouston 2y ago
hinkley 2y ago
Does npm even default to gzip -9? Wikipedia claims zopfli is 80 times slower under default settings.
- zahlman 2y ago
  My experience has been that past -6 or so, gzip files get only a tiny bit smaller in typical cases. (I think I've even seen them get bigger with -9.)
- EvanHahn 2y ago
  Yes: https://github.com/npm/pacote/blob/bf1f60f58bb61f053262f54724edcacaadb221ce/lib/util/tar-create-options.js#L12 https://github.com/npm/pacote/blob/bf1f60f58bb61f053262f5472...
jefozabuss 2y ago
I wonder what is the tarball size difference on average if you'd for example download everything in one tarball (full package list) instead of 1-by-1 as the gzip compression would work way better in that case. Also for bigger companies this is not really a "big" problem as they usually have in-house proxies (as you cannot rely on a 3rd party repository in CI/CD for multiple reasons (security, audit, speed, etc)).
- Dylan16807 2y ago
  You might save a little bit by putting similar very small files next to each other in the same tarball, but in general I would not expect significant improvements. Gzip can only compress repetitions that are within 32KB of each other.
snizovtsev 2y ago
Yes, but it was expected. It's like prioritising code readability over performance everywhere but the hot path. Earlier in my career, I managed to use Zopfli once to compress gigabytes of PNG assets into a fast in-memory database supporting a 50K+ RPS web page. We wanted to keep it simple and avoid the complexity of horizontal scaling, and it was OK to drop some rarely used images. So the more images we could pack into a single server, the more coverage we had. In that sense Zopfli was beneficial.
1337shadow 2y ago
Ok but why doesn't npm registry actually recompress the archives? It can even apply that retroactively, wouldn't require zopli in npm CLI
- aseipp 2y ago
  Hashes of the tarballs are recorded in the package-lock.json of downstream dependants, so recompressing the files in place will cause the hashes to change and break everyone. It has to be done at upload time.
  - notpushkin 2y ago
    But it still can be done on the npm side, right?
  - bhouston 2y ago
    The hashes of the uncompressed tarballs would be great. Then the HTTP connection can negotiate a compression format for transfer (which can change over time at HTTP itself changes) rather than baking it into the NPM package standard (which is incredibly inflexible.)
tonymet 2y ago
I’m more concerned about the 40gb of node_modules. Why hasn’t node supported tgz node_modules. That would save 75% of the space or more.
- bhouston 2y ago
  isn't this a file system thing? Why bake it into npm?
  - tonymet 2y ago
    efficiency
    - bhouston 2y ago
      The os should do file deduplication and compression and decompression faster than npm. But I guess the issue is npm cannot give the os hints to compress a folder or not?
      - MBCook 2y ago
        Even if the file system does it that’s still per-file. You still need all the bookkeeping. And you’re not gonna get great compression if a lot of the files are tiny. The tar contains a whole bunch of related files that probably compress a lot better together, and it’s only one file on disk for the file system to keep track of. It’s gonna be a lot more efficient. When compiling you’re probably gonna touch all the source files anyway right? So if you load the zip into memory once and decompress it there it’s probably faster than loading each individual file and decompressing each of them, or just loading each individual file raw.
        bhouston 2y ago
        > And you’re not gonna get great compression if a lot of the files are tiny. The future of compression is likely shared dictionaries, basically recognize the file type and then use a dictionary that is optimized for that file type. E.g. JavaScript/TypeScript or HTML or Rust or C++, etc. That can offset the problems with small files to a large degree. But again, this is a complex problem and it should be dealt with orthogonally to npm in my opinion. I also checked and it should be possible for npm to enable file system compression at the OS level on both Windows and MacOS. Also if it was dealt with orthogonally to npm, then it could be used by pip, and other package systems.
nopurpose 2y ago
It reminds me of an effort to improve docker image format and make it move away from being just a tar file. I can't find links anymore, but it was a pretty clever design, which still couldn't beat dumb tar in efficiency.
- bhouston 2y ago
  Transferring around dumb tar is actually smart because the HTTPS connection can negotiate a compressed version of it to transfer - e.g. gzip, brotli, etc. No need to bake in an unchangable compression format into the standard.
woadwarrior01 2y ago
From the RFC on github[1]. > Zopfli is written in C, which presents challenges. Unless it was added to Node core, the CLI would need to (1) rewrite Zopfli in JS, possibly impacting performance (2) rely on a native module, impacting reliability (3) rely on a WebAssembly module. All of these options add complexity. Wow! Who's going to tell them that V8 is written in C++? :) [1]: https://github.com/npm/rfcs/pull/595 https://github.com/npm/rfcs/pull/595
- kmacdough 2y ago
  It's not about C per-se, as much as each native compiled dependency creates additional maintenance concerns. Changes to hardware/OS can require a recompile or even fixes. NPM build system already requires a JavaScript runtime, so is already handled as part of existing maintenance. The point is that Zopfli either needs to be rewritten for a platform-agnostic abstraction they already support, or else Zopfli will be added to a list of native modules to maintain.
  - woadwarrior01 2y ago
    > It's not about C per-se, as much as each native compiled dependency creates additional maintenance concerns. Changes to hardware/OS can require a recompile or even fixes. This is a canard. zopfli is written in portable C and is far more portable than the nodejs runtime. On any hardware/OS combo that one can build the nodejs runtime, they certainly can also build and run zopfli.
gweinberg 2y ago
I was under the impression that bzip compresses more than gzip, but gzip is much faster, so gzip is better for things that need to be compressed on the fly, and bzip is better fro things that get archived. Is this not true? Wouldn't it have been better to use bzip all along for this purpose?
bangaladore 2y ago
I think the main TLDR here [1]: > For example, I tried recompressing the latest version of the typescript package. GNU tar was able to completely compress the archive in about 1.2 seconds on my machine. Zopfli, with just 1 iteration, took 2.5 minutes. [1] https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 My question of course would be, what about LZ4, or Zstd or Brotli? Or is backwards compatibility strictly necessary? I understand that GZIP is still a good compressor, so those others may not produce meaningful gains. But, as the author suggests, even small gains can produce huge results in bandwidth reduction.
loeg 2y ago
It probably makes more sense to save more bytes and compressor time and just switch to zstd (a bigger scoped effort, sure).
imoverclocked 2y ago
> But the cons were substantial: ... > This wouldn’t retroactively apply to existing packages. Why is this substantial? My understanding is that packages shouldn't be touched once published. It seems likely for any change to not apply retroactively.
szundi 2y ago
This is a nice guy
omoikane 2y ago
> Integrating Zopfli into the npm CLI would be difficult. Is it possible to modify "gzip -9" or zlib to invoke zopfli? This way everyone who wants to compress better will get the extra compression automatically, in addition to npm. There will be an increase in compression time, but since "gzip -9" is not the default, people preferring compression speed might not be affected.
- bombcar 2y ago
  You'd have more problems here, but you could do it - if you let it take ages and ages to percolate though all environments. It's been almost 30 years since bzip2 was released and even now not everything can handle tar.bz2
  - arccy 2y ago
    probably because bzip2 isn't a very good format
tehjoker 2y ago
5% improvement is basically the minimum I usually consider worthwhile to pursue, but it's still small. Once you get to 10% or 20%, things become much more attractive. I can see how people can go either way on a 5% increase if there are any negative consequences (such as increased build time).
Alifatisk 2y ago
I wonder if there is a way to install the npm packages without the crap they come included with (like docs, tests, readme etc).
rafaelmn 2y ago
I wonder if you could get better results if you built a dictionary over entire npm. I suspect most common words could easily be reduced to 16k word index. Would be much faster, dictionary would probably fit in cache, can even optimize it in memory for cache prefetch.
- zahlman 2y ago
  This seems like a non-starter to me - new packages are added to npm all the time, and will alter the word frequency distribution. If you aren't prepared to re-build constantly and accept that the dictionary isn't optimal, then it's hard to imagine it being significantly better than what you build with a more naive approach. Basically - why try to fine-tune to a moving target?
  - arccy 2y ago
    would it really change that quickly? you might get significant savings from just having keywords, common variable names, standard library functions
  - rafaelmn 2y ago
    But is it really moving that fast ? I suspect most fundamental terms in programming and the variations do not change often. You will always have keywords, built ins and the most popular concepts from libs/frameworks. So it is basically downloading a few hundred kb dictionary every year ?
    - MBCook 2y ago
      You’d have language keywords and such, yeah. But past that the most common words in a React project are going to be very different from a Vue project right?
hinkley 2y ago
Pulling on this thread, there are a few people who have looked at the ways zopfli is inefficient. Including this guy who forked it, and tried to contribute a couple improvements back to master: https://github.com/fhanau/Efficient-Compression-Tool https://github.com/fhanau/Efficient-Compression-Tool These days if you’re going to iterate on a solution you’d better make it multithreaded. We have laptops where sequential code uses 8% of the available cpu.
- vardump 2y ago
  > These days if you’re going to iterate on a solution you’d better make it multithreaded. Repetition eliminating compression tends to be inherently sequential. You'd probably need to change the file format to support chunks (or multiple streams) to do so. Because of LZ back references, you can't LZ compress different chunks separately on different cores and have only one compression stream. Statistics acquisition (histograms) and entropy coding could be parallel I guess. (Not a compression guru, so take above with a pinch of salt.)
  - hinkley 2y ago
    There are gzip variants that break the file into blocks and run in parallel. They lose a couple of % by truncating the available history. But zopfli appears to do a lot of backtracking to find the best permutations for matching runs that have several different solutions. There’s a couple of ways you could run those in parallel. Some with a lot of coordination overhead, others with a lot of redundant calculation.
frabjoused 2y ago
This reminds me of a time I lost an argument with John-David Dalton about cleaning up/minifying lodash as an npm dependency, because when including the readme and license for every sub-library, a lodash import came to ~2.5MB at the time. This also took a lot of seeking time for disks because there were so many individual files. The conversation started and ended at the word cache.
- MBCook 2y ago
  > This also took a lot of seeking time for disks because there were so many individual files. The fact NPM keeps things in node_modules unzipped seems wild to me. Filesystems are not great at hundreds of thousands of little files. Some are bad, others are terrible. Zip files are easier to store, take up less space, and CPUs are faster than disks so the decompression in memory is probably faster reading the unzipped files. That was one of my favorite features of Yarn when I tried it - pnp mode. But since it’s not what NPM does it requires a shim that doesn’t work with all ps mage’s. Or at least didn’t a few years ago.
zahlman 2y ago
I'd love to see an effort like like this succeed in the Python ecosystem. Right now, PyPI is dependent upon Fastly to serve files, on the order of >1 petabyte per day. That's a truly massive in-kind donation, compared to the PSF's operating budget (only a few million dollars per year - far smaller than Linux or Mozilla).
- cozzyd 2y ago
  No problem, I'm sure if Fastly stopped doing it JiaTanCo would step up
JoeAltmaier 2y ago
These days technology moves so fast it's hard to keep up. The slowest link in the system is the human being. That's a strong argument that 'if it isn't broke, don't fix it." LOts of numbers being thrown around, you add up tiny things enough times you can get a big number. But is npm package download the thing that's tanking the internet? No? Then this is a second- or thirt-order optimization.
luzifer42 2y ago
I created once a maven plugin to recompress Java artefacts with zopfli. I rewrote it in Java and runs entirely in the JVM. This means, the speed is worse and may contain bugs: https://luccappellaro.github.io/2015/03/01/ZopfliMaven.html https://luccappellaro.github.io/2015/03/01/ZopfliMaven.html
Julien_r2 2y ago
A pro could have been an extra narrative about carbon footprint savings. I'm surprised it hasn't been raised when talking about saving 2Tb/year only for React. It represents costs, which doesn't seem to be an issue, but also computing power and storage. (Event with a higher/longer computing power due to slower compression, it's done once per version, which isn't really comprable to the amount of downloads anyway) Hard to calculate the exact saving, but it would represent a smaller CO2 footprint..
wallunit 2y ago
I used zopflipng in the past to optimize PNG images. It made sense since there was no better alternative to store lossless image data than the PNG format at the given time in the given environment. Zopfli is awesome when you are locked in on deflate compression. I feel like if the npm folks would want to optimize for smaller package size a better strategy would be switching to some more effective text compression (e.g. bzip2, xz). That would result into a larger file size reduction than 5% for a smaller CPU time increase compared to Zopfli. You would need to come up with some migration strategy though as this change isn't per-se backwards compatible, but that seems manageable if you are in control of the tooling and infrastructure.
vivzkestrel 2y ago
I ll give you an even better idea but it ll need atleast a 100 volunteers, maybe a 1000. Take each package and rewrite it without external dependencies. That will cut tech debt for that package significantly. Just like how we have a @types/xyz where some dude named DefinitelyTyped is busy making typescript packages for everything, lets make a namespace like @efficient/cors @efficient/jsdom @efficient/jest etc and eliminate all external dependencies completely for every library on npm
DannyBee 2y ago
zopfli is the wrong thing to use here. If you want an example of where these things go badly: the standard compression level for rpms on redhat distros is zstd level 19. This has all the downsides of other algorithms - it's super slow - often 75-100x slower than the default level of zstd[1]. It achieves a few percent more compression for that speed. compared to even level 10, it's like 0-1% higher compression, but 20x slower. This is bad enough - the kicker is that at this level, it's slower than xz for ~all cases, and xz is 10% smaller. The only reason to use zstd is because you want fairly good compression, but fast. So here, they've chosen to use it in a way that compresses really slowly, but gives you none of the benefit of compressing really slowly. Now, unlike the npm case, there was no good reason to choose level 19 - there were no backwards compatibility constraints driving it, etc. I went through the PR history on this change, it was not particularly illuminating (it does not seem lots of thought was given to the level choice). I mention all this because it has a real effect on the experience of building rpms - this is why it takes eons to make kernel debuginfo rpms on fedora. Or any large RPM. Almost all time is spent compressing it with zstd at level 19. On my computer this takes many minutes. If you switch it to use even xz, it will do it about 15-20x faster (single threaded. if you thread both of them, xz will win by even more, because of how slow the setting is for zstd. If you use reasonable settings for zstd, obviously, it achieves gigabytes/second in parallel mode) Using zopfli would be like choosing level 19 zstd for npm. While backwards compatibility is certainly painful to deal with here, zopfli is not likely better than doing nothing. You will make certain cases just insanely slow. You will save someone's bandwidth, but in exchange you will burn insane amounts of developer CPU. zopfli is worse than level 19, it can often be 100x-200x slower than gzip -9. Doesn't npm support insanely relaxed/etc scripting hooks anyway? If so, if backwards compatibility is your main constraint, you would be "better off" double compressing (IE embed xz or whatever + a bootstrap decompressor and using the hooks to decompress it on old versions of npm). Or just shipping .tar.gz's than, when run, fetch the .xz and decompress it on older npm. Or you know, fish in the right pond - you would almost certainly achieve much higher reductions by enforcing cleaner shipping packages (IE not including random garbage, etc) than by compressing the garbage more. [1] on my computer, single threaded level 19 does 7meg/second, the default does 500meg/second. Level 10 does about 130meg/second.
- Someone 2y ago
kavenkanum 2y ago
If OP wanted to shrink nom packages then npm could introduce two types of npm package - the build package and source one. This way a lot of packages would be smaller, because package code could be safely distributed through a separate package and not kept in the build package. There are a lot npm packages that explicitly include whole git repo in the package and they do so because there's only one type of package they can use
franga2000 2y ago
We wouldn't have to worry about over-the-wire package size if the modern DevOps approach wasn't "nuke everything, download from the Internet" every build. Back in my Java days, most even small-time dev shops had a local Maven registry that would pass through and cache the big ones. A CI job, even if the "container" was nuked before each build, would create maybe a few kilobytes of Internet traffic, possibly none at all. Now your average CI job spins up a fresh VM or container, pulls a Docker base image, apt installs a bunch of system dependencies, pip/npm/... installs a bunch of project dependencies, packages things up and pushes the image to the Docker registry. No Docker layer caching because it's fresh VM, no package manager caching because it's a fresh container, no object caching because...you get the idea.... Even if we accept that the benefits of the "clean slate every time" approach outweigh the gross inefficiency, why aren't we at least doing basic HTTP caching? I guess ingress is cheap and the egress on the other side is "someone else's money".
- Terr_ 2y ago
  Yeah, I would also note that in addition to speed/transfer-costs, having an organizational package proxy is useful for reproducibility and security.
- robertlagrant 2y ago
  Lots of places use a cache like Artifactory so they don't get slammed with costs, and are resilient to network outages and dependency builds vanishing.
- alimbada 2y ago
  After reading the article, this comment and the comment thread further down on pnpm[1], it feels to me like the NPM team are doing everyone a disservice by ignoring the inefficiencies in the packaging system. It may not be deliberate or malicious but they could easily have provided better solutions than the one proposed in the article which, in my opinion is a band-aid solution at best. The real fix would be to implement what you mention here: local registry and caching, and/or symlinking a la pnpm. [1] https://news.ycombinator.com/item?id=42841658 https://news.ycombinator.com/item?id=42841658
- 2y ago
animuchan 2y ago
This presents a great opportunity for alternative npm clients (such as bun, yarn). An almost free advantage over the mainline npm CLI.
jokethrowaway 2y ago
Nice exercise in bureaucracy. - I don't see the problem with adding a C dependency to a node project, native modules are one of the nicest things about node.js - Longer publishing time for huge packages (eg. typescript) is a bigger problem but I don't think it should impact a default. Just give the option to use gzip for the few outliers.

365 points by todsacerdoti 2y ago

huqedato 2y ago

Try to use this https://github.com/xthezealot/npmprune https://github.com/xthezealot/npmprune

phdelightful 2y ago
My reading of OP is that it’s less about whether zopfli is technically the best way to achieve a 5% reduction in package size, and more about how that relatively simple proposal interacted with the NPM committee. Do you think something like this would fare better or differently for some reason?

hartator 2y ago

I mean 4-5% the size for 10-100x the time is not worth it.

swiftcoder 2y ago
That's not actually so straightforward. You pay the 10-100x slowdown once on the compressing side, to save 4-5% on every download - which for a popular package one would expect downloads to be in the millions.
- philipwhiuk 2y ago
  The downloads are cached. The build happens on every publish for every CI build.
  - realityking 2y ago
    Most packages don‘t publish every CI build. Some packages, such as Typescript, publish a nightly build once a day. Even then a longer compression time dosen‘t seem too bad.
  - swiftcoder 2y ago
    Caching downloads on a CDN helps offload work from the main server, it doesn't meaningfully change the bandwidth picture from the client's perspective.
inglor_cz 2y ago
As the author himself said, just React was downloaded half a billion times; that is a lot of saved bandwidth on both sides, but especially so for the server. Maybe it would make sense to only apply this improvement in images that are a) either very big or b) get downloaded at least million times each year or so. That would cover most of the savings while leaving most packages and developers out of it.
bonzini 2y ago
Assuming download and decompression cost to be proportional to the size of the incoming compressed stream, it would break even at 2000 downloads. A big assumption I know, but 2000 is a really small number.
liontwist 2y ago
It absolutely is. Packages are zipped once and downloaded thousands of times.

sd9 2y ago

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't closed the RFC themselves. It could have been the sort of thing that eventually happens after 2 years, but then quietly makes everybody's lives better.

n4r9 2y ago
I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.
- IshKebab 2y ago
  > The proposal wasn't rejected! They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider <extremely unlikely issue>? Please go and run more tests." It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares if it takes an extra 2 minutes to publish Typescript. Kind of a wild decision considering NPM uses 4.5 PB of traffic per week. 5% of that is 225 TB/week, which according to my brief checks costs around $10k/week! I guess this is a "not my money" problem fundamentally.
  - lyu07282 2y ago
    > which according to my brief checks costs around $10k/week That's the market price though, for Microsoft its a tiny fraction of that.
  - johnfn 2y ago
    This doesn't seem quite correct to me. They weren't asking for "more validation than was reasonable". They were asking for literally any proof that users would benefit from the proposal. That seems like an entirely reasonable thing to ask before changing the way every single NPM package gets published, ever. I do agree that 10k/week is non-negligible. Perhaps that means the people responsible for the 10k weren't in the room?

orta 2y ago

Congrats on a great write-up. Sometimes trying to ship something at that sorta scale turns out to just not really make sense in a way that is hard to see at the beginning. Another personal win is that you got a very thorough understanding of the people involved and how the outreach parts of the RFC process works. I've also had a few fail, but I've also had a few pass! Always easier to do the next time

stabbles 2y ago

One thing that's excellent about zopfli (apart from being gzip compatible) is how easy it is to bootstrap: git clone https://github.com/google/zopfli.git cc -O2 zopfli/src/zopfli/*.c -lm It just requires a C compiler and linker.

stabbles 2y ago
The main downside though, it's impressively slow. Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`. For example, gzipping llvm.tar (624MB) takes less than a second for me: $ time /home/harmen/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-13.2.0/pigz-2.8-5ptdjrmudifhjvhb757ym2bzvgtcsoqc/bin/pigz -k hello.tar real 0m0.779s user 0m11.126s sys 0m0.460s At the same time, zopfli compiled with -O3 -march=native takes 35 minutes. No wonder it's not popular. It is almost 2700x slower than the state of the art for just 6.8% bytes saved.
- Levitating 2y ago
  > 2700x slower That is impressively slow. In my opinion even the 28x decrease in performance mentioned would be a no-go. Sure the package saves a few bytes but I don't need my entire pc to grind to a halt every time I publish a package. Besides, storage is cheap but CPU power draw is not. Imagine the additional CO2 that would have to be produced if this RFC was merged. > 2 gigabytes of bandwidth per year across all installations This must be a really rough estimate and I am curious how it was calculated. In any case 2 gigabytes over a year is absolutely nothing. Just my home network can produce a terabyte a day.
  - bonzini 2y ago
    2 GB for the author's package which is neither extremely common nor large; it would be 2 TB/year just for react core.
    - Levitating 2y ago

Demlmlm 2y ago

He should not have been closing this tbh. 5% sounds like a good deal

ape4 2y ago

Usually people require more than 5% to make a big change

hinkley 2y ago
That’s why our code is so slow. Dozens of poor decisions that each account for 2-4% of overall time lost, but 30-60% in aggregate.

fergie 2y ago

Props to anyone who tries to make the world a better place. Its not always obvious who has the most important use cases. In the case of NPM they are prioritizing the user experience of module authors. I totally see how this change would be great for module consumers, yet create potentially massive inconvenience for module authors. Interesting write-up

atiedebee 2y ago
I think "massive" is overstating it. I don't think deploying a new version of a package is something that happens many times a day, so it wouldn't be a constant pain point. Also, since this is a case of having something compressed once and decompressed potentially thousands of times, it seems like the perfect tool for the job.
- philipwhiuk 2y ago
  Every build in a CI system would probably create the package. This is changing every build in every CI system to make it slower.
  - mkesper 2y ago
    Just use it on the release build.
- fergie 2y ago
  Module authors generally have fairly large test suites which are run often- sometimes on each file save. If you have a 1 or 2 second build script its not a huge deal. If that script starts taking 30-60 seconds- you have just hosed productivity. Also you have massively increased the load on your CI server- possibly bumping you out of a free tier. The fix would then have to be some variation of: a) Stop testing (so often) b) Stop bundling before testing c) Publish to a different package manager - all of which would affect the overall quality and quantity npm modules.
  - atiedebee 2y ago

avodonosov 2y ago

My experiment on how to reduce javascript size of every web app by 30-50% : https://github.com/avodonosov/pocl https://github.com/avodonosov/pocl Working approach, but in the end I abandoned the project - I doubt people care about such js size savings.

dagelf 2y ago
Wdym?? 50% is a big deal
- bluGill 2y ago
  50% size savings isn't important to the people who pay for it. They pay at most pennies for 100% savings (that is somehow all the functionality in zero bytes - not worth anything to those paying the bills)
  - tyre 2y ago
    Size savings translates to latency improvements which directly affects conversion rates. Smaller size isn’t about reducing costs but increased revenue. People care.
    - soared 2y ago
      Agreed - often a CTO of an ecom site is very very focused on site speed and has it as their #1 priority since it directly increases revenue.
    - fwip 2y ago
      Note that this proof-of-concept implementation saves latency on first load, but may add latency at surprising points while using the website. Any user invoking a rarely-used function would see a delay before the javascript executes, without the traditional UI affordances (spinners etc) to indicate that the application was waiting on the network. Further, these secretly-slow paths may change from visit to visit. Many users know how to "wait for the app to be ready," but the traditional expectation is that once it's loaded, the page/app will work, and any further delays will be signposted. I'm sure it works great when you've got high-speed internet, but might break things unacceptably for users on mobile or satellite connections.

jsheard 2y ago

I wonder if it would make more sense to pursue Brotli at this point, Node has had it built-in since 10.x so it should be pretty ubiquitous by now. It would require an update to NPM itself though.

silverwind 2y ago
+1 to brotli. Newly published packages could use brotli by default, so old ones stay compatible.
- canucker2016 2y ago
  Here's the Brotli supporter's blog post about adding Brotli support to NPM packages. https://jamiemagee.co.uk/blog/honey-i-shrunk-the-npm-package/ https://jamiemagee.co.uk/blog/honey-i-shrunk-the-npm-package... and the related HN discussion from that time: https://news.ycombinator.com/item?id=37754489 https://news.ycombinator.com/item?id=37754489

pornel 2y ago

It only doesn't apply to existing versions of existing packages. Newer releases would apply Zopfli, so over time likely the majority of actively used/maintained packages would be recompressed.

choobacker 2y ago

Nice write up! > When it was finally my turn, I stammered. > Watching it back, I cringe a bit. I was wordy, unclear, and unconvincing. > You can watch my mumbling in the recording I watched this, and the author was articulate and presented well. The author is too harsh! Good job for trying to push the boundaries.

cedws 2y ago

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

MortyWaves 2y ago
Totally agree with you. I wish npm did a better job of filtering the crap files out of packages.
Alifatisk 2y ago
At least, switch to pnpm minimize the bloat
- jefozabuss 2y ago
  I just installed a project with pnpm about 120 packages mostly react/webpack/eslint/redux related with prod env: 700MB without prod env: 900MB sadly the bloat cannot be avoided that well :/
  - jeffhuys 2y ago
    pnpm stores them in a central place and symlinks them. You’ll see the benefits when you have multiple projects with a lot of the same packages.
    - syncsynchalt 2y ago
      You'll also see the benefit when `rm -rf`ing a `node_modules` and re-installing, as pnpm still has a local copy that it can re-link after validating its integrity.
- MBCook 2y ago
  As someone who mostly works in Java it continues to floor me that this isn’t the default. Why does every project I work on need an identical copy of possibly hundreds of packages if they’re the same version? I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping. Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.

glenjamin 2y ago

This strikes me as something that could be done for the highest traffic packages at the backend, rather than be driven by the client at pubish-time.

fastest963 2y ago
The article talks about this. There are hashes that are generated for the tarball so the backend can't recompress anything.

deleted 2y ago

[deleted]

BonoboIO 2y ago

In summary: It’s a nice feature, which gives nice benefits for often downloaded packages, but nobody at npm cares for the bandwidth?

nikeee 2y ago

I don't see why it wouldn't be possible to hide behind a flag once Node.js supports zopfli natively. In case of CI/CD, it's totally feasible to just add a --strong-compression flag. In that case, the user expects it to take its time. TS releases a non-preview version every few months, so using 2.5 minutes for compression would work.

jopsen 2y ago
Think of the complexity involved, think of having to fix bugs because something went wrong. Such effort would be better spent preparing npm/node for a future where packages with a lower-bound npm version constraint can be compressed with zstd or similar. Every feature you add to a program is complexity, you need to really decide if you want it.

abound 2y ago

A few people have mentioned the environmental angle, but I'd care more about if/how much this slows down decompression on the client. Compressing React 20x slower once is one thing, but 50 million decompressions being even 1% slower is likely net more energy intensive, even accounting for the saved energy transmitting 4-5% fewer bits on the wire.

web007 2y ago
It's very likely zero or positive impact on the decompression side of things. Starting with smaller data means everything ends up smaller. It's the same decompression algorithm in all cases, so it's not some special / unoptimized branch of code. It's yielding the same data in the end, so writes equal out plus or minus disk queue fullness and power cycles. It's _maybe_ better for RAM and CPU because more data fits in cache, so less memory is used and the compute is idle less often. It's relatively easy to test decompression efficiency if you think CPU time is a good proxy for energy usage: go find something like React and test the decomp time of gzip -9 vs zopfli. Or even better, find something similar but much bigger so you can see the delta and it's not lost in rounding errors.
sltkr 2y ago
For formats like deflate, decompression time doesn't generally depend on compressed size. (zstd is similar, though memory use can depend on the compression level used). This means an optimization like this is virtually guaranteed to be a net positive on the receiving end, since you always save a bit of time/energy when downloading a smaller compressed file.
DannyBee 2y ago
I can speak to this - there is no meaningful decompression effect across an insane set of tested data at Google and elsewhere. Zopfli was invented prior to brotli Zopfli is easiest to think of as something that just tries harder than gzip to find matches and better encodings. Much harder. decompression speed is linear either way. It's easiest to think of decompression as a linear time vm executor[1], where the bytecoded instructions are basically go back <distance> bytes, output the next <length> bytes you see, then output character <c> (outputting literal data is the instruction <0,0,{character to output}>) Assuming you did not output a file larger than the original uncompressed file (why would you bother?), you will, worst case, process N bytes during decompression, where N is the size of the original input file. The practical decompression speed is driven by cache behavior, but it thrashes the cache no matter what. In practice, reduction of size vs gzip occurs by either finding larger runs, or encodings that are smaller than the existing ones. After all, if you want the compressed file to shrink, you need output less instructions somehow, or make more of the instructions identical (so they can be represented in less bits by later huffman coding). In practice, this has almost exclusively positive effects on decompression speed - either the vm has less things to process (which is faster), or more of the things it does look the same (which has better cache behavior). [1] this is one way archive formats will sometimes choose to deal with multiple compression method support - encode them all to the same kind of bytecode (usually some form of copy + literal instruction set), and then decoding is the same for all of them. ~all compression algorithms output some bytecode like the above on their own already, so it's not a lot of work. This doesn't help you support other archive formats, but if you want to have a bunch of per-file compression options that you pick from based on what works best, this enables you to still only have to have one decoder.

adgjlsfhk1 2y ago

This seems like a place where the more ambitious version that switches to ZSTD might have better tradeoffs. You would get similar or better compression, with faster decompression and recompression than zstd.It would lose backward compatibility though...

bufferoverflow 2y ago
Brotli and lzo1b have good compression ratios and pretty fast decompression speeds. Compression speed should not matter that much, since you only do it once. https://quixdb.github.io/squash-benchmark/ https://quixdb.github.io/squash-benchmark/ There even more obscure options: https://www.mattmahoney.net/dc/text.html https://www.mattmahoney.net/dc/text.html
vlovich123 2y ago
Not necessarily - could retain backward compat by publishing both gzip and zstd variants and having downloaders with newer npm’s prefer to download zstd. Over time, you could require packages only upload zstd going forward and either generate zstd versions of the backlog of unmaintained packages or at least those that see some amount of traffic over some time period if you’re willing to drop very old packages. The ability to install arbitrary versions of packages probably means you’re probably better off reprocessing the backlog although that may cost more than doing nothing. The package lock checksum is probably a more solvable issue with some coordination. The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).
MBCook 2y ago
Thats a much higher hurdle to jump. I don’t blame the author for trying this first. If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think. So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.

chuckadams 2y ago

Switching to a shared cache in the fashion of pnpm would eliminate far more redundant downloads than a compression algorithm needing 20x more CPU.

liontwist 2y ago

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And I would have quit at the first roadblock. What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line.

dr-detroit 2y ago
[dead]
coliveira 2y ago
The problem is that this guy is treating open source as if it was a company where you need to direct a project to completion. Nobody in open source wants to be told what to do. Just release your work, if it is useful, the community will pick it up and everybody will benefit. You cannot force your improvement into the whole group, even if it is beneficial in your opinion.
- deleted 2y ago
  [deleted]
- liontwist 2y ago
  > where you need to direct a project to completion Do you want to get a change in, or not? Is this a project working with the community or not? > Just release your work What motivation exists to optimize package formats if nobody uses that package format? There are no benefits unless it’s in mainline. > Nobody in open source wants to be told what to do He’s not telling anybody to do work. He is sharing an optimization with clear tradeoffs - not a new architecture. > You cannot force your improvement into the whole group Nope, but communication is key. “put up a PR and we will let you know whether it’s something we want to pursue”. Instead they put him through several levels of gatekeepers where each one gave him incorrect feedback. “Why do we want to optimize bandwidth” is a question they should have the answer to. If this PR showed up on my project I would say “I’m worried about X,Y,Z” we will set up a test for X and Y and get back to you. Could you look into Z?

liontwist 2y ago

Why would a more complex zip slow down decompress? This comment seems to misunderstand how these formats work. OP is right.

dvh 2y ago

Imagine being in the middle of nowhere, in winter, on Saturday night, on some farm, knee deep in a cow piss, servicing some 3rd party feed dispenser, only to discover that you have possible solution but it's in some obscure format instead of .tar.gz. Nearest internet 60 miles away. This is what I always imagine happening when some new obscure format come into play, imagine the poor fella, alone, cold, screaming. So incredibly close to his goal, but ultimately stopped by some artificial unnecessary made-up bullshit.

tehjoker 2y ago
I believe zopfli compression is backwards compatible with DEFLATE, it just uses more CPU during the compression phase.
- DannyBee 2y ago
  Correct - it is just looking for matches harder, and encoding harder.

PaulHoule 2y ago

Years back I came to the conclusion that conda using bzip2 for compression was a big mistake. Back then if you wanted to use a particular neural network it was meant for a certain version of Tensorflow which expected you to have a certain version of the CUDA libs. If you had to work with multiple models the "normal" way to do things was use the developer unfriendly [1][2] installers from NVIDIA to install a single version of the libs at a time. Turned out you could have many versions of CUDA installed as long as you kept them in different directories and set the library path accordingly, it made sense to pack them up for conda and install them together with everything else. But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time. If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression. [1] click... click... click... [2] like they're really going to do something useful with my email address

zahlman 2y ago
>But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time. For Paper, I'm planning to cache both the wheel archives (so that they're available without recompressing on demand) and unpacked versions (installing into new environments will generally use hard links to the unpacked cache, where possible). > If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression. FWIW, in my testing LZMA is a big win (and I'm sure zstd would be as well, but LZMA has standard library support already). But there are serious roadblocks to adopting a change like that in the Python ecosystem. This sort of idea puts them several layers deep in meta-discussion - see for example https://discuss.python.org/t/pep-777-how-to-re-invent-the-wheel/67484 https://discuss.python.org/t/pep-777-how-to-re-invent-the-wh... . In general, progress on Python packaging gets stuck in a double-bind: try to change too little and you won't get any buy-in that it's worthwhile, but try to change too much and everyone will freak out about backwards compatibility.
- PaulHoule 2y ago

kittikitti 2y ago

Thank you so much for posting this. The original logic was clear and it had me excited! I believe this is useful because compression is very common and although it might not fit perfectly in this scenario, it could very well be a breakthrough in another. If I come across a framework that could also benefit from this compression algorithm, I'll be sure to give you credit.

bhouston 2y ago

What about a different approach - an optional npm proxy that recompresses popular packages with 7z/etc in the background? Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready. Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packages. Main tradeoff is additional verification steps, but they could be optional given a digital signature approach. Curious if others see major security holes in this approach?

ndriscoll 2y ago
This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller than a .gz.
- cesarb 2y ago
  > hash a .tar file, not a .tar.gz file For security reasons, it's usually better to hash the compressed file, since it reduces the attack surface: the decompressor is not exposed to unverified data. There have already been vulnerabilities in decompressor implementations which can be exploited through malformed compressed data (and this includes IIRC at least one vulnerability in zlib, which is the standard decompressor for .gz).
  - bhouston 2y ago
    This suggests one should just upload a tar rather than a compressed file. Makes sense because one can scan the contents for malicious files without risking a decompressor bug. BTW npm decompressed all packages anyhow because it lets you view the contents these days on its website.
- bhouston 2y ago

hinkley 2y ago

Does npm even default to gzip -9? Wikipedia claims zopfli is 80 times slower under default settings.

zahlman 2y ago
My experience has been that past -6 or so, gzip files get only a tiny bit smaller in typical cases. (I think I've even seen them get bigger with -9.)
EvanHahn 2y ago
Yes: https://github.com/npm/pacote/blob/bf1f60f58bb61f053262f54724edcacaadb221ce/lib/util/tar-create-options.js#L12 https://github.com/npm/pacote/blob/bf1f60f58bb61f053262f5472...

jefozabuss 2y ago

I wonder what is the tarball size difference on average if you'd for example download everything in one tarball (full package list) instead of 1-by-1 as the gzip compression would work way better in that case. Also for bigger companies this is not really a "big" problem as they usually have in-house proxies (as you cannot rely on a 3rd party repository in CI/CD for multiple reasons (security, audit, speed, etc)).

Dylan16807 2y ago
You might save a little bit by putting similar very small files next to each other in the same tarball, but in general I would not expect significant improvements. Gzip can only compress repetitions that are within 32KB of each other.

snizovtsev 2y ago

Yes, but it was expected. It's like prioritising code readability over performance everywhere but the hot path. Earlier in my career, I managed to use Zopfli once to compress gigabytes of PNG assets into a fast in-memory database supporting a 50K+ RPS web page. We wanted to keep it simple and avoid the complexity of horizontal scaling, and it was OK to drop some rarely used images. So the more images we could pack into a single server, the more coverage we had. In that sense Zopfli was beneficial.

1337shadow 2y ago

Ok but why doesn't npm registry actually recompress the archives? It can even apply that retroactively, wouldn't require zopli in npm CLI

aseipp 2y ago
Hashes of the tarballs are recorded in the package-lock.json of downstream dependants, so recompressing the files in place will cause the hashes to change and break everyone. It has to be done at upload time.
- notpushkin 2y ago
  But it still can be done on the npm side, right?
- bhouston 2y ago
  The hashes of the uncompressed tarballs would be great. Then the HTTP connection can negotiate a compression format for transfer (which can change over time at HTTP itself changes) rather than baking it into the NPM package standard (which is incredibly inflexible.)

tonymet 2y ago

I’m more concerned about the 40gb of node_modules. Why hasn’t node supported tgz node_modules. That would save 75% of the space or more.

bhouston 2y ago
isn't this a file system thing? Why bake it into npm?
- tonymet 2y ago
  efficiency
  - bhouston 2y ago
    The os should do file deduplication and compression and decompression faster than npm. But I guess the issue is npm cannot give the os hints to compress a folder or not?
    - MBCook 2y ago
      Even if the file system does it that’s still per-file. You still need all the bookkeeping. And you’re not gonna get great compression if a lot of the files are tiny. The tar contains a whole bunch of related files that probably compress a lot better together, and it’s only one file on disk for the file system to keep track of. It’s gonna be a lot more efficient. When compiling you’re probably gonna touch all the source files anyway right? So if you load the zip into memory once and decompress it there it’s probably faster than loading each individual file and decompressing each of them, or just loading each individual file raw.
      - bhouston 2y ago
        > And you’re not gonna get great compression if a lot of the files are tiny. The future of compression is likely shared dictionaries, basically recognize the file type and then use a dictionary that is optimized for that file type. E.g. JavaScript/TypeScript or HTML or Rust or C++, etc. That can offset the problems with small files to a large degree. But again, this is a complex problem and it should be dealt with orthogonally to npm in my opinion. I also checked and it should be possible for npm to enable file system compression at the OS level on both Windows and MacOS. Also if it was dealt with orthogonally to npm, then it could be used by pip, and other package systems.

nopurpose 2y ago

It reminds me of an effort to improve docker image format and make it move away from being just a tar file. I can't find links anymore, but it was a pretty clever design, which still couldn't beat dumb tar in efficiency.

bhouston 2y ago
Transferring around dumb tar is actually smart because the HTTPS connection can negotiate a compressed version of it to transfer - e.g. gzip, brotli, etc. No need to bake in an unchangable compression format into the standard.

woadwarrior01 2y ago

From the RFC on github[1]. > Zopfli is written in C, which presents challenges. Unless it was added to Node core, the CLI would need to (1) rewrite Zopfli in JS, possibly impacting performance (2) rely on a native module, impacting reliability (3) rely on a WebAssembly module. All of these options add complexity. Wow! Who's going to tell them that V8 is written in C++? :) [1]: https://github.com/npm/rfcs/pull/595 https://github.com/npm/rfcs/pull/595

kmacdough 2y ago
It's not about C per-se, as much as each native compiled dependency creates additional maintenance concerns. Changes to hardware/OS can require a recompile or even fixes. NPM build system already requires a JavaScript runtime, so is already handled as part of existing maintenance. The point is that Zopfli either needs to be rewritten for a platform-agnostic abstraction they already support, or else Zopfli will be added to a list of native modules to maintain.
- woadwarrior01 2y ago
  > It's not about C per-se, as much as each native compiled dependency creates additional maintenance concerns. Changes to hardware/OS can require a recompile or even fixes. This is a canard. zopfli is written in portable C and is far more portable than the nodejs runtime. On any hardware/OS combo that one can build the nodejs runtime, they certainly can also build and run zopfli.

gweinberg 2y ago

I was under the impression that bzip compresses more than gzip, but gzip is much faster, so gzip is better for things that need to be compressed on the fly, and bzip is better fro things that get archived. Is this not true? Wouldn't it have been better to use bzip all along for this purpose?

bangaladore 2y ago

I think the main TLDR here [1]: > For example, I tried recompressing the latest version of the typescript package. GNU tar was able to completely compress the archive in about 1.2 seconds on my machine. Zopfli, with just 1 iteration, took 2.5 minutes. [1] https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 My question of course would be, what about LZ4, or Zstd or Brotli? Or is backwards compatibility strictly necessary? I understand that GZIP is still a good compressor, so those others may not produce meaningful gains. But, as the author suggests, even small gains can produce huge results in bandwidth reduction.

loeg 2y ago

It probably makes more sense to save more bytes and compressor time and just switch to zstd (a bigger scoped effort, sure).

imoverclocked 2y ago

> But the cons were substantial: ... > This wouldn’t retroactively apply to existing packages. Why is this substantial? My understanding is that packages shouldn't be touched once published. It seems likely for any change to not apply retroactively.

szundi 2y ago

This is a nice guy

omoikane 2y ago

> Integrating Zopfli into the npm CLI would be difficult. Is it possible to modify "gzip -9" or zlib to invoke zopfli? This way everyone who wants to compress better will get the extra compression automatically, in addition to npm. There will be an increase in compression time, but since "gzip -9" is not the default, people preferring compression speed might not be affected.

bombcar 2y ago
You'd have more problems here, but you could do it - if you let it take ages and ages to percolate though all environments. It's been almost 30 years since bzip2 was released and even now not everything can handle tar.bz2
- arccy 2y ago
  probably because bzip2 isn't a very good format

tehjoker 2y ago

5% improvement is basically the minimum I usually consider worthwhile to pursue, but it's still small. Once you get to 10% or 20%, things become much more attractive. I can see how people can go either way on a 5% increase if there are any negative consequences (such as increased build time).

Alifatisk 2y ago

I wonder if there is a way to install the npm packages without the crap they come included with (like docs, tests, readme etc).

rafaelmn 2y ago

I wonder if you could get better results if you built a dictionary over entire npm. I suspect most common words could easily be reduced to 16k word index. Would be much faster, dictionary would probably fit in cache, can even optimize it in memory for cache prefetch.

zahlman 2y ago
This seems like a non-starter to me - new packages are added to npm all the time, and will alter the word frequency distribution. If you aren't prepared to re-build constantly and accept that the dictionary isn't optimal, then it's hard to imagine it being significantly better than what you build with a more naive approach. Basically - why try to fine-tune to a moving target?
- arccy 2y ago
  would it really change that quickly? you might get significant savings from just having keywords, common variable names, standard library functions
- rafaelmn 2y ago
  But is it really moving that fast ? I suspect most fundamental terms in programming and the variations do not change often. You will always have keywords, built ins and the most popular concepts from libs/frameworks. So it is basically downloading a few hundred kb dictionary every year ?
  - MBCook 2y ago
    You’d have language keywords and such, yeah. But past that the most common words in a React project are going to be very different from a Vue project right?

hinkley 2y ago

Pulling on this thread, there are a few people who have looked at the ways zopfli is inefficient. Including this guy who forked it, and tried to contribute a couple improvements back to master: https://github.com/fhanau/Efficient-Compression-Tool https://github.com/fhanau/Efficient-Compression-Tool These days if you’re going to iterate on a solution you’d better make it multithreaded. We have laptops where sequential code uses 8% of the available cpu.

vardump 2y ago
> These days if you’re going to iterate on a solution you’d better make it multithreaded. Repetition eliminating compression tends to be inherently sequential. You'd probably need to change the file format to support chunks (or multiple streams) to do so. Because of LZ back references, you can't LZ compress different chunks separately on different cores and have only one compression stream. Statistics acquisition (histograms) and entropy coding could be parallel I guess. (Not a compression guru, so take above with a pinch of salt.)
- hinkley 2y ago
  There are gzip variants that break the file into blocks and run in parallel. They lose a couple of % by truncating the available history. But zopfli appears to do a lot of backtracking to find the best permutations for matching runs that have several different solutions. There’s a couple of ways you could run those in parallel. Some with a lot of coordination overhead, others with a lot of redundant calculation.

frabjoused 2y ago

This reminds me of a time I lost an argument with John-David Dalton about cleaning up/minifying lodash as an npm dependency, because when including the readme and license for every sub-library, a lodash import came to ~2.5MB at the time. This also took a lot of seeking time for disks because there were so many individual files. The conversation started and ended at the word cache.

MBCook 2y ago
> This also took a lot of seeking time for disks because there were so many individual files. The fact NPM keeps things in node_modules unzipped seems wild to me. Filesystems are not great at hundreds of thousands of little files. Some are bad, others are terrible. Zip files are easier to store, take up less space, and CPUs are faster than disks so the decompression in memory is probably faster reading the unzipped files. That was one of my favorite features of Yarn when I tried it - pnp mode. But since it’s not what NPM does it requires a shim that doesn’t work with all ps mage’s. Or at least didn’t a few years ago.

zahlman 2y ago

I'd love to see an effort like like this succeed in the Python ecosystem. Right now, PyPI is dependent upon Fastly to serve files, on the order of >1 petabyte per day. That's a truly massive in-kind donation, compared to the PSF's operating budget (only a few million dollars per year - far smaller than Linux or Mozilla).

cozzyd 2y ago
No problem, I'm sure if Fastly stopped doing it JiaTanCo would step up

JoeAltmaier 2y ago

These days technology moves so fast it's hard to keep up. The slowest link in the system is the human being. That's a strong argument that 'if it isn't broke, don't fix it." LOts of numbers being thrown around, you add up tiny things enough times you can get a big number. But is npm package download the thing that's tanking the internet? No? Then this is a second- or thirt-order optimization.

luzifer42 2y ago

I created once a maven plugin to recompress Java artefacts with zopfli. I rewrote it in Java and runs entirely in the JVM. This means, the speed is worse and may contain bugs: https://luccappellaro.github.io/2015/03/01/ZopfliMaven.html https://luccappellaro.github.io/2015/03/01/ZopfliMaven.html

Julien_r2 2y ago

A pro could have been an extra narrative about carbon footprint savings. I'm surprised it hasn't been raised when talking about saving 2Tb/year only for React. It represents costs, which doesn't seem to be an issue, but also computing power and storage. (Event with a higher/longer computing power due to slower compression, it's done once per version, which isn't really comprable to the amount of downloads anyway) Hard to calculate the exact saving, but it would represent a smaller CO2 footprint..

wallunit 2y ago

I used zopflipng in the past to optimize PNG images. It made sense since there was no better alternative to store lossless image data than the PNG format at the given time in the given environment. Zopfli is awesome when you are locked in on deflate compression. I feel like if the npm folks would want to optimize for smaller package size a better strategy would be switching to some more effective text compression (e.g. bzip2, xz). That would result into a larger file size reduction than 5% for a smaller CPU time increase compared to Zopfli. You would need to come up with some migration strategy though as this change isn't per-se backwards compatible, but that seems manageable if you are in control of the tooling and infrastructure.

vivzkestrel 2y ago

I ll give you an even better idea but it ll need atleast a 100 volunteers, maybe a 1000. Take each package and rewrite it without external dependencies. That will cut tech debt for that package significantly. Just like how we have a @types/xyz where some dude named DefinitelyTyped is busy making typescript packages for everything, lets make a namespace like @efficient/cors @efficient/jsdom @efficient/jest etc and eliminate all external dependencies completely for every library on npm

DannyBee 2y ago

zopfli is the wrong thing to use here. If you want an example of where these things go badly: the standard compression level for rpms on redhat distros is zstd level 19. This has all the downsides of other algorithms - it's super slow - often 75-100x slower than the default level of zstd[1]. It achieves a few percent more compression for that speed. compared to even level 10, it's like 0-1% higher compression, but 20x slower. This is bad enough - the kicker is that at this level, it's slower than xz for ~all cases, and xz is 10% smaller. The only reason to use zstd is because you want fairly good compression, but fast. So here, they've chosen to use it in a way that compresses really slowly, but gives you none of the benefit of compressing really slowly. Now, unlike the npm case, there was no good reason to choose level 19 - there were no backwards compatibility constraints driving it, etc. I went through the PR history on this change, it was not particularly illuminating (it does not seem lots of thought was given to the level choice). I mention all this because it has a real effect on the experience of building rpms - this is why it takes eons to make kernel debuginfo rpms on fedora. Or any large RPM. Almost all time is spent compressing it with zstd at level 19. On my computer this takes many minutes. If you switch it to use even xz, it will do it about 15-20x faster (single threaded. if you thread both of them, xz will win by even more, because of how slow the setting is for zstd. If you use reasonable settings for zstd, obviously, it achieves gigabytes/second in parallel mode) Using zopfli would be like choosing level 19 zstd for npm. While backwards compatibility is certainly painful to deal with here, zopfli is not likely better than doing nothing. You will make certain cases just insanely slow. You will save someone's bandwidth, but in exchange you will burn insane amounts of developer CPU. zopfli is worse than level 19, it can often be 100x-200x slower than gzip -9. Doesn't npm support insanely relaxed/etc scripting hooks anyway? If so, if backwards compatibility is your main constraint, you would be "better off" double compressing (IE embed xz or whatever + a bootstrap decompressor and using the hooks to decompress it on old versions of npm). Or just shipping .tar.gz's than, when run, fetch the .xz and decompress it on older npm. Or you know, fish in the right pond - you would almost certainly achieve much higher reductions by enforcing cleaner shipping packages (IE not including random garbage, etc) than by compressing the garbage more. [1] on my computer, single threaded level 19 does 7meg/second, the default does 500meg/second. Level 10 does about 130meg/second.

Someone 2y ago

kavenkanum 2y ago

If OP wanted to shrink nom packages then npm could introduce two types of npm package - the build package and source one. This way a lot of packages would be smaller, because package code could be safely distributed through a separate package and not kept in the build package. There are a lot npm packages that explicitly include whole git repo in the package and they do so because there's only one type of package they can use

franga2000 2y ago

We wouldn't have to worry about over-the-wire package size if the modern DevOps approach wasn't "nuke everything, download from the Internet" every build. Back in my Java days, most even small-time dev shops had a local Maven registry that would pass through and cache the big ones. A CI job, even if the "container" was nuked before each build, would create maybe a few kilobytes of Internet traffic, possibly none at all. Now your average CI job spins up a fresh VM or container, pulls a Docker base image, apt installs a bunch of system dependencies, pip/npm/... installs a bunch of project dependencies, packages things up and pushes the image to the Docker registry. No Docker layer caching because it's fresh VM, no package manager caching because it's a fresh container, no object caching because...you get the idea.... Even if we accept that the benefits of the "clean slate every time" approach outweigh the gross inefficiency, why aren't we at least doing basic HTTP caching? I guess ingress is cheap and the egress on the other side is "someone else's money".

Terr_ 2y ago
Yeah, I would also note that in addition to speed/transfer-costs, having an organizational package proxy is useful for reproducibility and security.
robertlagrant 2y ago
Lots of places use a cache like Artifactory so they don't get slammed with costs, and are resilient to network outages and dependency builds vanishing.
alimbada 2y ago
After reading the article, this comment and the comment thread further down on pnpm[1], it feels to me like the NPM team are doing everyone a disservice by ignoring the inefficiencies in the packaging system. It may not be deliberate or malicious but they could easily have provided better solutions than the one proposed in the article which, in my opinion is a band-aid solution at best. The real fix would be to implement what you mention here: local registry and caching, and/or symlinking a la pnpm. [1] https://news.ycombinator.com/item?id=42841658 https://news.ycombinator.com/item?id=42841658
2y ago

animuchan 2y ago

This presents a great opportunity for alternative npm clients (such as bun, yarn). An almost free advantage over the mainline npm CLI.

jokethrowaway 2y ago

Nice exercise in bureaucracy. - I don't see the problem with adding a C dependency to a node project, native modules are one of the nicest things about node.js - Longer publishing time for huge packages (eg. typescript) is a bigger problem but I don't think it should impact a default. Just give the option to use gzip for the few outliers.