3 ms·

What about a different approach - an optional npm proxy that recompresses popular packages with 7z/etc in the background? Could verify package integrity by has

What about a different approach - an optional npm proxy that recompresses popular packages with 7z/etc in the background? Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready. Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packages. Main tradeoff is additional verification steps, but they could be optional given a digital signature approach. Curious if others see major security holes in this approach?

ndriscoll 2y ago
This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller than a .gz.
- cesarb 2y ago
  > hash a .tar file, not a .tar.gz file For security reasons, it's usually better to hash the compressed file, since it reduces the attack surface: the decompressor is not exposed to unverified data. There have already been vulnerabilities in decompressor implementations which can be exploited through malformed compressed data (and this includes IIRC at least one vulnerability in zlib, which is the standard decompressor for .gz).
  - bhouston 2y ago
    This suggests one should just upload a tar rather than a compressed file. Makes sense because one can scan the contents for malicious files without risking a decompressor bug. BTW npm decompressed all packages anyhow because it lets you view the contents these days on its website.
- bhouston 2y ago
  You are correct. They should be uploading and downloading dumb tar files and let the HTTP connection negotiate the compression method. All hashes should be based on the uncompressed raw tar dump. This would be proper separation of concerns.
- cozzyd 2y ago
  Enjoy zipbomb.js
  - bhouston 2y ago
    But npm already decompresses every package because it shows the contents on its website. So yeah it can be malicious but it already has dealt with that risk.

ndriscoll 2y ago

This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller than a .gz.

cesarb 2y ago
> hash a .tar file, not a .tar.gz file For security reasons, it's usually better to hash the compressed file, since it reduces the attack surface: the decompressor is not exposed to unverified data. There have already been vulnerabilities in decompressor implementations which can be exploited through malformed compressed data (and this includes IIRC at least one vulnerability in zlib, which is the standard decompressor for .gz).
- bhouston 2y ago
  This suggests one should just upload a tar rather than a compressed file. Makes sense because one can scan the contents for malicious files without risking a decompressor bug. BTW npm decompressed all packages anyhow because it lets you view the contents these days on its website.
bhouston 2y ago
You are correct. They should be uploading and downloading dumb tar files and let the HTTP connection negotiate the compression method. All hashes should be based on the uncompressed raw tar dump. This would be proper separation of concerns.
cozzyd 2y ago
Enjoy zipbomb.js
- bhouston 2y ago
  But npm already decompresses every package because it shows the contents on its website. So yeah it can be malicious but it already has dealt with that risk.