5 ms·

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find

by sd9 2y ago

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't closed the RFC themselves. It could have been the sort of thing that eventually happens after 2 years, but then quietly makes everybody's lives better.

n4r9 2y ago
I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.
- IshKebab 2y ago
  > The proposal wasn't rejected! They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider <extremely unlikely issue>? Please go and run more tests." It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares if it takes an extra 2 minutes to publish Typescript. Kind of a wild decision considering NPM uses 4.5 PB of traffic per week. 5% of that is 225 TB/week, which according to my brief checks costs around $10k/week! I guess this is a "not my money" problem fundamentally.
  - lyu07282 2y ago
    > which according to my brief checks costs around $10k/week That's the market price though, for Microsoft its a tiny fraction of that.
  - johnfn 2y ago
    This doesn't seem quite correct to me. They weren't asking for "more validation than was reasonable". They were asking for literally any proof that users would benefit from the proposal. That seems like an entirely reasonable thing to ask before changing the way every single NPM package gets published, ever. I do agree that 10k/week is non-negligible. Perhaps that means the people responsible for the 10k weren't in the room?
  - bombcar 2y ago
    Or another way to look at it is it's just (at most!) 5% off an already large bill, and it might cost more than that elsewhere. And I can buy 225 TB of bandwidth for less than $2k, I assume Microsoft can get better than some HN idiot buying Linode.
    - IshKebab 2y ago
      > And I can buy 225 TB of bandwidth for less than $2k Even so, $2k a week is at least one competent FTE.
  - arccy 2y ago
    massively increase the open source github actions bill for runners running longer (compute is generally more expensive) to publish for a small decrease in network traffic (bandwidth is cheap at scale)?
deleted 2y ago
[deleted]
alt227 2y ago
I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.
- scott_w 2y ago
  I agree, going from 1 second to 2.5 minutes is a huge negative change, in my opinion. I know publishing a package isn't something you do 10x a day but it's probably a big enough change that, were I doing it, I'd think the publish process is hanging and keep retrying it.
  - pletnes 2y ago
    If you’re working on the build process itself, you’ll notice it a lot!
- rererereferred 2y ago
  Since it's backwards compatible, individual maintainers could enable it in their own pipeline if they don't have issues with the slowdown. It sounds like it could be a single flag in the publish command.
- michaelmior 2y ago
  Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.
  - Null-Set 2y ago
    No, it can't because the checksums won't match.
    - michaelmior 2y ago
      I don't think that's actually a problem, but it would require continuing to host both versions (at distinct URLs) for any users who may have installed the package before the Zopfli-compressed version completed. Although I think you could also get around this by tracking whether the newly-released package was ever served by the API. If not, which is probably the common case, the old gzip-compressed version could be deleted.
macspoofing 2y ago
> I don't find the cons all that compelling to be honest I found it reasonable. The 5% improvement was balanced against the cons of increased cli complexity, lack of native JS zopfli implementation, and slower compression .. and 5% just wasn't worth it at the moment - and I agree. >or at least I think they warrant further discussion I think that was the final statement.
- sd9 2y ago
  Yes, but there’s a difference between “this warrants further discussion” and “this warrants further discussion and I’m closing the RFC”. The latter all but guarantees that no further discussion will take place.
  - philipwhiuk 2y ago
    No it doesn't. It only does that if you think discussion around future improvements belongs in RFCs.
    - mcherm 2y ago
      Where DOES it belong, if not there?
jerf 2y ago
"I don't find the cons all that compelling to be honest" This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm. You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to make any changes, until you experience your very own "oh man, I really thought this change was perfectly safe and now my entire customer base is trashed" moment, and then suddenly things like "hey, we need to consider how this affects old signatures and the speed of decompression and just generally whether this is worth the non-zero risks for what are in the end not really that substantial benefits". I do not say this as the wise Zen guru sitting cross-legged and meditating from a position of being above it all; I say it looking at my own battle scars from the Perfectly Safe things I've pushed out to my customer base, only to discover some tiny little nit caused me trouble. Fortunately I haven't caused any true catastrophes, but that's as much luck as skill. Attaining the proper balance between moving forward even though it incurs risk and just not changing things that are working is the hardest part of being a software maintainer, because both extremes are definitely bad. Everyone tends to start out in the former situation, but then when they are inevitably bitten it is important not to overcorrect into terrified fear of ever changing anything.
- sd9 2y ago
  I agree with everything you said, but it doesn’t contradict my point
  - jerf 2y ago
    I'm saying you probably don't find them compelling because from your point of view, the problems don't look important to you. They don't from my point of view either. But my point of view is the wrong point of view. From their point of view this would be plenty to make me think twice and several times over past that from changing something so deeply fundamental to the system for what is a benefit that nobody who is actually paying the price for the package size seems to be particularly enthusiastic about. If the people paying the bandwidth bill aren't even that excited about a 5% reduction, then the cost/benefits analysis tips over into essentially "zero benefit, non-zero cost", and that's not very compelling.
advisedwang 2y ago
The pros aren't all that compelling either. The npm repo is the only group that this would really be remotely significant for, and there seemed to be no interest. So it doesn't take much of a con to nix a solution to a non-problem.
- ForOldHack 2y ago
  Every single download, until the end of time is affected: It speeds up the servers, speeds up the updates, saves disk space on the update servers, and saves on bandwidth costs and usage. Everyone benefits, the only cost is a ultra microscopic time on the front end, and a tiny cost on the client end, and for a very significant number of users, time and money saved. The examples of compression here...
  - pclmulqdq 2y ago
    Plus a few years of a compression expert writing a JS implementation of what was likely some very cursed C. And someone auditing its security. And someone maintaining it.
mootothemax 2y ago
> I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion It needs a novel JS port of a C compresison library, which will be wired into a heavily-used and public-facing toolchain, and is something that will ruin a significant number of peoples' days if it breaks. For me, that kind of ask needs a compelling use case from the start.

by sd9 2y ago

n4r9 2y ago

I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.

IshKebab 2y ago
> The proposal wasn't rejected! They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider <extremely unlikely issue>? Please go and run more tests." It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares if it takes an extra 2 minutes to publish Typescript. Kind of a wild decision considering NPM uses 4.5 PB of traffic per week. 5% of that is 225 TB/week, which according to my brief checks costs around $10k/week! I guess this is a "not my money" problem fundamentally.
- lyu07282 2y ago
  > which according to my brief checks costs around $10k/week That's the market price though, for Microsoft its a tiny fraction of that.
- johnfn 2y ago
  This doesn't seem quite correct to me. They weren't asking for "more validation than was reasonable". They were asking for literally any proof that users would benefit from the proposal. That seems like an entirely reasonable thing to ask before changing the way every single NPM package gets published, ever. I do agree that 10k/week is non-negligible. Perhaps that means the people responsible for the 10k weren't in the room?
- bombcar 2y ago
  Or another way to look at it is it's just (at most!) 5% off an already large bill, and it might cost more than that elsewhere. And I can buy 225 TB of bandwidth for less than $2k, I assume Microsoft can get better than some HN idiot buying Linode.
  - IshKebab 2y ago
    > And I can buy 225 TB of bandwidth for less than $2k Even so, $2k a week is at least one competent FTE.
- arccy 2y ago
  massively increase the open source github actions bill for runners running longer (compute is generally more expensive) to publish for a small decrease in network traffic (bandwidth is cheap at scale)?

deleted 2y ago

[deleted]

alt227 2y ago

I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.

scott_w 2y ago
I agree, going from 1 second to 2.5 minutes is a huge negative change, in my opinion. I know publishing a package isn't something you do 10x a day but it's probably a big enough change that, were I doing it, I'd think the publish process is hanging and keep retrying it.
- pletnes 2y ago
  If you’re working on the build process itself, you’ll notice it a lot!
rererereferred 2y ago
Since it's backwards compatible, individual maintainers could enable it in their own pipeline if they don't have issues with the slowdown. It sounds like it could be a single flag in the publish command.
michaelmior 2y ago
Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.
- Null-Set 2y ago
  No, it can't because the checksums won't match.
  - michaelmior 2y ago
    I don't think that's actually a problem, but it would require continuing to host both versions (at distinct URLs) for any users who may have installed the package before the Zopfli-compressed version completed. Although I think you could also get around this by tracking whether the newly-released package was ever served by the API. If not, which is probably the common case, the old gzip-compressed version could be deleted.

macspoofing 2y ago

> I don't find the cons all that compelling to be honest I found it reasonable. The 5% improvement was balanced against the cons of increased cli complexity, lack of native JS zopfli implementation, and slower compression .. and 5% just wasn't worth it at the moment - and I agree. >or at least I think they warrant further discussion I think that was the final statement.

sd9 2y ago
Yes, but there’s a difference between “this warrants further discussion” and “this warrants further discussion and I’m closing the RFC”. The latter all but guarantees that no further discussion will take place.
- philipwhiuk 2y ago
  No it doesn't. It only does that if you think discussion around future improvements belongs in RFCs.
  - mcherm 2y ago
    Where DOES it belong, if not there?

jerf 2y ago

"I don't find the cons all that compelling to be honest" This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm. You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to make any changes, until you experience your very own "oh man, I really thought this change was perfectly safe and now my entire customer base is trashed" moment, and then suddenly things like "hey, we need to consider how this affects old signatures and the speed of decompression and just generally whether this is worth the non-zero risks for what are in the end not really that substantial benefits". I do not say this as the wise Zen guru sitting cross-legged and meditating from a position of being above it all; I say it looking at my own battle scars from the Perfectly Safe things I've pushed out to my customer base, only to discover some tiny little nit caused me trouble. Fortunately I haven't caused any true catastrophes, but that's as much luck as skill. Attaining the proper balance between moving forward even though it incurs risk and just not changing things that are working is the hardest part of being a software maintainer, because both extremes are definitely bad. Everyone tends to start out in the former situation, but then when they are inevitably bitten it is important not to overcorrect into terrified fear of ever changing anything.

sd9 2y ago
I agree with everything you said, but it doesn’t contradict my point
- jerf 2y ago
  I'm saying you probably don't find them compelling because from your point of view, the problems don't look important to you. They don't from my point of view either. But my point of view is the wrong point of view. From their point of view this would be plenty to make me think twice and several times over past that from changing something so deeply fundamental to the system for what is a benefit that nobody who is actually paying the price for the package size seems to be particularly enthusiastic about. If the people paying the bandwidth bill aren't even that excited about a 5% reduction, then the cost/benefits analysis tips over into essentially "zero benefit, non-zero cost", and that's not very compelling.

advisedwang 2y ago

The pros aren't all that compelling either. The npm repo is the only group that this would really be remotely significant for, and there seemed to be no interest. So it doesn't take much of a con to nix a solution to a non-problem.

ForOldHack 2y ago
Every single download, until the end of time is affected: It speeds up the servers, speeds up the updates, saves disk space on the update servers, and saves on bandwidth costs and usage. Everyone benefits, the only cost is a ultra microscopic time on the front end, and a tiny cost on the client end, and for a very significant number of users, time and money saved. The examples of compression here...
- pclmulqdq 2y ago
  Plus a few years of a compression expert writing a JS implementation of what was likely some very cursed C. And someone auditing its security. And someone maintaining it.

mootothemax 2y ago

> I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion It needs a novel JS port of a C compresison library, which will be wired into a heavily-used and public-facing toolchain, and is something that will ruin a significant number of peoples' days if it breaks. For me, that kind of ask needs a compelling use case from the start.