4 ms·

> This is a solid example of how things change at scale. 5% is 5% at any scale.

by pif 2y ago

michaelmior 2y ago
Yes and no. If I'm paying $5 a month for storage, I probably don't care about saving 5% of my storage costs. If I'm paying $50,000/month in storage costs, 5% savings is a lot more worthwhile to pursue
- PaulHoule 2y ago
  Doesn't npm belong to Microsoft? It must be hosted in Azure which they own so they must be paying a rock bottom rate for storage, bandwidth, everything.
  - cwmma 2y ago
    It's probably less about MS and more about the people downloading the packages
    - PaulHoule 2y ago
      For them it is 5% of something tiny.
      - sneak 2y ago
        The amount of modules my docker hosts download from npm is anything but tiny.
      - imoverclocked 2y ago
        Maybe, maybe not. If you are on a bandwidth limited connection and you have a bunch of NPM packages to install, 5% of an hour is a few minutes saved. It's likely more than that because long-transfers often need to be restarted.
        PaulHoule 2y ago
        A properly working cache and download manager that supports resume goes a long way. I could never get Docker to work on my ADSL when it was 2 Mbps (FTTN got it up to 20) though it was fine in the Montreal office which had gigabit.
gregmac 2y ago
5% off your next lunch and 5% off your next car are very much not the same thing.
- dgfitz 2y ago
  So what, instead of 50k for a car you spend 47.5k? If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it. 5% is 5%.
  - post-it 2y ago
    I wouldn't pick 5¢ up off the ground but I would certainly pick up $2500.
    - ggm 2y ago
      You'd keep 5c. A significant number of people who find sums up around $2500 give it back unconditionally, with no expectation of reward. Whoever lost $2500 is having a really bad day.
  - ziddoap 2y ago
    Why do so many people take illustrative examples literally? I'm sure you can use your imagination to substitute "lunch" and "car" with other examples where the absolute change makes a difference despite the percent change being the same. Even taking it literally... The 5% might not tip the scale of whether or not I can purchase the car, but I'll spend a few hours of my time comparing prices at different dealers to save $2500. Most people would consider it dumb if you didn't shop around when making a large purchase. On the other hand, I'm not going to spend a few hours of my time at lunch so that I can save an extra $1 on a meal.
  - kemitche 2y ago
    If it takes 1 hour of effort to save 5%: - Doing 1 hour of effort to save 5% on your $20 lunch is foolhardy for most people. $1/hr is well below US minimum wage. - Doing 1 hour of effort to save 5% on your $50k car is wise. $2500/hr is well above what most people are making at work. It's not about whether the $2500 affects my ability to buy the car. It's about whether the time it takes me to save that 5% ends up being worthwhile to me given the actual amount saved. The question is really "given the person-hours it takes to apply the savings, and the real value of the savings, is the savings worth the person-hours spent?"
horsawlarway 2y ago
5% of newly published packages, with a potentially serious degradation to package publish times for those who have to do that step. Given his numbers, let's say he saves 100Tb of bandwidth over a year. At AWS egress pricing... that's $5,000 total saved. And arguably - NPM is getting at least some of that savings by adding CPU costs to publishers at package time. Feels like... not enough to warrant a risky ecosystem change to me.
- AlotOfReading 2y ago
  How often are individuals publishing to NPM? Once a day at most, more typically once a week or month? A few dozen seconds of one person's day every month isn't a terrible trade-off. Even that's addressable though if there's motivation, since something like transcoding server side during publication just for popular packages would probably get 80% of the benefit with no client-side increase in publication time.
- true_religion 2y ago
  https://www.reddit.com/r/webdev/comments/1ff3ps5/these_5000_npm_packages_consume_45_pb_of_traffic/ https://www.reddit.com/r/webdev/comments/1ff3ps5/these_5000_... NPM uses at least 5 petabytes per week. 5% of that is 250 terabytes. So $15,000 a week, or $780,000 a year in savings could’ve been gained.
  - canucker2016 2y ago
    In a great example of the Pareto Principle (80/20), or actually even more extreme, let's only apply this Zopfli optimization if the package download total is equal or more than 1GiB (from the Weekly Traffic in GiB column of the Top 5000 Weekly by Traffic tab of the Google Sheets file from the reddit post). For reference, total bandwidth used by all 5000 packages is 4_752_397 GiB. Packages >= 1GiB bandwidth/week - That turns out to be 437 packages (there's a header row, so it's rows 2-438) which uses 4_205_510 GiB. So 88% of the top 5000 bandwidth is consumed by downloading the top 8.7% (437) packages. 5% is about 210 TiB. Limiting to the top 100 packages by bandwidth results in 3_217_584 GiB, which is 68% of total bandwidth used by 2% of the total packages. 5% is about 161 TiB.
knighthack 2y ago
Do you even know how absolute numbers work vis-à-vis percentages?
Aicy 2y ago
That's right, and 5% of a very small number is a very small number. 5% of a very big number is a big number.
syncsynchalt 2y ago
In some scenarios the equation flips, and the enterprise is looking for _more_ scale. The more bandwidth that Cloudflare needs, the more leverage they have at the peering table. As GitHub's largest repo (the @types / DefinitelyTyped repo owned by Microsoft) gets larger, the more experience the owner of GitHub (also Microsoft) gets in hosting the world's largest git repos. I would say this qualifies as one of those cases, as npmjs is hosted on Azure. The more resources that NPM needs, the more Microsoft can build towards parity with AWS's footprint.
deleted 2y ago
[deleted]

> This is a solid example of how things change at scale. 5% is 5% at any scale.

by pif 2y ago

> This is a solid example of how things change at scale. 5% is 5% at any scale.

Yes and no. If I'm paying $5 a month for storage, I probably don't care about saving 5% of my storage costs. If I'm paying $50,000/month in storage costs, 5% savings is a lot more worthwhile to pursue

PaulHoule 2y ago
Doesn't npm belong to Microsoft? It must be hosted in Azure which they own so they must be paying a rock bottom rate for storage, bandwidth, everything.
- cwmma 2y ago
  It's probably less about MS and more about the people downloading the packages
  - PaulHoule 2y ago
    For them it is 5% of something tiny.
    - sneak 2y ago
      The amount of modules my docker hosts download from npm is anything but tiny.
    - imoverclocked 2y ago
      Maybe, maybe not. If you are on a bandwidth limited connection and you have a bunch of NPM packages to install, 5% of an hour is a few minutes saved. It's likely more than that because long-transfers often need to be restarted.
      - PaulHoule 2y ago
        A properly working cache and download manager that supports resume goes a long way. I could never get Docker to work on my ADSL when it was 2 Mbps (FTTN got it up to 20) though it was fine in the Montreal office which had gigabit.

gregmac 2y ago

5% off your next lunch and 5% off your next car are very much not the same thing.

dgfitz 2y ago
So what, instead of 50k for a car you spend 47.5k? If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it. 5% is 5%.
- post-it 2y ago
  I wouldn't pick 5¢ up off the ground but I would certainly pick up $2500.
  - ggm 2y ago
    You'd keep 5c. A significant number of people who find sums up around $2500 give it back unconditionally, with no expectation of reward. Whoever lost $2500 is having a really bad day.
- ziddoap 2y ago
  Why do so many people take illustrative examples literally? I'm sure you can use your imagination to substitute "lunch" and "car" with other examples where the absolute change makes a difference despite the percent change being the same. Even taking it literally... The 5% might not tip the scale of whether or not I can purchase the car, but I'll spend a few hours of my time comparing prices at different dealers to save $2500. Most people would consider it dumb if you didn't shop around when making a large purchase. On the other hand, I'm not going to spend a few hours of my time at lunch so that I can save an extra $1 on a meal.
- kemitche 2y ago
  If it takes 1 hour of effort to save 5%: - Doing 1 hour of effort to save 5% on your $20 lunch is foolhardy for most people. $1/hr is well below US minimum wage. - Doing 1 hour of effort to save 5% on your $50k car is wise. $2500/hr is well above what most people are making at work. It's not about whether the $2500 affects my ability to buy the car. It's about whether the time it takes me to save that 5% ends up being worthwhile to me given the actual amount saved. The question is really "given the person-hours it takes to apply the savings, and the real value of the savings, is the savings worth the person-hours spent?"

horsawlarway 2y ago

5% of newly published packages, with a potentially serious degradation to package publish times for those who have to do that step. Given his numbers, let's say he saves 100Tb of bandwidth over a year. At AWS egress pricing... that's $5,000 total saved. And arguably - NPM is getting at least some of that savings by adding CPU costs to publishers at package time. Feels like... not enough to warrant a risky ecosystem change to me.

AlotOfReading 2y ago
How often are individuals publishing to NPM? Once a day at most, more typically once a week or month? A few dozen seconds of one person's day every month isn't a terrible trade-off. Even that's addressable though if there's motivation, since something like transcoding server side during publication just for popular packages would probably get 80% of the benefit with no client-side increase in publication time.
true_religion 2y ago
https://www.reddit.com/r/webdev/comments/1ff3ps5/these_5000_npm_packages_consume_45_pb_of_traffic/ https://www.reddit.com/r/webdev/comments/1ff3ps5/these_5000_... NPM uses at least 5 petabytes per week. 5% of that is 250 terabytes. So $15,000 a week, or $780,000 a year in savings could’ve been gained.
- canucker2016 2y ago
  In a great example of the Pareto Principle (80/20), or actually even more extreme, let's only apply this Zopfli optimization if the package download total is equal or more than 1GiB (from the Weekly Traffic in GiB column of the Top 5000 Weekly by Traffic tab of the Google Sheets file from the reddit post). For reference, total bandwidth used by all 5000 packages is 4_752_397 GiB. Packages >= 1GiB bandwidth/week - That turns out to be 437 packages (there's a header row, so it's rows 2-438) which uses 4_205_510 GiB. So 88% of the top 5000 bandwidth is consumed by downloading the top 8.7% (437) packages. 5% is about 210 TiB. Limiting to the top 100 packages by bandwidth results in 3_217_584 GiB, which is 68% of total bandwidth used by 2% of the total packages. 5% is about 161 TiB.

knighthack 2y ago

Do you even know how absolute numbers work vis-à-vis percentages?

Aicy 2y ago

That's right, and 5% of a very small number is a very small number. 5% of a very big number is a big number.

syncsynchalt 2y ago

In some scenarios the equation flips, and the enterprise is looking for _more_ scale. The more bandwidth that Cloudflare needs, the more leverage they have at the peering table. As GitHub's largest repo (the @types / DefinitelyTyped repo owned by Microsoft) gets larger, the more experience the owner of GitHub (also Microsoft) gets in hosting the world's largest git repos. I would say this qualifies as one of those cases, as npmjs is hosted on Azure. The more resources that NPM needs, the more Microsoft can build towards parity with AWS's footprint.

deleted 2y ago

[deleted]