4 ms·

> I'm sure SC serves up a lot of content per day, but how do you think they will react by suddenly having someone download all of their 900 TB or whatever it is

by pfg 9y ago

> I'm sure SC serves up a lot of content per day, but how do you think they will react by suddenly having someone download all of their 900 TB or whatever it is in one day? How much will Archiveteam be contributing to SC's downfall by suddenly causing them a huge unexpected bill? Obviously, there's more than just the bandwidth cost, but assuming they pay $0.02/GB for CDN traffic, we're talking about $18k. It's not nothing, but I doubt it'd change their outlook in any meaningful way. I should add that the ArchiveTeam doesn't download everything in a day, but rather uses a distributed crawler (ArchiveTeam Warrior) run by volunteers. They rate-limit the crawling rate as needed in order not to overwhelm the site being archived.

makerbraker 9y ago
That's making the incorrect assumption that all 900TB is hosted by the CDN... you typically only have a fraction of your content, known as your working set, in the CDN's working cache. So you should factor in the bandwidth cost for CDN traffic AND origin pulls, which if you're serving from AWS (and not using CloudFront), is $0.08/GB. The size of your working set also influences your overall CDN bill. Storage isn't free.
- pfg 9y ago
  I'd expect their S3 data transfer pricing to be closer to the assumed $0.02 CDN price. The price available to the public goes down to $0.05 once you hit 200 TB, and AWS is known to give significant discounts to large clients. The storage cost is a good point, and I don't know which CDN they use, but I expect the songs that weren't "hot" would be evicted from edge caches pretty fast, so that should be no significant increase. To put things into perspective, an article from 2009 estimates that Spotify used 84TB of traffic per day[1], back when they had 5 million users. Soundcloud claims to have had 40 million registered users in 2013 and 175 million unique monthly listeners in 2014. These numbers aren't easy to compare, but either way I think it's safe to say that 900TB should be a small fraction of their monthly traffic, and shouldn't make a huge dent in their financials. [1]: https://www.theguardian.com/technology/blog/2009/oct/08/spotify-internet https://www.theguardian.com/technology/blog/2009/oct/08/spot...
- posguy 9y ago
  Is it just me, or is charging pennies per gigabyte usurious? If I paid even a penny per gig on my home internet, I'd have nearly a $150 bill just in gig charges, and I know I'm one of the lower bandwidth customers on my block. Why stay with a host that bends you over a barrel for transit? Even Netflix doesn't use Amazon to deliver their data heavy content, as their pricing is pie in the sky high.
  - qq66 9y ago
    What block do you live on that 15TB is on the low end?
    - pessimizer 9y ago
      They replaced all lighting with streaming 1080p video.

by pfg 9y ago

makerbraker 9y ago

That's making the incorrect assumption that all 900TB is hosted by the CDN... you typically only have a fraction of your content, known as your working set, in the CDN's working cache. So you should factor in the bandwidth cost for CDN traffic AND origin pulls, which if you're serving from AWS (and not using CloudFront), is $0.08/GB. The size of your working set also influences your overall CDN bill. Storage isn't free.

pfg 9y ago
I'd expect their S3 data transfer pricing to be closer to the assumed $0.02 CDN price. The price available to the public goes down to $0.05 once you hit 200 TB, and AWS is known to give significant discounts to large clients. The storage cost is a good point, and I don't know which CDN they use, but I expect the songs that weren't "hot" would be evicted from edge caches pretty fast, so that should be no significant increase. To put things into perspective, an article from 2009 estimates that Spotify used 84TB of traffic per day[1], back when they had 5 million users. Soundcloud claims to have had 40 million registered users in 2013 and 175 million unique monthly listeners in 2014. These numbers aren't easy to compare, but either way I think it's safe to say that 900TB should be a small fraction of their monthly traffic, and shouldn't make a huge dent in their financials. [1]: https://www.theguardian.com/technology/blog/2009/oct/08/spotify-internet https://www.theguardian.com/technology/blog/2009/oct/08/spot...
posguy 9y ago
Is it just me, or is charging pennies per gigabyte usurious? If I paid even a penny per gig on my home internet, I'd have nearly a $150 bill just in gig charges, and I know I'm one of the lower bandwidth customers on my block. Why stay with a host that bends you over a barrel for transit? Even Netflix doesn't use Amazon to deliver their data heavy content, as their pricing is pie in the sky high.
- qq66 9y ago
  What block do you live on that 15TB is on the low end?
  - pessimizer 9y ago
    They replaced all lighting with streaming 1080p video.