6 ms·

Storage price reductions slowing down or even coming to a halt makes me wonder what this means for "infinite storage" companies. I'm talking services like Face

by fleddr 4y ago

Storage price reductions slowing down or even coming to a halt makes me wonder what this means for "infinite storage" companies. I'm talking services like Facebook, Youtube, that are "free" (ad supported) where on a daily basis an absurd amount of new content is added, yet almost nothing ever removed. If storage needs grow endlessly yet storage costs stopped going down, wouldn't that mean that the model in the long term is unsustainable? Sure you can delay the inevitable (compress content, move old stuff to cold storage) but ultimately storage costs per user goes up whilst income likely does not.

prepend 4y ago
If storage lifetime is increasing it might not be so bad. Or compression/deduplication. As long as they have consistent ad revenue, they’ll be ok.
- flatiron 4y ago
  I wouldn’t believe dedupe would be a big thing with social media user created content.
  - deleted 4y ago
    [deleted]
  - vineyardmike 4y ago
    I suspect (but have no data for) that a significant chunk of content on social media is memes and still-image-video. Those are likely to be easy to dedupe or compress aggressively
Spooky23 4y ago
I disagree, I’m still in the datacenter storage business and our service delivery costs have been going down. You have to look at the service as a whole… nvme let us retire frames, etc. I think hyperscale margins are high, and I think between the many tiers of storage and backend technology they are constantly cooking cost out.
- fleddr 4y ago
  But still content keeps growing, and income per user won't. Doesn't that mean that at one point you're out of runway?
  - winkeltripel 4y ago
    Not really. As costs go up, you can mitigate by pushing more of your data out to slower mediums, like hdd.
  - Spooky23 4y ago
    I don’t think so. The highest speed storage is cheaper than ever and getting cheaper. That’s the expensive stuff. Our “big” high speed storage array was like $5M in 2012. Now it’s like 10x bigger, 5x faster, and 40% of the cost. Bulk spinning disk prices drop at a slower rate but historically they are performance bound. The marginal byte of cold data has no marginal cost, as you always have lots of space on disks where you’re short on IO budget. Really cold workloads that need to live on-prem for reasons and need to be retained for 30+ years can go even cheaper. That stuff lives on tape at a lower price point, like a half penny per raw gig.
  - deleted 4y ago
    [deleted]
andreimackenzie 4y ago
Once ancient content no longer produces enough ad revenue for the companies storing it, I suspect they will support it only in paid tiers for users who really want it. Something like this is already happening in Google's paid tiers: each user is valuable as part of Google's advertising audience, but the economics no longer make sense once those users keep many GB on Google's servers.
- Robotbeat 4y ago
  Ancient content can be moved away from edge servers and more to archival.
  - pyrolistical 4y ago
    already happening. ever notice youtube videos with low view counts take a really long time to load?
    - ineedasername 4y ago
      Presages an interesting digital future in which nothing ever truly goes away, it just gets slower and slower. An endless inner migration that never quite hits the stopping point, and which could theoretically be reversed if only the slowed content garnered enough attention.
      - vermilingua 4y ago
        Endlessly falling towards an event horizon of obscurity, but never reaching it (from the perspective of an external observer).
      - Dylan16807 4y ago
        I'd expect a significantly different scale. Specifically, I'd expect normal tiers ranging from "instant" to "several seconds", then a huge gap, then a rock bottom tape archive. From a more zoomed out look, you could simplify it to only two speeds. The fast speed taking 0-15 seconds, and the slow speed taking minutes to hours. Any content that's been accessed once or twice in the last week or month would be on the normal tiers. Extremely dead content could fall to the tape tier, but it has nowhere further to fall, and it would take only a tiny amount of activity to rescue it. I don't really see a reason for there to be a continuous falloff in speed. There's not really anything between hard drives and tape for responsiveness, either existing or proposed, that I'm aware of. Nor is there anything slower than tapes.
pclmulqdq 4y ago
This is a huge concern for me too. I suspect that FB and Youtube will start auto-deleting old content that people haven't seen in the last X months.
- Gigachad 4y ago
  There would be exabytes of content on youtube over 5 years old with under 20 views. Stuff no one will miss when its gone.
  - hsbauauvhabzb 4y ago
    Except people that use YT as a family video archive
    - Gigachad 4y ago
      I'm certain that in the ToS they reserve the right to remove any video at any time for any reason. Dumping your family videos on a public video sharing site that you don't pay for and expecting them to exist there forever for free was never a good idea.
      - hsbauauvhabzb 4y ago
        Tell that to the people who lose videos that way. Google would consider that a cost (loss of goodwill with a minute set of people). Idk about YouTube but I know many people whose sole copy of their treasured images is on Facebook.
    - deleted 4y ago
      [deleted]
    - vineyardmike 4y ago
      They’ll probably migrate those people to a “Google Photos” or “Google Drive” storage option so they pay for their usage but still have archive and sharing options.
  - 4y ago
notatoad 4y ago
storage price continues to decrease. the point is that S3 price doesn't drop at close to the same rate. facebook and youtube aren't hosting on S3, so they get to experience the real decreases in storage cost.
AlexC04 4y ago
I don't think the article indicates that storage isn't getting cheaper. Equally possible is that amazon's S3 margins are getting better. For companies like Facebook and YouTube it is possible that they're not affected in the same way that a regular S3 customer would be.
- h0l0cube 4y ago
  > Equally possible is that amazon's S3 margins are getting better. Which is acceptable if they're providing better value in features. From casually looking around it seems that Backblaze's B2 is much cheaper, but doesn't have a good story around reliability, namely uptime guarantees and simple multi-region redundancy. If Backblaze could match AWS in this regard, and with their Bandwidth Alliance with Cloudflare, they could provide better downward pressure on pricing.
  - sieabahlpark 4y ago
staticassertion 4y ago
That content has extreme temporal properties. An image that's uploaded is increasingly less likely to be accessed over time. You can probably store it on very cheap hardware with pretty serious compression. Not to mention that it's mostly text and images that get uploaded to these sites. Videos are the worst case. Let's say there are 1 billion facebook accounts, each one uploads 1GB of data (seems huge), and let's assume that compression just cancels out replication. That's 1,000 petabytes. On S3 that's in the low 6 figures per year, obviously ignoring the exfil/access costs. That's not that much. Obviously you want to keep some of it "hot" - profile pictures, recently uploaded pictures, etc. Hand waving, assuming 1% of the data needs to stay hot (seems high), that's 10PB of data. Certainly you're in "big data" land but it's not like there aren't databases that'll handle it. Not to mention FB invests in tech like zstd. Cold storage is definitely still getting much cheaper. Dropbox SMR drives are pretty crazy: https://dropbox.tech/infrastructure/smr-what-we-learned-in-our-first-year https://dropbox.tech/infrastructure/smr-what-we-learned-in-o...
femto113 4y ago
I'm not worried about the ad supported free storage providers. On YouTube the ratio is something like 10,000 views for every 1 upload, so they amortize that storage cost across a lot of other users, not just the person who uploads it. Ratio will be lower on Facebook but even active users probably don't average more than 1 or 2 uploads per day. I've had the same gmail address for over 10 years, never delete anything, and I'm still under 50% of their free storage limit.

by fleddr 4y ago

prepend 4y ago

If storage lifetime is increasing it might not be so bad. Or compression/deduplication. As long as they have consistent ad revenue, they’ll be ok.

flatiron 4y ago
I wouldn’t believe dedupe would be a big thing with social media user created content.
- deleted 4y ago
  [deleted]
- vineyardmike 4y ago
  I suspect (but have no data for) that a significant chunk of content on social media is memes and still-image-video. Those are likely to be easy to dedupe or compress aggressively

Spooky23 4y ago

I disagree, I’m still in the datacenter storage business and our service delivery costs have been going down. You have to look at the service as a whole… nvme let us retire frames, etc. I think hyperscale margins are high, and I think between the many tiers of storage and backend technology they are constantly cooking cost out.

fleddr 4y ago
But still content keeps growing, and income per user won't. Doesn't that mean that at one point you're out of runway?
- winkeltripel 4y ago
  Not really. As costs go up, you can mitigate by pushing more of your data out to slower mediums, like hdd.
- Spooky23 4y ago
  I don’t think so. The highest speed storage is cheaper than ever and getting cheaper. That’s the expensive stuff. Our “big” high speed storage array was like $5M in 2012. Now it’s like 10x bigger, 5x faster, and 40% of the cost. Bulk spinning disk prices drop at a slower rate but historically they are performance bound. The marginal byte of cold data has no marginal cost, as you always have lots of space on disks where you’re short on IO budget. Really cold workloads that need to live on-prem for reasons and need to be retained for 30+ years can go even cheaper. That stuff lives on tape at a lower price point, like a half penny per raw gig.
- deleted 4y ago
  [deleted]

andreimackenzie 4y ago

Once ancient content no longer produces enough ad revenue for the companies storing it, I suspect they will support it only in paid tiers for users who really want it. Something like this is already happening in Google's paid tiers: each user is valuable as part of Google's advertising audience, but the economics no longer make sense once those users keep many GB on Google's servers.

Robotbeat 4y ago
Ancient content can be moved away from edge servers and more to archival.
- pyrolistical 4y ago
  already happening. ever notice youtube videos with low view counts take a really long time to load?
  - ineedasername 4y ago
    Presages an interesting digital future in which nothing ever truly goes away, it just gets slower and slower. An endless inner migration that never quite hits the stopping point, and which could theoretically be reversed if only the slowed content garnered enough attention.
    - vermilingua 4y ago
      Endlessly falling towards an event horizon of obscurity, but never reaching it (from the perspective of an external observer).
    - Dylan16807 4y ago
      I'd expect a significantly different scale. Specifically, I'd expect normal tiers ranging from "instant" to "several seconds", then a huge gap, then a rock bottom tape archive. From a more zoomed out look, you could simplify it to only two speeds. The fast speed taking 0-15 seconds, and the slow speed taking minutes to hours. Any content that's been accessed once or twice in the last week or month would be on the normal tiers. Extremely dead content could fall to the tape tier, but it has nowhere further to fall, and it would take only a tiny amount of activity to rescue it. I don't really see a reason for there to be a continuous falloff in speed. There's not really anything between hard drives and tape for responsiveness, either existing or proposed, that I'm aware of. Nor is there anything slower than tapes.

pclmulqdq 4y ago

This is a huge concern for me too. I suspect that FB and Youtube will start auto-deleting old content that people haven't seen in the last X months.

Gigachad 4y ago
There would be exabytes of content on youtube over 5 years old with under 20 views. Stuff no one will miss when its gone.
- hsbauauvhabzb 4y ago
  Except people that use YT as a family video archive
  - Gigachad 4y ago
    I'm certain that in the ToS they reserve the right to remove any video at any time for any reason. Dumping your family videos on a public video sharing site that you don't pay for and expecting them to exist there forever for free was never a good idea.
    - hsbauauvhabzb 4y ago
      Tell that to the people who lose videos that way. Google would consider that a cost (loss of goodwill with a minute set of people). Idk about YouTube but I know many people whose sole copy of their treasured images is on Facebook.
  - deleted 4y ago
    [deleted]
  - vineyardmike 4y ago
    They’ll probably migrate those people to a “Google Photos” or “Google Drive” storage option so they pay for their usage but still have archive and sharing options.
- 4y ago

notatoad 4y ago

storage price continues to decrease. the point is that S3 price doesn't drop at close to the same rate. facebook and youtube aren't hosting on S3, so they get to experience the real decreases in storage cost.

AlexC04 4y ago

I don't think the article indicates that storage isn't getting cheaper. Equally possible is that amazon's S3 margins are getting better. For companies like Facebook and YouTube it is possible that they're not affected in the same way that a regular S3 customer would be.

h0l0cube 4y ago
> Equally possible is that amazon's S3 margins are getting better. Which is acceptable if they're providing better value in features. From casually looking around it seems that Backblaze's B2 is much cheaper, but doesn't have a good story around reliability, namely uptime guarantees and simple multi-region redundancy. If Backblaze could match AWS in this regard, and with their Bandwidth Alliance with Cloudflare, they could provide better downward pressure on pricing.
- sieabahlpark 4y ago

staticassertion 4y ago

That content has extreme temporal properties. An image that's uploaded is increasingly less likely to be accessed over time. You can probably store it on very cheap hardware with pretty serious compression. Not to mention that it's mostly text and images that get uploaded to these sites. Videos are the worst case. Let's say there are 1 billion facebook accounts, each one uploads 1GB of data (seems huge), and let's assume that compression just cancels out replication. That's 1,000 petabytes. On S3 that's in the low 6 figures per year, obviously ignoring the exfil/access costs. That's not that much. Obviously you want to keep some of it "hot" - profile pictures, recently uploaded pictures, etc. Hand waving, assuming 1% of the data needs to stay hot (seems high), that's 10PB of data. Certainly you're in "big data" land but it's not like there aren't databases that'll handle it. Not to mention FB invests in tech like zstd. Cold storage is definitely still getting much cheaper. Dropbox SMR drives are pretty crazy: https://dropbox.tech/infrastructure/smr-what-we-learned-in-our-first-year https://dropbox.tech/infrastructure/smr-what-we-learned-in-o...

femto113 4y ago

I'm not worried about the ad supported free storage providers. On YouTube the ratio is something like 10,000 views for every 1 upload, so they amortize that storage cost across a lot of other users, not just the person who uploads it. Ratio will be lower on Facebook but even active users probably don't average more than 1 or 2 uploads per day. I've had the same gmail address for over 10 years, never delete anything, and I'm still under 50% of their free storage limit.