5 ms·

"In 2006 a 320 GB hard drive cost $120. Today (Thailand floods aside) that much money will snag you a 3 TB drive." Floods or not, the current price isn't $120

"In 2006 a 320 GB hard drive cost $120. Today (Thailand floods aside) that much money will snag you a 3 TB drive." Floods or not, the current price isn't $120 dollars. It's 50% higher than that. http://camelcamelcamel.com/Western-Digital-Caviar-Green-Desktop/product/B004RORMF6?active=amazon http://camelcamelcamel.com/Western-Digital-Caviar-Green-Desk... Shows one of the cheapest 3tb non enterprise drives. It looks like 3tb was $120 for ~2 weeks. Looking at enterprise drives, 3tb is closer to $300. This article is basically advocating RAID 5 across many storage providers. *edit: From the pictures, article is advocating RAID 10. Nonetheless, RAID5 would be just as feasible for additional storage.

pandemicsyn 14y ago
People always conveniently forget as well that when you store data at S3 or Rackspace they're doing 3 replica's. If you store 3TB's, its not just the cost of a single 3TB drive..its 3x. To do it at home you'd need to buy 3x3TB Drives @300$ each.
- adamsmith 14y ago
  That's right. The main point is that these services still benefit from the falling cost of storage.
- revolutions 14y ago
  Suppose the prices of _x_ amount of storage was halved recently. If they stored their data in three replicas, shouldn't their price become (1/2)^3, or 1/8th the original price? That would only serve to prove his point further, or I'm missing something.
  - thereallurch 14y ago
    provider1=[d1,1][d1,2][d1,3] provider2=[d2,1][d2,2][d2,3] provider3=[d3,1][d3,2][d3,3] Each drive cost $x. You have 9$x = cost to store 1 drive of data, across 3 providers, who each store 3 copies. If drive prices halve, it's still 9$x.
    - wmf 14y ago
      I don't think anyone is proposing 3 providers, who each store 3 copies. Either use one provider who stores 3 copies (Amazon) or three providers who each store one copy.
      - thereallurch 14y ago
        "People always conveniently forget as well that when you store data at S3 or Rackspace they're doing 3 replica's. If you store 3TB's, its not just the cost of a single 3TB drive..its 3x. To do it at home you'd need to buy 3x3TB Drives @300$ each." Existing companies store multiple copies transparently...Either way, 1 copy or 3 copies, I dont think the math is wrong. Just change from 9$x to 3$x.
adamsmith 14y ago
> This article is basically advocating RAID 5 across many storage providers. That is correct. And to be more precise, I'm advocating RAID 5 across storage providers as a service, so people who just want to store data don't have to manage anything.
- thereallurch 14y ago
  Wouldn't the next issue become bandwidth? Sure, you have cheap storage, but you still need to manipulate the data somehow. I know S3 only charges for outbound data. What about the other companies? Is bandwidth free and plenty?
  - adamsmith 14y ago
    This is a really interesting question. At scale, this could be managed, I think, through a combination of (a) shipping hard drives around, (b) caching, and (c) peering between storage providers. For example, shipping hard drives around would be expedient if you wanted to switch out a raw storage provider. The optimal strategy also depends on the access patterns and latency requirements. It seems solvable, but not trivial.
- Terretta 14y ago
  >> This article is basically advocating RAID 5 across many storage providers. > That is correct. The diagram in the article and text description is RAID 1+0 aka RAID 10.
  - Dylan16807 14y ago
    No, it's RAID 5. Well, specifically, it's RAID 6 but customizable. You can lose any k drives.
- notmyname 14y ago
  If this were built on top of (or in conjunction with) Openstack swift (http://swift.openstack.org http://swift.openstack.org), it could be done with a common storage backend. There are already several public cloud storage providers using swift (Rackspace, HP, Softlayer, Internap, KT, and others), and a growing number of private deployments as well (Wikipedia probably being the most recognizable name). If you'd like to talk more about this, send me an email (address in profile).

pandemicsyn 14y ago

People always conveniently forget as well that when you store data at S3 or Rackspace they're doing 3 replica's. If you store 3TB's, its not just the cost of a single 3TB drive..its 3x. To do it at home you'd need to buy 3x3TB Drives @300$ each.

adamsmith 14y ago
That's right. The main point is that these services still benefit from the falling cost of storage.
revolutions 14y ago
Suppose the prices of _x_ amount of storage was halved recently. If they stored their data in three replicas, shouldn't their price become (1/2)^3, or 1/8th the original price? That would only serve to prove his point further, or I'm missing something.
- thereallurch 14y ago
  provider1=[d1,1][d1,2][d1,3] provider2=[d2,1][d2,2][d2,3] provider3=[d3,1][d3,2][d3,3] Each drive cost $x. You have 9$x = cost to store 1 drive of data, across 3 providers, who each store 3 copies. If drive prices halve, it's still 9$x.
  - wmf 14y ago
    I don't think anyone is proposing 3 providers, who each store 3 copies. Either use one provider who stores 3 copies (Amazon) or three providers who each store one copy.
    - thereallurch 14y ago
      "People always conveniently forget as well that when you store data at S3 or Rackspace they're doing 3 replica's. If you store 3TB's, its not just the cost of a single 3TB drive..its 3x. To do it at home you'd need to buy 3x3TB Drives @300$ each." Existing companies store multiple copies transparently...Either way, 1 copy or 3 copies, I dont think the math is wrong. Just change from 9$x to 3$x.

adamsmith 14y ago

> This article is basically advocating RAID 5 across many storage providers. That is correct. And to be more precise, I'm advocating RAID 5 across storage providers as a service, so people who just want to store data don't have to manage anything.

thereallurch 14y ago
Wouldn't the next issue become bandwidth? Sure, you have cheap storage, but you still need to manipulate the data somehow. I know S3 only charges for outbound data. What about the other companies? Is bandwidth free and plenty?
- adamsmith 14y ago
  This is a really interesting question. At scale, this could be managed, I think, through a combination of (a) shipping hard drives around, (b) caching, and (c) peering between storage providers. For example, shipping hard drives around would be expedient if you wanted to switch out a raw storage provider. The optimal strategy also depends on the access patterns and latency requirements. It seems solvable, but not trivial.
Terretta 14y ago
>> This article is basically advocating RAID 5 across many storage providers. > That is correct. The diagram in the article and text description is RAID 1+0 aka RAID 10.
- Dylan16807 14y ago
  No, it's RAID 5. Well, specifically, it's RAID 6 but customizable. You can lose any k drives.
notmyname 14y ago
If this were built on top of (or in conjunction with) Openstack swift (http://swift.openstack.org http://swift.openstack.org), it could be done with a common storage backend. There are already several public cloud storage providers using swift (Rackspace, HP, Softlayer, Internap, KT, and others), and a growing number of private deployments as well (Wikipedia probably being the most recognizable name). If you'd like to talk more about this, send me an email (address in profile).