4 ms·

> This article is basically advocating RAID 5 across many storage providers. That is correct. And to be more precise, I'm advocating RAID 5 across storage pro

> This article is basically advocating RAID 5 across many storage providers. That is correct. And to be more precise, I'm advocating RAID 5 across storage providers as a service, so people who just want to store data don't have to manage anything.

thereallurch 14y ago
Wouldn't the next issue become bandwidth? Sure, you have cheap storage, but you still need to manipulate the data somehow. I know S3 only charges for outbound data. What about the other companies? Is bandwidth free and plenty?
- adamsmith 14y ago
  This is a really interesting question. At scale, this could be managed, I think, through a combination of (a) shipping hard drives around, (b) caching, and (c) peering between storage providers. For example, shipping hard drives around would be expedient if you wanted to switch out a raw storage provider. The optimal strategy also depends on the access patterns and latency requirements. It seems solvable, but not trivial.
Terretta 14y ago
>> This article is basically advocating RAID 5 across many storage providers. > That is correct. The diagram in the article and text description is RAID 1+0 aka RAID 10.
- Dylan16807 14y ago
  No, it's RAID 5. Well, specifically, it's RAID 6 but customizable. You can lose any k drives.
notmyname 14y ago
If this were built on top of (or in conjunction with) Openstack swift (http://swift.openstack.org http://swift.openstack.org), it could be done with a common storage backend. There are already several public cloud storage providers using swift (Rackspace, HP, Softlayer, Internap, KT, and others), and a growing number of private deployments as well (Wikipedia probably being the most recognizable name). If you'd like to talk more about this, send me an email (address in profile).
- wmf 14y ago
  Aren't most Swift providers already using 2x-3x duplication? Putting parity on top of that would be like RAID15 (the reason you've never heard of RAID15 is that it would be crazy expensive).
  - notmyname 14y ago
    Yes, swift provides multiple redundant copies of the data. It depends on what you are protecting against: failure of a piece of your data or failure of a hosting provider. If you simply rely on "dumb disks" spread across multiple providers to provide availability, then you may be interested in looking at the nova-volumes part of openstack (it provides block storage attached to nova VMs). As part of openstack, it's an open system that is seeing rapid adoption. However, one of the most-requested features for swift is to provide support for logical clusters that span a wide geographic area. This could potentially allow multiple providers to collaborate in providing a multi-provider storage system. However, I'd guess that the technical problems are much simpler than the business problems in setting up multi-provider clusters.

thereallurch 14y ago

Wouldn't the next issue become bandwidth? Sure, you have cheap storage, but you still need to manipulate the data somehow. I know S3 only charges for outbound data. What about the other companies? Is bandwidth free and plenty?

adamsmith 14y ago
This is a really interesting question. At scale, this could be managed, I think, through a combination of (a) shipping hard drives around, (b) caching, and (c) peering between storage providers. For example, shipping hard drives around would be expedient if you wanted to switch out a raw storage provider. The optimal strategy also depends on the access patterns and latency requirements. It seems solvable, but not trivial.

Terretta 14y ago

>> This article is basically advocating RAID 5 across many storage providers. > That is correct. The diagram in the article and text description is RAID 1+0 aka RAID 10.

Dylan16807 14y ago
No, it's RAID 5. Well, specifically, it's RAID 6 but customizable. You can lose any k drives.

notmyname 14y ago

If this were built on top of (or in conjunction with) Openstack swift (http://swift.openstack.org http://swift.openstack.org), it could be done with a common storage backend. There are already several public cloud storage providers using swift (Rackspace, HP, Softlayer, Internap, KT, and others), and a growing number of private deployments as well (Wikipedia probably being the most recognizable name). If you'd like to talk more about this, send me an email (address in profile).

wmf 14y ago
Aren't most Swift providers already using 2x-3x duplication? Putting parity on top of that would be like RAID15 (the reason you've never heard of RAID15 is that it would be crazy expensive).
- notmyname 14y ago
  Yes, swift provides multiple redundant copies of the data. It depends on what you are protecting against: failure of a piece of your data or failure of a hosting provider. If you simply rely on "dumb disks" spread across multiple providers to provide availability, then you may be interested in looking at the nova-volumes part of openstack (it provides block storage attached to nova VMs). As part of openstack, it's an open system that is seeing rapid adoption. However, one of the most-requested features for swift is to provide support for logical clusters that span a wide geographic area. This could potentially allow multiple providers to collaborate in providing a multi-provider storage system. However, I'd guess that the technical problems are much simpler than the business problems in setting up multi-provider clusters.