55 ms·

The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and m

The cloud storage way is to avoid RAID 5 or 10 etc completely and just use triple JBOD instead - store each file / chunk of data on at least 3 JBOD drives and manage replication and whatnot in software. I think Google first popularized this technique. The whole point of this excercise is reducing cost across the board and upgrading consumer SATA disks to enterprise ones will make this setup a whole lot more expensive since the drives make up most of the hardware cost.

lsc 17y ago
But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.
- Devilboy 17y ago
  In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time. With RAID10 again you have to build large arrays to beat triple JBOD for storage efficiency - a 12 x 1TB array yields 5TB usable space while triple JBOD gives you 4TB. BUT you still have a single point of failure - your RAID card can give out and your array is unavailable. Using triple JBOD you'll be able to have your data on different physical machines or even geographically separate data centers if you wish. The advantages outweigh the reduction of storage density when you're dealing with petabytes of storage - why else would Google et al be doing it?
  - lsc 17y ago
    raid5 sucks hard when a drive fails, you are right. But, see, for me, what I need is something that can be mounted as a block device by a xen DomU. Something that performs reasonably well. On top of that, I am kindof dumb compared to the sort of person I'd want writing my block device drivers. I want to take well-tested, open-source software components and plug them together in a clear manner. I don't know of anything off the shelf that will give me a filesystem with reasonable performance on a '3 jbod' system, unless you mean running md in a raid1 with 3 drives. (I'm actually doing that on my more remote servers; the idea is that I can wait longer before replacing a bad disk.) The system I proposed above is basically that you can export drives that may fail to clients, who can then do their own redundancy. Because the drive is specified as 'may fail' write-back caching may be utilized. (God, ram has become cheap.) The client, in my case, will be the Dom0 of the DomU that wants the space, but if I was selling this space to random people on the internets, it seems that the client could be some box running md that treats it's iscsi devices syncronously, meaning it waits for the write to return from both MD devices before it returns it's write. If the intermediary client device did no caching at all, it seems that I may be able to setup IP failover. (though that part... sounds dangerous. I'd need to be very careful with 'fencing' it or what have you, so that the two nodes were not active at once.)
    - Devilboy 17y ago
      Yea I see your point. I think in the end it still comes down to scale, for a couple of racks it's not worth the risk and effort while on a larger scale it might be worth brewing your own meta-filestorage system.
      - lsc 17y ago
        Yeah. that's my thought. I'm at a couple racks now, but I want a solution that will scale beyond that. Thing is, to be honest, the coraid crap will probably be more reliable (and cheaper if you include my time) than stuff I build myself for the first few units, until I learn what I'm doing, but long-term, I could be significantly better off if my idea works as planned, as my units are potentially much cheaper and have a whole lot more cache (and cache redundancy) than the coraid. On the other hand, if I handle the 'beta' poorly, like taking it out of beta early, there won't be a long term to worry about.
- wmf 17y ago
  To get reliability without replication, it's not enough to use enterprise drives; you need redundant controllers that are multipathed to the disks. This is fairly expensive. If you want a shared-nothing cluster with less than 3x overhead you can use erasure codes. The software complexity is significant, but at scale it should be the cheapest option. http://cleversafe.org/ http://cleversafe.org/ or http://allmydata.org/trac/tahoe http://allmydata.org/trac/tahoe
  - lsc 17y ago
    One solution I'm looking at is essentially mirrored raid5 over iscsi but it gets around your raid card single point of failure; the idea is that I have 2 zfs boxes (to begin/for testing these are OpenSolaris xen guests that control the spare drive slots I have in boxes that are primarily doing other things.) on each of those, i zraid the available drives. I then export iscsi luns of some standard size. The client box takes an iscsi lun from each opensolaris box and mirrors them with whatever software raid stack the client likes. The big win here is caching... each of the opensolaris boxes can be using write-back caching, because all data is mirrored to the other opensolaris box. much like a NetAPP or EMC with dual heads, I don't have to worry about disk inconsistencies caused by write-back cache unless both drives fail. (of course, your 'real' dual-headed SAN will switch to write-through caching if one head dies, and doesn't require a full rebuild when the other head comes back online, but I can't afford a 'real' dual-headed SAN.) Rebuild times, I imagine, will be quite significant. Now, I am very leery of the performance of this system (being as these are xen instances, it wouldn't surprise me if it was unsuitable for anything but tape replacement) but I haven't tried yet. It seems, though that it would work just fine if I was not too cheap to use real hardware. If you wanted to add complexity, you could have 3 or more of these opensolaris boxes, and use software raid5 on the client to save space.
    - gstar 17y ago
      Are your target boxes virtual machines? I always prefer the strategy where mirroring the data happens before the disk is presented to the VM. Have a look at GlusterFS and see if that might be an option? I know it wasn't playing nice with zfs for a while there, though.
uhgygghhj 17y ago
Does anyone use raid if they could use ZFS ? These people can be on BSD and so could use ZFS for free

lsc 17y ago

But 3 jbod takes more space than raid5- in terms of disk cost, that's worse than raid10, zraid, or mirroring. The enterprise drives are maybe 20%-30% more expensive than consumer disk, while replicating your data 3 times is 50% more expensive than a mirror or raid 10. Zfs, if it does deal with consumer drives as well as it claims to, would solve the problem at the same disk space cost as raid5.

Devilboy 17y ago
In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time. With RAID10 again you have to build large arrays to beat triple JBOD for storage efficiency - a 12 x 1TB array yields 5TB usable space while triple JBOD gives you 4TB. BUT you still have a single point of failure - your RAID card can give out and your array is unavailable. Using triple JBOD you'll be able to have your data on different physical machines or even geographically separate data centers if you wish. The advantages outweigh the reduction of storage density when you're dealing with petabytes of storage - why else would Google et al be doing it?
- lsc 17y ago
  raid5 sucks hard when a drive fails, you are right. But, see, for me, what I need is something that can be mounted as a block device by a xen DomU. Something that performs reasonably well. On top of that, I am kindof dumb compared to the sort of person I'd want writing my block device drivers. I want to take well-tested, open-source software components and plug them together in a clear manner. I don't know of anything off the shelf that will give me a filesystem with reasonable performance on a '3 jbod' system, unless you mean running md in a raid1 with 3 drives. (I'm actually doing that on my more remote servers; the idea is that I can wait longer before replacing a bad disk.) The system I proposed above is basically that you can export drives that may fail to clients, who can then do their own redundancy. Because the drive is specified as 'may fail' write-back caching may be utilized. (God, ram has become cheap.) The client, in my case, will be the Dom0 of the DomU that wants the space, but if I was selling this space to random people on the internets, it seems that the client could be some box running md that treats it's iscsi devices syncronously, meaning it waits for the write to return from both MD devices before it returns it's write. If the intermediary client device did no caching at all, it seems that I may be able to setup IP failover. (though that part... sounds dangerous. I'd need to be very careful with 'fencing' it or what have you, so that the two nodes were not active at once.)
  - Devilboy 17y ago
    Yea I see your point. I think in the end it still comes down to scale, for a couple of racks it's not worth the risk and effort while on a larger scale it might be worth brewing your own meta-filestorage system.
    - lsc 17y ago
      Yeah. that's my thought. I'm at a couple racks now, but I want a solution that will scale beyond that. Thing is, to be honest, the coraid crap will probably be more reliable (and cheaper if you include my time) than stuff I build myself for the first few units, until I learn what I'm doing, but long-term, I could be significantly better off if my idea works as planned, as my units are potentially much cheaper and have a whole lot more cache (and cache redundancy) than the coraid. On the other hand, if I handle the 'beta' poorly, like taking it out of beta early, there won't be a long term to worry about.
wmf 17y ago
To get reliability without replication, it's not enough to use enterprise drives; you need redundant controllers that are multipathed to the disks. This is fairly expensive. If you want a shared-nothing cluster with less than 3x overhead you can use erasure codes. The software complexity is significant, but at scale it should be the cheapest option. http://cleversafe.org/ http://cleversafe.org/ or http://allmydata.org/trac/tahoe http://allmydata.org/trac/tahoe
- lsc 17y ago
  One solution I'm looking at is essentially mirrored raid5 over iscsi but it gets around your raid card single point of failure; the idea is that I have 2 zfs boxes (to begin/for testing these are OpenSolaris xen guests that control the spare drive slots I have in boxes that are primarily doing other things.) on each of those, i zraid the available drives. I then export iscsi luns of some standard size. The client box takes an iscsi lun from each opensolaris box and mirrors them with whatever software raid stack the client likes. The big win here is caching... each of the opensolaris boxes can be using write-back caching, because all data is mirrored to the other opensolaris box. much like a NetAPP or EMC with dual heads, I don't have to worry about disk inconsistencies caused by write-back cache unless both drives fail. (of course, your 'real' dual-headed SAN will switch to write-through caching if one head dies, and doesn't require a full rebuild when the other head comes back online, but I can't afford a 'real' dual-headed SAN.) Rebuild times, I imagine, will be quite significant. Now, I am very leery of the performance of this system (being as these are xen instances, it wouldn't surprise me if it was unsuitable for anything but tape replacement) but I haven't tried yet. It seems, though that it would work just fine if I was not too cheap to use real hardware. If you wanted to add complexity, you could have 3 or more of these opensolaris boxes, and use software raid5 on the client to save space.
  - gstar 17y ago
    Are your target boxes virtual machines? I always prefer the strategy where mirroring the data happens before the disk is presented to the VM. Have a look at GlusterFS and see if that might be an option? I know it wasn't playing nice with zfs for a while there, though.

uhgygghhj 17y ago

Does anyone use raid if they could use ZFS ? These people can be on BSD and so could use ZFS for free