2 ms·

In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During th

In a RAID 5 set you lose a lot of throughput when a drive fails, and with TB drives it takes many hours for a hot spare to replicate the missing data. During that time another drive failure is catastrophic for your data and the more disks in your RAID 5 set the more likely this scenario becomes. When you get to petabytes of data such catastrophic failures are just a matter of time. With RAID10 again you have to build large arrays to beat triple JBOD for storage efficiency - a 12 x 1TB array yields 5TB usable space while triple JBOD gives you 4TB. BUT you still have a single point of failure - your RAID card can give out and your array is unavailable. Using triple JBOD you'll be able to have your data on different physical machines or even geographically separate data centers if you wish. The advantages outweigh the reduction of storage density when you're dealing with petabytes of storage - why else would Google et al be doing it?

lsc 17y ago
raid5 sucks hard when a drive fails, you are right. But, see, for me, what I need is something that can be mounted as a block device by a xen DomU. Something that performs reasonably well. On top of that, I am kindof dumb compared to the sort of person I'd want writing my block device drivers. I want to take well-tested, open-source software components and plug them together in a clear manner. I don't know of anything off the shelf that will give me a filesystem with reasonable performance on a '3 jbod' system, unless you mean running md in a raid1 with 3 drives. (I'm actually doing that on my more remote servers; the idea is that I can wait longer before replacing a bad disk.) The system I proposed above is basically that you can export drives that may fail to clients, who can then do their own redundancy. Because the drive is specified as 'may fail' write-back caching may be utilized. (God, ram has become cheap.) The client, in my case, will be the Dom0 of the DomU that wants the space, but if I was selling this space to random people on the internets, it seems that the client could be some box running md that treats it's iscsi devices syncronously, meaning it waits for the write to return from both MD devices before it returns it's write. If the intermediary client device did no caching at all, it seems that I may be able to setup IP failover. (though that part... sounds dangerous. I'd need to be very careful with 'fencing' it or what have you, so that the two nodes were not active at once.)
- Devilboy 17y ago
  Yea I see your point. I think in the end it still comes down to scale, for a couple of racks it's not worth the risk and effort while on a larger scale it might be worth brewing your own meta-filestorage system.
  - lsc 17y ago
    Yeah. that's my thought. I'm at a couple racks now, but I want a solution that will scale beyond that. Thing is, to be honest, the coraid crap will probably be more reliable (and cheaper if you include my time) than stuff I build myself for the first few units, until I learn what I'm doing, but long-term, I could be significantly better off if my idea works as planned, as my units are potentially much cheaper and have a whole lot more cache (and cache redundancy) than the coraid. On the other hand, if I handle the 'beta' poorly, like taking it out of beta early, there won't be a long term to worry about.

lsc 17y ago

raid5 sucks hard when a drive fails, you are right. But, see, for me, what I need is something that can be mounted as a block device by a xen DomU. Something that performs reasonably well. On top of that, I am kindof dumb compared to the sort of person I'd want writing my block device drivers. I want to take well-tested, open-source software components and plug them together in a clear manner. I don't know of anything off the shelf that will give me a filesystem with reasonable performance on a '3 jbod' system, unless you mean running md in a raid1 with 3 drives. (I'm actually doing that on my more remote servers; the idea is that I can wait longer before replacing a bad disk.) The system I proposed above is basically that you can export drives that may fail to clients, who can then do their own redundancy. Because the drive is specified as 'may fail' write-back caching may be utilized. (God, ram has become cheap.) The client, in my case, will be the Dom0 of the DomU that wants the space, but if I was selling this space to random people on the internets, it seems that the client could be some box running md that treats it's iscsi devices syncronously, meaning it waits for the write to return from both MD devices before it returns it's write. If the intermediary client device did no caching at all, it seems that I may be able to setup IP failover. (though that part... sounds dangerous. I'd need to be very careful with 'fencing' it or what have you, so that the two nodes were not active at once.)

Devilboy 17y ago
Yea I see your point. I think in the end it still comes down to scale, for a couple of racks it's not worth the risk and effort while on a larger scale it might be worth brewing your own meta-filestorage system.
- lsc 17y ago
  Yeah. that's my thought. I'm at a couple racks now, but I want a solution that will scale beyond that. Thing is, to be honest, the coraid crap will probably be more reliable (and cheaper if you include my time) than stuff I build myself for the first few units, until I learn what I'm doing, but long-term, I could be significantly better off if my idea works as planned, as my units are potentially much cheaper and have a whole lot more cache (and cache redundancy) than the coraid. On the other hand, if I handle the 'beta' poorly, like taking it out of beta early, there won't be a long term to worry about.