6 ms·

I worked at a company that migrated to SSDs on hundreds of HP ProLiant servers. The performance was great, it really improved our DB latency. Unfortunately the

by spudlyo 10y ago

I worked at a company that migrated to SSDs on hundreds of HP ProLiant servers. The performance was great, it really improved our DB latency. Unfortunately the RAID 1+0 setup that was carried over from the spinning rust setup didn't work for our most common failure case; corrupted writes. The HP RAID controller failed to detect corrupted writes on what I seem to remember were Intel SSDs. The only way we learned about failed SSD drives is when we started catching large numbers of DB page checksum errors, and by that time it was too late to do anything about it. Operationally it sucked, swapping out drives is a lot easier maintenance than having to rebuild a DB from backup.

Veratyr 10y ago
Is there a reason you were using hardware RAID rather than a software based RAID with checksumming like ZFS? (I mean this as an honest question, I'm interested to know if there are performance or reliability gains to be had from a hardware controller)
- spudlyo 10y ago
  The oft repeated performance reason is that hardware RAID controllers have battery backed write cache, which really improves durable write performance; something traditional relational DBs do quite a lot of. Also, these machines were built by DB people, who have traditionally placed a lot of value in hardware RAID controllers, and to be fair, the HP controllers worked well with HDDs. Also ZFS wasn't a realistic option for production Linux in 2013. Me personally, I'm not a fan of exotic proprietary hardware, I much prefer working with JBOD disk setups and software RAID where appropriate.
  - creshal 10y ago
    > The oft repeated performance reason is that hardware RAID controllers have battery backed write cache, which really improves durable write performance; something traditional relational DBs do quite a lot of. Don't all enterprise SSDs come with their own backup capacitors?
    - baruch 10y ago
      They do. You get an ack for a write before it reaches the media as long as the supercaps are deemed ok.
    - virtualwhys 10y ago
      > Don't all enterprise SSDs come with their own backup capacitors? Yes, but the cost of such drives makes rolling the dice with consumer grade drives appealing to some[1]. From the same thread, here's a word of warning from a Stackoverflow devops engineer[2] wrt to lack of capicitors on consumer grade drives and the perfect storm that can occur when UPS goes down in a power failure (unlike with spinning disks hardware RAID is unable prevent data corruption/loss in this case). [1] https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/ https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/ [2] https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/#comment-121887 https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/#commen...
      - kpil 10y ago
        I think my consumer-grade Intel S3500 SSD have some kind of power-loss protection. I don't think that professional devices are 100% fail-proof either. I vaguely remember some test where they setup a test with forced power failures during heavy writes, and they zapped all drives but a few eventually. Something that worried me was that the failure mode quite often seemed to be a totally bricked SSD.
        creshal 10y ago
        > I think my consumer-grade Intel S3500 SSD have some kind of power-loss protection. The fine-print is important here. Crucial (IIRC) advertises some of its low-end consumer series with power-loss protection… against internal data corruption of the Flash Translation Layer. AKA "We won't completely brick the entire device if you pull power!" Err… thank you? Why are you advertising this as feature?
        imtringued 10y ago
pcunite 10y ago
Interesting. Can you provide more details?
amelius 10y ago
So based on this, should filesystems include a "verify-writes" mount option? Or would this functionality somehow be counteracted by a cache inside the SSD? Perhaps filesystems should include a "ssd" mount option, because there might be more things to worry about (for example frequent writes wearing out the device).
- eru 10y ago
  The block device abstraction needs to leak more.
- rsync 10y ago
  "So based on this, should filesystems include a "verify-writes" mount option?" They do. You can, for instance, mount a UFS filesystem "sync", which means that all writes are synchronous. I assume extX has a similar option ...
  - amelius 10y ago
    Afaik, "sync" means that the writes are immediate. "Verify-writes" would mean that the writes are not necessarily immedate, but that they are verified. Also be careful with sync on SSD media; this is what the mount manpage says about the sync option: > In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening.
Keyframe 10y ago
When you have that many spinning rust drives replaced with SSDs - do you notice the change in power consumption?
- imtringued 10y ago
  That could go either way. Less power consumption because you don't need 10k rpm disks or more power consumption because the density still isn't there.

by spudlyo 10y ago

Veratyr 10y ago

Is there a reason you were using hardware RAID rather than a software based RAID with checksumming like ZFS? (I mean this as an honest question, I'm interested to know if there are performance or reliability gains to be had from a hardware controller)

spudlyo 10y ago
The oft repeated performance reason is that hardware RAID controllers have battery backed write cache, which really improves durable write performance; something traditional relational DBs do quite a lot of. Also, these machines were built by DB people, who have traditionally placed a lot of value in hardware RAID controllers, and to be fair, the HP controllers worked well with HDDs. Also ZFS wasn't a realistic option for production Linux in 2013. Me personally, I'm not a fan of exotic proprietary hardware, I much prefer working with JBOD disk setups and software RAID where appropriate.
- creshal 10y ago
  > The oft repeated performance reason is that hardware RAID controllers have battery backed write cache, which really improves durable write performance; something traditional relational DBs do quite a lot of. Don't all enterprise SSDs come with their own backup capacitors?
  - baruch 10y ago
    They do. You get an ack for a write before it reaches the media as long as the supercaps are deemed ok.
  - virtualwhys 10y ago
    > Don't all enterprise SSDs come with their own backup capacitors? Yes, but the cost of such drives makes rolling the dice with consumer grade drives appealing to some[1]. From the same thread, here's a word of warning from a Stackoverflow devops engineer[2] wrt to lack of capicitors on consumer grade drives and the perfect storm that can occur when UPS goes down in a power failure (unlike with spinning disks hardware RAID is unable prevent data corruption/loss in this case). [1] https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/ https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/ [2] https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/#comment-121887 https://laur.ie/blog/2015/06/ssds-a-gift-and-a-curse/#commen...
    - kpil 10y ago
      I think my consumer-grade Intel S3500 SSD have some kind of power-loss protection. I don't think that professional devices are 100% fail-proof either. I vaguely remember some test where they setup a test with forced power failures during heavy writes, and they zapped all drives but a few eventually. Something that worried me was that the failure mode quite often seemed to be a totally bricked SSD.
      - creshal 10y ago
        > I think my consumer-grade Intel S3500 SSD have some kind of power-loss protection. The fine-print is important here. Crucial (IIRC) advertises some of its low-end consumer series with power-loss protection… against internal data corruption of the Flash Translation Layer. AKA "We won't completely brick the entire device if you pull power!" Err… thank you? Why are you advertising this as feature?
        imtringued 10y ago

pcunite 10y ago

Interesting. Can you provide more details?

amelius 10y ago

So based on this, should filesystems include a "verify-writes" mount option? Or would this functionality somehow be counteracted by a cache inside the SSD? Perhaps filesystems should include a "ssd" mount option, because there might be more things to worry about (for example frequent writes wearing out the device).

eru 10y ago
The block device abstraction needs to leak more.
rsync 10y ago
"So based on this, should filesystems include a "verify-writes" mount option?" They do. You can, for instance, mount a UFS filesystem "sync", which means that all writes are synchronous. I assume extX has a similar option ...
- amelius 10y ago
  Afaik, "sync" means that the writes are immediate. "Verify-writes" would mean that the writes are not necessarily immedate, but that they are verified. Also be careful with sync on SSD media; this is what the mount manpage says about the sync option: > In case of media with limited number of write cycles (e.g. some flash drives) "sync" may cause life-cycle shortening.

Keyframe 10y ago

When you have that many spinning rust drives replaced with SSDs - do you notice the change in power consumption?

imtringued 10y ago
That could go either way. Less power consumption because you don't need 10k rpm disks or more power consumption because the density still isn't there.