4 ms·

Something I learned the hard way is that SSD performance can nosedive if DISCARD/TRIM commands are not sent to the device. Up to 50% lower throughput on our Sam

Something I learned the hard way is that SSD performance can nosedive if DISCARD/TRIM commands are not sent to the device. Up to 50% lower throughput on our Samsung DC drives. Through metrics I noticed that some SSD in a cluster were much slower than others despite being uniform hardware. After a bit of investigation it was found that the slow devices had been in service longer, and we were mot sending DISCARDs to the SSDs due to a default in dm-crypt: https://wiki.archlinux.org/title/Dm-crypt/Specialties#Discard/TRIM_support_for_solid_state_drives_(SSD) https://wiki.archlinux.org/title/Dm-crypt/Specialties#Discar... The performance penalty for our drives (Samsung DC drives) was around 50% if TRIM was never run. We now run blkdiscard when provisioning new drives and enable discards on the crypt devices and things seem to be much better now. Reflecting a bit more, this makes me more bullish on system integrators like Oxide as I have seen so many times software which was misconfigured to not use the full potential of the hardware. There is a size of company between a one person shop and somewhere like facebook/google where they are running their own racks but they don’t have the in house expertise to triage and fix these performance issues. If for example you are getting 50% less performance out of your DB nodes, what is the cost of that inefficiency?

p_ing 1y ago
While not the same issue, I took four 500GB Samsung 850 EVO drives and created a Storage Space out of them for Hyper-V VMs. Under any sort of load the volume would reach ~1 second latency. This was on a SAS controller in JBOD mode. Switched to some Intel 480GB DC drives and performance was in the low milliseconds as I would have thought any drive should be. Not sure if I was hitting the DRAM limit of the Samsungs or what, spent a bit of time t-shooting but this was a home lab and used Intel DCs were cheap on eBay. Granted, the Samsung EVOs weren't targeted to that type of work.
- __turbobrew__ 1y ago
  850 EVO is basically the lowest tier consumer device, from what I have read those devices can only handle short bursts of IOs and do not perform well under sustained load.
- sitkack 1y ago
  Could be garbage collection pauses. You could try wiping them again with zeros or doing a drive specific reset and see if the performance is normative.
- pkaye 1y ago
  The Samsung 850 EVO drives probably used an SLC write cache. A small portion of the NAND is configured to use as an SLC write buffer so they can handle a burst of writes faster and later move them the the MLC/TLC region. This is sufficient for typical consumer workloads. Another thing you will notice is the 850 EVO is 500GB capacity while the Intel one is 480GB. The difference is capacity is put towards overprovisioning which reduces write amplification. The idea is if you have sufficient free space available, whole NAND blocks will naturally get invalidated before you run out of free blocks.
- ADefenestrator 1y ago
  The Samsung consumer drives definitely don't do well under sustained high write workloads. The SLC cache fills up after a while, and write speeds drop drastically. The They also have some variety of internal head-of-line blocking type issue, where read latency goes way up when the writes saturate. I can't say I've ever seen 1s latency out of them, though. Consumer drives can definitely have some quirks. The 2TB 960 Pro also just had weird write latency, even under relatively moderate load. Like 2-4ms instead of <1ms. It didn't really get much worse with extra load and concurrency, except that if there's writes enqueued the reads end up waiting behind them for some reason and also seeing the latency penalty. They can also be weird behind RAID controllers, though I'm not sure if JBOD counts there. For whatever reason, the 860 EVO line wouldn't pass TRIM through the SAS RAID controller, but the 860 PRO would.
lathiat 1y ago
The fun part is that for a bunch of SSD drives (especially older ones), sending discard/trim may also tank the performance. Due to firmware bugs.
- loeg 1y ago
  You still might need to pace how fast you send discard/trim to modern drives, FWIW.

p_ing 1y ago

While not the same issue, I took four 500GB Samsung 850 EVO drives and created a Storage Space out of them for Hyper-V VMs. Under any sort of load the volume would reach ~1 second latency. This was on a SAS controller in JBOD mode. Switched to some Intel 480GB DC drives and performance was in the low milliseconds as I would have thought any drive should be. Not sure if I was hitting the DRAM limit of the Samsungs or what, spent a bit of time t-shooting but this was a home lab and used Intel DCs were cheap on eBay. Granted, the Samsung EVOs weren't targeted to that type of work.

__turbobrew__ 1y ago
850 EVO is basically the lowest tier consumer device, from what I have read those devices can only handle short bursts of IOs and do not perform well under sustained load.
sitkack 1y ago
Could be garbage collection pauses. You could try wiping them again with zeros or doing a drive specific reset and see if the performance is normative.
pkaye 1y ago
The Samsung 850 EVO drives probably used an SLC write cache. A small portion of the NAND is configured to use as an SLC write buffer so they can handle a burst of writes faster and later move them the the MLC/TLC region. This is sufficient for typical consumer workloads. Another thing you will notice is the 850 EVO is 500GB capacity while the Intel one is 480GB. The difference is capacity is put towards overprovisioning which reduces write amplification. The idea is if you have sufficient free space available, whole NAND blocks will naturally get invalidated before you run out of free blocks.
ADefenestrator 1y ago
The Samsung consumer drives definitely don't do well under sustained high write workloads. The SLC cache fills up after a while, and write speeds drop drastically. The They also have some variety of internal head-of-line blocking type issue, where read latency goes way up when the writes saturate. I can't say I've ever seen 1s latency out of them, though. Consumer drives can definitely have some quirks. The 2TB 960 Pro also just had weird write latency, even under relatively moderate load. Like 2-4ms instead of <1ms. It didn't really get much worse with extra load and concurrency, except that if there's writes enqueued the reads end up waiting behind them for some reason and also seeing the latency penalty. They can also be weird behind RAID controllers, though I'm not sure if JBOD counts there. For whatever reason, the 860 EVO line wouldn't pass TRIM through the SAS RAID controller, but the 860 PRO would.

lathiat 1y ago

The fun part is that for a bunch of SSD drives (especially older ones), sending discard/trim may also tank the performance. Due to firmware bugs.

loeg 1y ago
You still might need to pace how fast you send discard/trim to modern drives, FWIW.