14 ms·
SSD Endurance
- lvs 5y agoIf one is concerned about the impact of swap activity, is there a reason not to just use a dedicated swap device?
- walrus01 5y agoor throw money at RAM rather than have any swap space at all, I have seen used dual xeon servers (2 socket, 16-core per socket) with 512GB of RAM for sale very slightly under $2000 now, with no storage in them. By the time you add a few nice fast SSDs and a 10GbE NIC the all-in cost is looking under $2850.
- sn 5y agoWhat I have done, and will continue to recommend, is to not have all identical SSDs in a RAID - they should be a mix of part number or age.
- csdvrx 5y agoThis, so much. At a previous company, mixing manufacturers (not brands) was a policy: WD or Sandisk -> Hynix or Toshiba, etc. Funny enough, this plays very well with RAID10: with each leg made of 2 different brands, a firmware failure will not affect you.
- deleted 5y ago[deleted]
- UI_at_80x24 5y agoWhich is the same advice given for disk-based RAID arrays. It is good advice and should be followed. Just back sure they all use the same block size.
- sennight 5y agoThis is what lot numbers are for, the tracking (and in this case: leveraging) of variability between production runs. Spread multiple lots over more than one storage array and you're good to go.
- wtallis 5y agoMultiple lot numbers won't save you from firmware bugs.
- sennight 5y agoNeither will brand diversification, where they can not only independently implement their own vision of what a data eating binary blob should look like - they can also converge onto a single controller source. Wanna run down that potential failure mode on a dozen different drives, or just one?
- vernie 5y agoIsn't this the same advice with HDDs?
- etbe 5y agoSometimes that's a good idea, sometimes it might make things worse. A previous server that had a RAID-1 of SSDs was limited by SSD write speed. As the SSDs got old the write speed dropped significantly. If I had 2 servers that each had the same brand of SSDs then one server would need both SSDs replaced while if I had 2 servers with different brands then each server would need a single SSD replaced which would have been more hassle.
- walrus01 5y agoI am not sure why this person feels compelled to use btrfs raid1 when mdadm raid1 exists and works just fine at the block device level.
- trillic 5y agoExactly, especially if he is using snapshots, BTRFS can cause serious write amplification for frequently snapshotted and changed data.
- etbe 5y agoI'm not sure it makes a lot of difference unless you have hundreds of snapshots. The way changes work to BTRFS is that every change without snapshots does most of the same stuff that a change with snapshots does. If however you have many snapshots (more than 500 according to rumours, I've seen it with over 1000) performance can get really bad. If your script to make snapshot backups works and the script to remove old ones doesn't work properly then the system runs fine for months and suddenly goes ridiculously slow. A VM with a BTRFS filesystem running on a host using BTRFS gives serious write amplification. But if the real storage is SSD then it's not too bad.
- MertsA 5y agoWhat's nice is that as of Linux 5.16 BTRFS now supports ZNS (https://zonedstorage.io/ https://zonedstorage.io/) to drastically reduce write amplification compared to traditional SSDs by bypassing a lot of the FTL and moving to an interface that's similar to host managed SMR drives.
- j16sdiz 5y agoIf a bit is flipped on one of the SSD, btrfs can recover using checksum. Not sure how useful it is, but this is what btrfs/zfs people say
- tuetuopay 5y agoHappened on my NAS: had a corruption due to a bitflip, BTRFS recovered fine.
- smarx007 5y agoHow does one write 1TB per day on a server that hosts a blog? Or is it all due to the BTRFS setup?
- nightfly 5y agoI've got 73 vms running on a production hypervisor (lvm-backed storage) and using their methods it's averaging 433.1 GiB/day over the disks for the VM storage. I don't know how they're managing double that hosting a blog.
- mhh__ 5y agoMaybe they're backing up the SHA256 hash of every byte it sends or receives?
- barsonme 5y agoThat’s only 256 hashes, though…
- etbe 5y agoThe "server which runs my blog (among many other things)" probably has most of it's writes from a mail server for dozens of domains and has some commercial web sites that are very interactive and database backed.
- amarshall 5y agoInstead of using swap (on disk) at all, try deleting the swap partition and just use zram [1]. It essentially puts swap in-memory but compressed. This is likely faster than disk anyway, and avoids worrying about endurance at all. I don’t bother with swap partitions anymore. [1]: https://www.kernel.org/doc/html/latest/admin-guide/blockdev/zram.html https://www.kernel.org/doc/html/latest/admin-guide/blockdev/...
- etbe 5y agozram is a good thing. However in my investigations in 2014 "my conclusion was that swap wouldn’t be a problem as no normally operating systems that I run had swap using any significant fraction of total disk writes". During the last 8 years the amount of RAM in all my systems has increased significantly so swap is even less of an issue.
- tpetry 5y agoSure but setting up swap is still recommended. I guess some internal linux memory algorithms prefer to have some safety net? Setting up a 1GB zram swap was effective for me, it‘s not much wasted memory as servers have so much memory these days and because of compression it can fit more than 1GB.
- wahern 5y agoWhat I've seen is that under memory pressure kernel tasks trying to evict or free pages to satisfy an allocation request can race with other tasks dirtying and filling pages even faster, especially via the buffer cache. This can induce patterns of lock contention on low-level VM data structures and flushing procedures that effectively behave like a deadlock. Eventually various loop limits and lock timeouts will help unstick things, but in the worst cases the system gets caught in higher order loops and I've seen systems lock up for minutes. The systems I've seen this on never had any swap. Some of the heuristics designed to minimize pathological contention latency seem to implicitly assume that the swapping subsystem--both in its ability to help free space, and the latency it introduces when loading and evicting pages--will help mitigate the chance tasks will get caught in a tight contention loop. IOW, the I/O latency of swap effectively induces back pressure on load, helping operations freeing pages to progress faster than operations consuming pages. (Predictably, the faster your swap, the less well this works. When people began putting swap on SSDs, heuristics had to be retuned.) Arguably the root of the problem is the legacy of overcommit. Even though it can be nominally disabled, many aspects of the kernel were designed with the notion that the only direction to move under memory pressure is forward, relying on the promise of the OOM killer eventually freeing up enough memory to maintain forward progress on the current operation, rather than unwinding and returning a failure condition. The dynamic seems similar to buffer bloat.
- smackeyacky 5y agoWhen I set up Debian on this machine (an old Proliant ML330G6 with 96Gb of memory) I did think "what do you do with swap?". In the before times, when I was a SunOS/Solaris admin in the 1990s we used to have a rule of thumb handed down from the boss that swap should be 2x physical memory but in the intervening years this seems like massive overkill. The Debian installer gave it 1Gb of swap as a default. Probably not enough to get you into trouble if something decided to go truly rogue but you can always hit the off switch. I did toy with the idea of giving it 200Gb of swap but watching the process monitor with the dev environments running and a few VMs I struggle to break 64Gb of memory usage. It is running hybrid disks and not SSDs though. If I ever bother to put SSDs in it (unlikely) I guess I'll think about it again but swap seems almost irrelevent on a workstation now. On a server...would probably have to get a new rule of thumb.
- LeoPanthera 5y agoDon't you still need at least as much swap as RAM if you want to use hibernate? It used to be that hibernate mode simply dumped the RAM into the swap. Does it still work that way?
- smackeyacky 5y agoI don't know. The idea of hibernating a deskside server that has no sleep capability never crossed my mind. Laptops would be a different story I guess.
- rhn_mk1 5y agoIt still works this way on Linux. It's better to overprovision the swap though, if you actually intend to use it as swap at runtime. Otherwise there may not be enough free space for the hibernation.
- Dylan16807 5y agoSwap is impressively workload-dependent. I'm typing this on a Windows system with 32GB of memory and an idle swap file. What I currently have open would probably fit into 16GB. But at the same time, I have over 30GB swapped out right now. 2x physical memory is about right for my desktop, when accounting for web browsers and electron apps and other programs the soak up gobs of memory and leave that data alone for days or weeks on end. Then there's the committed but unused memory on top of that, representing another 20GB of page file I need to have just to make it happy. And then I have linux servers with 1-4 gigabytes of swap that use either none at all or a couple hundred megabytes of it.
- ksec 5y agoOn my MacBook Pro 2015 I am averaging about 100GB+ per day simply because every time I trip the MagSafe Cable macOS will start paging or doing lots of write on Kernel_Task. Generally speaking I dont think macOS as a whole dont put much thought into SSD Endurance. That is a problem because you cant swap your SSD on Mac.
- dannyw 5y agoThis sells Apple more logic board replacements or new Macs.
- londons_explore 5y agoAre there many reports of SSD failures on M1 Mac's? I suspect Apple might have tweaked SSD firmware to behave better in various ways and increase endurance or performance near the end of the ssd's life, because if they hadn't I think we'd be seeing a flurry of people complaining about SSD failures by now.
- fomine3 5y agoSeagate's consumer SSDs are good about TBW rating. Check them, if workload is write heavy but not worth enterprise drives.
- dannyw 5y agoRated TBW is not actual endurance. My 980 Pro 1TB has about 3 PBW (about 5 times its rated warranty) and it still operates great.
- etbe 5y agoMost things last longer than their warranty, if even 5% of an item fail to reach the warranty then it's a disaster for the manufacturer. Planning for replacement is probably a good idea when you are 5* the warranty.
- dannyw 5y agoI plan for replacements by always assuming storage to be unreliable, and using RAID (plus backups).
- rasz 5y agopower it down for a week and recheck data integrity
- otterpro 5y agoI intentionally use spinning hard drive on any data that are transient or have heavy write operations: swap, logs (ie /var/log/), etc. The spinning hard drive should last a long time and they are fairly cheap, while the performant and main data stays on NVMe or SSD.
- etbe 5y agoIf you use the - option in syslogd configuration for non-synchronous writes then /var/log will be only a tiny portion of writes. If you have a mail server then the many log messages about a message delivery will have hardly any write load compared to all the synchronous writes to the mail queue and mail store. The only time I've seen a significant part of write load coming from /var/log is when the systemd-journald log database or a database server replication log or rollback log is there.