9 ms·
1-2 year SSD wear on build boxes has been minimal
- kryptiskt 12y ago"At some point in the next few years we are going to start getting HDD failures on our blade server. It has ~30 hard drives plugged into it after all (and ~12 SSDs as well). When that begins to happen I will probably do a wholesale replacement of all HDDs with SSDs. Once that is done I really expect failure rates to drop to virtually nil for the next ~20-30 years. And the blade server is so ridiculously fast that we probably won't have any good reason to replace it for at least a decade, or ever (though perhaps we will add a second awesome blade server at some point, who knows?)." That's taking the long view. :-)
- Jolijn 12y agoIs hardware (motherboard, CPU, memory) that good nowadays that one can expect it to last 30 years. I don't think it's designed with that kind of lifetime in mind.
- dperny 12y agoI mean, this is server hardware. One of the major differences between server hardware and desktop hardware is build quality. I've got a ten-year-old 1/2U rack server sitting in a closet that I bought for pennies at a surplus auction that still runs great.
- deleted 12y ago[deleted]
- jethro_tell 12y agoServer hardware is usually obsolete long before it stops working.
- mahyarm 12y agoBut would it really last 30 years with such things as lead-free solder?
- moe 12y agoWe will probably have to wait 30 years to really know. FWIW, lots of hardware from ~30 years ago still works. I have a 27 year old Amiga500 that still boots fine (many of the floppy disks have become unreadable, though). You can buy fully working vintage computers much older than that on eBay.
- vacri 12y ago30 years of constant/daily use is different from a few years of use, then pulling out of mothballs as a curio every now and then.
- sokoloff 12y agoA server that's 10 years old now was likely made with leaded solder.
- eitland 12y agoI've seen DL380 G5 giving in a few years ago already. (I still love those machines except they seem to boot slower each generation.)
- sliverstorm 12y agoTo my knowledge, the most common guarantee target is 5 years. Equipment can last much longer or much shorter, both as a function of chance and of workload. 30 years would be exceptional if the machine spent its life at the 5-year-target load.
- beachstartup 12y agowe don't exactly run the biggest operation, but in our experience the most common failure items in thousands-of-years-of-cumulative-uptime is network interface cards (or on-motherboard network interfaces) and hdd's. RAID controllers fail left and right. we keep tons of spares around. ssd's fail few and far between, cpus basically do not fail, and memory can go bad but it's exceedingly rare and easy to fix. psu's fail but are easy to fix in modern computers as well (slide-out, redundant, etc.) having said all that, heat is the primary killer of hardware. if you run a lot of equipment in a dense environment, get a laser thermometer and find your hot spots and fix them with some industrial fans or move your gear around. once your stuff gets hot anything can fail in weird and mysterious ways.
- Matumio 12y agoHow do network cards fail? Simply as if you cut the link? Or can you see error counters increasing and sporadic frame loss that gets worse over time?
- danpat 12y agoDepends on which bit fails, but increases in packet loss are a common early symptom of small components no longer acting within their specs. Network cards are subject to lots of signal phenomena that are rare inside the chassis. Long cables are pretty good antennas for certain types of RF signals, so there are all kinds of electrical noises, induced power spikes and other miscellaneous garbage that the network card has to tolerate. Well-shielded cables can help protect the card, but it's definitely one interface that's subject to a bit more electrical abuse than the rest. Components that have been stressed beyond their tolerances a few times can result in things like signal filters having a lower noise threshold, which makes it harder for the card to pick out the signal from the noise, which leads to more packet loss. After enough abuse, the threshold drops below the usable level and communications halt. There are lots of factors involved, such as shielding, proximity to nearby radiators, bend radius in cables, cable length, temperature, etc, etc. Whenever I delve into this world, I'm often amazed that anything works at all.
- beachstartup 12y ago
- rasz_pl 12y agoNo, thats naive view. SSDs do fail, and they tend to do it without any warning. Author thinks he discovered magic. Paradoxically SSDs from two years ago have better endurance than brand new drives now, and a lot better than SSDs in 5 years (unless something big happens).
- vacri 12y agoIt reminds me of the "burnable CDs will keep your data for 30 years!" phase we went through. Turns out that's not so reliable after all.
- dillondf 12y agoNo, nobody is magicking up any fairy dust here. The author (which is me) is looking at things realistically. My maintenance costs just from a time perspective are already radically lower. In anycase, so as long as the SSD is powered and thus able to rewrite blocks as bits start to go bad (creating some wear in the process but not a whole lot compared to normal operation), the actual data will be retained for significantly longer than 10 years. Unpowered data retention (where there is nothing there checking for and rewriting blocks whos bits start to go bad) depends on cell wear. I didn't pull up the chip specs but my recollection is that it is 10 years for new cells and 1-2 years for a relatively worn cell. Depends on temperature of course. That means I can pull a SSD and shelve it without too much worry for a relatively long time, something that cannot be said for any hard drive. CD's corrode from the inside out, primarily due to air trapped in the holes I believe. Even though it is laser burned there is no real care taken in constructing the plastic sides or metal film to keep up air so when it gets vaporized the oxygen sticks around and starts corroding the metal. Or the plastic gets fuzzy from UV exposure or age. Totally different ballgame. Hard drives have even worse problems if left unpowered. I used to pull backup drives and shelve them, but their life spans are radically lowered if left unpowered on a shelf for 6 months and then replugged into a machine at a later time. I'm sure some HDD expert can say why. So I stopped doing that. Now the backup drives stay powered on 24x7. Hard drives seem to have a limited life span whether you use them or not. I have tons of old HDDs sitting around, some well over 20 years old in fact. If I plug an old HDD in I can sometimes read the data off before the whole thing turns into a brick. I mostly use them to test disk driver error handling. SSDs left sitting around can always be reformatted (i.e. just using TRIM to clean out the whole thing then start writing to it fresh). Their wear limit is based on rewrite cycles rather than how long they've been sitting around. HDDs are physically destroyed, you don't get a fresh start by reformatting an old HDD. -Matt
- justizin 12y agoLook. :) Yeah, the build machine churns a lot. But that work should be primarily done by the FS cache, by buffers. Yes, it's going to write out those small files, but if DragonflyBSD has any kind of respectable kernel, though should be a solid curve, not lots of bursts. I would love if my old colleague Przemek from Wikia would talk about the SSD wear on our MySQL servers which had about 100k-200k databases per shard. We wore the _fuck_ out of some SSDs. You should replace your HDDs with SSDs, though, for a number of reasons, and take the long view, as kryptistk noted the OP is doing. Really compare the cost of SAS 15k drives and Intel 320s or 530s. But I think in his place, you can take the words of the inimitable Mr. Bergman: https://www.youtube.com/watch?v=H7PJ1oeEyGg Stop wasting your life. But don't expect a machine that does lots of random IO, like a database, to have 1-2% SSD wear after two years. It might not last two years. If it does, use it more. Aren't you making money with these drives? ;)
- engendered 12y agoBut don't expect a machine that does lots of random IO, like a database, to have 1-2% SSD wear after two years. It rather depends upon the database, no? A 512GB Samsung 850 Pro would last 29 years with 100GB of writes a day in a horrendous, worst case 3x write amplification scenario. Very, very few databases write more than 100GB of data a day, and most are magnitudes less.
- dillondf 12y agoTo be clear here, what the poster means is that piecemeal database writes of, say, 128-byte records can cause a huge amount of write amplification, so 100GB/day worth of database writes can end up being 1000GB/day worth of flash rewrites. This issue largely goes away if the database backend appends replacement records rather then rewriting them in place, and uses the index to point to the new copy of the record. At that point the battery-backed ram the RAID system has combined with the appends results in much lower write amplification and a SSD could probably handle it. -Matt
- engendered 12y ago
- unluckier 12y agoNone of those drives show evidence of regularly-scheduled self tests. They must not really care about them.
- wtallis 12y agoWhat sort of self-testing do you think is needed?
- unluckier 12y agoDaily short tests and weekly long tests seems reasonable. These drives show that they've gone thousands of hours since the last test.
- dillondf 12y agoIt's really unclear whether explicitly initiated tests actually help for a SSD. The SSD has its own internal mechanisms to scan the flash chips which (I presume) is unrelated to SMART, since they are required for normal operation... primarily detecting weak cells before the bits actually go bad. Whole chips can go bad out of the box but after an SSD has been running for a few months there isn't much left that can go south other than normal wear or a firmware failure. firmware issues dominated SSD problems in the early years. Those issues are far less prevalent today though Samsung got its ass bitten by not adding enough redundancy and having data go bad on some of its newer offerings. Strictly a firmware issue. Which is another way of saying that one should always buy the not-so-bleeding edge technologies that have had the kinks worked out of them rather than the bleeding-edge-technologies that haven't. If it starts to bite me I may change my tune. But until that happens I put it in the wasted-cycled category. -Matt
- wtallis 12y agoTests of what? The ability to read and write data without corruption? Ordinary usage with wear leveling takes care of that, and these SSDs are definitely being used full-time.
- dillondf 12y agoThere are certainly workloads that will wear out a SSD, random database writes being the most common. But that is only a very small portion of the storage ecosystem, not to mention that there are plenty of ways to retool SQL backends to not do random writes any more, particularly since it doesn't gain you anything on a modern copy-on-write style filesystem verses indexing the new record yourself as an append. So I expect this particular issue will take care of itself in the future. It's a matter of not blindly using someones database backend and expecting it to be nice. The vast majority of information stored these days is write-once-read-never, followed by write-once-read-occasionally. I expected our developer box which is chock full of uncompressed crash dumps and many, many copies of the source tree in various states to have more wear on it than it did, but after thinking about it a bit I realized that most of that data was write-once-read-hardly-at-all. In terms of hardware life, for servers there are only a few things which might cut short a computer's life, otherwise it would easily last 50 years. (1) Electrolytic capacitors. (2) Any spinning media or low frequency transformers. (3) On-mobo flash or e2 that cannot be updated. (1) Electrolytic capacitors have largely disappeared from motherboards in favor of solid caps which, if not over-volted, should last 30 years. Electrolytic caps are not sealed well and evaporate over time, as well as slowly burn holes in the insulator. They generally go bad 10-30 years later depending on how much you undervolt them (the more you undervolt, the longer they last). Even so I still have boards with 30+ year old electrolytics in them that work. (2) Spinning media obviously has a limited life. That's what we are getting rid of now :-). Low frequency transformers have mostly gone away. Transformers in general... anything with windings that is, have a limited life due to the wire insulation breaking down over time but most modern use cases in a computer (if there are any at all) likely have huge errors of margin. (3) Firmware stored in E2 and flash, or OTP eprom, rather than fuse-based proms, will become corrupt over time. 10 years is a minimum life, 20-30 years is more common. It depends on a number of factors. Other than that there isn't much left that can go bad. All motherboards these days have micro coatings which effectively seal even the chip leads, so corrosion isn't as big a factor as it was 20 years ago. The actual chip logic basically will not fail and since the high-speed clocks on the whole mobo can be controlled, so aging effects which degrade junction performance for most of the chip can be mitigated. I suppose an ethernet port might go bad if it gets hit by lightning but I've never had a mobo ethernet go bad in my life. Switch ethernet ports going bad is usually just due to poor parts selection or overvolting which would not be present in a colocation facility or machine room. In anycase, there is no reason under the sun that a modern computer with a SSD wouldn't last 30 years with only fan, real-time clock battery, and PSU replacements. -Matt
- kabdib 12y agoNot sure why you'd want to buy 10-year capable storage for a server. I buy for reliability. If I can plug something in and just forget about it (except for patches) for 3-4 years, we're done and whatever new thing I can buy will pay for itself in power savings. I'm happy that an SSD will last that long, but it's not something I worry about. I have had to rebuild servers because of abject SSD failure, and will no longer buy those brands. Failure seems to be quite highly correlated with trying to save a buck by getting non-top-tier drives (whereupon it's me, or someone like me, picking up the pieces when I could be writing code. Screw that). If a drive was happily at 90% wear after two years, I'd just provision more of the same drive. Yay! :-)
- dillondf 12y agoBasically only buy SSD brands who either are chip fabs or have a relationship with a single chip fab. So. Intel, Crucial, Samsung, maybe one or two others. And that's it. And frankly I have to say that only Intel and Crucial have been really proactive about engineered fixes for failure cases. Never buy SSDs from second-tier vendors such as Kingston who always use the cheapest third-party flash chips they can find. There are literally dozens of those sorts of vendors. Hundreds, even. -Matt
- gonzo 12y agoMicron is the name behind Crucial. Kingston has gotten very good (and they used to suck) at least in the eMMC space (not quite a SSD, but close).
- rasz_pl 12y ago>Kingston has gotten very good ...at ninja swapping components under same model name, google V300
- kabdib 12y agoCrucial are the drives I've been ripping out. Enterprise class. They just died after 2 years of service and not really all that much wear, won't talk to the outside world anymore. I nailed one to the wall of our IT lab as a warning not to buy any more of them.
- ChuckMcM 12y agoLove this quote: "This is the first time I've actually contemplated NOT replacing production hardware any time soon." There are two things that benefit from the turnover of machines, one is that Intel stays profitable, the other is that hardware standards have a chance to evolve. The move from ISA->PCI->VESA->AGP->PCIe on video cards would not have been possible had people been holding on to their machines for 3 - 5 years before buying new ones.
- diydsp 12y ago> ISA->PCI->VESA->AGP->PCIe That's one way to look at it. Another is that we might have gone ISA->PCI->PCIe in a shorter overall timeframe b/c there were no distractions to get short-term stuff to market.
- ChuckMcM 12y agoTrue but the relative market size growth has an impact. Had we skipped VESA and AGP for example, there would thousands more ISA slots (longer time in market) and so the next generation, PCI in your example, has to burn more cash getting "into" the space. The obvious principle here is that innovation happens more rapidly in a market where their his a large demand for improvement and a low friction for upgrades. Pull back on either of those and it slows down the rate of innovation.
- johnwfinigan 12y agoRunning a small datacenter for the last few years, I can totally identify with that quote. On the storage side, I can't help but wonder if it's partially due to incumbent vendors pushing flash as a high markup premium product. I suspect that I am not the only one who has thought, "Well, why replace this array with more already-obsolete magnetic disk that won't be any faster? Wait a year or three, flash markups will come down. Maintenance renewal is cheap." Now, I realize that cheap flash is dangerous to incumbent pricing structures, so $100K for an all flash tray makes sense when it can replace 10 trays of $30K disk. So maybe they've seen no other choice, business model wise. Maybe it even makes sense to milk a once-per-generation technology disruption for all it's worth for a couple of years before commoditizing it. But it has kept me happy with what I've already got, since the pricing of the alternative was, for my use cases, hilarious. As opposed to, say, this is somewhat more expensive but we can justify it.
- gojomo 12y agoSince it wasn't immediately clear to me how the percentages were pulled from the smartctl output, I'll note that "Perc_Rated_Life_Used" is the relevant readout. (Some online sources mention "Wear_Leveling_Count", and even misreport that as a percentage – but in fact that seems to be an absolute count of the number of times each single block has been rewritten. The percentage is presumably this Wear_Leveling_Count divided by the rated number of cycles.)
- jcampbell1 12y agoWear failure on SSDs is often a silly thing to worry about. The number of write-out cycles is roughly 10,000 and a modern spinning disk takes like 4 hours to write out, so it would take like 5 years of continuous writing to see an advantage for traditional HDs. In practice, you only have to worry about it, if the workload could never be considered on a traditional HD. When faced with the choice between a Fiesta and a Ferrari, there are many reasons to choose one over the other, but it is ridiculous to say "I picked the Fiesta because the Ferrari has a known tire problem when going 200+ MPH".
- dillondf 12y agoDurability has steadily decreased as flash density has increased. 10000, then 5000, then 2000 for standard MLC parts over the last few years as densities have increased. I'm not sure what Samsung's new 3D process is. At the same time, the voltage regulation and comparators used on-chip has gotten a lot better, making it easier to detect leaky cells so reliability has significantly improved for the erase cycles the flash does have. The original Intel 40G SSDs could handle an average of 10000 erase cycles (for each block of course), giving the 40G SSD around a 200TB minimum life if you divide it out and then divide by 2 for safety. (Intel spec'd 20TB or 40TB or something like that, 400TB @ 10000 erase cycles, divide by 2 gives you ~200TB or so). A modern 512G Crucial SSD sits somewhere around a 2000 erase cycle durability, or around 512TB of relatively reliable wear (1PB / 2 for safety). I would not necessarily trust an SSD all the way to the point where it says it is completely worn out, I would likely replace it well before that point or when the Hardware_ECC_Recovered counter starts to look scary. But I would certainly trust it at least through the 50% mark on the wear status. Remember that shelf life degrades as wear increases. I don't know what that curve looks like but we can at least assume somewhere in the ballpark of ~10 years new, and ~5 years at 50% wear. Below ~5 years and I would start to worry (but that's still better than a shelved HDD which can go bad in 6 months and is unlikely to last more than a year or two shelved). -Matt
- mrmondo 12y agoI'm currently working on a project to replace our older SANs with SSD-only storage servers - I've performed a few POCs with great results and am now documenting the build as I go. Not only is it going to give us some much needed IOP/s it's also going to save us hundreds of thousands of dollars on storage over the next 3 years. If you're interested take a look at: http://smcleod.net/building-a-high-performance-ssd-san http://smcleod.net/building-a-high-performance-ssd-san