4 ms·
As others have pointed out using consumer grade SSD drives for a database is a bad idea. Databases are going to do a lot of reads and writes so that's going to
by raiyu 8y ago
As others have pointed out using consumer grade SSD drives for a database is a bad idea. Databases are going to do a lot of reads and writes so that's going to lead to problems down the road.
The second issue, is that since this was setup in a RAID I'm assuming that all of the drives were purchased at the same time when it was originally setup which means depending on the supplier you are probably going to receive drives from the same batch of production. Because of striping with RAID if you have some fault in a batch and multiple drives from the same batch then you can have failures on multiple drives that occur at approximately the same time.
The simplest solution is of course to buy enterprise grade drives but the obvious issue there is the price is much higher. Especially if you are going to be purchasing the largest drives as you always pay a premium for max capacity drives and the price jump on large drives is not insignificant.
When we first built out our SSD cloud for DigitalOcean at the end of 2012 we went with consumer drives because we were bootstrapped and SSDs were also significantly more expensive so we couldn't justify the price jump without knowing if we had product market fit.
But rather quickly we ran into a whole host of issues. Another common issue is that performance can degrade when consumer drives get to near full disk capacity so you can't really use all of the available space if you plan to do a significant number of reads and writes, which is pretty much what a database server is designed for.
That was the first upgrade we did after raising money, which is switching to enterprise grade drives, unfortunately you have to pay the premium but it really reduces problems significantly.
- fabioborellini 8y agoI do believe your experience, but started wondering what makes an enterprise drive more reliable than a consumer-targeted one. What's preventing the manufacturers from selling the same drive with two different names and prices? Reputation?
- mirimir 8y agoOne difference is that enterprise drives have lots more spare capacity. Like >>100%.
- u02sgb 8y agoThis can be replicated in consumer drives by underprovisioning.
- mirimir 8y agoOK, fair point. Although I don't know enough about SSDs to know whether there's more to it. But what else is different?
- u02sgb 8y agoI think mainly warranty, although other posters are saying enterprise drives have capacitors to help with writing during power loss. I'd have thought a good battery backed raid controller would handle this but "enterprise"="belt and braces" so I can see why a lot of businesses pay the extra.
- mirimir 8y agoRight. Capacitors. Especially for write cache, I think. But yeah, battery-backed RAID plus UPS would be good enough for most, I guess.
- tinus_hn 8y agoSo the enterprise drives perform as specified and the consumer drives don’t?
- candiodari 8y agoIf you actually read the specification on consumer drives you'll see they're "guaranteed" for like 100 rewrites and you don't have any right to anything if they fail. Which should be a hint as to what's coming.
- u02sgb 8y agoSorry, late to reply! The warranty differences are things like with enterprise they'll send you a new one as soon as the SMART data suggests it's going to fail. Consumer is usually you post it back and they send you a new one.
- freddie_mercury 8y agoIt would quickly become very obvious. And then they'd be sued for a lot of money.
- xfitm3 8y agoEnterprise class disks have significant behavioral differences when dealing with a bad sector, among other differences. Intel has a good paper on this subject. See 3.2.1: https://www.intel.com/content/dam/support/us/en/documents/server-products/Enterprise_vs_Desktop_HDDs_2.0.pdf#page10 https://www.intel.com/content/dam/support/us/en/documents/se...
- dgquintas 8y agoNaive question, but is using old school spinning HDs completely out of the question? In other words, can they play a role in a modern deployment?
- jnwatson 8y agoSpinning hard drives are the tape drives of the modern era. Boot, logs, backups, but no databases.
- u02sgb 8y agoRandom reads are an order of magnitude slower so for fast access there's a notable difference. Still a role for hard drives with sequential reads/writes, such as the regular backups they should have been taking! Slower in all areas but cheaper for the same space.
- raiyu 8y agoUnfortunately for databases under a heavy load it's out of the question. You can have a server pegged at 100% CPU waiting on disk with spinning disks, and without changing anything else on the server RAM or CPU, and just replacing the drives with SSD you can quickly see that load drop down to 10%.
- zepolen 8y agoDisagree, Consumer grade hardware is a fantastic idea because it's so bad it forces you to write robust software. Enterprise hardware is 3x the cost of consumer hardware, but on average it's at the most twice as reliable. If your software robustness and redundancy is high enough, the total TCO will be lower and no less reliable. The real wtf with CamelCamelCamel is not that they were running consumer drives, it was they were running 3 replicas on consumer drives with what seemed to be a too heavy workload.
- xfitm3 8y agoProper infrastructure is expensive, along with competent ops. Clearly not prioritized. If infra was neglected I would wager security is too.
- zepolen 8y agoOh boy I got triggered, wall of text incoming. Competent ops being expensive wasn't part of the discussion, Good people are expensive period. In the long term, competent people provide better value. Competent ops would figure out to provide the same reliability guarantees at a lower price. Security likewise, nothing to do with it. That's just using circular logic to say using commodity hardware = bad security (wtf!?). Sofware and by extension infrastructure gets expensive because of incompetent dev/ops, who are too incompetent to implement efficient systems and use a 'hardware is cheap, throw hardware at the problem' mentality in ops. This especially shows today with the fact that we have have mobile phones with 1000x the processing power of desktops 20 years ago that are sluggish when running a simple TODO app. LET THAT SINK IN. Those same 20 year old computers running the first pentiums were running Quake 1(!). Back to ops, Google before they were "Google" were using custom built commodity hardware using a 'nodes die, deal with it' approach to software development. Obviously this pattern is a win with the popularity of k8s today / ephemeral nodes but it requires a more complicated development process. This extends to incompetent ops as well. Who are outsourcing all their work to the cloud, paying a crazy premium for it, calling it a day, whereas the competent guys are able to do it properly. Closer to the subject, Backblaze engineered a from scratch design to create a highly reliable/available/secure system based on non enterprise class storage that lets the offer storage at a fraction of existing solutions.