14 ms·
Farming hard drives: 2 years and $1M later
- pcunite 13y agoI personally (for home use) held off buying drives for two years, found out I did not really need so much space and discovered SSDs would do the trick.
- dublinben 13y agoSame here. I bought my last 2 TB for $80 shortly before the crisis, and was able to scrape by at 90% capacity until 4 TB drives became cheap enough.
- carlmcqueen 13y agoI've been trying to reduce my data hoarding. I had a larger back-up hard drive go down and take a lot of data with it. I felt sick knowing I potentially lost years of funny images, videos, music, writings, etc. What I found is I had back ups of back ups of the things I actually cared about and the things I lost I hadn't truly looked at in a long time. A lot of the music I hadn't listened to in years. I have found that I can fit on small SSD drives and as a result of the things I do wish I hadn't lost, I'm just keeping up with the practice of keeping multiple back ups. TL:DR: I have made an effort to keep a smaller data footprint instead of hoarding.
- alexkus 13y agoIf the only instance of a thing is on your "back-up" drive then it isn't a back-up.
- mikeash 13y agoData that isn't backed up (as in a second copy) doesn't really exist. Likewise: data that isn't backed up off site isn't really backed up.
- alexkus 13y agoUntested backups don't exist either. There are several horror stories of people going to their offsite tape backups to restore some lost data only to find that their backups didn't work due to a malfunctioning tape drive. Other stories about encrypted backups where the decryption key is lost (backup data is encrypted with a randomly chosen "session" key, session key is encrypted using a public key, no-one could find the appropriate private key to decrypt the session key. etc...
- tehwalrus 13y agoI heard about a private school that backed up its fee database (MS Access) to a USB stick every night - in fact what the accountant was doing was backing up the desktop shortcut to the database. In the event of the failure that revealed the problem, they lost about 7 months worth of data (which was the time since the computer had last been connected to the network, and had synced via Active Directory, or whatever they call it.)
- brazzy 13y ago> Data that isn't backed up (as in a second copy) doesn't really exist. An IMO more realistic statement: Data that is not backed up is implicitly classified as "nice to have" - in many cases a perfectly rational choice, if done consciously. > Likewise: data that isn't backed up off site isn't really backed up. You may as well argue that no data is ever "really" backed up because there is always the possibility of simultaneous failure. It's always a choice of how much risk you're willing to accept vs. how much money and effort it would cost to reduce it further.
- mikeash 13y agoYep. 1 copy of data is not enough, because it can be lost. Now, assuming that N copies of data is not enough, it's easy to see that N+1 copies is also not enough, because that extra copy can be lost, thus reducing to N copies, which is not enough. This proves by induction that no matter how many backups you have, it's not enough.
- TheLoneWolfling 13y agoAgreed. I currently have two external drives: an archive drive and a backup drive: both 2TB. (Most things on the backup drive are compressed and deduped, which is why I can fit everything.)
- post_break 13y agoWhy didn't they touch on why the Hitachi drives weren't the same quality as the other brands with this statement: "Switching hard drive manufacturers from Hitachi to Seagate and Western Digital. The assumption here is that all drives are the same and as we’ve learned that’s not really the case." Where is the footnote. Why aren't they the same?
- tachion 13y agoIt doesnt implicate that Hitachi were not the same quality as others. I undestand that the main reason to switch was the availability to price ratio, nothing else. What I think they meant when talking about drives to be different is the variety of sizes you get when purchasing 'a 3TB' drive.
- ewams 13y agoIn the parent article they said they switched because of cost: "When Hitachi drives got too expensive we switched to Seagate and Western Digital. When internal drives got too expensive we turned to external drives. When hard drive availability became an issue we turned to Drive Farming." From their article on v2 they said: "The Western Digital and Seagate equivalents we tested saw much higher rates of popping out of RAID arrays and drive failure. Even the Western Digital Enterprise Hard Drives had the same high failure rates. The Hitachi drives, on the other hand, perform wonderfully." http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v2-0revealing-more-secrets/ http://blog.backblaze.com/2011/07/20/petabytes-on-a-budget-v... There are other articles where they mention that they do testing and some brands perform better than others.
- recuter 13y agoAren't they big enough at this point to cut deals directly with the vendors? 10,000 drives at a discount, that sort of thing.
- SloopJon 13y agoIn the previous story, user brianwski of Backblaze said, "We have been told that until we buy blocks of 10,000 drives, we cannot deal directly with the drive manufacturers. We're not there yet." https://news.ycombinator.com/item?id=4633213 https://news.ycombinator.com/item?id=4633213
- grecy 13y agoCool. I know a few Backblaze people are reading/posting in these comments, so I'll ask here: Do you guys know the approximate price of when you can purchase them direct in 10,000 blocks?
- ChuckMcM 13y agoI'm not at Backblaze, but when I was at Google I did know what Google was paying for drives direct from the manufacturer and it was not as much lower than the distributor price as you might expect. Margins are pretty thin in the disk drive business.
- atYevP 13y agoYev from Backblaze here -> It really depends on the type of drive. One of the reasons we don't currently buy in chunks of 10,000 is that we like to forecast how many drives we'll need and not keep a bunch of inventory (if you look at the graphs, you'll see that prices used to drop a little bit each month, so keeping excess inventory was like leaving money on the table).
- brianwski 13y agobrianwski from Backblaze here again - this reminds me to check once again on this info. We heard a rumor that the number of drives might be lower because we only buy the largest and most expensive drive. The 10,000 number might be a rule of thumb for a retailer like Radio Shack that stocks a blend of different drive sizes. With all that said, EVERY MONTH we bid 20 different suppliers against each other and choose the lowest price. For goodness sake, anybody on earth who wants free money just has to undercut the other suppliers by 1 penny and you'll win our business. Always open to new suppliers, please, take our money!!
- ISL 13y agoThe original "Drive Farming" link is at least as interesting as the new article: http://blog.backblaze.com/2012/10/09/backblaze_drive_farming/ http://blog.backblaze.com/2012/10/09/backblaze_drive_farming...
- SloopJon 13y agoAnd here's the discussion of that story: https://news.ycombinator.com/item?id=4631027 https://news.ycombinator.com/item?id=4631027 I browsed the comments with two things in mind: was anyone uncomfortable with shucking and farming on this scale, and were the drives of equivalent quality? Regarding the latter, user rsync of rsync.net claimed that "the drives inside these enclosures are the worst spec'd, highest failure rate drives." User budmang of Backblaze, however, said, "The drives were literally the same inside the external enclosures." User atYevP of Backblaze posted a similar comment.
- jotm 13y agoHigh failure rates of external drives can also be attributed to the USB controllers - they're better than their IDE predecessors (which failed a lot, sometimes taking the drives with them - only low level utilities could revive them). Otherwise, I don't see how or why the hard drive manufacturers would bother to send the worst batches (which aren't that much worse) to external hard drive assemblers.
- sp332 13y agoIt could also be that they shut down more frequently. More stop-start cycles mean more wear on the moving parts of the disk.
- bentcorner 13y agoIsn't that a feature of the USB enclosure? (i.e., not an issue once a drive has been "shucked")
- ZoFreX 13y ago
- michaelt 13y agoThe extrapolations in the article were pretty optimistic to begin with. I mean, when you look at the dotted trendline in this graph: http://blog.backblaze.com/wp-content/uploads/2013/11/blog-cost-trend-1.jpg http://blog.backblaze.com/wp-content/uploads/2013/11/blog-co... you'd think by 2016 drives would cost $0 per gigabyte and by 2020, drive companies would be paying you to take drives off them.
- minikites 13y agoJust like the old razor blades graph: http://imgur.com/IBjL9tT http://imgur.com/IBjL9tT
- sitharus 13y agoIt scares me that we now have 6 blade razors: http://www.dorcousa.com/sxa1000-6-blade-system-for-men/ http://www.dorcousa.com/sxa1000-6-blade-system-for-men/ I'll stick with a single blade.
- Amadou 13y agoI'll just leave this right here: http://www.theonion.com/articles/fuck-everything-were-doing-five-blades,11056/ http://www.theonion.com/articles/fuck-everything-were-doing-...
- jleader 13y agoThat reminds me of Gordon Moore in a commencement address (at Caltech in 2001?) quoting the number of transistors the industry was producing annually per resident of the US (and how that number was growing), and telling the graduating engineers their job was to figure out what to do with all those transistors. The answers turned out to include things like multi-gigabyte smartphones. So who's going to figure out what to do with our future profusion of razor blades?
- atYevP 13y agoI believe my Dollar Shave Club Executive razors have 6 blades. So smooth.
- shmulkey18 13y agoI wonder why external drives are cheaper than internal drives. One would think that they would be more expensive, since the cost to the manufacturer of an external drive includes not only the drive itself but the adapter contained in the enclosure, the enclosure itself, cables, possibly a power supply, more elaborate packaging and the additional labor required to assemble the external drive. I assume that this consumer cost differential arises from differences in the markets for internal and external drives, but naively it seems like the internal drive market would be more price-sensitive and thus lower margin. Does anyone know why external drives are typically cheaper than internal drives?
- mangotree 13y agoMaybe, with the advent of laptops becoming more popular than desktops, there is just a bigger market for them. Cost is usually due to demand constraints.
- sliverstorm 13y agoI doubt it. Server farms consume an unreal quantity of drives. Desktops consume 1 drive per person.
- Retric 13y agoServer farms store data for actual people. So sit back and think about how much unique data you have on servers somewhere. Add it up and I dont't think the total data storage at all data centers everywhere in the US add up to more that 20GB per person in the US as we are just not that interesting even at 5x redundancy your still at 1/30th a drive per person. Don't forget at 300 million people that would still be 10 Million drives which is a lot of servers. PS: The NSA could be recording every phone call ever made and it's still not all that much actual data.
- sliverstorm 13y agoThe dropbox model is not the only place for server farms. How about render farms & their associated data stores. How about Google Maps & the associated scads of satellite imagery. You could be right, of course, but I'm not sure we can say with certainty.
- tedivm 13y agoWhy do they need hard drives when they don't even back up data? Seriously, you can find reports of this all over the internet. It's on reddit, twitter and elsewhere- these guys do not back files up properly, and their systems are so shoddy that they'll report they have files even when they don't. My girlfriend brought her computer in for repair and they changed the hard drive. She specifically checked before she left that the backups were set, and the dashboard said they were. When we went to restore, however, none of the files could be. We contacted support and they kept insisting we did something wrong. Finally they admitted that they had erased the files (yes, actually erased them). Due to a "bug" their dashboard reported we still had the files, even though they were gone. They did everything they could to hide their mistake. They lied to people on twitter about what I was saying (since the only way to get real support was to hound their twitter account). They lied about all sorts of things. I had to take screen shots of everything and post it online before they'd even admit to screwing up. Even at that point their dashboard was still saying it had all of these files, when it in fact had nothing. Backblaze is a completely untrustworthy company. If they spent half as much effort on backing up files as they do on marketing that may be a different case. Anyone who uses them for backups might as well not have backups at all.
- kalleboo 13y agoThe second rule of backups is, always test your backups. Never trust a cloud service, backup software or tape drive status. Test a restore periodically, or you can't be sure you're backed up at all.
- bruceboughton 13y agoVery true, but victim blaming gets us nowhere.
- kalleboo 13y agoAbsolutely, Backblaze are at fault here. Having a dashboard lie about files being there is a really, really serious bug. But computers always lie. Your disk says the file is there, you double click it, and it starts making noises and the file doesn't open. Backup systems are the same. It says the files are safe, but unless you test it, they may not be. In the end, all I want to say is, I wish ZFS was free software :(
- alayne 13y agoThis seems like something that should be part of their business instead of waiting for catastrophes.
- dublinben 13y agoSince the price of storage is consistently declining, buying any more than necessary is wasteful over the long term.
- clarky07 13y agoyeah, but it seems like you'd give yourself more than a few weeks of runway. Don't have to buy 5 years worth, but perhaps 3-6 months would be reasonable. The peace of mind has got to be worth the ever so slight cost increase.
- EGreg 13y agoSadly I read this as a precursor to shortages in water, fish and fruits/vegetables that are coming. The populations of monarch butterflies, bees and other insects are dropping like a stone year over year. The fish near asia's costs are disappearing. The culprits include pesticides, overfarming, overfishing, and pollution. I am not even as worried about the garbage anymore -- scandinavian countries have figured out ways to burn the garbage and "contain" the waste gases. However, the main problem of capitalism is the numerous externalities that are exploited until it's too late to fix. It's not just global warming. It's raping the ecosystems for profit. No futures are going to save us from that. We need sustainable practices but I really don't see a way that'll happen at this point.
- TrainedMonkey 13y agoToo fatalistic. In 19th century they predicted that Earth can't sustain population over 1 billion. In comes oil and agricultural revolution. I agree what we are doing with ecosystem in general is a crime, but it won't kill us. At least not anytime soon.
- xenonite 13y agoYes, it is true that the modern agriculture was enabled through abundance of oil. But that leads to a big problem as soon as the oil reserves diminish. How should one not be fatalistic then?
- EGreg 13y agoHow was modern agriculture enabled by an abundance of oil?
- EGreg 13y agoJust because advances were able to stave off collapse and allow billions of people to consume resources at an ever greater rate does not mean that the same can be done with the effects of overfarming, overfishing, and pollution. The fallacy is thinking "since we did it once we can always continue doing it." The probability exponentially rises that you can't, since this is basically the same as the probability of always succeeding in time to avert a disaster. Actually garbage / messing up ecosystems is a different type of problem than obtaining resources. It is somewhat related to the second law of thermodynamics. Once you introduce so much entropy into a system the genie is out of the bottle.of course this isn't exactly thermodynamics but I can say it this way -- putting humpty dumpty together again is different than figuring out how to make humpty give you more food for decades.
- sliverstorm 13y agoI don't know a whole lot about backblaze; as they are a backup service, is there a good reason why they couldn't ramp up usage of tape drives? Perhaps keeping recently uploaded backups on-disk, and shuttling older ones to tape?
- petemc_ 13y agoYes. Tapes and tape drives are expensive and awkward to use.
- hrrsn 13y agoYou also have to store them, test them and then find them when a customer wants their data. With drives in a pod they're always online and in a RAID so when a drive dies they can be alerted quickly and customers can access data essentially 24/7.
- smackfu 13y agoTheir backups are progressive, so a single disk image might be spread across 100 tapes. A restore would be a nightmare. Tapes really only work if you are doing a full image each time.
- JoeAltmaier 13y agoor a program of partial backups interleaved with full backups, say 3-to-1. You balance a reasonable restore time with a reasonable storage requirement.
- ghshephard 13y agoI have a colleague who works in the industry and purchases a few petabytes of spinning disk a quarter. I love reading the backblaze blogs, and constantly forward them to him. He thinks the backblaze guys are pretty crazy with their "shucking strategy", finds it hard to believe seagate/WD guys will honor warranties on those "shucked" drives, but if they will, all the more power to backblaze. He does note that his costs per drive didn't go up dramatically after the Thailand floods, and they were pretty close to what backblaze was playing for their "shucked drives" - though, they didn't go down signficantly either. I'm guessing Backblaze is still at the volume levels where they can't negotiate directly with Drive Vendors - I'm not sure where that level is - 10 Petabbytes/Quarter? 100 Petabytes/quarter? Regardless, I continue to be very impressed by how backblaze manages to circumvent what others might consider insurmountable hurdles. Truly hacking the system in a good way.
- atYevP 13y agoYev w/ Backblaze here -> Yea unfortunately when we shuck the drives the warranties are voided, but we add that factor in to the costs that we are willing to pay for the hard drives. Truth was, it was cheaper for us to buy the externals and shuck them, even with the voided warranty, than pay the larger fees for internal drives. Part of the is price sensitivity. External drives are geared towards consumers, who simply will not buy the drives if they go up too much in cost. The internal drives we were buying are targeted at business' who have a choice to make, either buy at the higher price, go out of business, or get creative. We try to go the go creative route because we really don't want to go out of business, and paying the higher prices for drives, especially in this case, would have gotten us there. As for the negotiation, BrianWski mentioned it in another reply on here, but we keep getting weird answers from vendors and manufacturers. Typically they want a minimum order of about 10,000 hard drives in order to work directly with them and we're simply not willing to buy that many at a time, especially since normally the price goes down monthly, meaning carrying inventory is leaving money on the table. Love that you enjoy your blog posts, stay tuned ;-)
- digitalzombie 13y agoStill... no linux.
- pwg 13y ago> When the Drive Crisis started, industry pundits estimated that the hard drive market would take anywhere from 3 months to 1 year to recover. No one guessed two years. Was the delay simply an issue in rebuilding and/or relocating the manufacturing and assembly facilities? This is a typical response in all markets that are commodities (or approaching commodity status). Once makers have had an excuse to move prices up, they want to hold onto the extra profit for as long as they can, so prices are fast to rise, but slow to fall. The most direct effect can be witnessed in gasoline prices. Notice how a news report that a camel farted in the desert of Saudi Arabia results in the price of gasoline at your local station rising by 25 cents a gallon (or more) nearly instantly. Even though the gas in the underground tanks was the same gas (at the same cost to the station) that it was 15 minutes ago. But then, it takes weeks before that 25 cent increase goes away when no more camel farts occur over those same weeks. Same effect here, just with hard drives rather than gasoline.
- brianwski 13y agoSix months after the Thailand drive crisis, drive prices "peaked" at about double their earlier prices and began falling. Just about then, an article came out featuring a bunch of panic inducing quotes from a Seagate VP loudly claiming it might be a year before they could get enough drives made, shortages would be terrible. It felt so disgusting to me, none of it was true, the VP was just trying to encourage pre-buying and drive hoarding to artificially pump sales.