16 ms·
Backblaze Durability Is Eleven 9s – And Why It Doesn’t Matter
- garettmd 8y agoThis was an interesting read, both the points made about durability, as well as the in-depth math. However, what stood out to me most was the line: Because at these probability levels, it’s far more likely that: - An armed conflict takes out data center(s). - Earthquakes / floods / pests / or other events known as “Acts of God” destroy multiple data centers. - There’s a prolonged billing problem and your account data is deleted. The point that once you get to a certain point of durability (at least as far as hardware/software is concerned) you're chasing diminishing returns for any improvement. But the risks that are still there (and have been big issues for people lately) are billing issues. I think it's an important point that the operational procedures (even in non-technical areas like billing and support) are critical factors in data "durability"
- schoen 8y agoI wonder if there's a general term in engineering for the case where a particular risk has been reduced well below the likelihood of more serious but exotic risks. I've heard about this most in cryptography where we can sometimes say that the risk of, say, a system failing due to an unexpected hash collision is drastically less than the risk of the system failing due to cosmic radiation or various natural disasters. At that point it doesn't seem important or worthwhile to consider this risk, because it's dwarfed by the others. This seems like a form of the same argument, and I wonder where else this arises and how people describe it.
- matthewmcg 8y agoI think in other contexts, you might say that these signals are "below the noise floor."
- entropicdrifter 8y agoAs an audio engineer (as well as a dev), I definitely agree with this use of that phrase.
- matt4077 8y agoIt’s not a perfect match, but Rumsfeldian “unknown unknowns” come to mind. Specifically: every X-nines durability design will be compromised by some failure mode you didn’t think of.
- macintux 8y agoI tend to assume pessimistically that the durability design will itself cause a problem. Redundant switches to survive a hardware failure, e.g., strikes me as inviting trouble.
- jzwinck 8y agoIndeed. I once had critical systems routed via a large redundant Cisco switch which claimed to be 1+1. Turns out there was a single "supervisor" component which failed (after just a year or two) and made the pair of switches useless. Apparently the designer worked in a team where nobody does anything when the boss is out.
- schoen 8y agoI don't think unknown unknowns are what I'm thinking of here. In this case the argument involves a very specific risk, and sometimes a very specific lower bound for its probability. For example, in the hash collision case the argument says that it's not worth worrying about the (known) probability of a software error due to an unexpected hash collision because it's dominated by the (known) probability of a comparable error due to cosmic radiation. (The former probability can be calculated using the birthday paradox formula, and the latter has been characterized experimentally in different kinds of semiconductor chips.) This kind of argument doesn't rely on the idea that there are other risks that we can't identify or quantify. It's about comparing two failure modes that we did think of, in order to argue that one of them is acceptable or at least not worth further attempts to mitigate.
- fanpuns 8y agoAlthough Rumsfeld often gets credit for this statement, it has been around for a long time before him https://en.wikipedia.org/wiki/There_are_known_knowns https://en.wikipedia.org/wiki/There_are_known_knowns
- pixl97 8y agohttps://en.wikipedia.org/wiki/Black_swan_theory https://en.wikipedia.org/wiki/Black_swan_theory
- jorangreef 8y agohttps://en.wikipedia.org/wiki/Information_theory https://en.wikipedia.org/wiki/Information_theory
- klodolph 8y agoI've posted the math here before but if we assume that an asteroid hits the earth every 65 million years and wipes out the dominant life forms, then this fact alone puts your yearly durability at a maximum of ~8 nines. The point about billing is better, though. My other concern is that a software bug, operator error, or malicious operator deletes your data.
- GauntletWizard 8y agoThe event that wiped out the Dinosaurs was 65 million years ago, but the Mesozoic era lasted for ~200 million years. Your point stands, though.
- klodolph 8y agoThe question of how many data points to use is a subtle one, though. I can say that I picked one data point because I was lazy and doing back of the envelope math, which is reasonable because we can be somewhat assured that I didn't choose a number of data points that was convenient for the hypothesis. But if you're choosing two data points, my question is... why two? If you are choosing whether or not to reply based on whether or not the second data point fits with the first, then you're introducing selection bias. The chance that the second data point disagrees with the first by at least as much as the 200 My interval disagrees with the 1/65 My rate is equal to 1-(exp(-65/200)-exp(200/65)) = 0.32, which is not especially high.
- tjoff 8y agoThat's why no sane entity would use one earth. Use two and you can quickly recover.
- klodolph 8y agoOur goal is N+2, that way when one Earth is down for planned maintenance, you can endure unplanned loss of a second Earth. N, of course, is always 1.
- metalliqaz 8y ago
- TheAceOfHearts 8y agoFor a consumer the cheapest and easiest way to backup important documents or files is to encrypt it and store it across multiple storage providers, e.g. Dropbox and Google Drive. They usually give you a reasonable amount of free storage, and it's unlikely all accounts would be terminated or locked at the same time. And of course, you should always have your local backups as well.
- e12e 8y agoAssume you want access to files over the next 20 years. What are the odds Google will have bought Dropbox in that time; and what are the odds an automated system monitor at Google ad words will have disabled your Google account in that time span? Replace with Amazon and/or crashplan as appropriate..
- scarface74 8y agoBackblaze suggests a “3-2-1” backup strategy. You should always have at least one backup on site. If a remote backup becomes inaccessible, you could move over to another remote access provider.
- eloff 8y agoThis was an interesting read from a technical point of view, but also well written and refreshingly transparent. I found the discussion about why it doesn't matter when you start talking about 11 nines of reliability to be hilariously true. At the end of the day we're still flawed humans living in a hostile universe, and no matter how foolproof we make the technology, there are some weaknesses that just can't be eliminated.
- kraftman 8y agoIt's 'foolproof', like dog proof but for fools.
- eloff 8y agoHah, good catch, I know that but somehow made the error anyway. I corrected it.
- brianwski 8y agoDisclaimer: I'm the author of the blog post. :-) > well written and refreshingly transparent Thank you! In the interests of full transparency, the blog post was a collaborative affair and was proof read and edited for clarity by several people at Backblaze. > discussion about why it doesn't matter One of the philosophies Backblaze uses is to build a reliable component out of several inexpensive and unrelated components. So combine 20 cheap drives into a ultra reliable vault. We have two or three inexpensive network connections into each datacenter instead of buying one REALLY expensive connection for 8x the price. Etc. Personally, I recommend customers do the same. Instead of storing two copies of your data in two regions in Amazon for 2x the price, store your data in one region of Amazon and put one copy in Backblaze B2 for 1.25x the price. We believe this will result in higher availability and higher durability that two copies in Amazon because Amazon S3 and Backblaze B2 don't share datacenters in common (that we know of), don't share network links, don't share the software stack, etc. For bonus points, use different credit cards to pay for each, and have different IT people's credentials (alert email address) on each. That way if one IT person leaves your company and you don't get an alert that the credit card has expired, hopefully your other copy will be Ok.
- ujjain 8y agoIt still wouldn't upload my 1TB in back-ups in an entire month. Amazon Drive back-up completed in 3 days. Their pricing is amazing, but saving money on a back-up solution that doesn't seem as good as the other cloud storage providers is a dangerous game.
- DenisM 8y agoHow do you mean? Is upload too slow?
- GuB-42 8y agoNot that dangerous if you do your own maths. I don't really trust cloud backup. As they said, the 11 9s doesn't matter. You are more likely to encounter a billing problem (as they said) but also get hacked, have a problem with your internet connection, many things can go wrong. That's why you need another solution if you are serious about your data, maybe a set of external hard drives (local backup). This way, you have redundancy and little correlation in failure, which greatly improves your general durability. That local storage may be paid with the money you save by getting an "inferior" cloud backup provider.
- anderspitman 8y agoI've found the upload speeds to B2 to be fantastic. Especially if running multiple connections.
- 3pt14159 8y agoI use B2 and I have no problems uploading large files quickly. Is this for the consumer option?
- fencepost 8y agoWhere are you geographically? Was your Amazon upload to a datacenter physically much nearer to you than California? 30MBit/s sustained for 3 days isn't unreasonable for a business connection, but seems high compared to most of what I see available at least in the US.
- brianwski 8y ago
- londons_explore 8y agoThis analysis is simplistic. Correlated failures are common in drives. That could be a power surge taking out a whole rack, a firmware bug in the drives making them stop working in the year 2038, an errant software engineer reformatting the wrong thing, etc. When calculating your chance of failure, you have to include that, or your result is bogus. Eg. Model A of drive has a failure rate of 1% per year, but when failed the symptom is failure of the drive to spin up from cold, however if already spinning it will keep working as normal. 3 years later, the datacenter goes down due to a grid power outage and a dispute with diesel suppliers so the generators go down. It's a controlled shutdown, so you believe no data is lost. 2 days later when grid power is back on, you boot everything back up, only to find out that 3% of drives have failed. Not a problem. Our 17 out of 20 redundancy can recover up to 15% failure! However, each customers data is split into files around 8MB, which are in turn split into the 20 redundancy chunks. Each customer stores say 1TB with you. That means each customer has ~100k files. The chances that you only have 16 good drives for a file is about (0.97^16 * 0.03^4)2019*18 = 0.3% Yet your customer has 100k files! The chance they can recover all their data is only (1-0.003)^100000... Which means every customer suffers data loss :-(
- ChuckMcM 8y agoI was going to post essentially the same thing, so here is an upvote :-) While I always find storage analysis interesting (I spent 5 years at NetApp where it was sort of a religion :-)) some of the assumptions that Brian was tossing out are not good ones to make. (like the lack of correlation, or that Drive Savers will exist as a company 10 years from now). Still it does help you to understand the they take data availability seriously which is the underlying message.
- brianwski 8y agoDisclaimer: I'm the author of the blog post. :-) > some of the assumptions that Brian was tossing out are not good ones to make. We COMPLETELY welcome other analysis and listing other assumptions. Internally, we argued endlessly about why this or that wasn't totally accurate, and finally decided to publish the math WITH all of our assumptions exposed so you could be the judge. If Amazon wants to publish their assumptions for S3 for comparison, we're all ears. > that Drive Savers will exist as a company 10 years from now Absolutely true, this calculation is only good RIGHT NOW. For example, one of the things that came up internally was "well, when drives get more dense the rebuild time rises, so this calculation will no longer be accurate in two years". But at the same time, we have some additional tricks and optimizations to make which we have not done yet to cut the 6 day average drive rebuild time down to 3 days. Also, drives last us about 5 years, so your data will be migrated to totally new drives 5 years from now. Those drives will absolutely have a different drive failure rate (maybe higher, maybe lower) so the calculation will no longer have the same result 5 years from now.
- riku_iki 8y ago> if you store 1 million objects in B2 for 10 million years, you would expect to lose 1 file. Can this be reformulated: you store 10 trln objects (e.g. 100TB of 10 byte records), you lose 1 record each year. Also curious what are the stats from other providers.
- jsjohnst 8y ago> Can this be reformulated Roughly, yes. > Also curious what are the stats from other providers. As to other providers, most are 6+ 9s that I’ve looked at, with many in the 8-9 range. Anything over 8 is (as they admitted) essentially marketing porn and not a useful metric (for reasons they mentioned as well as ones said by other comments here).
- riku_iki 8y agoAnd yet, no much complains about lost GMail letters in internet because of disc failures.. And GMail is likely stores much more than 10 trln objects and 100TB of data.
- sdmike1 8y agoI would be curious to see what the chance any given email is read and how much it drops off after the first reading. I would be willing to bet that out of 1000 people who receive a reasonable amount of mail you could probably find missing emails.
- brianwski 8y agoA number of years ago a Gmail "insider" that I know admitted they had lost customer emails and their policy (at that time at least) was to simply ignore it and not tell anybody because most customers never notice if it is a small number. I think a much bigger scandal is that all major laptop Operating System vendors (Microsoft and Apple) absolutely know when your laptop drive loses files or even is starting to go bad in some cases, and they NEVER tell the customer. I think an excellent product offering would be a 3rd party piece of software and cloud service which was a "verification service". It wouldn't store your files offsite, it would store the name, size, and SHA1 offsite and periodically check that no bits have been flipped on your local drive unless you intended it. For example, a week after I take a photo, I absolutely never want the photo to change. Ever. Same with music I (legally) download.
- londons_explore 8y agoI've very disappointed their recovery time is 6 days! Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under a second. Maybe 10 seconds with some sluggish tail machines. Why do you need it done in under a second? While the data is down one replica, it is at dramatically higher risk. Also, drive failures can be dramatically accelerated, for example in the case of a bad software release erasing data - you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release takes 3 hours!
- ChuckMcM 8y agoBe careful here, it isn't 6 days until data is recovered it is 6 days until it is fully protected again, there is a big difference. During the 6 days the data would be available it just might have to be reconstructed on the fly by the error correcting rather than read directly. In most systems we assume that "primary traffic" (read/write stuff) is prioritized over "rebuild traffic" which is recovering lost shards. So when you specify these things it is best to specify "how long to rebuild a shard while the array is providing storage services at its maximum specified rate." This assures the customer that if they have a 24/7/365 non-stop traffic pattern their data will still stay protected in the face of drive failures.
- brianwski 8y agoDisclaimer: I'm the author of the blog post. :-) > During the 6 days the data would be available it just might have to be reconstructed on the fly by the error correcting rather than read directly. Correct. More specifically, the FIRST time the data is accessed in any 24 hour period it must ALWAYS be reconstructed from the Reed-Solomon encoded parts on 17 other drives on 17 other machines. Any 17 is fine, so it's totally fine if 1 or 2 drives are not available. Once reconstructed it is stored in a set of front end cache computers that have fast SSDs for this purpose. The second time the same file is accessed in a 24 hour period, it will be fetched out of the SSD cache layer so it won't even hit the spinning drives and won't care if all 20 drives are offline. > "primary traffic" (read/write stuff) is prioritized over "rebuild traffic" Yes. Backblaze balances between the two if only one drive has failed, but as a tome (20 drive group spread across 20 computers) becomes more badly degraded Backblaze begins favoring the rebuild. When two drives have failed out of 20, Backblaze stops allowing any writes to that tome because more writes will tend to fail yet another drive. Fewer writes offloads the tome. But we still allow reads. At Backblaze, we have never been 3 drives degraded out of 20 (knock on wood), but if this ever occurs the 20 drive tome is now running without parity -> so in that case we even stop allowing reads AT ALL until we are returned to at least 1 drive of fully redundant parity.
- londons_explore 8y agoI've very disappointed their recovery time is 6 days! Recovery workload should be spread across the whole cluster, so that the recovered data gets distributed evenly. In that case, assuming 10,000 drives, to recover one dead 12TB drive and a recovery rate of even 10 MB/secs per machine, recovery of one drive should be done in under 2 minutes. Maybe 10 minutes with some sluggish tail machines. Why do you need it done so fast? Drive failures can be dramatically accelerated, for example in the case of a bad software release erasing data - you need to be able to move data faster than bad software gets released. And releasing software at a rate of one machine per second still means a release takes 3 hours!
- Animats 8y agoFinancial failure or service shutdown by the provider is the highest risk for long term storage. The backup services CrashPlan, Dell DataSafe, Symantec, Ubuntu One, and Nirvanix all shut down. Nirvanix only gave two weeks notice for users to save their data.[1] [1] https://www.computerweekly.com/opinion/Nirvanix-failure-a-blow-to-the-cloud-storage-model https://www.computerweekly.com/opinion/Nirvanix-failure-a-bl...
- chx 8y agoJust for fun, here's my report of Nirvanix in 2008. I feel no problems sharing it since both NowPublic and Nirvanix is long gone. Let's just say there were reasons they went under: 1) Sideloading. I was unable to benchmark or even to get this to work. Images requested to be loaded from media-src.nowpublic.com to node4 @ Nirvanix never showed up. I showed my code to [nirvanixcontact] who said that the code looked OK but someone did a load testing on node4 without informing Nirvanix and that steps are made so that such a situation won't occur again. He told me that the requests are not lsot but they still have not landed. Again, my code can be at fault and I would be happy to run a sideload example code. 2) Upload speeds. I have uploaded from d2 to Nirvanix 100 images each between 180-8584 kbytes totalling almost exactly 100 MB (101 844 931 bytes). The upload was a single HTTP request. The uploads took 18-19 minutes (I repeated the experiment). To give us a comparison I changed the URL in the script to a one line PHP script on another server (at hostignition.com) which just echo'd the number of uploaded files. This took 16.25 seconds and echo'd 100 so seemingly the files landed. 2a) I tried to get another node via the LoginProxy method which we would need for uploads anyways. While LoginProxy itself did work, GetStorageNodeExtended https://services.nirvanix.com/ws/IMFS/GetStorageNodeExtended.ashx?sessionToken=e7b00d25-fc35-431c-9437-9a4302767f46&sizeBytes=102309888&consumerIP=89.133.186.28&destFolderPath=upload/test https://services.nirvanix.com/ws/IMFS/GetStorageNodeExtended... always fails with ResponseCode 80006, ErrorMessage: Session not found for ip = 67.15.102.70, token = e7b00d25-fc35-431c-9437-9a4302767f46. Seemingly, does not pick up the consumerIP. 3) The image conversion itself is blazing fast though. These images took a total of 101.58 seconds to convert and this includes 300 HTTP requests (200 sent to d2 from Nirvanix, 100 to Nirvanix).
- brador 8y agoI would add a third risk of an account ban by some type of future automated copyright content ID. Especially if it is silent/without warning.
- rafaelgarrido 8y agois the website down?
- hartator 8y agoIt was the 0.00000001%.
- cascom 8y agoOne simple question - what is the math if one data center goes down? https://www.nytimes.com/interactive/2018/05/24/us/disasters-hurricanes-wildfires-storms.html?hp&action=click&pgtype=Homepage&clickSource=g-artboard&module=second-column-region®ion=top-news&WT.nav=top-news https://www.nytimes.com/interactive/2018/05/24/us/disasters-...
- shub 8y agoJust one missing piece: the actual loss rate.
- mikeryan 8y agoThere’s another piece too. Practically they’re a backup service so knowing how often someone needs to recover their data and not had an opportunity to reset their backup based on their current state. I’ve used Backblaze for years and only needed it once. (I also back up to a time capsule as well since data recovery is more Practical/Easier from that)
- anderspitman 8y agoOff topic but I just want to say how happy I've been as a Backblaze customer. B2 is a fantastic product for my backup needs, and their hard drive stats have always been handy when selecting drives.
- atYevP 8y agoYev from Backblaze -> Thanks! Glad you're enjoying it :D
- toolslive 8y agoThe best durability is probably achieved by Amplidata, but it does not matter. You need to do the same calculation for your meta data, which is probably not erasure coded. If you lose this, you don't lose your data, but you no longer know where you put it. So you probably add your meta data to your data as well in some kind of recoverable format. That's fine, it means that you can harvest the meta data again. But how long does this take ?
- Pissompons 8y agoIt's a really nice blog post but coming from Backblaze, it would have been nice if they wrote it _after_ bringing the Phoenix DC fully online. When Amazon or Google say 11 9s, I can believe it but Backblaze still only has a single datacenter for most data. All it takes is an earthquake.
- sneak 8y agoOr an overzealous prosecutor.
- fencepost 8y agoOne question since I know some of the Backblaze folks respond to these threads: In addition to calling (and possibly getting blocked/ignored), does your customer service staff send text messages? I suspect that a big percentage of the phone numbers you have are for cell phones these days, and I see a lot less SMS spam than I do telemarketing. SMS would also allow you to get a bit of info visible to recipients (e.g. "Backblaze CC Expired") with more detail once a message is opened.
- atYevP 8y agoYev from Backblaze here -> I believe we do send SMSs in the case of a Cap or Alert getting reached, so yes that could be possible - though I'm not sure if an SMS is part of our billing failure process - that's an interesting question!
- u02sgb 8y agoBackblaze B2 customer here. My credit card stopped accepting your billing and no SMS for me. Took me a month or so to notice the emails and update my details. I've got SMS alerts active for Caps. Would be worth adding that as was a bit scary when I noticed the mail (think it was the third one you'd sent!).
- atYevP 8y agoVERY good to know! Thank you!
- hartator 8y ago> The math on calculating all this is extremely complex. Hum, I would be more reassured by past statistics than a probability evaluation. Did they happen to have loss data since their creation?
- duxup 8y agoI just wish Backblaze would not go "oh man a lot of your data changed, you should probabbly check for integrity and start your backup with us again later...." Oh gosh thanks Backblaze, I'll just dig through several TB of stuff....
- egonschiele 8y agowhy calculate poisson and binomial when both are ultimately gaussian?
- RA_Fisher 8y agoThey're not ultimately gaussian. :) The normal distribution is continuous and has unbounded support whilst neither the binomial or Poisson ate cts and unbounded.
- rossdavidh 8y agoFun read, but the whole time I was reading it I could hear Nassim Nicholas Taleb making exasperated noises of outrage.
- skybrian 8y agoThe billing problem is mentioned, but just left dangling. I wonder if anyone has done any interesting work at fixing this?
- Jabbles 8y agoto lose a file, we have to have four drives fail before we had a chance to rebuild the first one. I wonder how many times they've had 2 or 3 drives fail before they've rebuilt and if that matches their predictions.
- aikinai 8y agoI really want to like Backblaze and they seem to do a lot of good work, but whenever this comes up, I also feel responsible to let people know the dark side so they're informed at least. I've written in more detail before[0], but just to share the gotchas in case anyone here is thinking of switching to Backblaze: 1. They backup almost no file metadata. 2. The client is very slow (days or more) to add new files and there's no transparency (it claims everything is backed up when it's not). 3. There are still bugs in the client that can put your backup into an invalid state where it gets deleted. 4. Support is terrible, and won't be any help when you run into these bugs. [0] https://news.ycombinator.com/item?id=16301626 https://news.ycombinator.com/item?id=16301626
- Dylan16807 8y agoI didn't like their main client very much (though it was a while ago) but I'm still planning to use B2 for some things. So it depends on what you're buying from them.
- rendaw 8y agoAs much as this is off topic, I'd like to continue this conversation. Do you have any details/reference for point 1? I've been using rclone (got the recommendation here) which has been reliable. Also, does anyone know if Backblaze has any plans to offer u2f? I've switched dns and email providers to get u2f.
- aikinai 8y agoYeah, it's a bit old, but here's an article about Backblaze not supporting metadata. [0] "It fails all but one of the Backup Bouncer tests, discarding file permissions, symlinks, Finder flags and locks, creation dates (despite claims), modification date (timezone-shifted), extended attributes (which include Finder tags and the “where from” URL), and Finder comments." And I don't know if it supports U2F, but it does support TOTP. [0] https://mjtsai.com/blog/2014/05/22/what-backblaze-doesnt-back-up/ https://mjtsai.com/blog/2014/05/22/what-backblaze-doesnt-bac...
- zimbatm 8y agoanswer: because all the files are in the same, single data center which has a lot less nines
- contravariant 8y ago>The sub result for 4 simultaneous drive failures in 156 hours = 1.89187284e-13. That means the probability of it NOT happening in 156 hours is (1 – 1.89187284e-13) which equals 0.999999999999810812715 (12 nines). Minor nitpick, this ignores the possibility of more than 4 failures, although this error only affects the fourth digit after the nines. Much more egregious is the following: >there are 56 “156 hour intervals” in a given year This is too simplistic, there are in fact infinitely many 156-hour intervals in a year, some of them just happen to overlap. This overlap can't simply be ignored because even if none of their 56 disjoint intervals contain 4 events this does not rule out the possibility of there being 4 events in some 156 hour interval they didn't take into account. In fact failing to take into account even one of the infinitely many intervals creates a blind spot (consider what happens if the drives happen to fail precisely at the start and end of a particular interval). You can still get a lower bound by e.g. ensuring none of the 56 intervals contain more than 1 failure, or by adding more intervals and ensuring none of them have more than 2 failures etc. Their binomial calculation contains the same mistake. A quick improved lower bound can be obtained by calculating the probability that any failure is followed by (at least) 3 other failures within 156 hours. For one failure this probability is given by the Poisson distribution and is Pc = 1 -\sum_{k<3} e^-λ λ^k / k! = 5.18413e-10. Now we get into some trouble because the failures and the probability of a 'catastrophic' failure are dependent, however the probability that any particular failure turns catastrophic is constant, so the expected number of catastrophic failures can't be greater than the expected number of failures times that constant, this gives a lower bound of Pc (365·24·λ) = 6.63154e-9 this is a lower bound, but that's still three fewer nines left than their claim. Anyway let's just hope their data centres are more reliable than their statistics. Edit: This last calculation can be justified by noting that the probability that 1 critical failure starts in a particular time interval is Pc times the probability of 1 failure in that interval plus some constant times the probability of more than one interval. Similarly the probability of more than one critical failure is at most the probability of more than one failure. Now the probability of more than one failure in a time interval is dominated by the length of the interval, therefore if you calculate the density those parts fall away and you're left with a density of Pc λ critical failures per hour. This seems to be an exact expression for the expected number of critical failures, and not just a lower bound. Although it is still a lower bound for the probability of a critical failure, albeit a fairly tight one.
- sneak 8y agoThis article supposes that a meteor impact is more likely than disaster that renders northern California (their only data centers are in Oakland and Sacramento AFAIK) without power or civil order within ten million years. As someone else pointed out, it’s overly simplistic. They’re a great low-cost alternative to S3, sure. But keep a backup on another continent if you need your data 100 years from now.
- simonebrunozzi 8y agoThe Backblaze blog post points to Amazon's CTO (Werner Vogel)'s blog post, in which he states that "...These techniques allow us to design our service for 99.999999999% durability." (side note: Werner is a great person) Unfortunately, there is a difference, a huge difference, between a system "designed" for 11 9s of durability, and a system "offering" 11 9s of durability. I wish Backblaze, or Amazon, or anybody else, would clarify durability using very honest terms. An example? "This system offers X 9s of durability over a period of one year, on average. This is a technical paper that describes how we tested that durability", followed by measurements and test specifics. Any other claim has much less value to me.
- HelloNurse 8y ago"Our nines go to eleven" Well written, but there are other significant risks like losing access credentials (e.g. a password stored only on one device that is destroyed in the same accident in which its only user, who remembered the password by heart, dies) or being hacked by someone who gains access to cloud storage and intentionally erases or corrupts data. Specialization is good, but if Backblaze is strictly in the business of storing data on hard disks, who's going to help with designing and maintaining the reliable complete system on top of their service that users actually need?
- juancn 8y agoMy experience is that it's bullshit. I had a backup (damn I still have one) with Backblaze, when attempting to restore it, maybe 30% of the files survived restore. The rest are lost in smoke. They don't have any way to detect corruption in the data or if they have, the backup clients are oblivious to it. I lost about a 150GB of family photos and videos.