23 ms·
Backblaze Hard Drive Stats Q2 2019
- hombre_fatal 7y agoI just destroyed my Backblaze buckets recently and realized they didn't have a "delete" button. You have to make a deletion API request for every single item in each bucket before you can delete a bucket. I found that a bit ridiculous. My internet was so bad that I had to spin up an EC2 nano just to make a few billion requests and have them succeed. Took hours. If backblaze wants to charge you on the way out (I wonder how much all those requests cost me), okay fine. But please just make a "delete bucket" button that does all this for us. If I was in a worse mood, I would've considered letting the account accrue balance until they did it for me.
- IfOnlyYouKnew 7y agoI seem to remember the same experience (not completely sure if it was Backblaze, though). Good to know I wasn't stupid. Couldn't shake the feeling at the time that I was missing something.
- jcims 7y agoAmazon S3 does the same thing. You can make bulk requests but you still are responsible for deleting all objects before removing the bucket. Agree it's stupid.
- _Propaganda_ 7y agoThis isn't strictly true - If you use the S3 dashboard the Delete Bucket button will abstract all the individual object deletes away for you.
- PopeDotNinja 7y agoHere's a video on deleting a bucket with a large number objects I just watched & found informative. It basically says something like you disable versioning, expire objects after 1 day, delete deletion markers, and delete deleted deletion markers (getting a little meta there). https://youtu.be/TPavkqp-Ymo https://youtu.be/TPavkqp-Ymo
- Karunamon 7y agoNo, you can nuke a bucket with stuff in it. I just realize I've never done this before, but the deletion screen calls out that all the contents will go away too. https://i.imgur.com/j1av6iE.png https://i.imgur.com/j1av6iE.png
- kalleboo 7y agoThat must be a somewhat recent addition - a year or two ago it wasn't possible (and the suggestion was what was posted in another comment - set the lifecycle rules to expire all the objects in a day) edit: actually the "don't close your browser window" warning leads me to believe that's just a client-side JavaScript solution which doesn't work with massive buckets (we had one with millions of objects)
- jandrese 7y agoWhy would the Javascript solution be inherently unable to deal with large buckets? Do we have to assume that they're taking some incredibly naive approach and trying to load every single ID into memory at once instead of streaming them out in chunks? It seems like Amazon should be capable of hiring an engineer who knows what they are doing.
- kalleboo 7y agoIt just seems like an inherently slow process. I remember at the time there was a python script someone wrote, which used the API to delete them in 1000 object chunks, and even running that with 10 threads the delete was going to take days and days.
- jcims 7y agoThe problem is the duration and durability of the exercise when you have millions of objects in the bucket. It could easily take days.
- jandrese 7y agoAt least it's the kind of problem you can come back to and restart if the process dies in the middle. You're bulk deleting data so maintaining integrity isn't a concern.
- nijave 7y agoYou can nuke a whole S3 bucket. There's a switch in the cli and a checkbox in the GUI. I think there's another option for deletion protection that disables that somewhere else
- jcims 7y agoI can't find the CLI option, the docs still say that the bucket must be empty[1]. There's something available in the management console but that doesn't really help with automated deployments and testing. [1] - https://docs.aws.amazon.com/cli/latest/reference/s3api/delete-bucket.html https://docs.aws.amazon.com/cli/latest/reference/s3api/delet... Edit: Found aws s3 rb --force, but this just automates the process of listing/deleting objects then deleting the bucket. When you have 7 million objects in a cloudtrail bucket, that can take days.
- Nyandalized 7y agoLooking at the pricing at: https://www.backblaze.com/b2/b2-transactions-price.html https://www.backblaze.com/b2/b2-transactions-price.html It seems that calling b2_delete_key is free. It's just not conveniently abstracted. Listing the items however if you don't already have a catalog is $4/M items
- jorams 7y ago> Listing the items however if you don't already have a catalog is $4/M items That doesn't seem correct. Each call to b2_list_keys can return up to 10 000 results, so it's $0.40/M items.
- deleted 7y ago[deleted]
- mrlinx 7y agoI was able to delete everything using their bucket browser UI. did you try that?
- hombre_fatal 7y agoThe web UI button specifically says the bucket cannot be deleted because it has contents.
- mr_t 7y agoThey did an AMA on Reddit a few months ago[0]. I remember this AMA in particular because they thoroughly explained why they designed some features in certain ways. Overall they said their target customers are the ones with absolutely no IT knowledge. I guess this part from one of their answers fits your complaint quite well: But here is the thing -> YOU can work around this problem, the naive customers CANNOT. Honestly, they are too computer-illiterate. But even computer illiterate people deserve to have their files backed up, and they are the target market for Backblaze Personal Backup. [1] From this point of view, I guess a "delete" button could be fatal for some of the older folks out there. But I don't use Backblaze, I don't know if my comparison makes sense and we both talk about the same thing. The Reddit quotes refers to their "Backblaze Personal Backup". [0] https://www.reddit.com/r/IAmA/comments/b6lbew/were_the_backblaze_cloud_team_managing_750/ https://www.reddit.com/r/IAmA/comments/b6lbew/were_the_backb... [1] https://www.reddit.com/r/IAmA/comments/b6lbew/were_the_backblaze_cloud_team_managing_750/ejlawg8/ https://www.reddit.com/r/IAmA/comments/b6lbew/were_the_backb...
- hombre_fatal 7y agoThere are all sorts of ways to created a delete-bucket button that avoids tech-illiterate fat fingering. Like simply burying it in options. Requiring end user to make a billion individual requests is almost as bad as making me call a number to delete my bucket. I’m voicing this here because I was surprised at the UX. I was happy with my backblaze experience otherwise. My bill was incredibly cheap for the amount of data and access I was using. I’d consider using it in the future for things we generally default to S3 for.
- typon 7y agoThere are good UI solutions to this problem... For example, a dialogue box that asks you to type "Yes I am sure I want to delete all my data, forever". We do things like this for internal tools at work, seems to go pretty well.
- wlesieutre 7y agoGitLab makes you do something similar to delete a repo, but not as explicit with the message. I think you just have to type the repository name in a box (to confirm you read the instructions) and click the delete button.
- Wouter33 7y agoYou can set a lifecycle rule for the bucket to delete all objects after a number of days. This way they wil be deleted automatically.
- swarnie_ 7y agoI've been reading these reports since 2012(13?) and have used them multiple times to buy hard drives for my home storage systems, I haven't always had similar results however. Does anyone else use these posts to pick their own personal disks? If so do you see the data given as a good indicator of lifespan or is the home environment vs DC just too different?
- redorb 7y agocomparing the results of your small scale personal drive purchases to their larger scale won't perfectly align. Their numbers are an average over a large number of drives - your number is picking 1 of their drive randomly and giving the number etc.. it's shouldn't hurt to use their numbers to guide your decision it is no guarantee though.
- roddux 7y agoI've sworn by HGST drives since BackBlaze have repeatedly covered their impressively low failure rate. They've worked well for me.
- swarnie_ 7y agoMaybe i'll give these a try next, my slow lab environment is due a refresh around new years. Thanks!
- mbreese 7y agoThe statistics of small sample sizes comes into play here. The small sample size of a personal environment doesn't lend itself well to comparisons to a large sample size of a cloud storage provider. For example, their 2% failure rate of Vendor X drives doesn't mean anything to you if you don't have enough drives. Your few drives might be in the 2% or the 98%. The 2% failure rate is only applicable to the population of drives as a total. It doesn't tell you anything about your drives in particular. Now, on top of that, their usage patterns don't necessarily reflect what a typical user would see on their personal computer. I suspect that their read/write patterns skew towards the write-once, read-rarely category. Even for a typical enterprise datacenter, their usage patterns are going to be different.
- woliveirajr 7y agoSeems that Seagate are the worst, even if it's a 2% failure rate. And reality sucks: right yesterday I lost one hdd from Seagate and the second one is making noises. Ohh boy
- gruez 7y agoKeep in mind there's no WD drives in the comparison, only HGST (reputed to be high quality), and tosiba (low sample count).
- kijin 7y agoI've been following these Backblaze reports for a few years now, and Seagate always seemed to do worse than other brands. HGST has consistently been at the top, rivaled only by Toshiba lately. No wonder Backblaze has replaced most of their inventory with these two brands.
- atYevP 7y agoYev here -> Yea, the HGST drives continue to do really well for us, but at around 2% the Seagates are no slouches - especially when you consider the sample size. That's a pretty darn good ROI from drives that also tend to be a bit less expensive!
- kijin 7y agoYep, I suppose that's why you continue to use Seagate drives. They're often more than 2% cheaper than the competition! I think I saw double-digit failure rates from certain Seagate models a few years ago, though :(
- atYevP 7y agoYea they went through a bit of a rough patch, but it's honestly been years since that was the case, they're more in-line with other manufacturers now and the pricing is very competitive!
- lousken 7y agoSince people from backblaze will be reading this - what happened to the datacenter in europe? Last news I heard was Q2 2019 and we're already way past that
- c0l0 7y agoI would love to know that, too. Geographical distance to my data - and the potential bandwidth loss incurred by that - is what's keeping me personally (and, potentially, my current employer) from becoming a paying customer.
- miyuru 7y agonot a backblaze employee, just a keen eyed user Blackblaze uses f00{0..9}.backblazeb2.com which represents different data centers. Currently f003.backblazeb2.com is live and returns a IP based out of the Netherlands.
- deleted 7y ago[deleted]
- diblasio 7y agoThey're hiring here in the Netherlands too. https://www.linkedin.com/jobs/view/1351508643 https://www.linkedin.com/jobs/view/1351508643
- 7y ago
- Siecje 7y agoI have a few drives with video files and I use Backblaze for Windows. But there is no local redundancy. What is a better solution?
- OedipusRex 7y agoRAID
- pnutjam 7y agoRAID give you in time redundancy, so your system doesn't fail while your working. Most home users don't need this, it's cheaper and better to sync to a 2nd drive nightly. This gives you a local backup and redundancy, because you can swap to that drive if your first drive fails. If you're doing RAID, you still need a local and offsite backup.
- Siecje 7y agoHow do you sync nightly? If a file is corrupted or a bit is flipped are you copying over a good copy?
- pnutjam 7y agoI sync on linux, btrfs lets me take snapshots after each sync. I currently use rsync, but I'm thinking of setting my primary drive to btrfs so I can use btrfs sync. I used to do this with windows. Yeah, if you have corruption, you copy it, but so does a raid? At least this way your safe from fat finger errors and you have a quick access backup.
- bristleworm 7y agoAll my data is on a Synology NAS. Can't think of anything better.
- Siecje 7y agoDo you have an online/off-site backup?
- linsomniac 7y agoI used to run a small hosting service and we standardized on Ultrastar disks. We would regularly have drives fall out of the RAID arrays, but when we ran badblocks on them to verify they were having problems they would always show up as good. After we put them back in production, they would always be happy from that point. So we started going on the theory that there were some marginal parts of the disk during manufacture, and exercising them would allow them to be remapped to good parts of the disk. Once we started including "badblocks -svw -p 5" in the burnin procedure, our drive failure rates dropped to basically 0. I wonder if backblaze does anything similar to provision drives. Recently I replaced some drives in our old Dell R720 servers (5 years old). They were showing media errors over several months but never went degraded. During the replacement, each server had another drive fail. So I ran badblocks on the RAID arrays over the weekend, just to make sure any other marginal drives or sectors had an opportunity to remap. No other drive failures. Aside: I didn't lose any data, the arrays never went FAIL, but also I had migrated all our services off these machines. Love virtualization!
- chrisseaton 7y agoIs it easy to standardise on anything and then reliably get that product in the future? I would have thought supply would start and stop, prices would go up and down, quality would vary, etc.
- linsomniac 7y agoFor drives it was relatively easy, though we would have to eventually just use the next bigger size. All of our RAID systems would handle that just fine. Drive quality never really was an issue, even in the "Deathstar" days... The bigger standardizing issue was the system. We use Supermicro barebones, and between memory and CPU changes, we would have to qualify a couple-three new chassis every couple of years. And those chassis were much more expensive than a disk drive. Mostly, Supermicro quality was very good. With the exception of around a year or 18 months where the power supplies they were using all failed roughly a year after we bought them. We had older and newer servers with no problem, but around 30 in the middle all failed.
- 7y ago
- Unklejoe 7y agoA word of caution to anyone considering buying one of the 4 TB Hitachi drives listed in the report: There are a lot of vendors selling “refurbished” HGST drives. Do not buy these. As I understand it, there is no such thing as a refurbished hard drive (not economical). They just zero out the SMART data and sell it as new/refurbished. They even look new, but they’re not. I made the mistake of buying one. It had a ton of vibration and started reporting bad sectors immediately. If you want one of these drives, you’re better off buying an actual used one off eBay or something.
- iicc 7y ago"recertified" drives are a thing. https://www.blizzarddr.com/recertified-hard-disk-drives-refurbished-hdd/ https://www.blizzarddr.com/recertified-hard-disk-drives-refu...
- jaclaz 7y agoCannot say nowadays, but once defective multi-platters disks were "refurbished" (and "recertified") by simply disabling the couple of heads on the defective platter(s) in firmware and "downgrade" the capacity specs. It simply makes no sense to actually open a US$ 100 (when new) disk drive to repair it (replacing parts that even in the factory cost money for the cataloguing and storage) to re-sell it at US$ 30 or 40. If you go to the WD page mentioned in the article you linked: https://www.wd.com/en-gb/products/wd-recertified.html https://www.wd.com/en-gb/products/wd-recertified.html you will see how they are all "external disks in a case", those might be subject to a number of "returns" for reasons different from the actual disk inside (cosmetic damage, power adapter (if any), connectors, network card or wi-fi, etc.), so those may (while actually being defective as a whole) well have not a defective disk inside. On the other hand, if they are so good, why would the warranty be limited to 6 months (or less).
- someonehere 7y agoI worked for a startup ten years ago that stored a lot of user data. A batch of drives from the manufacturer had some sort of firmware issue. It affected a lot of users. Essentially their data was put into read only mode by our service. Two weeks went by before the manufacturer came out and worked on replacing the drives and migrating the data over for us. Unfortunately the data center admin at the time didn’t have backup servers and drives in place and lost his job over the snafu once the drives were replaced. Always have backups of your data and make sure the drives are good quality.
- ksec 7y agoCan someone explain on the situation of "Western Digital". I am confused. At first I thought Western Digital is retiring their WD brand for HDD, and focus on SSD, but that is not true I find no information on it. There are still Red, Blue, and others. As a matter of fact, it is the HGST brand that is getting killed, but the UltraStar range lives on. So I am assuming the article is saying, the specific breed of HDD, Western Digital used to provided is no longer available, and it will be replaced by UltraStar, which used to be a HGST brand but are now under Western Digital ? What happened to 20TB HDD which Seagate promised for 2020. Are we no where close to it? And MAMR Drive? Has Price / GB continued to decline, or given the situation of HDD market, the HDD maker are milking it? ( And I don't blame them )