30 ms·
The drive stats of Backblaze storage pods
- MeteorMarc 3y ago>>>All 40 of the Dell servers which make up these two Vaults were relocated to the top of 52U racks, and it appears that initially they did not like their new location. Or the way in which they were handled during the relocation?
- newsclues 3y agoMaybe they moved them again to confirm location was the issue
- lostlogin 3y ago“All 40 of the Dell servers which make up these two Vaults were relocated to the top of 52U racks, and it appears that initially they did not like their new location. Recent data indicates they are doing much better, and we’ll publish that data soon.” Italics mine. Seems like they changed something.
- anarazel 3y agoCooling or vibration issues seem like the first suspect?
- neonnomad 3y agoMy guess would be vibration at the top of the rack which is why when you relocate them they do much better. This is such a great thing to call out explicitly with data.
- willcodeforfoo 3y agoLooks like ~3,087 Petabytes of total storage!
- atYevP 3y agoYev from Backblaze here -> Loads of data :D
- stackskipton 3y agoIs there any information what lead to Backblaze to stop making their own hardware and going with COTS?
- mbrameld 3y ago> Over the last few years, we began using storage servers from Dell and, more recently, Supermicro, as they have proven to be economically and operationally viable in our environment. Sounds like it was money.
- wannacboatmovie 3y agoYes, building your own shit is expensive. Tale as old as time. Before someone uses Google's servers as an example, I would say that strapping together consumer grade components with zip ties and no case isn't what I'd consider a 'server'; rather, a loose collection of parts.
- hinkley 3y agoGoogle has 3 orders of magnitude more servers than I’ll ever need. With that sort of difference it’s not only a change of solution but also a change of rules. We simply do not witness the same bottlenecks.
- lesuorac 3y agoIIRC, parts for some of the garage servers were found from literally dumpster diving. However, it does cause you to write some very fault tolerant code.
- Analemma_ 3y agoThey may have just reached a scale where it was viable, where it wasn't before. The bigger you are, the bigger discounts you can negotiate. They also certainly have the leverage to walk away from a deal, since they have proof "hey, we'll just build our own if we don't like your offer."
- ericbarrett 3y agoMy sense is that storage server prices have had to come down to compete with both cloud (like S3 archival tiers) and roll-your-own solutions. The former are relatively recent and the latter have become more feasible as companies—Backblaze among them—have open sourced well-engineered designs. Storage server margin used to be insane, over 80% after COGS back in the 00s and only slightly less last decade, and there's still room to compete.
- dijit 3y agoI love these so I'm really sad to say this: What's missing is an AFR based on deployed time in increments, not total time. For example: there should be a sliding window (or a histogram) relating to AFR after deployment for a time frame. IE: AFR between 0-300 days, AFR between 301-600 days, AFR between 601-900 days (etc). Otherwise we're looking at failures historically for the entire period, which might hide a spat of failures that consistently occur 3 years in, giving a relatively unfair advantage in numbers to newer drives. That said, I really do love these and I hope they continue.
- ahazred8ta 3y agoBackblaze's SSD bathtub curve shows higher failure rates < 6mo and > 3yr, with a sweet spot around 1.5yr where the failure rate is 4-5 times lower. - https://regmedia.co.uk/2023/09/26/bathtub.jpg https://regmedia.co.uk/2023/09/26/bathtub.jpg The HDD afr is under 2% for the first 3.5 years. Due to factory testing, 'the bathtub now has no left side, which makes it hard to take a bath'. https://i.gzn.jp/img/2021/12/21/black-blaze-how-long-disk-drives-last/drive-failure-by-quarter.png https://i.gzn.jp/img/2021/12/21/black-blaze-how-long-disk-dr...
- okbrown 3y agoIf there is one company that could give accurate stats on what drives actually perform better and what drives are the worst and fail more often it is Black Blaze, but yet they don't. What a missed opportunity to provide some insight.
- jannyfer 3y agoLike this? https://www.backblaze.com/blog/backblaze-drive-stats-for-q2-2023/ https://www.backblaze.com/blog/backblaze-drive-stats-for-q2-...
- okbrown 3y agoMy first Facepalm!! For 2024!!
- okbrown 3y agoNo.... I take it back, I am a fool! They do publish the stats!! https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-2023/ https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-... My bad!!
- ranting-moth 3y agoYep, I never buy a drive for myself or recommend to other without having first looked the their stats.
- okbrown 3y agoOne thing for sure though, these stats have confirmed my disappointment with Seagate all these years. Utter garbage!
- apocalyptic0n3 3y agoThey've been publishing these stats for years and years now, too. It's a wealth of data they've provided and has helped me pick the dozens of drives I've ordered over the years
- pqdbr 3y agoI really wished Backblaze B2 worked better to be a real alternative to S3. The pricing is a middle finger to AWS and other providers. However, we were having 1 outage per month with B2, and in the middle of 2023 we decided to go back to AWS and only use S3 for replication. We still monitor both services simultaneously (S3 and B2), and every other month we are still having an episode where the latency rises from 15ms to something like 25 seconds for each write/read operation.
- nerdponx 3y agoIt works pretty well for low-volume ad-hoc data storage at least. I use it for personal data science projects.
- eropple 3y agoHave you looked at Cloudflare R2? I've had good luck there, but am exercising it only lightly.
- tempest_ 3y agoIt has been a while since I looked but CFs R2 was cheaper than s3 the last I looked, never used it though.
- Manouchehri 3y agoIt depends. Cloudflare R2 has unlimited free egress; if you’re using a Cloudflare Worker already and proxying the download, then Backblaze B2 becomes cheaper again because egress is free and unlimited. Here’s my rough pricing notes: https://gist.github.com/Manouchehri/733e6235457e60de24fdbb15046fba7f https://gist.github.com/Manouchehri/733e6235457e60de24fdbb15... One useful trick is if you’re already using a Cloudflare Worker, you can force cache fetches to Backblaze B2 (or any S3 provider). This is allowed by Cloudflare’s ToS as you’re using a Worker.
- chaikasumu 3y agoMy understanding is a worker isn't a way around that, they clarified that all in their blog here: https://blog.cloudflare.com/updated-tos/ https://blog.cloudflare.com/updated-tos/ > Finally, we made it clear that customers can serve video and other large files using the CDN so long as that content is hosted by a Cloudflare service like Stream, Images, or R2. > Video and large files hosted outside of Cloudflare will still be restricted on our CDN It is a bit confusing, espec when they only enforce it at super high usage/when it becomes an issue, but makes sense that a Worker isn't a magical free pass when you don't get charged Bandwidth on them either
- atesti 3y agoDoes anybody know which Dell product they use? Dell also has JBOD enclosures like Powervault MD2460
- mmmooo 3y agothey mention servers, so possibly not jbods, but R740xd2 (or newer R760xd2). The former are 26x3.5" so would match what they describe.
- metadaddy 3y agoFull disclosure - I am Chief Technical Evangelist at Backblaze. We use Dell PowerEdge R740xd2 rack servers in our Amsterdam data center.
- atesti 3y agoCool, thanks! Do you put official, expensive Dell drives in there or your own drives?
- metadaddy 3y ago> Do you put official, expensive Dell drives in there Absolutely not! The Dell servers have a similar mix of drives as we use in the Backblaze pods and SMC servers. We added a 'datacenter' field to the Drive Stats data in Q3 2023, so you can download the CSV files and see exactly what's deployed in ams5. See https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-2023/ https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-...
- gorkish 3y agoThe most interesting thing here is the discussion about using vendor hardware instead of their own bespoke stuff. My takeaway is that it's still essentially impossible to negotiate anywhere close to a fair price with commodity server vendors until you are buying hundreds of machines, and then only if you are capable of demonstrating that you are willing to design and build them yourself. And yet despite this, it's still cheaper than cloud even paying advertised prices. Where do regular people buy servers without insane markup if you need say 10-20? Used to be I could actually buy supermicro barebones, but that ended a really long time ago.
- prirun 3y ago> Where do regular people buy servers without insane markup if you need say 10-20? IMO, for a small number of servers, the insane markup isn't a huge cost of business so doesn't matter that much.
- aaronax 3y agoYou get two VARs, one with Dell and one with HP. Then a couple back and forth negotiation rounds and you will be paying 40-50% of MSRP. Or even try to do it direct with your Dell/HP reps, since VARs are of dubious value.
- gorkish 3y agoI know how to get to the price; it's just an incredible pain in the ass.
- mynameisnoone 3y agoBoooo! No drive manufacturer names and model numbers. This is totally useless data for anyone so it's absolutely pointless to blog about. https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-2023/ https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-... contains the necessary data, but it seems old. I don't know why they're buying COTS server gear because their whole premise was the economy-of-scale of having their own pods made. I don't understand why they don't go to Quanta or Foxxconn to build them whatever they need because Dell is really just a marketing front like CDW that relies heavily on third-party contract designers and major component manufacturing, and then only does final assembly itself.
- Sohcahtoa82 3y ago> https://www.backblaze.com/blog/backblaze-drive-stats-for-q3- https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-... contains the necessary data, but it seems old. ...is this a joke? That's Q3 2023. Assuming the quarters are calendar quarters, Q4 2023 ended literally 3 days ago. They're not going to have a report out right away.
- itsgrimetime 3y agoMaybe you misread it but that link is stats for Q3 2021 - but Q3 2023 is available here: https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-2023/ https://www.backblaze.com/blog/backblaze-drive-stats-for-q3-...
- Sohcahtoa82 3y agoNope. GP's link is indeed for Q3 2023. When I made my comment, I copy/pasted the message which included a truncated link which ends up going to Q3 2021.
- joshfee 3y agoI use B2 as the backend for my personal backups using restic (which I would highly recommend https://github.com/restic/restic https://github.com/restic/restic). I don't have a ton of data to backup, so even with hourly backups (restic only backs up when there are changes) I have ~100GB and it runs me a whopping $0.60/month. I almost feel guilty when I get the bill. But the minute I need to pick a storage platform in a professional context I know what my first choice will be. (I am _not_ affiliated with Backblaze in anyway. Just a happy user)
- qwertox 3y agoWe're so lucky that Backblaze creates these reports and shares them openly. That's not normal for companies to do and it's valuable for us consumers to get these insights. Actually I'm referring to their drive stats and not their storage pod stats like in this report. Thank you Backblaze.
- agumonkey 3y agoSeconded, it's one of the links that I will automatically read in great details. Thanks indeed.
- atYevP 3y agoYev from Backblaze here -> We love creating these and it's a team effort internally which a bunch of departments like to help with. Also appreciate the kudos and we share it with the org as well. You keep reading it and we'll keep making it! :)
- gadders 3y agoYeah yeah. Just make your client software suck less. Crashplan is much better but went professional only, sadly.
- CharlesW 3y ago> Crashplan is much better but went professional only, sadly. How do you mean? CrashPlan Essential (200GB) is $3/mo. CrashPlan Professional (unlimited) is $88/year.
- magicalhippo 3y agoPerhaps they reverted their decision, but they did indeed discontinue their Crashplan service for non-businesses[1]. I was using it at the time and had to find an alternative, which I never really did. I tried restic and similar programs, but they were all too slow finding what changed. The key to Crashplan was that it monitored the filesystem for changes, so it didn't have to do an expensive traversal. In the end I settled for just using the daily full-image backups I was already taking and forego the <15min backup points I had with Crashplan. Been missing it a few times, but overall surviving without. [1]: https://www.theverge.com/2017/8/22/16184430/crashplan-home-shutting-down https://www.theverge.com/2017/8/22/16184430/crashplan-home-s...
- CharlesW 3y agoAs the article notes, they just ended the "CrashPlan for Home" plan and decided not to do B2C marketing, but they remain happy to take money from individuals. At the time I just migrated to "CrashPlan for Business" and paid them $10/mo (because I have a NAS), but thanks to your note I'll be moving to CrashPlay Pro (which is cheaper on a yearly basis).
- gadders 3y agoAt the time of the migration, they seemed like they very much did not want my money as an individual.
- 3y ago
- sudhirkhanger 3y agoDo you guys have suggestions for backing up media (photos and videos)?
- FirmwareBurner 3y agoBackblaze
- sudhirkhanger 3y agoI guess it would lack a web interface for me to quickly download a single picture/video.
- ac29 3y agoB2 has a web interface that you can use to download files.
- neonnomad 3y agoBackblaze, Google Photos, iCloud, Dropbox, Onedrive or buy a Synology and use Synology Photos and then backup to one of them.
- quickthrower2 3y agoPricing can be funny! Total Data Stored Monthly Downloads Cost 1TB 1000TB $119,712/yr 1000TB 1000TB $72,000/yr What is the cheapest TDS for a given 1000TB monthly downloads? Seems to be: 334TB 1000TB $24,048/yr So there are situations where uploading dummy files that pad with zeros will reduce your bill. --- Edit: The 334 now seems obvious when you consider: * Egress is $10/TB * Store is $6/TB Free Egress is 3x monthly data store. So egress costs $6/TB when you get it 'free' vs $10/TB, so you need it all free to get the cheapest egress, which means you need 1/3 storage that is egress. Any higher and your wasting money on storage!
- throwaway2037 3y agoFrom the very top, this is great analysis. I wonder if they are willing to strike a deal if you need relatively high egress, but very low storage. Also, I do expect that Backblaze's lifetime egress costs per drive far exceed the TCO for the drive (minus egress). Zero trolling: It would great to see some commentary directly from Backblaze on this matter -- for example, explain that egress is very expensive. EDIT Above, you wrote that egress is 10 USD / TB. What are egress costs for Amazon, Google, and Microsoft?
- metadaddy 3y agoFull disclosure - I am Chief Technical Evangelist at Backblaze. Pulling the egress costs from the hyperscalers' storage pricing pages: * Amazon S3 ranges from $50-$90/TB depending on monthly volume. * Google Cloud Storage ranges from $80-$230/TB depending on monthly volume and where you're transferring data to/from. * Azure Blob Storage ranges from $40-$181/TB (with the First 100GB/month free) depending on monthly volume and whether you route data via the Microsoft Premium Global Network. Cloudflare published a blog post a couple of years ago explaining just how much money AWS makes on egress - customers are paying up to 80x Amazon's costs: https://blog.cloudflare.com/aws-egregious-egress https://blog.cloudflare.com/aws-egregious-egress > I wonder if they are willing to strike a deal if you need relatively high egress, but very low storage. It depends on what "very low" means. We have a capacity-based pricing option, Backblaze B2 Reserve, starting at 20 TB, that includes all egress and transaction fees.
- tunesmith 3y agoRecently got a Synology and have been considering cloud backup needs. Mine would roughly be a terabyte, and so far my thinking is shaking out as: For incrementally backing up laptop data, the process needs to run on my laptop anyway, so I may as well just use Arq. For NAS data that changes frequently enough to desire incrementals, choose Synology Hyperbackup. For NAS data that doesn't change frequently and can just be sync'd, choose Synology Cloud Sync. For NAS data that you don't need locally and only need to archive, choose Synology Cloud Sync with one-way-sync. (Less hassle than AWS Glacier.) As for the cloud provider itself, I'm think I'll probably go with Wasabi, at $6/TB. I heard that Backblaze occasionally has weird gotchas that surprise people, like auto-deleting files from your backups that you delete locally, so I just feel cautious. Main things giving me pause: minimum block size on Wasabi (I'm sure some of my files are smaller; don't know what sticker shock I'll experience), and unsure why I should consider a command line tool like restic instead of the above.
- MobileVet 3y agoSo many options… and the data is likely very precious so it is worth the time to review and test what you can. I tried several different configurations and finally landed on the following. I have a Synology that I have been running for a couple years now. - RAID 0 w 2 14TB drives, 400GB is highly critical The 400GB of highly critical is further backed up: - external 1 TB SSD that lives at my sisters house and is updated monthly or so using Sync w encrypted EXT4 formatting - B2 sync with 1 way, encrypted backup and nightly updates (using Backblaze’s encryption because Synology requires your device and if that goes… you are sol) I friggin love it. I have restored things several times to test the process and it just works. It only costs me $2/ mo for Backblaze B2 with that amount of data. Well worth it. Good luck
- metadaddy 3y agoFull disclosure - I am Chief Technical Evangelist at Backblaze. > I heard that Backblaze occasionally has weird gotchas that surprise people, like auto-deleting files from your backups that you delete locally, so I just feel cautious. We have two similar products, and it's easy to mix them up. To clarify: Backblaze Computer Backup (a different product from Backblaze B2) deletes old versions of files from your backup either 30 days or 1 year (you choose which) after you delete them locally. You also have the option to enable "Forever Version History", which costs $6/TB per month once your files age out of the backup. Backblaze B2 (S3-compatible cloud object storage) will never delete anything unless you tell it to do so, either via the API or a lifecycle rule.
- xwowsersx 3y agoWhat's my best bet for cost-effective and convenient external local storage? I have a ton of photos and large videos which are consuming tens of GBs on my main computer's hard drive. Just get a cheap thumb flash drive?
- etrautmann 3y agoYeah I have 3.5TB of photos. Backing them up redundantly is a bit of a pain but just a few external drives will do it, though requires manual syncing
- xwowsersx 3y agoThanks. Also...holy shit, I haven't kept up on how precipitously prices for storage have dropped. Just saw a 10 TB external drive for only $150!
- neonnomad 3y agoJust remember, local external drives can still burn up in fires or drown in floods. I thought about this recently a lot and went with a Synology where I can have all of my data locally but backup that data to B2 (Backblaze) automatically.
- thekombustor 3y agoThat 10TB drive is 100% fake! Amazon is full of fakes. A good rule of thumb right now is $60/TB, significantly less and you're approaching fake territory.
- thekombustor 3y agoI should be clearer - that drive is fake if it's an SSD, it's a normal price for HDDs. Cost per TB of HDD isn't linear, though, it's a bit of a bathtub based on drive capacity and feature set (e.g CMR vs SMR)
- rr808 3y ago
- DeathArrow 3y agoIf it's cheaper to use servers from Dell and Supermicro instead of their own design, why weren't they doing it from the start?
- GrayShade 3y agoProbably not cheaper today, but reasonably close. And certainly not cheaper 10 years ago.
- djfergus 3y agoBackblaze have just given super micro and dell a fantastic customer reference- hoping they got a huge discount in exchange. Dell/Supermicro sales guys: “look even the experts who build their own pods buy our servers”