9 ms·
Backblaze is building a 270 TB storage pod
- themoonbus 12y agoI really do like this level of communicativeness from my backup service.
- pezh0re 12y agoI was always a fan of Backblaze sharing details about how they built out their storage - it made me feel that if I wanted to, I could also build a redundant/massive storage array.
- toomuchtodo 12y agoI have Backblaze-based storage pods in production at an old employer for a broadcast video automation system (plays out over the air). Quite the sturdy design.
- ansible 12y agoJust put a 6TB drive in my media server. I was going to put in another 4TB, but the cost per TB was not that much worse, so I decided to give it a go. There's also the issue that I only had room for one more drive anyway, and I wanted to RAID-1 the OS and important data partitions too. I'm still learning the ins and outs of using btrfs too. I recently found this btrfs snapshot based backup utility that I'm going to give a try: https://github.com/ruediste1/btrbck https://github.com/ruediste1/btrbck This should be a lot, lot faster and more efficient than using rsync and rsnapshot for disk-to-disk backups with my current scheme.
- ruediste 12y agoHow is it going?
- ansible 12y agoI haven't tried that backup utility yet. I'm frankly a little nervous to trust it with valuable data. I'll run it and something based on rsnapshot for the near term. Now I just have to figure out if I've got enough external drives for that, or do I need to buy more. I'll likely first try to run things so that the btrfs snapshots are running over the Internet, which will really cut down on the time and bandwidth needed. And as a backup use rsnapshot locally.
- vamega 12y agoMan, I keep seeing awesome posts about Backblaze, and really would like to use them. However I run exclusively on linux these days, and they don't seem to have a Linux client. Public pledge, if backblaze releases a linux client, even if it's command line only and requires that I manually edit an XML file, I'll purchase a 1 year subscription.
- epaladin 12y agoThe only issue I have with them is the removal of deleted files from backup archives after 30 days. Crashplan doesn't have that restriction (and also has a Linux client) although the Java-based engine is pretty clunky, and it eats memory like crazy to do client-side dedupe if you have it enabled. They've been promising native clients for a long time, and a sales guy earlier this year said v4.0 was going to be out Q1 this year, which the engineering team later denied ever saying... So I still don't know when the non-Java version is coming with its improved memory efficiency.
- deno 12y agoI really wish at some point JVM will stop being synonymous with memory–hog on desktop. It’s certainly possible, e.g. http://hiroshiyamauchi.blogspot.com/2013/06/making-jvm-release-memory.html http://hiroshiyamauchi.blogspot.com/2013/06/making-jvm-relea...
- toomuchtodo 12y ago> Public pledge, if backblaze releases a linux client, even if it's command line only and requires that I manually edit an XML file, I'll purchase a 1 year subscription. Consider how many Linux users would need to do this to a) pay for the developer time to write/maintain the client and b) pay for the support time for Linux issues
- pyre 12y agoThere are other things to consider too: developer community goodwill, people that may not be on Linux now, but only use systems that are cross-platform (to curb lock-in to a particular OS), etc. How much revenue is Atom for Linux expected to generate for Github?
- jtbigwoo 12y agoI always look forward to backblaze's posts. Does anybody know if drive manufacturers pay attention to their posts about reliability? I know when I worked with mainframes we sent every bad drive back to IBM for failure analysis, but would Backblaze's aggregate data be useful to somebody like Seagate?
- bluedino 12y ago>> would Backblaze's aggregate data be useful to somebody like Seagate? Probably not. They posted before that they don't even buy enough drives to purchase directly from the manufacturers.
- atYevP 12y agoYev from Backblaze here -> We've since spoken to some of the device manufacturers after some of our more recent posts. :-p
- pbreit 12y agoSo straightforward but still so fascinating.
- ksec 12y agoSeagate promised 8 / 10TB HDD within the next 10 months. Cant wait to see that.
- atYevP 12y agoYev from Backblaze here -> We've seen an 8TB drive, can't wait to get enough of them to drop in to a pod and test :)
- jewel 12y agoFor certain workloads these storage pods are much, much cheaper than S3. Anything where you are storing files that rapidly become stale, but still need to be instantly accessible for the rare random request. I've only looked at it from the perspective of video files, though. Where I work we add a gigabyte of data per user per month. Eventually our S3 storage bill is going to be our largest cost due to compounding growth.
- res0nat0r 12y agoAny idea what the durability of Backblaze is? S3 is 99.999999999%, is Backblaze anywhere near that? I can't seem to find it in their FAQ.
- Retric 12y agoAmazon is no where near that. Software glitches have already distroyed s3 data. More importantly anything over 99.999% has to take into account vary low probability events like nuclear war.
- res0nat0r 12y agoTheir FAQ that has been in place for years needs to be updated: http://aws.amazon.com/s3/faqs/ http://aws.amazon.com/s3/faqs/
- jewel 12y agoS3's number is just marketing. It's going to be that durable until a black swan event wipes out 0.1% of all files. I imagine they keep multiple copies of every file, spread across multiple data centers. It's durable against hardware failure of all varieties, but it's still vulnerable to a software bug. It's also possibly vulnerable to certain types of natural disasters. IIRC backblaze employs a similar strategy. They have the advantage that your home computer is still storing a copy, so even if they have a minor catastrophe they can recover by simply having their client re-upload those files.
- 12y ago
- dchest 12y agoBackblaze still shows country block due to sanctions that were dropped by US many years ago, for the country which no longer exists.
- dublinben 12y agoCan you elaborate on this? Where are they blocking their website?
- brianwski 12y agoBrianW from Backblaze -> yes, this is a silly situation we need to find time to fix. We blocked a list of countries something like 6 years ago and geopolitical boundaries and alliances have since changed. (sigh) Did I mention we have two open reqs for datacenter employees right now? Anybody want to come join us to help? We're in San Mateo, California. Hit up our "jobs" page...
- KaiserPro 12y agoWhat is interesting is that backblaze still use the 45 drive in 4u config. We use Dell/LSI/Netapp/engenio md3260s which pack 60 drives into the same space. We use raid 6, as we want the storage to manage the redundancy, not the software. With four hot spares we achieve a density of 170 tb usable per 4u (technically its 5U as we have a fileserver sitting on top. however assuming 48u it works out the same density) With backblaze, assuming raid 6(unless you are using FEC then there is no other way.) you'll only get 120TB per 4u (but a file server as well) The big advantage of the Dell/netapp/etc raid is that the drives are easily hot swappable. Ironically at the scale that backblaze are doing it, they are probably cheaper as well.
- deleted 12y ago[deleted]
- wmf 12y agoWe also have some of those Engenio JBODs at my office for "big data", but I assume they're much more expensive than the Storage Pod. My general experience with enterprisey hot-swap JBODs is that the enclosure costs more than the drives. Supermicro also has a 4x12 drive in 4U box that looks a little less "homemade" than the Backblaze Storage Pod.
- KaiserPro 12y agoYes and no, ours are pretty cheap, and they come with 24x7 4 hour response, which means that we don't have to worry about keeping lots of spares.
- revelation 12y agoI wonder if the power savings going from 4TB -> 6TB could make up for the difference/GB. Spinning disks still operate at relatively high power (~8W), and I don't think this value would change with capacity (they don't add extra disk heads, I think its just more platters/higher capacity platters). Pretty much the major cost running servers nowadays is simply the cost of the necessary electricity to run them and the cooling systems. (Note that this doesn't apply for Backblazes usecase; they fill harddisks up, then pretty much power them down. That is why you can't instantly access your data stored with them)
- insertion 12y agoI'm a happy Backblaze customer, but I've noticed it typically uses around 240MB of memory. Is this normal? Why is it so high?
- darkr 12y agoThe current desktop client runs atop a JVM
- zo1 12y agoTheir web page disagress with you: "No Java Software Java is responsible for 91% of security attacks. Backblaze's code is native to Mac and PC and doesn't use Java."* It even has a big logo with Java logo crossed out.
- mrchucklepants 12y agoI'm pretty sure their clients are all native.
- brianwski 12y agoBackblaze engineer here -> we absolutely DO NOT use a JVM on the client running on laptops. We love Java and use it in the datacenter on every web server and every pod. The reason we don't use it on the client running on laptops is twofold: 1) Java doesn't deploy super smoothly - The initial download might be 30 MBytes to include the JVM instead of 1 or 2 MBytes for a 'C' executable. Also, you have to keep updating the JVM separately, etc. It's friction to customers. 2) Java is hard to make look "native". Macintosh/Apple customers especially are sensitive to the look and feel of applications and like them to feel extremely "native".
- brianwski 12y agoBrianW from Backblaze here. That is a little high, but is plausible in some installations. When running Backblaze on a laptop, you will notice a couple different processes running. "bzserv" should always be very very small, it doesn't do much. When you see "bztransmit", that is the process that reads your files from disk, encrypts them in RAM, and transmits them to the Backblaze datacenter. It tends to be about 30 MBytes of executable code, plus up to another 30 MBytes of the file being encrypted in RAM and transmitted. So most customers will see bztransmit bounce around at 60 MBytes RAM. HOWEVER, there are short moments when bztransmit must read a list of files into RAM, and it can reach larger sizes like 240 MBytes. bztransmit does the deduplication, which means it must store one SHA-1 of every file it has transmitted from your computer to our datacenter. If you have a lot of files that have been transmitted (over 1 million files is "a lot") then Backblaze might reach 200 MBytes in size.
- lucb1e 12y agoI don't understand this: 6TB drives are more expensive per gigabyte and they even mention it in their post. They say that they expect the drives to become cheaper over time and that's why they are switching now. Wait, what? Switching now when they are, I quote, "at the top of the curve"[1] does not make sense? It might be that the physical space reduction compensates for it, but they don't mention that so I don't get it. [1] https://www.backblaze.com/blog/wp-content/uploads/2014/08/blog-cost-per-gb-generic2.jpg https://www.backblaze.com/blog/wp-content/uploads/2014/08/bl...
- ghshephard 12y agoMy read was they they aren't switching over everything - they are testing new drives, to figure out what reliability looks like. Once the cost curve costs over, they will then know which vendors drives to go all in on.
- lucb1e 12y agoOh, that explains, thanks :)
- delucain 12y agoYeah, I don't think they replace drives in pods until they fail. Even if they're old 1TB models. The drive costs just outstrip their infrastructure costs by so much that 4 racks of 1TB drives is still cheaper than retrofitting even 1 rack with new 4TB drives.
- deleted 12y ago[deleted]
- timc3 12y agoNot just the physical space, but also the power and cooling and serving requirements might tip the balance in favour of a higher upfront cost, but lower lifetime cost.
- 12y ago
- brokentone 12y agoI'm wondering why Backblaze hasn't been moving toward cold storage? With a wait time on recovering/downloading files, I don't see an obvious reason why not. But I can see some cost and energy savings to be had.
- jonknee 12y agoLikely because they keep deleting stuff which is difficult in cold storage. They keep stuff for 30-days after you delete it. https://www.backblaze.com/remote-backup-everything.html https://www.backblaze.com/remote-backup-everything.html > Backblaze will keep versions of a file that changes for up to 30 days. However, Backblaze is not designed as an additional storage system when you run out of space. Backblaze mirrors your drive. If you delete your data, it will be deleted from Backblaze after 30 days.
- bumbledraven 12y agoI stopped using backblaze when I learned that they require you to TRANSMIT YOUR PRIVATE KEY TO THEIR SERVER in order to restore your files from backup.
- jonknee 12y agoI was considering signing up and just noticed this on their site... Definitely odd. I understand they want to be easy to use and feel seamless, but it defeats the purpose of the password protected private key. https://www.backblaze.com/backup-encryption.html https://www.backblaze.com/backup-encryption.html > To decrypt your data, you are required to enter your passphrase on our secure website. When you do so, it is passed over an encrypted connection to our datacenter where it is used to decrypt your private key, which in turn is used to decrypt your data. Your passphrase is never saved on disk and it is discarded once it is used. As before, once we decrypt your data on our secure restore servers we then zip it and send it over an encrypted SSL connection to your computer. Once it arrives on your computer, you can unzip it and you have your data back.
- brianwski 12y agoBrian from Backblaze here. To be clear, there are two levels of security/encryption at Backblaze: 1) The friendliest way we could design for people to restore their files was to allow customers to sign into a website with a username/password and recover one or more files. This is the default situation. 2) You can optionally turn on a "private encryption key" but if you do that, understand you MUST write down that key because if you lose it, you can never recover it, and Backblaze (nor any government organization) will EVER be able to recover your files. NEVER. LOSE THAT PASSWORD AND THEY ARE GONE GONE GONE! In the case of #2, as long as you don't need to recover from a crash, you don't enter your private encryption key and nobody will ever have access to your files, period. However, if you lose a file, you have to sign into the Backblaze website and provide your passphrase which is ONLY STORED IN RAM for a few seconds and your file is decrypted. Yes, you are now in a "vulnerable state" until you download then "delete" the restore at which point you are back to a secure state. If you are even more worried about the privacy of your data, we highly recommend you encrypt it EVEN BEFORE BACKBLAZE READS IT on your laptop! Use TrueCrypt. Backblaze backs up the TrueCrypt encrypted bundle having no idea at all what is in it (thank goodness) and you restore the TrueCrypted bundle to yourself later.
- Keyframe 12y agoThis is really great! Anyone has any idea what kind of RAID are they running on their pods and what kind of read and write speed one can get from one of these?