5 ms·

Earlier we were totally interested in ZFS, as it would replace RAID & LVM as well (and ZFS gets great reviews). But (to my understanding) native ZFS is not ava

Earlier we were totally interested in ZFS, as it would replace RAID & LVM as well (and ZFS gets great reviews). But (to my understanding) native ZFS is not available on Linux and we're not really looking to switch to OpenSolaris. ANOTHER option down this line of thinking is switching to btrfs, but we haven't played with it yet. By the way, at Backblaze I've felt like we have had to implement several things that I would have guessed would be standard "off the shelf". One example: When a customer wants to download a restore file with a web browser, are you aware there are no checksums for over-the-network transfers other than the built in (completely unacceptable) 16-bit TCP checksum? You are virtually guaranteed to have an undetected corruption within 30 Gbytes of download, which is basically what we like to call "a totally average customer restore". So Backblaze had to write our own custom reliable, restartable "downloader". It boggles my mind that the whole internet is throwing undetected errors on HTTP downloads and nobody cares to fix the protocol?!! Where the heck is Google, Apple, Facebook, Microsoft, or <insert standards body> defining a standard for web browser downloads larger than a few GBytes?

zerosanity 15y ago
We're using FreeBSD's ZFS support and haven't had any hiccups to complain about.
- thaumaturgy 15y ago
  We've had a backup server running for a client with FreeBSD's ZFS for about a year now, also with zero hiccups. We first tried to get this kind of stuff working about three or four years ago, and went the OpenSolaris/ZFS route -- what a pain that was. OpenSolaris somehow corrupted its own boot partition one day and just completely refused to start with a totally cryptic error code. The last time we tried Linux for this, it couldn't do it. Maybe that's gotten better though. We also tried DragonflyBSD, but it had hardware support issues, and OpenBSD, but unfortunately OpenBSD just simply cannot do large filesystems. At all. ZFS-on-FreeBSD is the way to go at the moment, I think.
- finnh 15y ago
  ZFS on FreeBSD works a charm - I was happy to switch to FreeBSD after having done OpenSolaris & Nexenta specifically for ZFS. I actually use ZFS on Mac OS, as well, and have been happy with it there (thanks Dustin Sallings!).
db48x 15y ago
Actually, zfs-on-linux is doing pretty well. It's not production ready yet, of course. As for HTTP, I agree with you entirely! There is actually a standard that lets you put a hash (SHA-1 or SHA-256 or something) on an http anchor link, and the browser will verify it when the download finishes. It hasn't really gained very wide adoption though. Personally, I think something akin to Bittorrent is a better solution, since it doesn't have to redownload the entire file when it detects an error. It's ironic that our videos often have better data integrity than the graphics driver installer that I just downloaded, or the browser that I downloaded yesterday.
silvestrov 15y ago
The OSX Lion download is +3 GB, so Apple customers must take care of TCP errors (or at least Apple Support would be swamped with support requests). Two possible ways of solving this: make a download into e.g. 16 MB chunks and append a SHA-1 checksum for each chunk (not enterely unlike BitTorrent). Then re-download the chunks where the SHA-1 doesn't match. Another solution would be to use e.g. Red Solomon error correction.
- brianwski 15y ago
  Yep, Apple solves this problem by using their "AppStore" application to reliably download the file in chunks (like you suggested). My complaint is that Apple should extend Safari to be able to download the OS X Lion download reliably. (Then let the rest of us use that protocol.) It seems a bit insane that the OS group at Apple had to write their own custom download application, but did not think to extend their web browser to allow for "restartable downloads that are no longer corrupted". Microsoft does the same thing -> they have a custom application to download their OS updates reliably. They do not use Internet Explorer's regular download capability because of the limitations. It baffles me that the big browser companies (Google with Chrome, Apple with Safari, Microsoft with IE) are not interested in building a standard reliable downloader. I mean, what else does a web browser really do?
andrewf 15y ago
gzip and zlib HTTP encodings (different containers for "deflate" compression) include a checksum of the uncompressed content (a CRC-32 variant and Adler-32 respectively). If compression isn't wanted, you can wrap uncompressed bytes with a deflate structure - although it'll have to be split into ~64k chunks with a few header bytes ahead of each. I'm not sure which HTTP client implementations reliably test those checksums, though. (Or whether CRC32/Adler32 would be adequate for 30 gigs!)
- mrb 15y ago
  And when using HTTPS, all corruption introduced by the network stack will be detected (thanks to the MAC in each TLS record). But brianwski's point remains that neither HTTP encodings, nor HTTPS can detect certain errors like in-memory corruption of the data after it has been read from disk, but before it has been sent out over the HTTP connection.
brmj 15y ago
Solaris isn't your only option. Various BSDs support ZFS quite well.
brmj 15y ago
Solaris isn't your only option. Various BSDs support ZFS quite well.

zerosanity 15y ago

We're using FreeBSD's ZFS support and haven't had any hiccups to complain about.

thaumaturgy 15y ago
We've had a backup server running for a client with FreeBSD's ZFS for about a year now, also with zero hiccups. We first tried to get this kind of stuff working about three or four years ago, and went the OpenSolaris/ZFS route -- what a pain that was. OpenSolaris somehow corrupted its own boot partition one day and just completely refused to start with a totally cryptic error code. The last time we tried Linux for this, it couldn't do it. Maybe that's gotten better though. We also tried DragonflyBSD, but it had hardware support issues, and OpenBSD, but unfortunately OpenBSD just simply cannot do large filesystems. At all. ZFS-on-FreeBSD is the way to go at the moment, I think.
finnh 15y ago
ZFS on FreeBSD works a charm - I was happy to switch to FreeBSD after having done OpenSolaris & Nexenta specifically for ZFS. I actually use ZFS on Mac OS, as well, and have been happy with it there (thanks Dustin Sallings!).

db48x 15y ago

Actually, zfs-on-linux is doing pretty well. It's not production ready yet, of course. As for HTTP, I agree with you entirely! There is actually a standard that lets you put a hash (SHA-1 or SHA-256 or something) on an http anchor link, and the browser will verify it when the download finishes. It hasn't really gained very wide adoption though. Personally, I think something akin to Bittorrent is a better solution, since it doesn't have to redownload the entire file when it detects an error. It's ironic that our videos often have better data integrity than the graphics driver installer that I just downloaded, or the browser that I downloaded yesterday.

silvestrov 15y ago

The OSX Lion download is +3 GB, so Apple customers must take care of TCP errors (or at least Apple Support would be swamped with support requests). Two possible ways of solving this: make a download into e.g. 16 MB chunks and append a SHA-1 checksum for each chunk (not enterely unlike BitTorrent). Then re-download the chunks where the SHA-1 doesn't match. Another solution would be to use e.g. Red Solomon error correction.

brianwski 15y ago
Yep, Apple solves this problem by using their "AppStore" application to reliably download the file in chunks (like you suggested). My complaint is that Apple should extend Safari to be able to download the OS X Lion download reliably. (Then let the rest of us use that protocol.) It seems a bit insane that the OS group at Apple had to write their own custom download application, but did not think to extend their web browser to allow for "restartable downloads that are no longer corrupted". Microsoft does the same thing -> they have a custom application to download their OS updates reliably. They do not use Internet Explorer's regular download capability because of the limitations. It baffles me that the big browser companies (Google with Chrome, Apple with Safari, Microsoft with IE) are not interested in building a standard reliable downloader. I mean, what else does a web browser really do?

andrewf 15y ago

gzip and zlib HTTP encodings (different containers for "deflate" compression) include a checksum of the uncompressed content (a CRC-32 variant and Adler-32 respectively). If compression isn't wanted, you can wrap uncompressed bytes with a deflate structure - although it'll have to be split into ~64k chunks with a few header bytes ahead of each. I'm not sure which HTTP client implementations reliably test those checksums, though. (Or whether CRC32/Adler32 would be adequate for 30 gigs!)

mrb 15y ago
And when using HTTPS, all corruption introduced by the network stack will be detected (thanks to the MAC in each TLS record). But brianwski's point remains that neither HTTP encodings, nor HTTPS can detect certain errors like in-memory corruption of the data after it has been read from disk, but before it has been sent out over the HTTP connection.

brmj 15y ago

Solaris isn't your only option. Various BSDs support ZFS quite well.

brmj 15y ago

Solaris isn't your only option. Various BSDs support ZFS quite well.