5 ms·

Good rant :) As the co-owner of a hosting provider with its own data centre I might be biased but I totally reject "You can’t trust your datacenter anymore whe

by mattbee 12y ago

Good rant :) As the co-owner of a hosting provider with its own data centre I might be biased but I totally reject "You can’t trust your datacenter anymore when you’re in the cloud" argument. You _need_ your backup server to do some lifting for you to make half of that wishlist work. If you can't trust your provider not to read your unencrypted some of the time data, you're using the wrong provider, and making backups much harder than they need to be. Also a lot of that wishlist seems to ignore network realities - you can do a lot more with a backup server in the next rack, or one that's the end of a private link, than you can with a very remote storage provider, so insisting on the same tools no matter where your backups are located seems a bit hopeful. I've worked on a backup system for our customers called "byteback" which I'm slowly finishing off and documenting - will only be 2-3000 lines of Ruby when finished. It currently leans on rsync, ssh and btrfs's copy-on-write snapshots to keep efficient copies of entire servers. There's a server-side pruning algorithm that allows the disc to fill up with daily snapshots for multiple servers, then prunes the least "useful" ones. It's trying to be zero-configuration, so it builds its list from a list of "local" filesystems, so that you can copy the whole snapshot back quickly to restore a system. The only other feature I'm going to need is to automatically drive "snapshot" functionality when it finds it on the server - e.g. LVs, btrfs subvolumes and other points on the filesystem where it can make a safe snapshot, it should do that automatically. The zero-configuration rationale is just that where we've had backup failures, it's been manual misconfigurations and misguided attempts to be "efficient" that caused important files to be missed. So I'm trying to bake in defaults and features to cover every mistake we've ever made :) As the name implies it's made for our customers, and our defaults, but I'm pretty sure it'll work for a lot of other server use cases as well. I'm still going over a few 10s of live backups fixing problems and adding new defaults as I find them. If anyone's interested, shout and I'll try to push on with the documentation and put it up in its current state.

mpasternacki 12y ago
Original ranter here, thanks :) Regarding "trusting the datacenter": it was an overstatement, but while I can risk my own data, I am double-wary with my clients' data, and triple-wary with my clients' users' data. And while I don't assume that cloud provider is necessarily evil and unreliable, I need to survive losing a provider (be it because provider went bankrupt, or because I tried out GAE and got locked out in a Google Checkout mishap, or because of whatever reason), and I cannot rule out possibility of a leak (think Dropbox: hindsight's 20/20, but I don't believe I can predict which one of cloud storages will be next Dropbox). An extra bonus of good encryption is that I can spread storage over cheaper, but less reliable providers. Your remark about ignoring network realities is good, thanks! I work mostly with cloud or remote servers, it's been at least 10 years since lat time I in the same room as a server I manage (and back then it was an office Samba file/print server). Maybe my perspective shows here, and it may be just as limiting as older tools just phrasing everything as "tapes" and "autochangers". I am closely looking at FreeBSD/ZFS right now, and this may not fit well with `zfs send`-based backups. I'd be definitely interested in taking a closer look at Byteback! Since it's based on btrfs' snapshots, it may also work well with zfs (and may be exactly the plumbing I am about to write soon). If you manage to publish it, please let me know. Not sure if my email is listed on my HN profile: it's maciej at pasternacki dot net. Thanks!
- ScottBurson 12y ago
  Have you looked at Tahoe-LAFS? I don't think it's a complete solution, but it might be part of one.
rev_bird 12y ago
> If you can't trust your provider not to read your unencrypted some of the time data, you're using the wrong provider, and making backups much harder than they need to be. This could just be me acting like a post-Snowden jaded American, but the providers themselves aren't whom I'm skeptical of anymore. When the author wrote that a company's "cloud" could be spread out between "five data centers, three continents, three hosting providers, and two storage providers," they seemed to be implying that there are a lot of opportunities for a company other than yours to hire someone just shady enough to do some damage. What I first thought of, though, was that were sooo many companies for a government to confront, demand access, and then threaten to ruin them if they told anybody.
- mattbee 12y ago
  Yeah, it's important to know where your service provider is, both physically and judicially. The majority of hosting providers are surely located in just one country. Amazon, Google, Microsoft are the exceptions here. Mainly because of http://www.rte.ie/news/business/2014/0904/641416-us-microsoft-data/ http://www.rte.ie/news/business/2014/0904/641416-us-microsof... - I wonder whether this is still being fought?

by mattbee 12y ago

mpasternacki 12y ago

Original ranter here, thanks :) Regarding "trusting the datacenter": it was an overstatement, but while I can risk my own data, I am double-wary with my clients' data, and triple-wary with my clients' users' data. And while I don't assume that cloud provider is necessarily evil and unreliable, I need to survive losing a provider (be it because provider went bankrupt, or because I tried out GAE and got locked out in a Google Checkout mishap, or because of whatever reason), and I cannot rule out possibility of a leak (think Dropbox: hindsight's 20/20, but I don't believe I can predict which one of cloud storages will be next Dropbox). An extra bonus of good encryption is that I can spread storage over cheaper, but less reliable providers. Your remark about ignoring network realities is good, thanks! I work mostly with cloud or remote servers, it's been at least 10 years since lat time I in the same room as a server I manage (and back then it was an office Samba file/print server). Maybe my perspective shows here, and it may be just as limiting as older tools just phrasing everything as "tapes" and "autochangers". I am closely looking at FreeBSD/ZFS right now, and this may not fit well with `zfs send`-based backups. I'd be definitely interested in taking a closer look at Byteback! Since it's based on btrfs' snapshots, it may also work well with zfs (and may be exactly the plumbing I am about to write soon). If you manage to publish it, please let me know. Not sure if my email is listed on my HN profile: it's maciej at pasternacki dot net. Thanks!

ScottBurson 12y ago
Have you looked at Tahoe-LAFS? I don't think it's a complete solution, but it might be part of one.

rev_bird 12y ago

> If you can't trust your provider not to read your unencrypted some of the time data, you're using the wrong provider, and making backups much harder than they need to be. This could just be me acting like a post-Snowden jaded American, but the providers themselves aren't whom I'm skeptical of anymore. When the author wrote that a company's "cloud" could be spread out between "five data centers, three continents, three hosting providers, and two storage providers," they seemed to be implying that there are a lot of opportunities for a company other than yours to hire someone just shady enough to do some damage. What I first thought of, though, was that were sooo many companies for a government to confront, demand access, and then threaten to ruin them if they told anybody.

mattbee 12y ago
Yeah, it's important to know where your service provider is, both physically and judicially. The majority of hosting providers are surely located in just one country. Amazon, Google, Microsoft are the exceptions here. Mainly because of http://www.rte.ie/news/business/2014/0904/641416-us-microsoft-data/ http://www.rte.ie/news/business/2014/0904/641416-us-microsof... - I wonder whether this is still being fought?