7 ms·
Bup: Efficient file backup system based on the git packfile format
- deleted 13y ago[deleted]
- mattdeboard 13y agoI'm assuming you're not the author. But just in case the author wanders by: How did you decide which parts to write in C?
- staticshock 13y agoThe author is https://news.ycombinator.com/user?id=apenwarr https://news.ycombinator.com/user?id=apenwarr, though it looks like he isn't the most active maintainer nowadays.
- nathancahill 13y agoNot the author either, but I've made that decision based on profiling the code. It's generally easy to see which functions are slowing down performance and can be refactored to a more performant language.
- apenwarr 13y agoIt's not too hard actually. A line of python is roughly 80x slower than a line of C (no exaggeration). But a typical line of python does a lot more than a typical line of C. So things you can do with a "loose" loop (like once per 64k block) is usually ok in python. Things you have to do with a "tight" loop (like once per byte) need to be in C. I once did a presentation about python performance optimization lessons from bup: http://lanyrd.com/2011/pycodeconf/sghxk/ http://lanyrd.com/2011/pycodeconf/sghxk/ And it's true, I'm not the most active maintainer anymore. The people who took over seem to be doing a pretty good job though.
- tekacs 13y agoI'd be curious to know if your stance on PyPy has changed at all since 2011 (if indeed it's something that you've taken any new long looks at since) given their progress in that time. I know that I would humbly submit at the least that my position has moved to believing that PyPy is a viable option for high-speed code (albeit in substantial part due to better interaction with C, nowadays).
- gcr 13y agoBup is lovely. I used it to back up my huge home folder and only switched away to rdiff-backup because (at the time) there was no support for deleting old revisions. Is there any support for that? (Of course, for a large enough hard drive, it's not much of a problem...)
- jitl 13y agoLooking at the README under "Stuff that is stupid": > bup currently has no way to prune old backups Thanks for the rdiff-backup shout-out. I'm looking for a nice way to do system backups to my NAS of large VM images without having to install Crashplan. Bup and rdiff-backup both look pretty good.
- apenwarr 13y agoPeople have been actively working on a "prune" feature but it seems to never quite get finished. This would indeed be nice to have, although it's less important than you might think, given really good deduplication (which bup has). Currently bup has a very simple model - never delete anything - which is hard to screw up, so you're very unlikely to lose data.
- bambambazooka 13y agoI really have to finish that feature...
- rlpb 13y ago
- gesman 13y agoThis is an outstanding project with great potential. Killer for many commercial, overpriced services.
- natch 13y agoHow do people who would use this kind of thing manage to have remote servers with terabytes of available disk space on them? Anything is possible with money, of course, but how is this anything other than really expensive? For example AWS S3 would be $235/month (that's $2,820/year!) for 3TB not even including any data-out transfer charges. Sure there are others that are cheaper but only marginally so. Is this really what people are doing? Makes the commercial services sound really cheap.
- apenwarr 13y agoMy suggestion is to backup your cloud servers, which are expensive and redundant and have good uplink speeds, to home servers which are cheap and have good downlink speeds. You don't need your backup file server to be ultra-reliable or even up all the time, so the cheapest possible PC sitting on a home internet connection is a pretty good choice. That way, 3TB is just $150 or so plus your electricity, and it's not a per-month fee.
- nisa 13y agoIf you ignore the cloud services and rent dedicated servers you can get up to 6 TB disk space for 50$ a month. edit: 45tb for 300€/month: https://www.hetzner.de/en/hosting/produkte_rootserver/xs29 https://www.hetzner.de/en/hosting/produkte_rootserver/xs29
- drdaeman 13y ago
- atmosx 13y agoThe most efficient backup system for operating systems I've used so far is 'tarsnap'. The only drawback is that restore is really slow.
- tekacs 13y agoI love Tarsnap, but S3 storage costs aren't exactly brilliant. Figuring out exactly how you wish to store keys can also be another thought, upfront (albeit one that arises from the increased security that you get 'for free'). Alternately, CrashPlan and other consumer-style services have a bad habit of using very slow, heavy, world-slowing systemwide file update scanning. :/ Having said this, a discussion of the merits and flaws of Tarsnap and similar backup services is something I'm fairly certain I've seen lengthy discussions of on similar HN posts. (https://news.ycombinator.com/item?id=5767116 https://news.ycombinator.com/item?id=5767116 is a good source for lots of that sort of discussion)
- tobias2014 13y agoAnother backup possibility I currently use: ZFS on a backup server (not necessarily ZFS on the system that should be backed up), pull data with rsync on the backup host to a ZFS, after that make a snapshot for an "incremental backup". So simplified it's like: rsync -avx remote:/etc /backup/ && zfs snapshot backup@`date` With zfSnap (https://github.com/graudeejs/zfSnap https://github.com/graudeejs/zfSnap) you can tell how long incremental backups/snapshots are kept, "rsync && zfSnap -d -a 1w backup" You can take advantage of the /backup/.zfs/snapshot directory to access all snapshots, built-in compression and possible data deduplication. If you also have ZFS on the remote host, you can use zfs send and zfs receive to transfer the snapshot directly to the backup server, instead of using rsync for the diff.
- tekacs 13y agorsync.net is an example of a host that does something like this (with daily snapshots) $> ssh rsyncnet ls .zfs/snapshot daily_2014-02-09 daily_2014-02-10 daily_2014-02-11 daily_2014-02-12 daily_2014-02-13 daily_2014-02-14 daily_2014-02-15 They allow you to customise the length of time for which these snapshots are kept, too (IIRC) (at the cost to you of the incremental extra storage)
- res0nat0r 13y agorsync.net looks awesome, but it is just way too expensive for me.
- cbhl 13y agoHave you asked for the HN discount? https://news.ycombinator.com/item?id=6554313 https://news.ycombinator.com/item?id=6554313 (I happen to get a discount for being a prgmr.com user, but IIRC the terms are similar.)
- rsync 13y agoYou should email us and ask about the "new customer HN reader discount". Also, note that 1TB accounts are 15c per GB, per month, and 10TB accounts are 7c per GB, per month - all with no traffic or usage costs. This compares very favorably with S3 and blows the mozy pro pricing out of the water[1]. [1] https://mozy.com/product/mozy/business https://mozy.com/product/mozy/business
- kimjotki2 13y agoa 'backup system' that runs on python. how oxymoron.
- jrockway 13y agoSeriously. I write all my critical software in assembly so that my super-fast disks and networks aren't bottlenecked by unnecessary CPU instructions! Backup software always values speed over correctness!
- isomorphic 13y agoYou mean like rdiff-backup, which I've been using in production against millions of files for more than half a decade without a single problem? It may be that some languages or runtimes host consistently more reliable software than others, but I'd bet that the individual programmer, coding style and practice have more of an effect on reliability.
- qwerta 13y agoHow is that different from shell script which calls programs?
- leephillips 13y agoNot really closely related, but another solution that uses git infrastructure to back up large files is git-annex: https://git-annex.branchable.com/ https://git-annex.branchable.com/
- rsync 13y ago... and there are some tangential benefits to using git-annex as well: http://rsync.net/products/git-annex-pricing.html http://rsync.net/products/git-annex-pricing.html
- bambambazooka 13y agoand git-annex has a bup backend :)
- atso 13y agoI hear about rdiff-backup, but I think its two main drawbacks are: * on the webpage, there is no new release since 2009. * has no de-duplication. I was considering moving my 5+ years old rdiff-backup system to any of those new, promising programs: * obnam [http://liw.fi/obnam/ http://liw.fi/obnam/] * attic [https://pythonhosted.org/Attic/ https://pythonhosted.org/Attic/] They both do automatic de-duplication, old backup deletion and remote encryption.
- limmeau 13y agoThe original rdiff-backup author went on to create duplicity. Maybe a big part of the rdiff-backup community has followed him? I'm using obnam now.
- sciurus 13y agoFor a link, http://duplicity.nongnu.org/ http://duplicity.nongnu.org/ I backup locally using duplicity, then I ship those files off to Amazon Glacier using mt-aws. https://github.com/vsespb/mt-aws-glacier https://github.com/vsespb/mt-aws-glacier
- brimstedt 13y agoso, its 2014 and still people use homegrown variations of tar, rsync, git and whatnot. Or a half done solution like this, or an abandoned solution like box backup. why on earth isnt there already a perfect cross platform open source backup program? :) I know,i know..why dont i make one myself? because we dont need a nother half done solution :-b
- brimstedt 13y ago(dont mean tobsay this is a bad product or anything, just not a finished product..)
- bambambazooka 13y agoyou can help finish it
- gcr 13y agoEvery "perfect cross platform open source program" had to start as a homegrown variation of tar, rsync, git, and whatnot. It's not like they fall out of the sky.
- brimstedt 13y agoof course things dont come out of nothing, i just find it so odd that we have open source top quality OSes, monitoring systems, programming languages, ides, browsers, graphics suites, etc etc, but no backup ..
- pjc50 13y agoNot having a backup isn't a "pain point". Until you have a disaster. And disasters are infrequent enough ..