4 ms·

This is why it's so important to take some simple steps to ensure that you don't bork your site. For my production sites, I usually have the following setup:

by mdasen 18y ago

This is why it's so important to take some simple steps to ensure that you don't bork your site. For my production sites, I usually have the following setup: * Replicated Database with one slave in a separate data center just as disaster recovery (not a performance thing with the separate data center). Heartbeat for the slave(s) in the same datacenter so that it stays up. * Nightly backups offsite. Used to be rsync, but recently thinking of using tarsnap for the ease of it (and S3 puts it in several data centers). * Files stored either in a MogileFS setup or S3 in multiple data centers. It doesn't have instant failover to another data center, but the offsite DB slave should mean no data loss beyond a second or two. Ma.gnolia is a decent sized site. Maybe they did have a decent infrastructure and it will only be a little while before they've gotten everything back. Of course, after the fiasco with the blog site that used RAID as their backup system, I've started to think that many people don't take data as seriously as it needs to be taken.

timf 18y ago
I like the idea of the offsite slave. I was also reading about another idea, forcing a slave delay to recover from oopses/corruption: http://www.rustyrazorblade.com/2008/05/07/mysql-time-delayed-replication/ http://www.rustyrazorblade.com/2008/05/07/mysql-time-delayed...
- eli 18y ago
  I do something like that for my outgoing email ;)
t0pj 18y ago
"take some simple steps to ensure that you don't bork your site" unborked[.com] is currently available. A site dedicated to stopping all of the borking going on out there. === I think some cheap, simple sql dumps from the db, burned to an external HD and taken home would've been a good first step -- your data would be dated but somehow restorable nonetheless without the need for off-site, replicated, heart-beat monitored databases. IMHO you really need an expert database admin (of which I am not) to get all of this up and keep all of it going; a caretaker who's only job in life is the maintenance of any and all persisted data. On another note, backups are really useless unless you have actually tested the restores.
- jacquesm 18y ago
  what does 'burning to an external harddrive' do ? is this some new tech that I'm unfamiliar with ;) ?
bprater 18y ago
I'd really like to store incrementals of my MySQL at S3. Anyone using something they really like? Or should I just break-down and do full dumps each night?
- fendale 18y ago
  Then you can still lose a days data. Why not switch and ship the mysql binary log every x minutes? Last backup plus all the logs gives you much better recoverability
- mdasen 18y ago
  Tarsnap. Tarsnap lets you say: tarsnap -c -f backup01302009 mysql_dir/ And you can just adjust the date each day. It gives you the luxury of a full dump (anytime you want to restore, just reference backup01302009), but it only actually stores the deltas (making sure not to duplicate data that might be in backup01292009 or backup01282009 and so on). Tarsnap stores the data to S3 so that it's replicated in multiple data centers. It costs a little more than S3 at 30 cents per GB, but it's metered out so that if you only use 1MB of storage, you'll only be charged 0.03 cents for that storage. You could try creating your own way of doing incrementals, but I doubt you'd get it as efficient as Colin (the math genius behind Tarsnap) and so I doubt you'd get it cheaper. Plus, this way you don't have to deal with it. And remember, it's hard to fill up a database.* As the Django Book notes: "LJWorld.com's database - including over half a million newspaper articles dating back to 1989 - is under 2GB." So, if they were using Tarsnap, they might be storing 5 or 10GB tops at a whopping $1.50-$3 per month plus whatever the transfer of their deltas was for the month. Oh, and tarsnap compresses the data too. So, maybe they'd be paying $1 or something lower. * Clearly, if you hit it big time, you might not want to continue paying for tarsnap. However, if you become the next big thing, you can hire someone to deal with it for you.
  - paddy_m 18y ago
    EBS does deltas too. Is anyone else using it? I like the ability to mount a volume or clone a volume almost instantly and mount it on another machine.
anuraggoel 18y ago
Reminds me of the Couchsurfing fiasco back in 2006. http://www.techcrunch.com/2006/06/29/couchsurfing-deletes-itself-shuts-down/ http://www.techcrunch.com/2006/06/29/couchsurfing-deletes-it... http://www.couchsurfing.com/crash_page.html http://www.couchsurfing.com/crash_page.html Couchsurfing had a backup process in place, but failed due to human error. It's highly likely that magnolia had backups of some sort. Before we all start berating magnolia (and website owners in general) for not backing up their data, we need to know more about the real reason for this failure. Couchsurfing was able to come back from the dead because of limited competition. In this case, unfortunately, it will be easier for users to look elsewhere.
- jodrellblank 18y ago
  Reminds me of LeafyHost. (Summary: Small hosting company advertising multiple backups; their single server crashed and they had no backups at all. Long story involving terrible customer service ensues).
- eli 18y ago
  You're right that we probably shouldn't jump to conclusions, but it doesn't look good. And human error shouldn't really be possible if your backups are automated and you periodically do test restores.

by mdasen 18y ago

timf 18y ago

I like the idea of the offsite slave. I was also reading about another idea, forcing a slave delay to recover from oopses/corruption: http://www.rustyrazorblade.com/2008/05/07/mysql-time-delayed-replication/ http://www.rustyrazorblade.com/2008/05/07/mysql-time-delayed...

eli 18y ago
I do something like that for my outgoing email ;)

t0pj 18y ago

"take some simple steps to ensure that you don't bork your site" unborked[.com] is currently available. A site dedicated to stopping all of the borking going on out there. === I think some cheap, simple sql dumps from the db, burned to an external HD and taken home would've been a good first step -- your data would be dated but somehow restorable nonetheless without the need for off-site, replicated, heart-beat monitored databases. IMHO you really need an expert database admin (of which I am not) to get all of this up and keep all of it going; a caretaker who's only job in life is the maintenance of any and all persisted data. On another note, backups are really useless unless you have actually tested the restores.

jacquesm 18y ago
what does 'burning to an external harddrive' do ? is this some new tech that I'm unfamiliar with ;) ?

bprater 18y ago

I'd really like to store incrementals of my MySQL at S3. Anyone using something they really like? Or should I just break-down and do full dumps each night?

fendale 18y ago
Then you can still lose a days data. Why not switch and ship the mysql binary log every x minutes? Last backup plus all the logs gives you much better recoverability
mdasen 18y ago
Tarsnap. Tarsnap lets you say: tarsnap -c -f backup01302009 mysql_dir/ And you can just adjust the date each day. It gives you the luxury of a full dump (anytime you want to restore, just reference backup01302009), but it only actually stores the deltas (making sure not to duplicate data that might be in backup01292009 or backup01282009 and so on). Tarsnap stores the data to S3 so that it's replicated in multiple data centers. It costs a little more than S3 at 30 cents per GB, but it's metered out so that if you only use 1MB of storage, you'll only be charged 0.03 cents for that storage. You could try creating your own way of doing incrementals, but I doubt you'd get it as efficient as Colin (the math genius behind Tarsnap) and so I doubt you'd get it cheaper. Plus, this way you don't have to deal with it. And remember, it's hard to fill up a database.* As the Django Book notes: "LJWorld.com's database - including over half a million newspaper articles dating back to 1989 - is under 2GB." So, if they were using Tarsnap, they might be storing 5 or 10GB tops at a whopping $1.50-$3 per month plus whatever the transfer of their deltas was for the month. Oh, and tarsnap compresses the data too. So, maybe they'd be paying $1 or something lower. * Clearly, if you hit it big time, you might not want to continue paying for tarsnap. However, if you become the next big thing, you can hire someone to deal with it for you.
- paddy_m 18y ago
  EBS does deltas too. Is anyone else using it? I like the ability to mount a volume or clone a volume almost instantly and mount it on another machine.

anuraggoel 18y ago

Reminds me of the Couchsurfing fiasco back in 2006. http://www.techcrunch.com/2006/06/29/couchsurfing-deletes-itself-shuts-down/ http://www.techcrunch.com/2006/06/29/couchsurfing-deletes-it... http://www.couchsurfing.com/crash_page.html http://www.couchsurfing.com/crash_page.html Couchsurfing had a backup process in place, but failed due to human error. It's highly likely that magnolia had backups of some sort. Before we all start berating magnolia (and website owners in general) for not backing up their data, we need to know more about the real reason for this failure. Couchsurfing was able to come back from the dead because of limited competition. In this case, unfortunately, it will be easier for users to look elsewhere.

jodrellblank 18y ago
Reminds me of LeafyHost. (Summary: Small hosting company advertising multiple backups; their single server crashed and they had no backups at all. Long story involving terrible customer service ensues).
eli 18y ago
You're right that we probably shouldn't jump to conclusions, but it doesn't look good. And human error shouldn't really be possible if your backups are automated and you periodically do test restores.