4 ms·
Ma.gnolia.com crashes hard -- no backups?
- sam_in_nyc 18y agoHis note says it's January 31st... What will get fixed first? The date, or the data? I'm betting on data.
- whatusername 18y agoI don't know about you - but it's January 31st for me.. :P (although I do believe it was just a typo)
- jjs 18y agoTell us what the future is like! ;-)
- whatusername 18y agovery hot down here.. Just had 3 days in a row over 43 (109) and todays pretty warm as well. (my house still hasn't cooled down for sure)
- quoderat 18y agoThe backup strategy, in my experience, is the first or second thing you need to be thinking about when you set up a server, service or anything IT-related. The first thing I ask when my boss tells me to set up something new: how are we going to back that up?
- eli 18y agoIt's a classic dilemma -- most people don't care about backups until immediately after it's too late.
- mdasen 18y agoThis is why it's so important to take some simple steps to ensure that you don't bork your site. For my production sites, I usually have the following setup: * Replicated Database with one slave in a separate data center just as disaster recovery (not a performance thing with the separate data center). Heartbeat for the slave(s) in the same datacenter so that it stays up. * Nightly backups offsite. Used to be rsync, but recently thinking of using tarsnap for the ease of it (and S3 puts it in several data centers). * Files stored either in a MogileFS setup or S3 in multiple data centers. It doesn't have instant failover to another data center, but the offsite DB slave should mean no data loss beyond a second or two. Ma.gnolia is a decent sized site. Maybe they did have a decent infrastructure and it will only be a little while before they've gotten everything back. Of course, after the fiasco with the blog site that used RAID as their backup system, I've started to think that many people don't take data as seriously as it needs to be taken.
- timf 18y agoI like the idea of the offsite slave. I was also reading about another idea, forcing a slave delay to recover from oopses/corruption: http://www.rustyrazorblade.com/2008/05/07/mysql-time-delayed-replication/ http://www.rustyrazorblade.com/2008/05/07/mysql-time-delayed...
- eli 18y agoI do something like that for my outgoing email ;)
- t0pj 18y ago"take some simple steps to ensure that you don't bork your site" unborked[.com] is currently available. A site dedicated to stopping all of the borking going on out there. === I think some cheap, simple sql dumps from the db, burned to an external HD and taken home would've been a good first step -- your data would be dated but somehow restorable nonetheless without the need for off-site, replicated, heart-beat monitored databases. IMHO you really need an expert database admin (of which I am not) to get all of this up and keep all of it going; a caretaker who's only job in life is the maintenance of any and all persisted data. On another note, backups are really useless unless you have actually tested the restores.
- timf 18y agoI'm planning on burning time on building a full failure solution. Records snapshotted at least daily and any single node/service can die entirely and there is an exact, tested manual recovery checklist or automatic rollover option in place for each permutation. This runs counter to the more cavalier "release early, polish later" advice I keep seeing. Maybe I am doubly freaked out because the things I'm storing are not easily recovered or re-imported by the users themselves or any kind of algorithm/redux.
- bprater 18y agoIt'll be a time sink. Why not do it as dirty as possible now and go back and tune it as time goes on?
- timf 18y agoSee the original submission, that's why not :-) I want to not only have a backup scheme but also make sure it's restore-tested. Maybe I wasn't totally clear, not planning on a beautiful failover in each place in the beginning (planning failover for the DB at least). Just a tested (even if manual) restore procedure in each situation.
- jodrellblank 18y agoIt also runs counter to "do the simplest thing that could possibly work" and "KISS" and "YAGNI". Doesn't mean it's a bad idea though. But if it's that good, make sure you announce it and market it as a significant feature...
- palish 18y agoI wish Twitter allowed comments on tweets. Companies (like Magnolia) use it for important announcements / status updates, and yet there's no way to see feedback from the community.
- tptacek 18y agoThat's what search.twitter.com is for.
- jodrellblank 18y agoHope Twitter have backups...
- nickb 18y agoFrom reading the notice, I get the feeling that they don't have any backups! Not very re-assuring at all.
- bprater 18y agoGut wrenching, for sure. To see your whole model explode, it's just not worth it. If you don't have a backup plan -- close your browser, stop surfing Y!Hacker, and go write a shell script to dump your database and rsync to anywhere else on earth.
- eli 18y ago...and then do a test restore!
- jacquesm 18y agoamen to that! You should see the confused looks I get when doing technical due diligence somewhere and they get that question: "when did you last try to restore a backup ?" then once they get the point they want to leave the room asap...
- rarrrrrr 18y agoYES! This comment should be modded up to the top. We talk to a lot of customers about their backups every day. I'd say more than half of the backup failure stories we hear come from failing to adequately test that data can actually be restored. Some have been comically tragic. Like creating an offsite backup using encryption keys that are only archived as part of said backup... :(
- eli 18y agoA former colleague was attempting to replace a failed hard drive in a server. So he installed a fresh OS, put the tape in the drive, installed the backup software and watched as the backup software started to reformat the tape...
- timf 18y ago"creating an offsite backup using encryption keys that are only archived as part of said backup..." Ugh, that is awful.
- jodrellblank 18y agoI doubt Google have backups, take a copy home on a CD, have a massive third party database with dedicated certified people, or backup to Amazon S3. I suspect their file storage system is simply good enough to replicate data intelligently across many machines/sites like a giant RAID array. I also think that it's a truly massive benefit they have over other companies, particularly small companies, and that it's rarely discussed as such.
- eli 18y agoMagnolia ain't exactly indexing every page on the internet though, ya know? I agree with you that that is one of Google's key strengths, though. PageRank is neat, but the real edge is that the site is fast and the infrastructure is based on commodity hardware, so it's cheap.
- jodrellblank 18y agoThe real edge is not that it's based on commodity hardware, but that it's based on scalable low-management software (I imagine all this, and skimmed their Google File System paper once. I don't work for Google). It wouldn't matter if they were blades or Sun UltraWhatevers, the real benefit is that if one dies, nobody need panic - plug another in and "the system" will rebuild it. If a rack dies, plug another in and the system will rebuild it. If a datacenter dies, the others will cover for it. No backup tapes, no manual tweaking each new build.
- mikeyur 18y agoAs its been said a million times before, hardware is cheap. Creating a system that can utilize the cheapest Celeron servers you have with the flick of a switch is what you really need to create. Adding a few more servers is much cheaper than downtime.
- eli 18y agoWhat gets me is how people even do normal development without some sort of backup -- even an informal one. Maybe I'm just lucky enough to work with relatively small datasets, but our QA server contains a complete copy of the production database and is at most a week old. So even if we didn't run any other backups (which we do), we could restore from that server and be back up and running within hours.
- billturner 18y agoThey have a page/section at GetSatisfaction, but it doesn't seem to be updated anytime soon (though there are 3 employees set to answer questions): http://getsatisfaction.com/magnolia http://getsatisfaction.com/magnolia
- raghus 18y agoEarly on the West-coast morning of Friday, January 31st, - poor guys must be frazzled. Jan 31 is Saturday.
- seldo 18y agoOr else this is a dire warning from the future! There's still time to save the data; this accident won't happen until tomorrow!
- rarrrrrr 18y agoThe trouble with convincing humans to back up is that it's important, but not urgent. For any given day of procrastination, the likelihood of consequences is small. Our motto is "If it's not backed up, it's down." (Applies doubly so for us as a backup company.) For sufficiently seasoned administrators, non-redundant storage should cause sleeping difficulties.
- deleted 18y ago[deleted]
- Locke 18y agoIt's easy to say, "What? No backups... how stupid can you be?" But, without knowing the particulars, I wouldn't necessarily jump to that conclusion. It's very easy to have regular automated backups fail in some way. Unfortunately, it's not enough just to have backups. You have to actually verify that they're correct and up-to-date. Verification is easy when your database is small (for example). You can just load it in your development environment occasionally. But, how do you verify your backups if you have hundreds of gigs or even terabytes of data? As an example, I've seen cases where backups were successful every night... but, they were being run against a slave db and replication had failed. The result: Excellent backups of weeks old data.
- timf 18y agoThat's nasty. Maybe you could automatically inject test data into a fake account and run test queries on the backup. That will not verify you are getting everything but at least that you got something recent.
- Locke 18y agoVerifying backups is almost always many times more difficult than setting up the backups themselves. Checking that replication was in fact working turned out to be fairly easy to automate (although, short of checksumming all the records in all the tables you can't really be 100% sure everything is okay). That's the thing with backups, they can fail or become corrupt in many ways. If you don't use them for something on a regular basis or have some regular verification process you may not know what you have until it's too late. And, of course, I've also seen situations where subtle data corruption in the master database leads to weeks of subtly corrupt backups. By the time the corruption was discovered we faced the choice of rolling back several weeks to the last good backup or fixing the corruption. In the end we had to do a kind of merge -- it was a real pain.
- bemmu 18y agoIf I just have one MySQL machine and take daily database dumps, what would be good way of testing that my backups are OK?
- rarrrrrr 18y agoAn easy way to have a space efficient, perpetual backup of a database: Run the dump file every N units of time. Compress the dump file using gzip with the --rsyncable option. This increases size by 1% but makes it efficiently diff/patchable. Use diff to make a patch going back to the previous compressed dump of the file. Keep the patch, discard the previous dump if you like. You now can apply the patches in succession to go back however far you like. Finally, and most importantly, use par2 to store parity along with your backup files to protect against silent bitrot. Note: --rsyncable is in newer gzip versions. It's too new to be included in OS X Leopard.
- yeahit 18y agoWouldnt these two solutions be even nicer? * Backup the Binlog of the DB (At least its called Binlog in MySql) or * Make a diff of the uncompressed dump and zip that. This diff would even be human readable.
- philfreo 18y ago... glad I'm on del.icio.us, but going to export all my bookmarks now anyways.
- timf 18y agoLet's say you had an insane amount of bookmarks (or something precious like it). Would you pay to have DVDs burned and sent to you? It seems like a convenient re-assurance for users to get DVD backups mailed to them every so often -- but it would be a total pain to support this as a small business...
- timf 18y agoFor example: http://www.smugmug.com/help/backups http://www.smugmug.com/help/backups But how do I offer this when we're in colocation or slicehost etc.? Can't be driving to the colocation center every time someone wants a DVD...
- joshu 18y agoI have an insane amount of bookmarks. I can still back them up in a few seconds via wget. Of course photos and whatnot are much harder.
- timf 18y agoI am of course talking about something much harder. With big file sizes, the likelihood that users will regularly (much less in an automated way) download their own backups decreases. I broke this out... http://news.ycombinator.com/item?id=458921 http://news.ycombinator.com/item?id=458921
- tlrobinson 18y agoI don't know if I'd pay for just my list of bookmarks (I don't have that many), but I might pay for a monthly DVD archive of the site's content for all of my bookmarks, plus a searchable index. I hate when I can't find a bookmark, or a bookmark has gone offline.
- rbanffy 18y ago
- vaksel 18y agosounds like the same thing as Journal Space http://www.techcrunch.com/2009/01/03/journalspace-drama-all-data-lost-without-backup-company-deadpooled/ http://www.techcrunch.com/2009/01/03/journalspace-drama-all-...
- sam_in_nyc 18y agoDoes anybody use this site? I can never, ever remember the URL, and that bothers me a lot. The way I see it, there are too many social bookmarking sites out there. If there's now one less, I wouldn't mind.
- decadentcactus 18y agoWell, I finally did it. I stopped what I was doing, and worked out my backup strategy, which includes (finally) getting a script put together, and now to adjust it to email me copies and also (contemplating) using S3 to store copies. Maybe also box.net, depending on a few things.
- acangiano 18y agoBacking up is like testing, everyone knows that they should do it, but very few people actually do. That said, I'm sorry to hear about their troubles. They are good people genuinely trying their best.
- critic 18y agoIf YC crashes and nulls my karma, there's gonna be a lawsuit!
- ahoyhere 18y agoWe worked out our backup strategy for our time tracking system (http://letsfreckle.com/ http://letsfreckle.com/) while in beta. While we're still small, we do full db dumps to off-site, once an hour. This does not require a lot of sophistication. And yes, we might have some downtime while we restore in the event of a catastrophe, but we do have the data. I can't even imagine the nerve of shipping a product to the public that doesn't have even something this rudimentary in place.