3 ms·

I've seen many of my clients set up their own backup systems and have those fail at the worst times. Last month a large client of ours called our managed suppor

by XERQ 12y ago

I've seen many of my clients set up their own backup systems and have those fail at the worst times. Last month a large client of ours called our managed support team at 3AM saying they hired the wrong developer who completely trashed their database and hosed their entire application. They had their own backup system in place and it silently failed, but luckily they ordered our internal backup solution as a secondary. We were able to get them restored in 5 minutes, if they didn't have our solution in place they would've had to spend weeks fixing what the developer broke. Current Linux backup solutions are not made for humans. Have a look at the mondorescue guide[1], nobody is going to read that and comprehend it with full mastery, meaning you're leaving yourself open to losing data. VPS providers offer backups that are usually in the same datacenter, which means you're SOL if there's a disaster. Those same providers also don't allow you to restore single files/directories from snapshots, usually you have to launch a new instance or revert everything back to snapshot. [plug] We ended up creating a simple Linux backup solution[2] that's as simple as copying and pasting a single command to get installed, notifies you if your backups aren't running, handles snapshots, and is secure. Restoring your data is a single command away, so you can focus instead on building your startup rocketship. Our mission is to make data loss a thing of the past. [/plug] [1] http://www.mondorescue.org/docs/mondorescue-howto.html http://www.mondorescue.org/docs/mondorescue-howto.html [2] https://jarvys.io https://jarvys.io

donmcronald 12y ago
How do you handle a scenario like this... I have a Redmine install that I want to backup. It uses both a database and the file system. If I attach a file to an issue, the attachment is stored on the FS with a reference in the DB. I don't know how Redmine deals with keeping the FS vs the DB consistent, but assume it uses some type of transaction across both when an attachment is added. How do you back that up without possibly getting the FS and the DB out of sync. Ex: What if you snapshot the FS right after an attachment is written, but before the reference is added to the DB? The transaction in the app will succeed, but your backup is inconsistent compared to what the app expects. How can you get a truly consistent backup without either a) stopping the app or b) integrating with the app to make sure you're not breaking assumptions needed for consistency? Basically, almost every backup solution I've ever seen is crash consistent at best. How is yours different?
- derekp7 12y ago
  I've been struggling with this myself in various cases. The conclusion I've come down to, is that the app has to take some responsibility to provide for consistent backups, either via checkpoints / write barriers, or via a quiesce command that can finalize in-flight transactions, and pause / buffer operations long enough for a volume snapshot to take place. The other thing I've seen apps do is provide a data export functionality, so you end up backing up the export file not the live data. But that requires extra time and disk space. Another option is to run the app in a VM, and do a live VM snapshot which includes the running state of the VM. Bottom line is if you can't live backup your app, file a bug report with the app vendor. Although it would be really nice if there were standards in this area, so that a backup tool would just need to call one operation to put all supported apps into a hot-backup mode.
- XERQ 12y ago
  Our core value is around the simplicity of use, so instead of having to read a novel of a manual written by a crusty UNIX sysadmin, along with combining it with storage, you can create an account and copy-and-paste a single command that does all the work for you. We've even tested it with people who've never used Linux before and they were able to install it and restore a file without much direction. With that said, we do provide the ability to hook your own scripts and commands into the backup process itself, for instance backing up MySQL you'd just put in the mysqldump command (with relevant DB and user/pass info) into the hook script uncleverly titlted 'run-before-backup.sh' in the config directory. I personally don't have any experience with Redmine. Application-specific backups are outside the scope of our initial launch, but as we get more user feedback we'll be able to build plugins that integrate with specific applications. In the mean time, I'd both: 1) pester the developers to provide a simple backup solution for Redmine, and 2) look into either putting it on a VM that you can snapshot, or use something like ZFS snapshots and send/receive it to a remote location.
  - moe 12y ago
    Um. From your docs: C. RESTORE FROM ANOTHER SERVER'S BACKUP This feature is currently in development. So... My server goes up in a puff of smoke and I can't access the backup?
mercurial 12y ago
> Last month a large client of ours called our managed support team at 3AM saying they hired the wrong developer who completely trashed their database and hosed their entire application. Meaning that they didn't have a QA process in place before putting things in production, or that "dev" is also "production"?
- acveilleux 12y ago
  That's surprisingly common for "internal" applications. And quite frankly, the type of dev that screws everything up like that is also the type of dev that doesn't know any better, doesn't version things, doesn't have an issue tracker and doesn't set up DB backups, etc., etc.
  - mercurial 12y ago
    That's an institutional failure in this case. I certainly wouldn't expect a dev to setup backups for the company database, for instance. A VCS and issue tracker should be a company-wide (or at least department-wide) thing, not something you setup (or not) from project to project.

by XERQ 12y ago

donmcronald 12y ago

How do you handle a scenario like this... I have a Redmine install that I want to backup. It uses both a database and the file system. If I attach a file to an issue, the attachment is stored on the FS with a reference in the DB. I don't know how Redmine deals with keeping the FS vs the DB consistent, but assume it uses some type of transaction across both when an attachment is added. How do you back that up without possibly getting the FS and the DB out of sync. Ex: What if you snapshot the FS right after an attachment is written, but before the reference is added to the DB? The transaction in the app will succeed, but your backup is inconsistent compared to what the app expects. How can you get a truly consistent backup without either a) stopping the app or b) integrating with the app to make sure you're not breaking assumptions needed for consistency? Basically, almost every backup solution I've ever seen is crash consistent at best. How is yours different?

derekp7 12y ago
I've been struggling with this myself in various cases. The conclusion I've come down to, is that the app has to take some responsibility to provide for consistent backups, either via checkpoints / write barriers, or via a quiesce command that can finalize in-flight transactions, and pause / buffer operations long enough for a volume snapshot to take place. The other thing I've seen apps do is provide a data export functionality, so you end up backing up the export file not the live data. But that requires extra time and disk space. Another option is to run the app in a VM, and do a live VM snapshot which includes the running state of the VM. Bottom line is if you can't live backup your app, file a bug report with the app vendor. Although it would be really nice if there were standards in this area, so that a backup tool would just need to call one operation to put all supported apps into a hot-backup mode.
XERQ 12y ago
Our core value is around the simplicity of use, so instead of having to read a novel of a manual written by a crusty UNIX sysadmin, along with combining it with storage, you can create an account and copy-and-paste a single command that does all the work for you. We've even tested it with people who've never used Linux before and they were able to install it and restore a file without much direction. With that said, we do provide the ability to hook your own scripts and commands into the backup process itself, for instance backing up MySQL you'd just put in the mysqldump command (with relevant DB and user/pass info) into the hook script uncleverly titlted 'run-before-backup.sh' in the config directory. I personally don't have any experience with Redmine. Application-specific backups are outside the scope of our initial launch, but as we get more user feedback we'll be able to build plugins that integrate with specific applications. In the mean time, I'd both: 1) pester the developers to provide a simple backup solution for Redmine, and 2) look into either putting it on a VM that you can snapshot, or use something like ZFS snapshots and send/receive it to a remote location.
- moe 12y ago
  Um. From your docs: C. RESTORE FROM ANOTHER SERVER'S BACKUP This feature is currently in development. So... My server goes up in a puff of smoke and I can't access the backup?

mercurial 12y ago

> Last month a large client of ours called our managed support team at 3AM saying they hired the wrong developer who completely trashed their database and hosed their entire application. Meaning that they didn't have a QA process in place before putting things in production, or that "dev" is also "production"?

acveilleux 12y ago
That's surprisingly common for "internal" applications. And quite frankly, the type of dev that screws everything up like that is also the type of dev that doesn't know any better, doesn't version things, doesn't have an issue tracker and doesn't set up DB backups, etc., etc.
- mercurial 12y ago
  That's an institutional failure in this case. I certainly wouldn't expect a dev to setup backups for the company database, for instance. A VCS and issue tracker should be a company-wide (or at least department-wide) thing, not something you setup (or not) from project to project.